Traffic light recognition method, device, equipment, storage medium and program product
By combining deep learning object detection and color extraction algorithms, and comprehensively analyzing the indicator light area and color detection box, the problems of false detection and missed detection in traffic light recognition in autonomous vehicles are solved, and traffic light recognition with higher accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- VANJEE TECHNOLOGY CO LTD
- Filing Date
- 2022-11-25
- Publication Date
- 2026-06-26
AI Technical Summary
The lack of accurate traffic light recognition methods in existing technologies leads to missed and false detections when autonomous vehicles identify traffic lights, affecting driving safety.
By combining deep learning object detection algorithms and color extraction algorithms, object detection and color extraction are performed on the image to be detected to obtain the indicator light area and color detection box. The indicator light object detection box is determined by comprehensive analysis and recognition processing is performed to obtain the recognition result.
This improves the accuracy of indicator light target detection, ensures the accuracy of traffic light recognition results, reduces false detections and missed detections, and guarantees the safe operation of autonomous vehicles.
Smart Images
Figure CN118097607B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a traffic light recognition method, apparatus, device, storage medium, and program product. Background Technology
[0002] With the development of technology, automobiles have gradually become the main mode of transportation. However, due to driver fatigue, inattention, and other reasons, numerous traffic accidents occur. Therefore, traffic light recognition is an essential technology for ensuring the safety and efficiency of autonomous vehicles, and a crucial prerequisite for their deployment on public roads. Based on this, traffic light recognition in driving scenarios is of paramount importance.
[0003] However, there is a lack of methods in the relevant technologies that can accurately identify traffic lights. Summary of the Invention
[0004] Therefore, it is necessary to provide a traffic light recognition method, device, equipment, storage medium, and program product that can accurately recognize traffic lights, addressing the aforementioned technical problems.
[0005] Firstly, this application provides a traffic light recognition method. The method includes:
[0006] Based on the image to be detected in the target traffic scene, the region where the indicator lights are located in the image to be detected is detected to obtain the indicator light region detection box; and the color of the image to be detected is extracted to obtain the indicator light color detection box.
[0007] Based on the indicator light area detection box and the indicator light color detection box, determine the indicator light target detection box;
[0008] The indicator light target detection box is processed for recognition to obtain the indicator light recognition results in the target traffic scene.
[0009] In one embodiment, color extraction is performed on the image to be detected to obtain an indicator light color detection frame, including:
[0010] Color extraction is performed on the image to be detected to obtain the indicator light area of the image;
[0011] Based on the location distribution of the indicator lights, the indicator light area is expanded to determine the indicator light color detection frame.
[0012] In one embodiment, color extraction is performed on the image to be detected to obtain the indicator light area of the image to be detected, including:
[0013] Boundary extraction is performed on the image to be detected to determine the color bounding boxes in the image;
[0014] The indicator light area of the image to be detected is determined based on the color bounding box.
[0015] In one embodiment, boundary extraction is performed on the image to be detected to determine the color bounding boxes in the image to be detected, including:
[0016] Each color region in the image to be detected is mapped onto its corresponding binary image;
[0017] Obtain the connected components in each binary graph;
[0018] Extract the minimum regular bounding box of the connected components in each binary graph, and determine the minimum regular bounding box as the color bounding box of the image to be detected.
[0019] In one embodiment, determining the indicator light area of the image to be detected based on the color bounding box includes:
[0020] If the area of the color bounding box is within the preset area range, then the color bounding box is determined as the indicator light area of the image to be detected.
[0021] In one embodiment, determining the indicator light target detection box based on the indicator light area detection box and the indicator light color detection box includes:
[0022] Calculate the overlap between the indicator light area detection box and the indicator light color detection box;
[0023] The target detection frame for the indicator light is determined based on the degree of overlap between regions.
[0024] In one embodiment, determining the indicator light target detection box based on the region overlap includes:
[0025] If the overlap of regions is greater than the preset overlap threshold, the indicator light region detection box is determined as the indicator light target detection box;
[0026] If the overlap of regions is less than or equal to the overlap threshold, then the indicator area detection box and the indicator color detection box are determined as the indicator target detection box.
[0027] In one embodiment, the indicator light target detection box is recognized to obtain the indicator light recognition result in the target traffic scene, including:
[0028] Identify the indicator light target detection box and determine the indicator light category of the indicator light target detection box;
[0029] Determine the indicator light category of the image to be inspected based on the indicator light category;
[0030] Based on the indicator light categories in the image to be detected and the indicator light categories in historical detected images, the indicator light recognition results in the target traffic scene are determined.
[0031] In one embodiment, determining the indicator light category of the image to be detected based on the indicator light category includes:
[0032] Obtain the number of each indicator light category in the image to be detected;
[0033] The category with the most indicator lights is determined as the indicator light category of the image to be detected.
[0034] In one embodiment, the method further includes:
[0035] If the indicator light category is the first color category, then the indicator light category of the indicator light target detection box is determined to be the second color category; the first color category and the second color category meet the preset display conditions.
[0036] In one embodiment, the indicator light recognition result in the target traffic scene is determined based on the indicator light category of the image to be detected and the indicator light categories of historical detected images, including:
[0037] Based on the indicator light categories of the image to be detected and the indicator light categories of historical detected images, obtain the number of indicator lights corresponding to each indicator light category;
[0038] The indicator light category with the most indicator lights is identified as the indicator light recognition result in the target traffic scenario.
[0039] Secondly, this application also provides a traffic light recognition device, which includes:
[0040] The detection module is used to detect the area where the indicator lights are located in the image to be detected based on the target traffic scene, and obtain the indicator light area detection box; and to extract the color of the image to be detected to obtain the indicator light color detection box.
[0041] The detection frame determination module is used to determine the target detection frame of the indicator light based on the indicator light area detection frame and the indicator light color detection frame;
[0042] The recognition module is used to recognize the target detection box of the indicator light and obtain the recognition result of the indicator light in the target traffic scene.
[0043] Thirdly, embodiments of this application provide a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method provided in any of the embodiments of the first aspect described above.
[0044] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in any of the embodiments of the first aspect described above.
[0045] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method provided in any of the embodiments of the first aspect described above.
[0046] The aforementioned traffic light recognition method, apparatus, device, storage medium, and program product, based on a target traffic scene image to be detected, detects the area where the traffic lights are located in the image to be detected, obtaining a traffic light area detection box; extracts color from the image to be detected, obtaining a traffic light color detection box; and determines the traffic light target detection box based on the traffic light area detection box and the traffic light color detection box. The traffic light target detection box is then recognized to obtain the traffic light recognition result in the target traffic scene. This method, by obtaining the traffic light area detection box and the traffic light color detection box of the image to be detected in the target traffic scene, determines the traffic light target detection box of the image to be detected. By analyzing the detection boxes in the image to be detected from two dimensions, it avoids false detections or missed detections of traffic lights by the target detection method and the color extraction method, improving the accuracy of the traffic light target detection box extraction in the image to be detected, thereby ensuring the accuracy of the traffic light recognition result in the target traffic scene. Attached Figure Description
[0047] Figure 1 This is an application environment diagram of the traffic light recognition method in one embodiment;
[0048] Figure 2 This is a flowchart illustrating a traffic light recognition method in one embodiment;
[0049] Figure 3 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0050] Figure 4 This is a schematic diagram of the expanded area of the indicator light region in one embodiment;
[0051] Figure 5 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0052] Figure 6 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0053] Figure 7 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0054] Figure 8 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0055] Figure 9 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0056] Figure 10 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0057] Figure 11 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0058] Figure 12 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0059] Figure 13 This is a flowchart illustrating the traffic light recognition method in another embodiment;
[0060] Figure 14 This is a structural block diagram of a traffic light recognition device in one embodiment;
[0061] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0063] The traffic light recognition method provided in this application can be applied to vehicles such as electric vehicles and autonomous vehicles. For ease of explanation, the following embodiments use a vehicle from this application as an example. Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of the vehicle 100 provided in the embodiments of this application, wherein the control system 101 is the control system of the vehicle 100.
[0064] Traffic light recognition is an essential technology for ensuring the safety and efficiency of autonomous vehicles in autonomous driving technology.
[0065] One approach typically uses deep learning object detection to identify traffic lights on roads, but this can lead to missed detections and false detections, which can be dangerous during driving. Another approach uses traditional color extraction methods to identify traffic lights on roads, but color extraction methods are greatly affected by color and have poor environmental adaptability, resulting in more false detections.
[0066] Furthermore, the indicator lights in the detection image are detected, and the indicator lights are classified based on the detection results. Due to the influence of the first step of detection, if the first step does not detect any indicator lights, the classification results in the second step will be meaningless.
[0067] Based on this, this application proposes a traffic light recognition method, device, equipment, storage medium, and program product, which integrates a deep learning-based object detection algorithm and a color extraction algorithm to achieve accurate recognition of traffic lights.
[0068] The following explanation uses the control system in a vehicle as the implementing entity to illustrate the traffic light recognition method.
[0069] In one embodiment, such as Figure 2 As shown, a traffic light recognition method is provided, including the following steps:
[0070] S201, based on the image to be detected of the target traffic scene, detect the area where the indicator lights are located in the image to be detected to obtain the indicator light area detection box; and extract the color of the image to be detected to obtain the indicator light color detection box.
[0071] Taking autonomous vehicles as an example, the target traffic scene is the scene in which the autonomous vehicle is driving on the road. During the autonomous vehicle's operation, it can acquire images of the target traffic scene, which are the images to be detected.
[0072] The image to be detected can be acquired by the Advanced Driver Assistance System (ADAS) in the vehicle; the ADAS includes a variety of sensors that can acquire images of the scene around the vehicle.
[0073] After obtaining the image to be detected, target detection is performed on the area where the indicator light is located in the image to obtain the indicator light area detection box. At the same time, color extraction is performed on the image to be detected to obtain the indicator light color detection box.
[0074] One method for target detection in the area where the indicator lights are located in the image to be detected is to use a preset indicator light detection model to detect the indicator lights in the image to be detected, thereby obtaining the indicator light area detection box in the image to be detected. The indicator light detection model can be a YOLO network model.
[0075] Another way to perform target detection in the area where the indicator lights are located in the image to be detected is to use a target detection algorithm to detect the indicator lights in the image to be detected, and obtain the detection box of the indicator light area in the image to be detected.
[0076] The color detection box of the indicator light in the image to be detected can be obtained by a color extraction algorithm. The color extraction algorithm can be a color quantization method, a clustering algorithm, or a color modeling method, etc.
[0077] Optionally, the indicator lights may include red, yellow, green, black, etc.
[0078] S202, determine the indicator light target detection box based on the indicator light area detection box and the indicator light color detection box.
[0079] Since both the indicator light area detection box and the indicator light color detection box are extracted from the image to be detected, the indicator light target detection box in the image to be detected can be determined by comprehensively analyzing the indicator light area detection box and the indicator light color detection box. The indicator light target detection box is the indicator light detection box obtained by comprehensively considering the target detection method and the color extraction algorithm.
[0080] If the indicator light area detection box is within the preset range of the indicator light color detection box, then the indicator light area detection box is determined as the indicator light target detection box. The preset range can be within a preset area centered on the indicator light area detection box.
[0081] If the indicator light area detection box is not within the preset range of the indicator light color detection box, then both the indicator light area detection box and the indicator light color detection box will be determined as the indicator light target detection box.
[0082] S203, perform recognition processing on the indicator light target detection box to obtain the indicator light recognition result in the target traffic scene.
[0083] Based on the preset indicator light recognition model, the indicator light target detection box is recognized and processed to obtain the indicator light recognition result corresponding to the indicator light target detection box. This indicator light recognition result is the indicator light recognition result in the target traffic scene.
[0084] The indicator light recognition result can be the type of indicator light in the target traffic scene, such as yellow light, red light, green light, or black light; and the indicator light recognition result can also include whether the indicator light is a motor vehicle signal light, a non-motor vehicle signal light, a pedestrian crossing signal light, a directional indicator light (arrow signal light), a lane signal light, a flashing warning signal light, or a road and railway level crossing signal light, etc.
[0085] The aforementioned traffic light recognition method, based on the target traffic scene image to be detected, detects the area where the traffic lights are located in the image to obtain the traffic light region detection box, and extracts the color of the image to obtain the traffic light color detection box. Based on the traffic light region detection box and the traffic light color detection box, the target detection box of the traffic light is determined. This target detection box is then processed for recognition to obtain the traffic light recognition result in the target traffic scene. This method determines the target detection box of the traffic lights in the image to be detected by obtaining the traffic light region detection box and the traffic light color detection box. By analyzing the detection boxes in the image to be detected from two dimensions, it avoids false detections or missed detections of traffic lights by the target detection method and the color extraction method, improving the accuracy of the traffic light target detection box extraction and thus ensuring the accuracy of the traffic light recognition result in the target traffic scene.
[0086] The following is a detailed description of the specific method for obtaining the indicator light color detection box through an embodiment. In one embodiment, such as... Figure 3 As shown, color extraction is performed on the image to be detected to obtain the indicator light color detection box, including:
[0087] S301, extract colors from the image to be detected to obtain the indicator light area of the image to be detected.
[0088] Color extraction of the image to be detected can be performed using the YCbCr color space to determine the indicator light area of the image.
[0089] YCbCr is a color space that uses mathematical methods to describe a set of colors. The original image to be detected is in R (red), G (green), and B (blue) format. The RGB format of the image to be detected can be converted to YCbCr format. YCbCr format is represented by triplets, which consist of Y (Luminance), Cb (Chrominance-Blue), and Cr (Chrominance-Red). Y represents the brightness and intensity of the color, while Cb and Cr represent the blue intensity offset and red intensity offset of the color, respectively. The relationship between the RGB values and the Y, Cb, and Cr components can be expressed as shown in Equation (1).
[0090]
[0091] From the converted image to be detected, the indicator light area of the image is obtained according to the color distribution and the color category of the indicator light.
[0092] S302, based on the location distribution of the indicator lights, expand the indicator light area to determine the indicator light color detection frame.
[0093] Since the indicator area for color recognition is the area of one indicator, but in normal circumstances, the detection frame of the indicator includes the area of three indicator. Therefore, in order to ensure the integrity and size consistency of the indicator detection area, the indicator area is expanded after obtaining the indicator area.
[0094] Typically, indicator lights are distributed in both horizontal and vertical directions.
[0095] Therefore, after obtaining the indicator light area in the image to be detected, the area of the indicator light can be expanded according to the position distribution of the indicator light to obtain the indicator light color detection box.
[0096] In one embodiment, if the indicator lights are distributed horizontally, the indicator light area is directly expanded horizontally by 2 times to the left or right, or by 1 time to both the left and right, to obtain the indicator light color detection frame; if the indicator lights are distributed vertically, the indicator light area is directly expanded vertically by 2 times to the down or up, or by 1 time to both the up and down, to obtain the indicator light color detection frame.
[0097] In another embodiment, such as Figure 4 As shown, based on the positional distribution of the indicator lights and the state of the indicator light area, the indicator light area is expanded to determine the indicator light color detection frame. Taking a horizontal distribution of indicator lights as an example, if the horizontal color distribution of the indicator lights is red, yellow, and green; if the indicator light area is red, the indicator light area can be expanded horizontally to the right by a factor of 2 to obtain the indicator light color detection frame, as shown. Figure 4 As shown in (a); if the indicator light area is yellow, the indicator light area can be expanded horizontally by 1 unit to the left and horizontally by 1 unit to the right to obtain the indicator light color detection frame, as shown in (a). Figure 4 As shown in (b); if the indicator light area is green, the indicator light area can be expanded horizontally to the left by 2 times to obtain the indicator light color detection frame, as shown in (b). Figure 4 As shown in (c).
[0098] Optionally, the indicator lights are arranged vertically, with their colors ranging from red, yellow, and green from highest to lowest, and the height of each indicator light is three times its width. The original size of the indicator light area is (x, y, w1, w2), where x and y represent the top-left vertex of the indicator light area, w1 represents the width of the indicator light area, and w2 represents the height of the indicator light area. At this point, w1 is approximately equal to w2. When the indicator light area is red, the area is expanded, resulting in an indicator light color detection frame size of (x, y, w1, 3w1); when the indicator light area is green, the area is expanded, resulting in an indicator light color detection frame size of (x, y - 2w1, w1, 3w1).
[0099] The traffic light recognition method described above extracts colors from the image to be detected, obtains the indicator light region of the image, and expands the indicator light region according to the positional distribution of the indicator lights to determine the indicator light color detection box. In this method, by expanding the indicator light region after color extraction, the size of the indicator light color detection box is ensured, thus improving the accuracy of traffic light recognition.
[0100] When extracting colors from an image to be detected, the indicator light area in the image can be determined by observing color changes and identifying color boundaries. This will be explained in detail below with an example. In one example, as shown... Figure 5 As shown, color extraction is performed on the image to be detected to obtain the indicator light area of the image, including the following steps:
[0101] S501, perform boundary extraction on the image to be detected to determine the color bounding box in the image to be detected.
[0102] By converting the image to be detected from RGB format to YCbCr format, and using the YCbCr color space to identify the indicator lights, the color bounding box in the image to be detected is determined.
[0103] In one embodiment, such as Figure 6 As shown, boundary extraction of the image to be detected, determining the color bounding boxes in the image to be detected, includes the following steps:
[0104] S601, map each color region in the image to be detected onto the corresponding binary image.
[0105] First, create multiple binary images of the same size as the image to be detected to store the results of each color region, and set the range of each color region according to the Cr value characteristics of each color region.
[0106] Iterate through the Cr values in the YCbCr image corresponding to the image to be detected. For any color region, if the Cr value is within the set color region range, set the pixel value corresponding to that color region to 255, otherwise set it to 0. This process yields the binary image of each color region.
[0107] For example, if the color region includes red and green, iterate through the Cr values in the YCbCr image. If the Cr value is within the set red range, set the corresponding pixel of the binary image of the red region to 255, otherwise set it to 0, thus obtaining the binary image corresponding to the red region. If the Cr value is within the set green range, set the corresponding pixel of the binary image of the green region to 255, otherwise set it to 0, thus obtaining the binary image corresponding to the green region.
[0108] S602, obtain the connected components in each binary graph.
[0109] Analyze the connected regions in each binary image, identify and label each connected region, and determine the connected components in each binary image.
[0110] A connected region is an image region consisting of foreground pixels that have the same pixel value and are adjacent in position. As can be seen from the definition of a connected region, a connected region is a set of pixels consisting of adjacent pixels with the same pixel value. Therefore, for any binary image, we can find connected regions in the binary image by using these two conditions. For each connected region found, we assign a unique label to distinguish it from other connected regions.
[0111] Alternatively, the connected components in a binary graph can be obtained by a two-pass scanning method, that is, scanning the binary graph twice to find and mark all connected regions in the image, so as to obtain the connected components of each binary graph.
[0112] Optionally, before obtaining the connected components of each binary image, the corresponding shapes in the image can be measured and extracted using structuring elements of a certain shape for boundary extraction of the image. The morphological operations include binarization erosion and dilation operations, and the structuring elements can include squares.
[0113] Therefore, based on the square structuring element, the dilation operation is first applied to each binary graph, and then the erosion operation is applied to the binary graph after the dilation operation to obtain the morphologically processed binary graph. Then, the connected components in each morphologically processed binary graph are obtained.
[0114] S603, extract the minimum regular bounding box of the connected components in each binary graph, and determine the minimum regular bounding box as the color bounding box of the image to be detected.
[0115] The connected components of each binary image obtained above may be an irregular polygon. Therefore, the polygon can be circumscribed to form a regular square, which is the smallest regular bounding box of the connected components in each binary image. Then, the smallest regular bounding box is determined as the color bounding box of the image to be detected.
[0116] It should be noted that there can be one or more color bounding boxes in the image to be detected.
[0117] S502, determine the indicator light area of the image to be detected based on the color bounding box.
[0118] The color bounding boxes of the image to be detected obtained above may contain some unreasonable bounding boxes. For example, if the area of the obtained color bounding box is too large or too small, it is impossible for it to be the indicator light area in the image to be detected.
[0119] Therefore, in one embodiment, determining the indicator light area of the image to be detected based on the color bounding box includes: if the area of the color bounding box is within a preset area range, then the color bounding box is determined as the indicator light area of the image to be detected.
[0120] Based on the area of the color bounding box, remove color bounding boxes that are too large or too small, and determine the color bounding boxes with an area within the preset area range as the indicator light area of the image to be detected.
[0121] The preset area range is between the maximum area and the minimum area. The maximum area is the maximum area allowed for the indicator light area, and the minimum area is the minimum area allowed for the indicator light area.
[0122] In one embodiment, determining the indicator light area of the image to be detected based on the color bounding box includes: if the aspect ratio of the color bounding box is within a preset ratio range, then the color bounding box is determined as the indicator light area of the image to be detected.
[0123] Normally, the indicator light area is square. Considering the error during detection, the maximum and minimum aspect ratios of the indicator light area can be set. The range between the minimum and maximum aspect ratios is the preset aspect ratio range. The color bounding box with aspect ratios within the preset aspect ratio range is defined as the indicator light area of the image to be detected.
[0124] The traffic light recognition method described above extracts the boundaries of the image to be detected, determines the color bounding boxes in the image, and then determines the indicator light area based on the color bounding boxes. This method obtains the color bounding boxes of the image to be detected and, based on the area of the color bounding boxes, removes detection boxes that do not conform to the area of the indicator light area, making the obtained indicator light area more reasonable and improving the accuracy of color extraction.
[0125] Based on the indicator light area detection box and indicator light color detection box obtained above, the following example will describe in detail how to determine the indicator light target detection box according to the indicator light area detection box and indicator light color detection box. In one example, as follows: Figure 7 As shown, the indicator light target detection box is determined based on the indicator light area detection box and the indicator light color detection box, including the following steps:
[0126] S701, calculate the overlap between the indicator light area detection box and the indicator light color detection box.
[0127] The intersection-union ratio (IUGR) is calculated between the indicator light region detection box obtained from target detection and the indicator light color detection box obtained from color extraction. This IUGR is the ratio of the intersection to the union of the indicator light region detection box and the indicator light color detection box. This ratio is then used to determine the region overlap between the indicator light region detection box and the indicator light color detection box.
[0128] Optionally, the region overlap degree ranges from 0 to 1. The closer the region overlap degree is to 0, the smaller the overlap range between the indicator light region detection box and the indicator light color detection box; the closer the region overlap degree is to 1, the larger the overlap range between the indicator light region detection box and the indicator light color detection box.
[0129] S702, determine the indicator light target detection frame based on the area overlap.
[0130] The overlap between the indicator light area detection box and the indicator light color detection box is compared with a preset overlap threshold, and the indicator light target detection box is determined based on the comparison result; in one embodiment, such as Figure 8 As shown, the indicator light target detection box is determined based on the region overlap, including the following steps:
[0131] S801, if the overlap of the regions is greater than the preset overlap threshold, then the indicator area detection box is determined as the indicator target detection box.
[0132] S802, if the region overlap is less than or equal to the overlap threshold, then the indicator area detection box and the indicator color detection box are determined as the indicator target detection box.
[0133] If the overlap between the indicator light area detection box and the indicator light color detection box is greater than the preset overlap threshold, it is determined that the indicator light area detection box and the indicator light color detection box overlap too much. The indicator light area detection box and the indicator light color detection box are then identified as the same detection box. The duplicate detection box (indicator light color detection box) is then removed, and the indicator light area detection box is identified as the indicator light target detection box.
[0134] If the overlap between the indicator light area detection box and the indicator light color detection box is less than or equal to the preset overlap threshold, then it is determined that the overlap between the indicator light area detection box and the indicator light color detection box is small, and the indicator light area detection box and the indicator light color detection box are two different detection boxes. Therefore, both the indicator light area detection box and the indicator light color detection box are indicator light target detection boxes, and the indicator light area detection box and the indicator light color detection box are determined as indicator light target detection boxes.
[0135] The traffic light recognition method described above calculates the overlap between the indicator light area detection box and the indicator light color detection box, and determines the indicator light target detection box based on the overlap. This method removes duplicate detection boxes by calculating the overlap between the indicator light area detection box and the indicator light color detection box, and determines whether any detection boxes were missed, thus improving the accuracy of the indicator light target detection box.
[0136] The above embodiments describe how to determine the indicator light target detection box. The following embodiment will provide a detailed explanation of how to identify and process the indicator light target detection box. In one embodiment, such as... Figure 9 As shown, the indicator light target detection box is processed for recognition to obtain the indicator light recognition result in the target traffic scene, including the following steps:
[0137] S901 identifies the indicator light target detection box and determines the indicator light category of the indicator light target detection box.
[0138] The indicator light target detection box in the image to be detected is identified by a preset indicator light classification model. Specifically, the indicator light target detection box is used as the input of the preset indicator light classification model. The indicator light classification model analyzes the indicator light target detection box to obtain the indicator light category corresponding to the indicator light target detection box.
[0139] Optionally, the indicator light categories include red light, yellow light, green light, black light, or none.
[0140] S902, determine the indicator light category of the image to be detected based on the indicator light category.
[0141] Based on the indicator light category, determine the indicator light category of the image to be detected. It should be noted that the target detection box of the indicator light in the image to be detected may be one or more.
[0142] If there is only one indicator light target detection box in the image to be detected, then the indicator light category of the indicator light target detection box is the indicator light category of the image to be detected.
[0143] If there are multiple indicator light target detection boxes in the image to be detected, the indicator light category of the image to be detected can be determined according to the indicator light category corresponding to each indicator light target detection box. In one embodiment, such as... Figure 10 As shown, determining the indicator light category of the image to be detected based on the indicator light category includes the following steps:
[0144] S1001, Obtain the number of each indicator light category in the image to be detected.
[0145] Obtain the number of each indicator light category in the image to be detected. For example, if the image to be detected includes 5 indicator light target detection boxes, and the indicator light categories corresponding to the indicator light target detection boxes are red light, red light, yellow light, red light and black light, then determine that there are 3 red lights, 1 yellow light and 1 black light in the image to be detected.
[0146] S1002, the category with the most indicator lights is determined as the indicator light category of the image to be detected.
[0147] Based on the categories of the indicator lights mentioned above, the category with the most indicator lights is determined as the indicator light category of the image to be detected.
[0148] For example, if there are 3 red lights, 1 yellow light, and 1 black light in the image to be detected, then the number of red lights is the largest, and red lights are identified as the indicator light category of the image to be detected.
[0149] In one embodiment, for driving safety, if the indicator light category of the indicator light target detection frame is yellow, the yellow light can be treated as a red light; if the indicator light category is a first color category, the indicator light category of the indicator light target detection frame is determined to be a second color category; the first color category and the second color category meet preset display conditions; wherein, the first color category is yellow and the second color category is red.
[0150] If the indicator light category of the indicator light target detection box is yellow, then the yellow light in the indicator light target detection box will be determined as red.
[0151] S903, based on the indicator light category of the image to be detected and the indicator light category of the historical detected images, determines the indicator light recognition result in the target traffic scene.
[0152] The indicator light category for historical detection images can be the detection image acquired at the moment before the time when the image to be detected was acquired. If the image to be detected is the image acquired at the current time, then the historical detection image can be the detection image acquired at the previous time.
[0153] Based on the indicator light categories in the image to be detected and the indicator light categories in historical detected images, the indicator light recognition results in the target traffic scene are determined.
[0154] If the indicator light category in the historical detection image is red and the indicator light category in the image to be detected is green, then in order to ensure the safety of driving on the road, the indicator light recognition result in the target traffic scene will be determined as red.
[0155] If the indicator light category in the historical detection image is green, and the indicator light category in the image to be detected is also green, then the indicator light recognition result in the target traffic scene can be determined as green.
[0156] The aforementioned traffic light recognition method identifies the traffic light target detection box, determines the traffic light category of the target detection box, determines the traffic light category of the image to be detected based on the traffic light category, and determines the traffic light recognition result in the target traffic scene based on the traffic light category of the image to be detected and the traffic light categories of historical detection images. This method determines the traffic light recognition result in the target traffic scene by comprehensively analyzing the traffic light category of the image to be detected and the traffic light categories of historical detection images. It considers the recognition results of historical detection images and avoids using the recognition result of a single detection image as the traffic light recognition result in the target traffic scene, thus ensuring road driving safety.
[0157] In one embodiment, such as Figure 11 As shown, the indicator light recognition result in the target traffic scene is determined based on the indicator light category of the image to be detected and the indicator light categories of historical detected images, including the following steps:
[0158] S1101, based on the indicator light category of the image to be detected and the indicator light category of the historical detected images, obtain the number of indicator lights corresponding to each indicator light category.
[0159] Among them, the historical detection images are a preset number of consecutive frames preceding the image to be detected, and the image to be detected and the historical detection images are detection images of consecutive frames.
[0160] Therefore, the number of indicator lights corresponding to each indicator light category is determined based on the indicator light category of the image to be detected and the indicator light category of the historical detected images.
[0161] For example, if there are 5 historical detection images, and the indicator light categories of the historical detection images are green, green, green, green and red, and the indicator light category of the image to be detected is red, then it can be determined that there are 2 red lights and 4 green lights in the image to be detected and the historical detection images.
[0162] S1102, the indicator light category with the most indicator lights is determined as the indicator light recognition result in the target traffic scene.
[0163] The indicator light category with the most indicator lights in the image to be detected and the historical detection images is determined as the indicator light recognition result in the target traffic scene.
[0164] For example, if the number of red lights is the largest in the image to be detected and the historical detection images, the traffic light recognition result in the target traffic scene is red; if the number of green lights is the largest in the image to be detected and the historical detection images, the traffic light recognition result in the target traffic scene is green.
[0165] Optionally, if the number of red lights is the same as the number of green lights, the indicator light recognition result in the target traffic scene will be determined as a red light.
[0166] In one embodiment, in the image to be detected and the historical detection images, if the number of red lights is greater than 0 and the number of red lights is greater than the number of green lights, then the indicator light recognition result in the target traffic scene is red; if the number of green lights is greater than 0 and the number of green lights is greater than the number of red lights, then the indicator light recognition result in the target traffic scene is green; if the number of red lights is greater than 0 and the number of red lights is equal to the number of green lights, then it can be determined that the indicator light recognition result in the target traffic scene is red; otherwise, the indicator light recognition result in the target traffic scene is no light.
[0167] The traffic light recognition method described above obtains the number of traffic lights corresponding to each category based on the traffic light category in the image to be detected and the traffic light categories in historical detected images. The category with the highest number of traffic lights is then identified as the traffic light recognition result in the target traffic scene. This method determines the traffic light recognition result in the target traffic scene by detecting the traffic light categories in multiple consecutive frames of images, providing reaction time for driving vehicles in traffic light scenarios and ensuring driving safety.
[0168] In one embodiment, such as Figure 12 As shown, this embodiment includes the following steps:
[0169] S1201, acquire the image to be detected of the target traffic scene.
[0170] S1202, Perform image target detection on the area where the indicator light is located in the image to be detected, and obtain the indicator light area detection box.
[0171] S1203, extract the color from the image to be detected to obtain the indicator light color detection frame.
[0172] S1204: Based on the indicator light area detection box and the indicator light color detection box, remove duplicate detection boxes to obtain the indicator light target detection box.
[0173] Calculate the intersection over Union (IOU) of the indicator light area detection box and the indicator light color detection box. If the intersection over Union is greater than the preset overlap threshold, the indicator light area detection box is determined as the indicator light target detection box; if the intersection over Union is less than or equal to the overlap threshold, the indicator light area detection box and the indicator light color detection box are determined as the indicator light target detection box.
[0174] S1205 uses a classification network to classify the indicator light target detection boxes and determine the indicator light category of the image to be detected.
[0175] If the indicator light category of the image to be detected is yellow, then the indicator light category of the image to be detected will be treated as red.
[0176] S1206, Obtain the indicator light categories of the 5 historical detection images preceding the image to be detected, and count the number of each indicator light category in the historical detection images and the image to be detected.
[0177] S1207, if the number of red lights in the historical detection image and the image to be detected is greater than 0, and the number of red lights is greater than or equal to the number of green lights, then the indicator light recognition result in the target traffic scene is determined to be a red light; if the number of green lights in the historical detection image and the image to be detected is greater than 0, and the number of green lights is greater than the number of red lights, then the indicator light recognition result in the target traffic scene is determined to be a green light; otherwise, it means there are no lights.
[0178] In one embodiment, such as Figure 13 As shown, Figure 13 for Figure 12 In step S1203, color extraction is performed on the image to be detected to obtain a detailed description of the indicator light color detection frame. This embodiment includes the following steps:
[0179] S1301 converts the RGB image of the image to be detected into a YCbCr image.
[0180] S1302, construct a binary image corresponding to red and green in the image to be detected based on the Cr value in the YCbCr image;
[0181] Specifically, two binary images of the same size as the RGB image of the image to be detected are created. One image is used to store the result of the red light, and the other image is used to store the result of the green light. The Cr values in the YCrCb image are traversed. If the Cr value is within the set red range, the corresponding pixel in the binary image of the red light is set to 255, otherwise it is set to 0. If the Cr value is within the set green range, the corresponding pixel in the binary image of the green light is set to 255, otherwise it is set to 0.
[0182] S1303, after performing dilation operation on each binary image, perform erosion operation on the dilated binary image to obtain the eroded binary image.
[0183] S1304: Calculate the connected components based on the binary images after each erosion operation, and obtain the minimum bounding box of the connected components.
[0184] S1305: Based on the preset minimum and maximum areas, remove unreasonable bounding boxes from the minimum bounding box to obtain the bounding box.
[0185] S1306, based on the location and status of the traffic lights, expand the bounding box to obtain the indicator light color detection box.
[0186] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0187] Based on the same inventive concept, this application also provides a traffic light recognition device for implementing the traffic light recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more traffic light recognition device embodiments provided below can be found in the limitations of the traffic light recognition method described above, and will not be repeated here.
[0188] In one embodiment, such as Figure 14 As shown, a traffic light recognition device is provided, including: a detection module 1401, a detection frame determination module 1402, and a recognition module 1403, wherein:
[0189] The detection module 1401 is used to detect the area where the indicator lights are located in the image to be detected based on the image to be detected of the target traffic scene, and obtain the indicator light area detection box; and to extract the color of the image to be detected and obtain the indicator light color detection box.
[0190] The detection frame determination module 1402 is used to determine the target detection frame of the indicator light based on the indicator light area detection frame and the indicator light color detection frame;
[0191] The recognition module 1403 is used to perform recognition processing on the indicator light target detection box and obtain the indicator light recognition result in the target traffic scene.
[0192] In one embodiment, the detection module 1401 includes:
[0193] The color extraction unit is used to extract colors from the image to be detected and obtain the indicator light area of the image to be detected.
[0194] The area expansion unit is used to expand the indicator light area according to the position distribution of the indicator lights and determine the indicator light color detection frame.
[0195] In one embodiment, the color extraction unit includes:
[0196] The boundary extraction subunit is used to extract the boundary of the image to be detected and determine the color bounding box in the image to be detected.
[0197] The region determination subunit is used to determine the indicator light area of the image to be detected based on the color bounding box.
[0198] In one embodiment, the boundary extraction subunit includes:
[0199] The mapping subunit is used to map each color region in the image to be detected onto the corresponding binary image;
[0200] The first acquisition subunit is used to acquire the connected components in each binary graph;
[0201] Extraction sub-units are used to extract the minimum regular bounding boxes of connected components in each binary graph, and the minimum regular bounding boxes are determined as the color bounding boxes of the image to be detected.
[0202] In one embodiment, the region determination subunit includes:
[0203] The first judgment subunit is used to determine the color boundary box as the indicator light area of the image to be detected if the area of the color boundary box is within a preset area range.
[0204] In one embodiment, the detection frame determination module 1402 includes:
[0205] The calculation unit is used to calculate the overlap between the indicator light area detection box and the indicator light color detection box.
[0206] The detection frame determination unit is used to determine the target detection frame for the indicator light based on the overlap of the regions.
[0207] In one embodiment, the detection box determination unit includes:
[0208] The second judgment subunit is used to determine the indicator light area detection box as the indicator light target detection box if the area overlap is greater than the preset overlap threshold.
[0209] The third judgment subunit is used to determine the indicator area detection box and the indicator color detection box as the indicator target detection box if the region overlap is less than or equal to the overlap threshold.
[0210] In one embodiment, the identification module 1403 includes:
[0211] The identification unit is used to identify the indicator light target detection box and determine the indicator light category of the indicator light target detection box;
[0212] The category determination unit is used to determine the category of the indicator lights in the image to be detected based on the indicator light category.
[0213] The recognition result determination unit is used to determine the recognition result of the indicator lights in the target traffic scene based on the indicator light category of the image to be detected and the indicator light category of the historical detected images.
[0214] In one embodiment, the category determination unit includes:
[0215] The second acquisition subunit is used to acquire the number of each indicator light category in the image to be detected;
[0216] The category determination subunit is used to determine the category of the most numerous indicator lights as the indicator light category of the image to be detected.
[0217] In one embodiment, the device 1400 further includes:
[0218] The first judgment unit is used to determine the indicator light category of the indicator light target detection box as the second color category if the indicator light category is the first color category; the first color category and the second color category meet the preset display conditions.
[0219] In one embodiment, the identification result determination unit includes:
[0220] The third acquisition subunit is used to acquire the number of indicator lights corresponding to each indicator light category based on the indicator light category of the image to be detected and the indicator light category of the historical detected images;
[0221] The identification result determination subunit is used to identify the indicator light category with the most indicator lights as the indicator light identification result in the target traffic scene.
[0222] Each module in the aforementioned traffic light recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.
[0223] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores traffic light recognition data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a traffic light recognition method.
[0224] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0225] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0226] The implementation principles and technical effects of each step in this embodiment are similar to those of the traffic light recognition method described above, and will not be repeated here.
[0227] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0228] The implementation principles and technical effects of each step in this embodiment when the computer program is executed by the processor are similar to those of the traffic light recognition method described above, and will not be repeated here.
[0229] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0230] The implementation principles and technical effects of each step in this embodiment when the computer program is executed by the processor are similar to those of the traffic light recognition method described above, and will not be repeated here.
[0231] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0232] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0233] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0234] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for recognizing traffic lights, characterized in that, The method includes: Based on the image to be detected of the target traffic scene, the area where the indicator lights are located in the image to be detected is detected to obtain the indicator light area detection box; and the color of the image to be detected is extracted to obtain the indicator light color detection box. The indicator light target detection box is determined based on the indicator light area detection box and the indicator light color detection box; The indicator light target detection box is processed for recognition to obtain the indicator light recognition result in the target traffic scene; The step of extracting colors from the image to be detected and obtaining the indicator light color detection frame includes: Color extraction is performed on the image to be detected to obtain the indicator light area of the image to be detected; Based on the location distribution of the indicator lights, the indicator light area is expanded to determine the indicator light color detection frame.
2. The method according to claim 1, characterized in that, The step of extracting colors from the image to be detected to obtain the indicator light area of the image to be detected includes: Boundary extraction is performed on the image to be detected to determine the color bounding boxes in the image to be detected; The indicator light area of the image to be detected is determined based on the color bounding box.
3. The method according to claim 2, characterized in that, The step of extracting the boundary of the image to be detected and determining the color bounding box in the image to be detected includes: Each color region in the image to be detected is mapped onto its corresponding binary image; Obtain the connected components in each of the binary graphs; Extract the minimum rule bounding box of each connected component in the binary graph, and determine the minimum rule bounding box as the color bounding box of the image to be detected.
4. The method according to claim 2, characterized in that, Determining the indicator light area of the image to be detected based on the color bounding box includes: If the area of the color bounding box is within a preset area range, then the color bounding box is determined as the indicator light area of the image to be detected.
5. The method according to claim 1, characterized in that, The step of determining the indicator light target detection box based on the indicator light area detection box and the indicator light color detection box includes: Calculate the overlap between the areas of the indicator light area detection frame and the indicator light color detection frame; The target detection frame for the indicator light is determined based on the overlap of the regions.
6. The method according to claim 5, characterized in that, The step of determining the indicator light target detection frame based on the region overlap includes: If the overlap of the regions is greater than a preset overlap threshold, then the indicator light region detection box is determined as the indicator light target detection box; If the overlap of the regions is less than or equal to the overlap threshold, then the indicator area detection box and the indicator color detection box are determined as the indicator target detection box.
7. The method according to claim 1, characterized in that, The step of recognizing the indicator light target detection box to obtain the indicator light recognition result in the target traffic scene includes: The indicator light target detection box is identified to determine the indicator light category of the indicator light target detection box; Based on the indicator light category, determine the indicator light category of the image to be detected; Based on the indicator light category of the image to be detected and the indicator light category of the historical detected images, the indicator light recognition result in the target traffic scene is determined.
8. The method according to claim 7, characterized in that, Determining the indicator light category of the image to be detected based on the indicator light category includes: Obtain the number of each indicator light category in the image to be detected; The category of indicator lights with the highest number of occurrences is determined as the indicator light category of the image to be detected.
9. The method according to claim 8, characterized in that, The method further includes: If the indicator light category is the first color category, then the indicator light category of the indicator light target detection box is determined to be the second color category; the first color category and the second color category meet the preset display conditions.
10. The method according to claim 7, characterized in that, The step of determining the indicator light recognition result in the target traffic scene based on the indicator light category of the image to be detected and the indicator light categories of historical detected images includes: Based on the indicator light category of the image to be detected and the indicator light category of the historical detected images, obtain the number of indicator lights corresponding to each indicator light category; The category of indicator lights with the highest number of indicator lights is determined as the indicator light recognition result in the target traffic scenario.
11. A traffic light recognition device, characterized in that, The device includes: The detection module is used to detect the area where the indicator lights are located in the image to be detected based on the image to be detected of the target traffic scene, and obtain the indicator light area detection box; and to extract the color of the image to be detected to obtain the indicator light color detection box. The detection frame determination module is used to determine the indicator target detection frame based on the indicator area detection frame and the indicator color detection frame; The recognition module is used to recognize the indicator light target detection box and obtain the indicator light recognition result in the target traffic scene; The detection module includes: The color extraction unit is used to extract colors from the image to be detected and obtain the indicator light area of the image to be detected. The area expansion unit is used to expand the indicator light area according to the position distribution of the indicator lights and determine the indicator light color detection frame.
12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.
14. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.