Traffic signal light recognition method, electronic device, and storage medium
By combining dynamic frame capture and deep learning models with traffic light and sign detection, the problem of low traffic light recognition efficiency in existing technologies has been solved, achieving efficient traffic signal recognition in autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG LEAPMOTOR TECH CO LTD
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing traffic light detection methods rely on high-precision maps or complex image processing, which are inefficient and highly dependent, making it difficult to accurately identify traffic light status in autonomous driving.
Road images are acquired using dynamic frame capture, and traffic light and lane sign detection are combined. Regions of interest are determined by the confidence scores of traffic lights and lane signs. A deep learning model is then used for traffic light recognition, reducing the area to be recognized in the image.
Traffic signals are accurately identified when the conditions in the current frame image are met; otherwise, the region of interest is determined by detecting signs, which improves the efficiency of traffic light recognition and reduces the consumption of computing resources.
Smart Images
Figure CN116129398B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of traffic signal recognition technology, and in particular to a traffic signal recognition method, electronic device and storage medium. Background Technology
[0002] In the field of autonomous driving, the ability to accurately identify the status of traffic lights is crucial to the safety of autonomous vehicles; that is, automatic traffic light recognition plays a key role in the field of autonomous driving.
[0003] However, current traffic light detection methods either rely on high-precision maps or require complex image processing of road images, resulting in low efficiency and strong dependence. Summary of the Invention
[0004] This application provides a traffic light recognition method, electronic device, and storage medium. When the current frame of the road image meets preset conditions, the traffic light recognition method can accurately identify the traffic signal result. When the current frame of the road image does not meet the preset conditions, it uses sign detection to determine a small region of interest when the sign meets the conditions. Based on the region of interest, traffic light detection is performed on the next frame of the road image, thereby reducing the area to be recognized and improving recognition efficiency.
[0005] To address the aforementioned technical problems, this application provides a traffic light recognition method, comprising: acquiring a current frame road image and performing traffic light detection on the current frame road image; in response to the current detection result meeting a first preset requirement, performing signal recognition on the current frame road image to obtain a traffic signal result corresponding to the current frame road image; or, in response to the current traffic light detection result not meeting the first preset requirement, performing lane sign detection on the current frame road image to obtain a lane sign detection result; determining a region of interest in the next frame road image based on the lane sign detection result, and performing traffic light detection on the next frame road image based on the region of interest to obtain a next traffic light detection result; and in response to the next traffic light detection result meeting the first preset requirement, performing signal recognition on the next frame road image to obtain a traffic signal result corresponding to the next frame road image.
[0006] The process of detecting traffic lights in the current frame of the road image includes: using a detection model to detect traffic lights in the current frame of the road image and obtaining the current detection result, which includes the confidence level of the traffic lights.
[0007] Specifically, in response to the current detection result meeting the first preset requirement, signal recognition is performed on the current frame road image to obtain the traffic signal result corresponding to the current frame road image, including: in response to the traffic light confidence meeting the first preset requirement, determining the traffic light area image from the current frame road image; performing signal recognition on the traffic light area image to obtain the traffic signal result corresponding to the current frame road image.
[0008] The process of performing signal recognition on the traffic light area image to obtain the traffic signal result corresponding to the current frame road image includes: using a direction recognition model to recognize the traffic light area image to obtain the corresponding traffic light direction; and using a signal recognition model corresponding to the traffic light direction to perform signal recognition on the traffic light area image to obtain the traffic signal result corresponding to the current frame road image.
[0009] The signal recognition model includes an encoding network, a decoding network, and an output network. It uses a recognition model corresponding to the direction of the traffic lights to perform signal recognition on the traffic light area image and obtain the traffic signal result corresponding to the current frame road image. This includes: encoding the traffic light area image using the encoding network to obtain the corresponding encoded features; decoding the encoded features using the decoding network to obtain the corresponding decoded features; and recognizing the decoded features using the output network to obtain the traffic signal result corresponding to the current frame road image.
[0010] The encoding network consists of several sequential encoding blocks. Each encoding block includes a residual block, a multi-head attention layer, a convolutional layer, and a pooling layer. In the multi-head attention layer, different channels have different dependency weights on different positions.
[0011] The detection model and signal recognition model are trained using road training images. The labels of the road training images are composed of a two-dimensional array. The first column of the two-dimensional array includes red, yellow and green signals, and the first row includes U-turn, left turn, straight, right turn, pie chart and digital signals.
[0012] The process of determining the region of interest in the next frame of the road image based on the sign detection results includes: determining the region of interest in the next frame of the road image in response to the sign detection results meeting a second preset requirement; wherein the region of interest is obtained by expanding the sign region in the current frame image.
[0013] The next frame road image is the road image after a preset interval from the current frame road image.
[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned traffic light recognition method.
[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the above-mentioned traffic light recognition method.
[0016] The beneficial effects of this application are as follows: Unlike the prior art, the traffic light recognition method provided by this application can accurately identify traffic signal results when the current frame of the road image meets the preset conditions, and when the current frame of the road image does not meet the preset conditions, it can determine a small region of interest by means of sign detection when the sign meets the conditions, and then perform traffic light detection on the next frame of the road image based on the region of interest, thereby reducing the area of image recognition and improving recognition efficiency. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:
[0018] Figure 1 This is a flowchart illustrating the first embodiment of the traffic light recognition method provided in this application;
[0019] Figure 2 This is a schematic diagram of an embodiment of the traffic light detection model provided in this application;
[0020] Figure 3 This is a schematic diagram of an embodiment of the relationship between road images provided in this application;
[0021] Figure 4 This is a flowchart illustrating the second embodiment of the traffic light recognition method provided in this application;
[0022] Figure 5 This is a schematic diagram of an embodiment of the signal recognition model provided in this application;
[0023] Figure 6 This is a schematic diagram of an embodiment of the multi-head attention layer provided in this application;
[0024] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;
[0025] Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation
[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0027] Traffic light detection falls under the category of small object detection. Traffic lights, such as red and green lights, occupy relatively small pixels in an image, and feature extraction from them is limited. Currently, there are three main methods for traffic light detection: traditional vision-based methods, deep learning-based methods, and methods combining high-precision maps.
[0028] Traffic light detection methods based on traditional vision approaches can be categorized into color- and edge-information-based methods and background suppression-based methods. Color- and edge-information-based methods primarily involve four steps: image acquisition; image preprocessing, converting the RGB image to the CIE Lab color space to enhance the red-green color difference, followed by hole filling; candidate region detection; and candidate region verification, including spatiotemporal persistence verification. Background suppression-based methods comprise two modules: a candidate region extraction module, which uses adaptive background suppression to highlight the foreground to obtain candidate boxes; and a recognition module, where the features of each candidate region are fed into a recognition network for identification. However, this method requires complex image processing before detection, making it complex and time-consuming, and unsuitable for the autonomous driving field.
[0029] The method of combining high-precision maps with traffic light detection works as follows: First, a detection algorithm determines the location and category of the traffic light in the image. Then, it is compared with the traffic lights recorded on the high-precision map. If a match is found, the world coordinates of the traffic light are obtained, thereby determining the corresponding road and helping traffic control make correct decisions. This method relies heavily on high-precision maps and is highly dependent on them.
[0030] Furthermore, traffic light detection differs from obstacle detection. Traffic light detection doesn't require real-time monitoring, so a fixed detection cycle is typically set. Traffic lights are usually LEDs. Firstly, LED flicker is an inherent phenomenon in digital imaging technology; the flicker captured is due to the exposure spanning the LED's AC drive brightness cycle. Secondly, traffic lights also flicker slightly when their color is about to change. When the sensor exposure time is very short, the traffic light may appear completely dark, though this darkness is not caused by the flickering of the color change. In most areas, the AC frequency is a fixed 50Hz. Setting the sensor's exposure time to an integer multiple of this, i.e., setting the sensor's operating frequency to 25fps or 50fps, should solve the flicker problem. However, in reality, AC frequencies vary by region, ranging from 50Hz to 60Hz. Additionally, sensors supplied by vendors may operate at 20fps, 25fps, or 30fps. Moreover, some traffic lights beyond their lifespan are still in use. Therefore, the frequency of flickering issues is relatively high.
[0031] To address any of the aforementioned technical problems, this application provides a novel traffic light recognition method, see reference. Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the traffic light recognition method provided in this application. The method includes:
[0032] Step 11: Obtain the current frame road image and perform traffic light detection on the current frame road image.
[0033] It is understandable that road images can be obtained through vehicle-mounted cameras, meaning that multiple frames of road images can be collected while the vehicle is in motion.
[0034] In some embodiments, a vehicle-mounted camera is used to acquire a forward-view image, and the forward-view image data is input into an on-board information processing system. When the target vehicle approaches an intersection, the information processing module needs to perform traffic light detection on the acquired forward-view image data to determine whether there are traffic lights at the intersection ahead. Traffic light detection is performed at intersections because intersections are usually equipped with traffic lights (such as red and green lights) and lane markings.
[0035] In some embodiments, a traffic light detection model can be used to detect traffic lights in the current frame of the road image, such as... Figure 2As shown, the traffic light detection model uses the FCOS (Fully Convolutional One-Stage Object Detection) algorithm. FCOS is a per-pixel, anchor-free one-stage object detection algorithm that can detect objects by regressing the distances from each position on the feature map to the target bounding box (top, bottom, left, and right). The backbone layer of the traffic light detection model is ResNet50, and the bottleneck layer is FPN (Feature Pyramid Networks). The prediction layer consists of two branches: a classification branch and a regression branch. The classification branch outputs two object classes: guide signs and traffic lights. The regression branch outputs four channels, i.e., it uses a regression form with unknown classes.
[0036] To avoid the presence of completely dark traffic light groups in the captured images due to flickering of traffic lights (LED lights), this application employs a dynamic frame-taking method to acquire the current frame of the road image. Dynamic frame-taking means that the next frame of the road image is the road image at a preset interval from the current frame, where the preset interval can be 1, 2, 3, or other values, and the preset interval can be random.
[0037] For example, if the preset interval is fixed at 1, the current frame of the road image is frame 1, the next frame of the road image is frame 3, and the next frame of the road image is frame 5.
[0038] For example, the current frame of the road image is frame 10, the next frame of the road image is frame 15, and the next frame of the road image is frame 17.
[0039] In some embodiments, when the data comes from different sensors, since the operating frequencies of different sensors are different, the traffic lights do not need to identify every frame. Instead, they can periodically select video frames, such as taking 1 frame in the 1st second, 2 frames in the 2nd second, 3 frames in the 3rd second, and so on. Assuming the sensor output is 30 FPS (Frames Per Second), the first frame of the 30 frames is taken in the 1st second, the 1st and 16th frames are taken in the 2nd second, and the 1st, 11th, and 21st frames are taken in the 3rd second. This allows for the input of as much rich data as possible without increasing computational performance excessively.
[0040] By using dynamic frame capture, we can reduce the number of image frames to be recognized, thereby reducing the resource consumption of the vehicle system. On the other hand, we can effectively improve the problem of traffic light flashing causing the inability to be recognized.
[0041] After detecting traffic lights in the current frame of the road image, it is necessary to determine whether the traffic light detection result corresponding to the current frame image meets the conditions. If the conditions are met, proceed to step 12; otherwise, proceed to steps 13 to 15.
[0042] Step 12: In response to the current detection result meeting the first preset requirement, perform signal recognition on the current frame road image to obtain the traffic signal result corresponding to the current frame road image.
[0043] In some embodiments, a confidence threshold can be used as a first preset condition. In response to a confidence level greater than or equal to the confidence threshold in the identified traffic lights in the road image, the current detection result is considered to meet the first preset condition, and then signal identification is performed on the traffic lights in the road image.
[0044] In some embodiments, when the current detection result meets a first preset condition, the current frame road image is input to the signal recognition module of the target vehicle, so that the signal recognition module can perform signal recognition on the current frame road image to obtain the traffic signal result corresponding to the current frame road image. The target vehicle can make driving preparations such as stopping, changing lanes, or moving forward based on the traffic signal result.
[0045] Understandably, traffic light detection is to confirm whether there are traffic lights in the road image, and signal recognition is to identify the traffic signals in the detected traffic lights, such as whether it is a red light, green light or yellow light, and whether it is a straight, left turn or right turn, etc.
[0046] When performing signal recognition, the acquired traffic signal results must meet the relevant requirements of the road traffic light setting and installation specifications. For example, if a red left-turn signal is illuminated, a green left-turn signal should not be illuminated simultaneously. In some embodiments, after acquiring the traffic signal results, the results can be re-evaluated according to traffic rules to determine whether the signal recognition results comply with the relevant traffic rules. If the signal recognition results comply with the relevant traffic rules, the target vehicle can prepare to stop, change lanes, or proceed based on the traffic signal results.
[0047] Step 13: In response to the current traffic light detection result not meeting the first preset requirement, lane sign detection is performed on the current frame road image to obtain the sign detection result.
[0048] In some embodiments, a confidence threshold can be used as a first preset condition. If the confidence level of a traffic light group in the road image is less than the threshold, the current detection result is considered not to meet the preset condition, i.e., no traffic light is detected in the current frame of the road image. Therefore, lane sign detection is required on the road image to obtain the lane sign detection result.
[0049] Lane sign detection involves determining whether lane signs exist in the current frame of the road image. Before a target vehicle approaches an intersection, there are usually corresponding lane signs indicating its presence. Lane sign detection can be performed on the road image to determine if lane signs are present in the current frame.
[0050] If the sign detection result shows that there is a lane sign in the current frame of the road image, then proceed to step 14.
[0051] Step 14: Determine the region of interest in the next frame of the road image based on the sign detection results, and perform traffic light detection on the next frame of the road image according to the region of interest to obtain the next traffic light detection results.
[0052] In some embodiments, in response to the sign detection result meeting a second preset requirement, a region of interest in the next frame of the road image is determined; wherein the region of interest is obtained by expanding the sign region in the current frame image.
[0053] In some embodiments, a confidence threshold for lane signage can be used as a second preset requirement. When the confidence level corresponding to the lane signage detection result is greater than or equal to the confidence threshold, it is considered that a lane signage exists in the current frame of the road image. At this point, a region of interest (ROI) can be determined based on the current frame of the road image, and the ROI determined from the current frame can be directly used as the ROI in the next frame of the road image. It is understood that the field of view of the vehicle-mounted camera is fixed; therefore, after determining the ROI, it is also a fixed area in the image, and can therefore be directly applied to the next frame of the road image.
[0054] If no traffic light is detected during traffic light detection in the current frame of the road image, it's necessary to determine if lane signs are detected in the current frame. If lane signs are detected but traffic lights are not, it indicates that the vehicle's location is too far from the traffic light. Referring to the design speeds of different road levels (e.g., expressways have design speeds of 100km / h, 80km / h, and 60km / h, while arterial roads have design speeds of 60km / h, 50km / h, and 40km / h), we can see that the position of lane signs in the road image doesn't change significantly within one second. Therefore, the lane sign area in the current frame can be expanded to obtain the corresponding region of interest (ROI). When the next frame of the road image is acquired, traffic light detection is performed based on the ROI to obtain the next traffic light detection result. It's understood that lane signs and traffic lights are usually on the same side; therefore, after expanding to determine the ROI, the traffic lights will be within the ROI in subsequently acquired road images. Therefore, traffic light detection can be performed on the next frame of the road image based on the region of interest to obtain the detection result of the next traffic light, thereby reducing the detection of other areas of the next frame of the road image and improving detection efficiency.
[0055] The detection area of the next frame of the road image is determined by the expanded area of the lane signs detected in the current frame of the road image. In some embodiments, such as... Figure 3 As shown, the current frame of the road image is A. When a lane sign area B is detected in A, B can be expanded outwards to obtain the region of interest C. After acquiring the next frame of the road image, the region of interest C can be directly determined in the next frame of the road image.
[0056] The expansion can be centered on the lane sign, simultaneously expanding it by a factor of N in the left, right, top, and bottom directions. This means expanding the lane sign's width by a factor of N in the left and right, and its height by a factor of N in the top and bottom, thus obtaining the region of interest. Alternatively, other expansion methods can be used, but the resulting region of interest must include the lane sign.
[0057] Step 15: In response to the next traffic light detection result meeting the first preset requirement, perform signal recognition on the next frame of the road image to obtain the traffic signal result corresponding to the next frame of the road image.
[0058] In some embodiments, the first preset condition is a threshold for the confidence level of the road image. In response to the next traffic light detection result meeting the first preset requirement, the confidence level of the traffic light group in the next frame of the road image is greater than or equal to the threshold, and then signal recognition is performed on the road image.
[0059] In some embodiments, in response to the next traffic light detection result not meeting the first preset requirement, the confidence level of the traffic light group in the next frame of the road image is less than the threshold, and lane sign detection can be performed on the next frame of the road image.
[0060] In this embodiment, when the current frame road image meets the preset conditions, the traffic signal result can be accurately identified. When the current frame road image does not meet the preset conditions, a small region of interest is determined by means of sign detection when the sign meets the conditions. Based on the region of interest, traffic light detection is performed on the next frame road image, reducing the area of image recognition and improving recognition efficiency.
[0061] See Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of the traffic light recognition method provided in this application. The method includes:
[0062] Step 41: Obtain the current frame road image, use the detection model to detect traffic lights in the current frame road image, and obtain the current detection result, which includes the traffic light confidence level.
[0063] In some embodiments, an image acquisition device is used to acquire a current frame of road image, wherein the image acquisition device may be a forward-looking telephoto camera. After acquiring the current frame of road image, a detection model can be used to detect traffic lights in the current frame of road image. The detection model is a traffic light detection model, which can be used to detect whether traffic lights exist in the image.
[0064] After detecting traffic lights in the current frame of the road image, it is necessary to determine whether the obtained traffic light detection results of the current frame meet the conditions. If the conditions are met, proceed to steps 42 to 43; otherwise, proceed to steps 44 to 46.
[0065] Specifically, the presence or absence of traffic lights in the current frame of the road image can be determined by comparing the confidence level of the current detection result with a preset threshold. For example, if the confidence level of the current detection result is greater than or equal to the preset threshold, it indicates that the current frame of the road image contains traffic lights, and step 42 can be executed. If the confidence level of the current detection result is less than the preset threshold, it indicates that the current frame of the road image does not contain traffic lights, and step 44 can be executed.
[0066] Step 42: In response to the traffic light confidence level meeting the first preset requirement, determine the traffic light area image from the current frame road image.
[0067] In some embodiments, a confidence threshold can be used as a first preset condition. When the confidence level of a traffic light group in a road image is greater than or equal to the threshold, the current detection result is considered to meet the preset condition, and the traffic light area image can then be determined from the current frame of the road image.
[0068] Step 43: Perform signal recognition on the traffic light area image to obtain the traffic signal results corresponding to the current frame of the road image.
[0069] In some embodiments, a direction recognition model can be used to identify the traffic light area image to obtain the corresponding traffic light direction. Then, a signal recognition model corresponding to the traffic light direction can be used to identify the signal in the traffic light area image to obtain the traffic signal result corresponding to the current frame road image.
[0070] The direction recognition model and signal recognition model used in this application are both deep learning models. The direction recognition model adopts an OCR (Optical Character Recognition) model to recognize text in images; that is, each bulb in a traffic light group can be interpreted as a type of colored text in the image. The aspect ratio can be used to determine whether the traffic light group is horizontal or vertical, and then separate independent signal recognition models are used for each.
[0071] For example, the lateral signal recognition model is based on the MASTER model in the form of "CNN + Transformer". Figure 5 As shown, the signal recognition model includes an encoding network, a decoding network, and an output network. The encoding network encodes the traffic light area image to obtain corresponding encoded features. Then, the decoding network decodes these encoded features to obtain corresponding decoded features. Finally, the output network recognizes the decoded features to obtain the traffic signal result corresponding to the current frame's road image. The encoding network consists of several sequential encoding blocks. Each encoding block includes a residual block, a multi-aspect attention layer (Multi-Aspect GCAttention, MAGC), a convolutional layer (conv), and a max pooling layer. In the multi-aspect attention layer, different channels have different weights for different locations. For example... Figure 6 As shown, in the C*1*1 features of the GCNet output, each channel depends on the same weights of size H and W. MAGC improves upon the original GCNet by adopting a multi-head mechanism so that the dependency weights of different channels on different positions are no longer the same.
[0072] Furthermore, the encoding network has three basic blocks, each consisting of three modules: one Masked Multi-HeadAttention, one Multi-Head Attention, and one feedforward network.
[0073] A signal recognition network is used to identify traffic signals in the traffic light area image to obtain the traffic signal result corresponding to the current frame of the road image. The traffic signal result is related to the attribute of the lit bulb in the traffic light group. For example, the traffic signal result is "green left turn + red straight + green right turn".
[0074] In addition, the detection model and signal recognition model are trained using road training images. The labels of the road training images are constructed in the form of a two-dimensional array, as shown in the table below. The first column of the two-dimensional array includes red signals, yellow signals, and green signals, and the first row of the two-dimensional array includes U-turn signals, left turn signals, straight signals, right turn signals, pie chart signals, and digital signals.
[0075] Two-dimensional array table:
[0076] U-turn Turn left straight Turn right round cake number red 1 0 0 0 0 0 yellow 0 0 1 0 0 0 green 0 0 0 1 0 0
[0077] The traffic lights / signal light groups used in this application can provide semantic layer information. By translating the labels of motor vehicle signal light groups into a two-dimensional array, when only focusing on the lit bulbs in the signal lights, each value in the two-dimensional array can only be 0 / 1, where 0 indicates non-existence and 1 indicates presence. Furthermore, each column can have at most one 1, and under the restriction of column information, the minimum number of 1s in each row is 0, and the maximum is 5. If a signal light group malfunctions, it is labeled as a "malfunctioning signal light group." As shown in the table above, the semantic information corresponding to the label information "red U-turn + yellow straight + green right turn" is "U-turn not allowed + straight-ahead time is about to end + right turn allowed." When the labeler completes the labeling of a signal light group, the labeling platform performs a logical check. In addition, the signal lights in the signal light groups of this application cannot be simply rotated into a vertical form. For example, if a horizontal signal light group is rotated 90° clockwise around its center point, and there is a left-turn arrow in the horizontal signal light group, it will become a straight-ahead arrow after rotation. Except for circular patterns, other patterns cannot be arbitrarily rotated 90°.
[0078] Before training, a sufficient number of road images need to be collected as road training images. For example, the data collection period should cover every time period of 24 hours. Specifically, for the collected video segments, the portion near intersections should be retained. The video should be saved as images at 5 frames per second before being sent for annotation. Annotation objects include lane signs and traffic lights. Lane signs are annotated with the minimum bounding rectangle for each individual sign; traffic light groups are annotated with the bounding rectangle of the group, and then filled with the information in the table above, with a default value of 0.
[0079] The detection and recognition models used in this application are both supervised neural network models, which can be trained by collecting a preset amount of training data, and then the trained models can be deployed and used.
[0080] In other embodiments, the detection and recognition models used in this application may also be unsupervised neural network models.
[0081] Step 44: In response to the current traffic light detection result not meeting the first preset requirement, lane sign detection is performed on the current frame road image to obtain the sign detection result.
[0082] Step 45: Determine the region of interest in the next frame of the road image based on the sign detection results, and perform traffic light detection on the next frame of the road image according to the region of interest to obtain the next traffic light detection results.
[0083] Step 46: In response to the next traffic light detection result meeting the first preset requirement, perform signal recognition on the next frame of the road image to obtain the traffic signal result corresponding to the next frame of the road image.
[0084] Steps 44 to 46 may have the same or similar technical features as any of the above embodiments, and will not be repeated here.
[0085] Unlike existing technologies, the traffic light recognition method provided in this application proposes a novel two-dimensional array for traffic light group attribute annotation. Furthermore, by dynamically capturing the current and next frame of the road image, it effectively addresses the problem of traffic light flickering. When the current frame of the road image meets preset conditions, it can accurately identify the traffic signal result. When the current frame does not meet the preset conditions, it uses sign detection. If the sign meets the conditions, it identifies a smaller region of interest, and then uses this region of interest to detect traffic lights in the next frame of the road image, reducing the area to be recognized and improving recognition efficiency.
[0086] See Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 70 includes a memory 701 and a processor 702. The memory 701 is used to store computer programs, and the processor 702 is used to execute the computer programs to implement the traffic light recognition method of any of the above embodiments, which will not be described in detail here.
[0087] Among them, electronic device 70 can be in-vehicle electronic device, which can be applied in the field of autonomous driving.
[0088] See Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 80 is used to store a computer program 801. When the computer program 801 is executed by a processor, it is used to implement the traffic light recognition method of any of the above embodiments, which will not be described in detail here.
[0089] In summary, the traffic light recognition method provided in this application addresses the issue of traffic light flickering through dynamic frame capture and proposes a novel method for annotating traffic light group attributes, which can improve traffic light recognition efficiency. When the current frame of the road image meets preset conditions, it can accurately identify traffic signal results. When the current frame does not meet the preset conditions, it uses sign detection to identify a smaller region of interest (ROI) when the sign meets the conditions. Based on this ROI, traffic light detection is performed on the next frame of the road image, reducing the area to be recognized and improving recognition efficiency.
[0090] The processor involved in this application may be referred to as a CPU (Central Processing Unit), which may be an integrated circuit chip, or a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.
[0091] The storage media used in this application include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), or optical discs.
[0092] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A traffic light recognition method, characterized in that, The method includes: Acquire the current frame road image and perform traffic light detection on the current frame road image; In response to the current detection result meeting the first preset requirement, it is determined that the traffic light is detected in the current frame road image, and signal recognition is performed on the current frame road image to obtain the traffic signal result corresponding to the current frame road image; In response to the current detection result not meeting the first preset requirement, it is determined that the traffic light was not detected in the current frame road image, and lane sign detection is performed on the current frame road image to obtain the lane sign detection result; and In response to the sign detection result meeting the second preset requirement, a region of interest (ROI) is determined in the next frame of the road image; wherein, the ROI is obtained by expanding the sign region in the current frame image; the expansion is centered on the lane sign, and is simultaneously expanded by N times to the left, right, top, and bottom; the lane sign and the traffic light are on the same side; the next frame of the road image is a road image after a preset interval from the current frame of the road image; the preset interval is a random interval; Based on the region of interest, traffic light detection is performed on the next frame of the road image to obtain the next traffic light detection result; In response to the next traffic light detection result satisfying the first preset requirement, it is determined that the traffic light is detected in the next frame of the road image, and signal recognition is performed on the next frame of the road image to obtain the traffic signal result corresponding to the next frame of the road image.
2. The method according to claim 1, characterized in that, The step of detecting traffic lights in the current frame of the road image includes: The current frame road image is used to detect traffic lights using a detection model to obtain the current detection result, wherein the current detection result includes traffic light confidence. The step of responding to the current detection result satisfying a first preset requirement, determining that the traffic light is detected in the current frame road image, performing signal recognition on the current frame road image, and obtaining the traffic signal result corresponding to the current frame road image includes: In response to the traffic light confidence level meeting a first preset requirement, it is determined that the traffic light is detected in the current frame road image, and the traffic light area image is determined from the current frame road image; The traffic signal results corresponding to the current frame road image are obtained by performing signal recognition on the traffic light area image.
3. The method according to claim 2, characterized in that, The step of performing signal recognition on the traffic light area image to obtain the traffic signal result corresponding to the current frame road image includes: The direction recognition model is used to identify the traffic light area image to obtain the corresponding traffic light direction; The traffic signal result corresponding to the current frame road image is obtained by using a signal recognition model corresponding to the direction of the traffic light to perform signal recognition on the traffic light area image.
4. The method according to claim 3, characterized in that, The signal recognition model includes an encoding network, a decoding network, and an output network. The step of using the recognition model corresponding to the direction of the traffic light to perform signal recognition on the traffic light area image to obtain the traffic signal result corresponding to the current frame road image includes: The signal light area image is encoded using the encoding network to obtain corresponding encoded features; The encoded features are decoded using the encoding network to obtain the corresponding decoded features; The output network is used to identify the decoded features to obtain the traffic signal result corresponding to the current frame road image.
5. The method according to claim 4, characterized in that, The encoding network includes a number of sequential encoding blocks. Each encoding block includes a residual block, a multi-head attention layer, a convolutional layer, and a pooling layer. In the multi-head attention layer, different channels have different dependency weights on different positions.
6. The method according to claim 3, characterized in that, The detection model and the signal recognition model are trained using road training images. The labels of the road training images are constructed in the form of a two-dimensional array. The first column of the two-dimensional array includes red signals, yellow signals, and green signals, and the first row of the two-dimensional array includes U-turn signals, left turn signals, straight signals, right turn signals, pie chart signals, and digital signals.
7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program and the processor being used to execute the computer program to implement the method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a processor, is used to implement the method as described in any one of claims 1-6.