Traffic light recognition methods, computer equipment, storage media, and motor vehicles

Abstract

This invention discloses a traffic light recognition method, computer device, readable storage medium, and motor vehicle, relating to the field of autonomous driving technology. The method includes the following steps: acquiring predicted bounding boxes of traffic lights in the current frame image; classifying the predicted bounding boxes using a Refine model, and identifying the color, pattern, and countdown status of the traffic lights using a recognition model; filtering the predicted bounding boxes of traffic lights that are not at intersections in the current frame, and combining and outputting the retained traffic lights. This invention only requires image information acquired by a front-mounted vehicle camera to achieve traffic light recognition in any scene and at any time.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a traffic light recognition method, computer equipment, readable storage medium, and motor vehicle. Background Technology

[0002] Autonomous driving technology is developing rapidly. During operation, autonomous driving systems need to recognize traffic lights and make driving decisions based on these signals and traffic regulations. Current technologies involve the following process for traffic light recognition: acquiring current image information using an onboard front-facing camera, matching the image with a high-precision map in the vehicle's coordinate system, obtaining the approximate location and pattern information of the traffic lights from the map, and using a neural network to recognize the colors. However, current solutions rely heavily on high-precision maps, which are costly to acquire, have low update frequencies, and are subject to numerous limitations, making it impossible to recognize traffic lights in any scenario or at any time. Furthermore, most current traffic light recognition schemes only recognize color information and not the arrow light patterns, thus limiting the recognition of left turns, U-turns, right turns, and straight-ahead maneuvers. Summary of the Invention

[0003] This invention aims to address one of the technical problems in related technologies to a certain extent. To this end, this invention provides a traffic light recognition method that only requires image information acquired by a front-mounted vehicle camera to achieve traffic light recognition in any scenario and at any time.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A traffic light recognition method includes the following steps:

[0006] Obtain the predicted bounding boxes of traffic lights in the current frame image;

[0007] The Refine model classifies the predicted bounding boxes, and the recognition model identifies the color, pattern, and countdown status of the traffic lights.

[0008] Filter the predicted bounding boxes of traffic lights that are not at intersections in the current frame, and combine and output the remaining traffic lights.

[0009] Optionally, the Refine model includes several sequentially connected convolutional blocks, each convolutional block including a convolutional layer, a normalization layer, and an activation layer; the last convolutional block connects the first output layer, the second output layer, and the third output layer. The first output layer is used to output the predicted boxes into red traffic lights for motor vehicles, pedestrian crossing lights, and non-motor vehicle traffic lights. The second output layer is used to divide the traffic lights into horizontal light groups, vertical light groups, and square light groups. The third output layer is used to correct the position of the predicted boxes.

[0010] Optionally, the Refine model has at least 5 convolutional blocks. The first convolutional block has 8 output channels, a size of 5 pixels * 5 pixels, and a stride of 2 pixels. The other convolutional blocks have 16 output channels, a size of 3 pixels * 3 pixels, and a stride of 2 pixels. The first output layer has a convolutional kernel size of 4 pixels * 4 pixels and 3 output channels. The second output layer has a convolutional kernel size of 4 pixels * 4 pixels and 3 output channels. The third output layer has a convolutional kernel size of 4 pixels * 4 pixels and 4 output channels.

[0011] Optionally, the position of the prediction box can be corrected using the following formula:

[0012] X_center'=X_center+a*w_img

[0013] Y_center'=Y_center+b*h_img

[0014] W'=(c+1)*W

[0015] H'=(d+1)*H

[0016] Where X_center is the X-axis coordinate of the center point of the predicted bounding box, X_center' is the corrected X-axis coordinate of the center point of the predicted bounding box, a is the correction amount of the X-axis coordinate of the center point of the predicted bounding box, Y_center is the Y-axis coordinate of the center point of the predicted bounding box, Y_center' is the corrected Y-axis coordinate of the center point of the predicted bounding box, b is the correction amount of the Y-axis coordinate of the center point of the predicted bounding box, w_img is the width of the predicted bounding box, h_img is the height of the predicted bounding box, W is the width of the traffic light, W' is the corrected width of the traffic light, c is the correction amount of the width of the traffic light, H is the height of the traffic light, H' is the corrected height of the traffic light, and d is the correction amount of the height of the traffic light.

[0017] Optionally, when the Refine model classifies the predicted bounding boxes, the length and width of the predicted bounding boxes are first expanded to 128 pixels * 128 pixels before being input into the first convolutional block of the Refine model.

[0018] Optionally, the Refine model for classifying the predicted bounding boxes is a trained Refine model. The data used to train the Refine model consists of images that have been cleaned and labeled with ground truth values, and the length and width of the predicted bounding boxes of traffic lights have been randomly enlarged by a factor of 1-1.2. The regression loss function used to train the Refine model is the smooth L1 loss function, and the classification loss function is the focal loss function.

[0019] Optionally, the recognition model includes a recognition model for horizontal traffic lights and a recognition model for vertical traffic lights. The image size of the input image for the vertical traffic light recognition model is 32 pixels * 64 pixels, and the image size of the input image for the horizontal traffic light recognition model is 64 pixels * 32 pixels. The recognition model includes at least 5 convolutional blocks, each with 32 output channels. Each convolutional block includes one convolutional layer, one normalization layer, and one activation layer. Following the data flow order, for the vertical traffic light recognition model, the output layer size of the first convolutional block is 32 channels * 56 channels * 1 pixel * 8 pixels; for the horizontal traffic light recognition model, the output layer size of the first convolutional block is 32 channels * 56 channels * 8 pixels * 1 pixel. The second and fourth convolutional blocks are downsampled by 1 / 2. The output of the recognition model is Batch size * 56 * 1 * 16, where batch size... size represents the number of traffic lights input, 56 = 3*(10+8)+1(black)+1(blank), and 16 represents the longest bulb of the traffic lights that can be recognized.

[0020] Optionally, the trained recognition model identifies the color, pattern, and countdown status of traffic lights. When training the recognition model, the loss function is the CTC Loss function.

[0021] Optionally, the detection model used to obtain the predicted bounding box of the traffic light in the current frame includes an encoding module, a decoding module, and a detection head connected in sequence. The encoding module includes 5 layers, each layer is formed by stacking several residual networks. Each residual network includes a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, a BatchNorm layer, and a ReLU activation layer. The last convolutional block of each layer performs a 1 / 2 downsampling operation on the output features. The decoding module is an FPN network, including a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, and an upsampled 2-pixel * 2-pixel deconvolutional layer. The detection head is an FCOS network, including 4 layers of 3-pixel * 3-pixel convolutional layers and a classification and regression layer.

[0022] Optionally, the detection model is a trained detection model. When training the detection model, the data used is the data after cleaning and labeling the collected images with ground truth values. The regression loss function is the GIOU loss function, and the classification loss function is the focal loss function.

[0023] Optionally, before the Refine model classifies the predicted bounding boxes, the predicted bounding boxes are corrected using historical trajectories, including the following steps:

[0024] Expand the length and width of the detection boxes in the historical trajectory by at least 20%, expand the length and width of the prediction boxes in the current frame by at least 20%, and then perform IoU matching between the two.

[0025] For prediction boxes that fail to match, rematch is performed according to the following conditions: the prediction box of the current frame overlaps with the prediction box of the previous frame, and the length and width of the prediction box of the previous frame change by no more than 10% relative to the length and width of the prediction box with overlapping area in the previous frame; otherwise, it is considered a mismatch.

[0026] Unmatched prediction boxes are updated into the historical trajectory, and no more than 30 new trajectories are retained.

[0027] Optionally, for retained traffic lights, those meeting the following condition are considered to be in the same pole group, where the pole group consists of all traffic lights on a single traffic light pole at an intersection:

[0028] The difference between the length and width of the current traffic light's prediction frame is less than 15% of the prediction frame of the matched traffic light, and the difference in the coordinates of the center point is less than 5%.

[0029] The pixel distance between traffic lights in the same pole group is positively correlated with the prediction box size. The pixel distance of the current traffic light is less than twice the pixel distance of the longer side of the prediction box of the matched traffic light.

[0030] Optionally, the retained traffic lights can be combined and output after timing correction. Timing correction includes:

[0031] The red, yellow, and green colors change in sequence; the countdown timer is decreasing.

[0032] When a certain pattern of a certain color is displayed continuously on a traffic signal light, no other colors and / or patterns appear.

[0033] Furthermore, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the traffic light recognition method described in any one of the above claims.

[0034] Meanwhile, a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the traffic light recognition method described in any one of the above claims.

[0035] In addition, the present invention also provides a motor vehicle having a navigation-assisted cruise function or an automatic driving function, wherein when the motor vehicle is operating the navigation-assisted cruise function or the automatic driving function, the traffic light is identified by the traffic light recognition method described in any one of the above.

[0036] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0037] The present invention will be further described below with reference to the accompanying drawings:

[0038] Figure 1 A flowchart illustrating one embodiment of the traffic light recognition method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the detection model in the traffic light recognition method provided by the present invention;

[0040] Figure 3 This is a schematic diagram of step S120 in the traffic light recognition method provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the Refine module in the traffic light recognition method provided by the present invention;

[0042] Figure 5 A schematic diagram of the electronic device provided by the present invention;

[0043] Figure 6 This is a schematic diagram of a computer-readable medium provided by the present invention. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the present invention.

[0045] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this patent. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0046] As a first aspect of the present invention, a traffic signal light recognition method is provided, wherein, as Figure 1 As shown, the methods for determining driving status include:

[0047] In step S110, the predicted bounding box of the traffic light in the current frame image is obtained;

[0048] In step S120, the Refine model classifies the predicted bounding boxes, and the recognition model identifies the color, pattern, and countdown status of the traffic lights.

[0049] In step S130, the predicted bounding boxes of traffic lights that are not at intersections in the current frame are filtered, and the retained traffic lights are combined and output.

[0050] The in-vehicle front camera is a basic configuration for vehicles with autonomous driving functions. In this disclosure, based on this basic configuration, traffic lights are initially identified through image recognition detection. Then, the identified traffic lights are classified through the Refine model. The recognition model further identifies different colors of traffic lights, such as red, yellow, and yellow, as well as circular signals, arrow signals, etc. Finally, the complete recognition results are presented by combining the images, without relying on high-precision maps.

[0051] In this disclosure, no special limitations are made on how step S110 is performed. As an optional implementation, such as... Figure 2 As shown, in step S110, the detection model obtains the predicted bounding boxes of traffic lights in the current frame. The input to the detection model is the image captured by the vehicle's front-facing camera. The predicted bounding box contains (xmin, ymin, xmax, ymax) information, that is, the coordinates of the top-left and bottom-right corners of the box in the image coordinate system. The detection model includes an encoding module, a decoding module, and a detection head connected in sequence. The encoding module includes 5 layers, each layer is formed by stacking several residual networks. Each residual network includes a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, a BatchNorm layer, and a ReLU activation layer. The last convolutional block of each layer performs a 1 / 2 downsampling operation on the output features. The decoding module is an FPN network, including a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, and an upsampled 2-pixel * 2-pixel deconvolutional layer. The detection head is an FCOS network, including 4 layers of 3-pixel * 3-pixel convolutional layers and a classification and regression layer. After decoding and NMS (non-maximum suppression), the predicted bounding box is output in the form of (xmin, ymin, xmax, ymax), which corresponds to the coordinates of the top left and bottom right corners of the box.

[0052] Before deployment, the detection model needs to be trained using a standard training method common in this field, which will not be elaborated upon here. The training data consists of cleaned and ground truth labeled images. Specifically, the images are cleaned by removing invalid or static images. The remaining images are then labeled with ground truth values, which include the image coordinates of the traffic lights and the category attributes of the light groups: motor vehicle traffic lights, non-motor vehicle traffic lights, and pedestrian crossing traffic lights. The labels also include the recognition status, specifically pattern information, color status, and countdown information. The regression loss function is the GIOU loss function, and the classification loss function is the focal loss function.

[0053] In this disclosure, no special limitation is made on how step S120 is performed. As an optional implementation, the predicted bounding box is corrected using historical trajectories before step S120. Due to vehicle bumps or other reasons, the position of the traffic light in the previous frame may change from the position in the current frame, and there may also be frame drops from the vehicle's front-facing camera. Therefore, to ensure data accuracy, the predicted bounding box needs to be corrected. The correction includes the following steps:

[0054] Expand the length and width of the detection boxes in the historical trajectory by at least 20%, expand the length and width of the prediction boxes in the current frame by at least 20%, and then perform IoU matching between the two.

[0055] For prediction boxes that fail to match, rematch is performed according to the following conditions: the prediction box of the current frame overlaps with the prediction box of the previous frame, and the length and width of the prediction box of the previous frame change by no more than 10% relative to the length and width of the prediction box with overlapping area in the previous frame; otherwise, it is considered a mismatch.

[0056] Unmatched prediction boxes are updated into the historical trajectory, and no more than 30 new trajectories are retained.

[0057] In this disclosure, no special limitations are made on how step S120 is performed. As an optional implementation, after the correction of the prediction box is completed, as follows: Figure 3 As shown, the position of the prediction box is further corrected using the Refine module with the following formula:

[0058] X_center'=X_center+a*w_img

[0059] Y_center'=Y_center+b*h_img

[0060] W'=(c+1)*W

[0061] H'=(d+1)*H

[0062] Where X_center is the X-axis coordinate of the center point of the predicted bounding box, X_center' is the corrected X-axis coordinate of the center point of the predicted bounding box, a is the correction amount of the X-axis coordinate of the center point of the predicted bounding box, Y_center is the Y-axis coordinate of the center point of the predicted bounding box, Y_center' is the corrected Y-axis coordinate of the center point of the predicted bounding box, b is the correction amount of the Y-axis coordinate of the center point of the predicted bounding box, w_img is the width of the predicted bounding box, h_img is the height of the predicted bounding box, W is the width of the traffic light, W' is the corrected width of the traffic light, c is the correction amount of the width of the traffic light, H is the height of the traffic light, H' is the corrected height of the traffic light, and d is the correction amount of the height of the traffic light.

[0063] After further refinement, the predicted bounding boxes were expanded to 128 pixels x 128 pixels each before being input into the Refine model, resulting in more accurate predicted bounding boxes, category information, and horizontal / vertical state classification. Specifically, the expanded bounding boxes were input into the first convolutional block of the Refine model. Figure 4 As shown in this disclosure, the Refine model includes at least five sequentially connected convolutional blocks, preferably five in this embodiment. Each convolutional block includes a convolutional layer, a normalization layer, and an activation layer. The first convolutional block has eight output channels, a size of 5 pixels * 5 pixels, and a stride of 2 pixels. The other convolutional blocks have 16 output channels, a size of 3 pixels * 3 pixels, and a stride of 2 pixels. The last convolutional block connects the first output layer, the second output layer, and the third output layer. The first output layer has a convolutional kernel size of 4 pixels * 4 pixels and three output channels, used to output the predicted bounding boxes into red traffic lights for motor vehicles, pedestrian crossing lights, and non-motor vehicle traffic lights. The second output layer has a convolutional kernel size of 4 pixels * 4 pixels and three output channels, used to output the traffic lights into horizontal light groups, vertical light groups, and square light groups. The third output layer has a convolutional kernel size of 4 pixels * 4 pixels and four output channels, used to correct the position of the predicted bounding boxes.

[0064] In this disclosure, the Refine model requires training before deployment. The SGD optimizer is used for training, and the data used consists of images that have been cleaned and labeled with ground truth values. The predicted bounding boxes of traffic lights are then randomly enlarged by a factor of 1-1.2 to simulate the offset of the predicted bounding boxes around the ground truth boxes in the first stage. Image cleaning involves removing invalid and static images. The remaining images are then labeled with ground truth values, including the image coordinates of the traffic lights and the category attributes of the light groups. Specifically, the category attributes are for motor vehicle traffic lights, non-motor vehicle traffic lights, and pedestrian crossing traffic lights, as well as horizontal and vertical state classification branches. The recognition state is also included, specifically pattern information, color status, and countdown information. The training method is a common method in the field and will not be elaborated here. The regression loss function used to train the Refine model is the smooth L1 loss function, and the classification loss function is the focal loss function.

[0065] After the predicted bounding boxes are classified by Refine, the recognition model identifies the color, pattern, and countdown status of the traffic lights. In this disclosure, the recognition model is divided into two types: a horizontal traffic light recognition model and a vertical traffic light recognition model. The image size input to the vertical traffic light recognition model is 32 pixels * 64 pixels, with a width of 32 pixels and a height of 64 pixels. The image size input to the horizontal traffic light recognition model is 64 pixels * 32 pixels, with a width of 64 pixels and a height of 32 pixels. The network structures of the two models are identical, both including at least 5 convolutional blocks, each with 32 output channels. Following the data flow order, the output layer size of the first convolutional block in the vertical traffic light recognition model is 32 channels * 56 channels * 1 pixel * 8 pixels, and the output layer size of the first convolutional block in the horizontal traffic light recognition model is 32 channels * 56 channels * 8 pixels * 1 pixel. The second and fourth convolutional blocks are downsampled by 1 / 2, i.e., stride = 2. The output of the recognition model is Batch size * 56 * 1 * 16, where batch size is the number of input traffic lights, 56 = 3 * (10 + 8) + 1(black) + 1(blank), and 16 is the longest bulb of the traffic lights that can be recognized. In the above formula, 3 represents the three colors of traffic lights: red, yellow, and green; 10 represents the ten Arabic numerals from 0 to 9; 8 represents the eight patterns: straight arrow light, left turn arrow light, right turn arrow light, U-turn arrow light, straight arrow light with left turn arrow light, straight arrow light with right turn arrow light, left turn arrow light with U-turn arrow light, and circular light; 1(black) represents the state where all the bulbs of the traffic light are off; and 1(blank) represents the state where the color is unknown.

[0066] Before the recognition model can be put into use, it needs to be trained. The training method is the general training method in this field, which will not be elaborated here. The loss function is the CTC Loss loss function.

[0067] In this disclosure, no special limitations are made on how step S130 is performed. As an optional implementation, for the retained traffic lights, traffic lights belonging to the same pole group are selected if they meet the following condition: [The pole group refers to all traffic lights on a single traffic light pole at an intersection].

[0068] The difference between the length and width of the current traffic light's prediction frame is less than 15% of the prediction frame of the matched traffic light, and the difference in the coordinates of the center point is less than 5%.

[0069] The pixel distance between traffic lights in the same pole group is positively correlated with the size of the prediction box. The larger the prediction box, the closer it is to the vehicle. At the same time, the horizontal pixel distance between traffic lights is greater. The pixel distance of the current traffic light is less than twice the pixel distance of the longer side of the prediction box of the matched traffic light.

[0070] As another optional combination condition, the rod group is generally located in the center of the image.

[0071] Typically, one pole group is used for traffic control at each intersection. After combining traffic lights into pole groups based on the above logic, one pole group is selected for recognition. The recognition information comes from the recognition module, and the final output is the status of U-turn, left turn, straight ahead, right turn, and countdown.

[0072] As another optional combination condition, after combination, the output is processed through temporal correction. The temporal correction module stores recognition information from several historical frames. Based on the recognition result of the current frame, it corrects the recognition result of the current frame using prior knowledge, including:

[0073] The order of change for red, yellow, and green. For example, red cannot turn into yellow, and yellow cannot turn into green, etc.

[0074] The countdown is decreasing. Based on prior knowledge, the countdown can only be decreasing.

[0075] When a certain color and a certain pattern on a traffic light are in effect, no other colors and / or patterns should appear. For example, while a left-turn arrow light remains red, there should be no situation where the left-turn arrow light is red in the previous frame, green in the current frame, and then turns back to red in the next frame.

[0076] Meanwhile, this embodiment also provides

[0077] An electronic device, such as Figure 5 As shown, it includes:

[0078] One or more processors 101;

[0079] The memory 102 stores one or more computer programs that, when executed by the one or more processors 101, cause the one or more processors 101 to implement the driving state determination method according to the first aspect of the invention.

[0080] The electronic device may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.

[0081] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the first memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the processor and the memory, including but not limited to a data bus (Bus).

[0082] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0083] As a third aspect of the present invention, a computer-readable medium, such as... Figure 6 As shown, a computer program is stored thereon, which, when executed by a processor, implements the driving state determination method provided in the first aspect of this disclosure.

[0084] As a fourth aspect of the present invention, this embodiment also provides a motor vehicle having a navigation-assisted cruise function or an autonomous driving function. When the motor vehicle provided in this embodiment operates the navigation-assisted cruise function or the autonomous driving function, it identifies traffic lights according to the aforementioned traffic light recognition method.

[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0086] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A traffic light recognition method, characterized in that, The traffic light recognition method includes the following steps: Obtain the predicted bounding boxes of traffic lights in the current frame image; Historical trajectories are used to correct the predicted bounding boxes. After the predicted bounding boxes are corrected, the Refine model classifies and corrects their positions. The recognition model identifies the color, pattern, and countdown status of the traffic lights based on the horizontal and vertical status classification of the light group, the category attributes, and the position-corrected predicted bounding boxes output by the Refine model. The category attributes include at least one of motor vehicle traffic lights, non-motor vehicle traffic lights, and pedestrian crossing traffic lights. Filter the predicted bounding boxes of traffic lights that are not at intersections in the current frame, and combine and output the retained traffic lights; wherein, for the retained traffic lights, traffic lights that meet the following condition are traffic lights of the same pole group, and the pole group is all traffic lights on a traffic light pole at an intersection: The difference between the length and width of the current traffic light's prediction frame is less than 15% of the prediction frame of the matched traffic light, and the difference in the coordinates of the center point is less than 5%. The pixel distance between traffic lights in the same pole group is positively correlated with the prediction box size. The pixel distance of the current traffic light is less than twice the pixel distance of the longer side of the prediction box of the matched traffic light.

2. The traffic light recognition method according to claim 1, characterized in that, The Refine model includes several sequentially connected convolutional blocks. Each convolutional block includes a convolutional layer, a normalization layer, and an activation layer. The last convolutional block connects the first output layer, the second output layer, and the third output layer. The first output layer is used to output the predicted boxes into red traffic lights for motor vehicles, pedestrian crossing lights, and non-motor vehicle traffic lights. The second output layer is used to divide the traffic lights into horizontal light groups, vertical light groups, and square light groups. The third output layer is used to correct the position of the predicted boxes.

3. The traffic signal light recognition method according to claim 2, characterized in that, The Refine model has at least 5 convolutional blocks. The first convolutional block has 8 output channels, a size of 5 pixels * 5 pixels, and a stride of 2 pixels. The other convolutional blocks have 16 output channels, a size of 3 pixels * 3 pixels, and a stride of 2 pixels. The first output layer has a convolutional kernel size of 4 pixels * 4 pixels and 3 output channels. The second output layer has a convolutional kernel size of 4 pixels * 4 pixels and 3 output channels. The third output layer has a convolutional kernel size of 4 pixels * 4 pixels and 4 output channels.

4. The traffic signal light recognition method according to claim 2, characterized in that, The position of the prediction box is corrected using the following formula: X_center'=X_center+a*w_img Y_center'=Y_center+b*h_img W'=(c+1)*W H'=(d+1)*H Where X_center is the X-axis coordinate of the center point of the predicted bounding box, X_center' is the corrected X-axis coordinate of the center point of the predicted bounding box, a is the correction amount of the X-axis coordinate of the center point of the predicted bounding box, Y_center is the Y-axis coordinate of the center point of the predicted bounding box, Y_center' is the corrected Y-axis coordinate of the center point of the predicted bounding box, b is the correction amount of the Y-axis coordinate of the center point of the predicted bounding box, w_img is the width of the predicted bounding box, h_img is the height of the predicted bounding box, W is the width of the traffic light, W' is the corrected width of the traffic light, c is the correction amount of the width of the traffic light, H is the height of the traffic light, H' is the corrected height of the traffic light, and d is the correction amount of the height of the traffic light.

5. The traffic signal light recognition method according to claim 2, characterized in that, When the Refine model classifies the predicted bounding boxes, the length and width of the predicted bounding boxes are first expanded to 128 pixels * 128 pixels before being input into the first convolutional block of the Refine model.

6. The traffic signal light recognition method according to claim 2, characterized in that, The Refine model used to classify the predicted bounding boxes is a trained Refine model. The data used to train the Refine model consists of images that have been cleaned and labeled with ground truth values, and the length and width of the predicted bounding boxes of traffic lights have been randomly enlarged by a factor of 1-1.

2. The regression loss function used to train the Refine model is the smooth L1 loss function, and the classification loss function is the focal loss function.

7. The traffic signal light recognition method according to any one of claims 1 to 6, characterized in that, The recognition model includes recognition models for horizontal and vertical traffic lights. The image size input to the vertical traffic light recognition model is 32 pixels * 64 pixels, and the image size input to the horizontal traffic light recognition model is 64 pixels * 32 pixels. The recognition model includes at least 5 convolutional blocks, each with 32 output channels. Each convolutional block includes one convolutional layer, one normalization layer, and one activation layer. Following the data flow order, the output layer size of the first convolutional block in the vertical traffic light recognition model is 32 channels * 56 channels * 1 pixel * 8 pixels, and the output layer size of the first convolutional block in the horizontal traffic light recognition model is also 32 channels * 56 channels * 8 pixels * 1 pixel. The second and fourth convolutional blocks are downsampled by 1 / 2. The output of the recognition model is Batch size * 56 * 1 * 16, where batch size... size represents the number of traffic lights input, 56 = 3*(10+8)+1(black)+1(blank), and 16 represents the longest bulb of the traffic lights that can be recognized.

8. The traffic signal light recognition method according to any one of claims 1 to 6, characterized in that, The trained recognition model identifies the color, pattern, and countdown status of traffic lights. The loss function used during the training of the recognition model is the CTC Loss function.

9. The traffic signal light recognition method according to any one of claims 1 to 6, characterized in that, The detection model used to obtain the prediction bounding box of the traffic light in the current frame includes an encoding module, a decoding module, and a detection head connected in sequence. The encoding module includes 5 layers, each layer is formed by stacking several residual networks. Each residual network includes a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, a BatchNorm layer, and a ReLU activation layer. The last convolutional block of each layer performs a 1 / 2 downsampling operation on the output features. The decoding module is an FPN network, including a 1-pixel * 1-pixel convolutional layer, a 3-pixel * 3-pixel convolutional layer, and an upsampled 2-pixel * 2-pixel deconvolutional layer. The detection head is an FCOS network, including 4 layers of 3-pixel * 3-pixel convolutional layers and a classification and regression layer.

10. The traffic signal light recognition method according to claim 9, characterized in that, The detection model is a trained detection model. When training the detection model, the data used is the collected images after cleaning and labeling with ground truth values. The regression loss function is the GIOU loss function, and the classification loss function is the focal loss function.

11. The traffic signal light recognition method according to any one of claims 1 to 6, characterized in that, Before the Refine model classifies the predicted bounding boxes, historical trajectories are used to refine the predicted boxes, including the following steps: Expand the length and width of the detection boxes in the historical trajectory by at least 20%, expand the length and width of the prediction boxes in the current frame by at least 20%, and then perform IoU matching between the two. For prediction boxes that fail to match, rematch is performed according to the following conditions: the prediction box of the current frame overlaps with the prediction box of the previous frame, and the length and width of the prediction box of the previous frame change by no more than 10% relative to the length and width of the prediction box with overlapping area in the previous frame; otherwise, it is considered a mismatch. Unmatched prediction boxes are updated into the historical trajectory, and no more than 30 new trajectories are retained.

12. The traffic signal light recognition method according to any one of claims 1 to 6, characterized in that, The retained traffic lights are combined and then output after timing correction. The timing correction includes: The red, yellow, and green colors change in sequence; the countdown timer is decreasing. When a certain pattern of a certain color is displayed continuously on a traffic signal light, no other colors and / or patterns appear.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the traffic light recognition method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the traffic light recognition method according to any one of claims 1 to 12.

15. A motor vehicle, characterized in that, The motor vehicle has a navigation-assisted cruise function or an automatic driving function. When the motor vehicle is operating the navigation-assisted cruise function or the automatic driving function, the traffic light is identified by the traffic light identification method according to any one of claims 1 to 12.

Images