Traffic light recognition method based on deep learning

By improving the yolov5 model, enhancing the small object recognition capability and feature extraction layer, the problem of inaccurate traffic light recognition in the yolov3 algorithm is solved, and high-precision and high-real-time traffic light recognition is achieved, which is suitable for unmanned driving technology.

CN113989781BActive Publication Date: 2024-11-01DAOJIANYOUXING (CHONGQING) TECH CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111291491.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-01
Publication Date
2024-11-01
Estimated Expiration
2041-11-01

AI Technical Summary

Technical Problem

The existing traffic light recognition algorithm based on yolov3 has problems with inaccurate detection and low confidence, which leads to vehicles that may make incorrect judgments, affecting autonomous driving and path planning.

Method used

Improve the yolov5 model and increase the ability to recognize small objects. By adding multi-scale feature extraction layer and feature fusion layer after the feature extraction layer, and adding a small object detection layer before the object detection layer, replacing the BottleNeckCSP module in the PANet architecture as the Ghostbottleneck module. ReLU function of GhostModule2 module is removed and filtered and denoised to improve recognition accuracy.

Benefits of technology

It improves the accuracy and real-time nature of traffic light recognition, enhances the ability to identify small targets, solves the problems of slow recognition speed and low accuracy in traditional recognition methods, and is suitable for unmanned driving technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989781B_ABST
    Figure CN113989781B_ABST
Patent Text Reader

Abstract

The present invention relates to a traffic light recognition method based on deep learning, including marking a traffic light picture set and training to obtain a traffic light model; increasing the small target recognition capability of a yolov5 model; enabling the yolov5 model to read the image of a camera in real time; filtering and denoising the image collected by the camera; adding the data set and the traffic light weight file in the trained traffic light model to an improved yolov5 model, identifying the real boundary frame of the traffic light from the image collected by the camera through the model, and outputting the recognition result. In the present invention, a traffic light recognition method based on yolov5 is adopted, and by improving yolov5, the problems of slow image recognition speed and low accuracy of traditional recognition methods are solved; and the recognition capability of small targets is enhanced, the target recognition accuracy is high, and the real-time performance is strong, which is very suitable for application in unmanned driving technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned driving and relates to a traffic light recognition method based on deep learning. Background Art

[0002] Traffic light recognition is an important part of autonomous driving. At present, the algorithm based on yolov3 has problems of inaccurate detection and low confidence when detecting traffic lights. Once the wrong boundary box of the traffic light is given, the vehicle will make a wrong judgment. In recent years, with the rise of autonomous driving research, the technology of accurate object recognition for vehicles has become the focus, especially the recognition of traffic lights is crucial; accurate recognition of traffic lights is of great significance for autonomous driving and path planning. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a traffic light recognition method based on deep learning.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A traffic light recognition method based on deep learning includes the following steps:

[0006] Step S1, obtaining and annotating a set of traffic light images, and obtaining a traffic light model capable of recognizing the shape and color of traffic lights through training;

[0007] Step S2, improving the yolov5 model to increase the small target recognition capability of the yolov5 model;

[0008] Step S3, adding the data set and weight file in the trained traffic light model to the improved yolov5 model to obtain a traffic light automatic recognition model, and training the traffic light automatic recognition model so that the traffic light automatic recognition model first recognizes the traffic light boundary box, and then recognizes the ID number corresponding to the traffic light from the traffic light boundary box;

[0009] Step S4, enabling the yolov5 model to read the image of the camera in real time;

[0010] Step S5: using the trained traffic light automatic recognition model to identify the real boundary box of the traffic light from the image captured by the camera, and output the recognition result.

[0011] Furthermore, in step S1, traffic lights are assigned ID numbers according to their shapes and colors. When annotating a set of traffic light images, a bounding box is firstly marked at the position of the traffic light on the traffic light image, and then the ID number corresponding to the shape and color of the traffic light in each bounding box is marked.

[0012] Furthermore, in step S2, improving the yolov5 model includes:

[0013] A multi-scale feature extraction layer and a feature fusion layer are added after the original feature extraction layer, and a small target detection layer is added before the target detection layer. The feature information output by the original feature extraction layer is sent to the multi-scale feature extraction layer and the feature fusion layer respectively. The multi-scale feature extraction layer is used to filter the feature information output by the original feature extraction layer and extract obvious feature information; the feature fusion layer is used to fuse the feature information output by the original feature extraction layer and the feature information output by the multi-scale feature extraction layer; the small target detection layer is used to enhance the fine-grained features of small targets by upsampling.

[0014] Furthermore, in step S2, improving the yolov5 model further includes:

[0015] In the PANet architecture of the yolov5 model, the BottleNeckCSP module is replaced with the Ghostbottleneck module, and the order of the submodules of the Ghostbottleneck module is set from front to back to GhostModule 1 module, depthwise module, SEModule module, GhostModule2 module, and Identity mapping module.

[0016] Furthermore, the ReLU function of the GhostModule2 module is removed.

[0017] Furthermore, in step S4, after reading the image captured by the camera, filtering and denoising are performed, and the filtering and denoising method is as follows:

[0018] First, the color image is converted into a grayscale image through grayscale image conversion, then binarization is performed, and finally Gaussian filtering is performed for denoising; the filter function selects FIR windowed filter PtByPt VI.

[0019] Furthermore, in step S3, the formula for model training is:

[0020] t x =(G x -P x )

[0021] t y =(G y -P y )

[0022] t w =log(G w / P w )

[0023] t h =log(G h / P h )

[0024] In the formula, G x The horizontal coordinate of the center point of the marked bounding box; G y The vertical coordinate of the center point of the labeled bounding box; G w Indicates the width of the bounding box of the annotation; G h represents the height of the bounding box of the annotation; P x represents the horizontal coordinate of the center point of the predicted bounding box; P y Represents the ordinate of the center point of the predicted bounding box; P w Indicates the width of the predicted bounding box; P h Represents the height of the predicted bounding box; t x Represents the lateral offset of the model training output; t y It represents the longitudinal offset of the model training output; tw represents the horizontal scaling scale of the model training output; th represents the vertical scaling scale of the model training output.

[0025] Furthermore, in step S5, the output recognition result includes the number of bounding boxes recognized in the image, the position information of the four corner points of each bounding box, the ID number corresponding to the traffic light information in each bounding box, and the confidence of each bounding box.

[0026] Furthermore, in step S5, the method for identifying the traffic light from the image captured by the camera is:

[0027] First, divide the traffic light image into SxS grids, then identify each grid in turn and predict the bounding box; then calculate the confidence of the bounding box, and filter out the predicted bounding box with the highest confidence for each traffic light based on the confidence of the predicted bounding box and non-maximum suppression; finally, identify the real bounding box of the traffic light and output the recognition result.

[0028] The calculation formula of the true bounding box is as follows:

[0029] b x =σ(t x )+c x

[0030] b y =σ(t y )+c y

[0031]

[0032]

[0033] Where b x Represents the horizontal coordinate of the center point of the real bounding box; b y represents the ordinate of the center point of the true bounding box; σ() represents the sigmoid function; c x Indicates a multiple of the grid width; b w Indicates the width of the true bounding box; b h Indicates the width of the ground-truth bounding box.

[0034] Furthermore, the confidence of the predicted bounding box is calculated as:

[0035]

[0036] Among them, D conf is the confidence of the predicted bounding box; P(o) is the probability of the detected object. If the center of the object falls within the grid, P(o) is 1, otherwise it is 0; is the intersection-over-union ratio of the predicted bounding box and the true bounding box.

[0037] In the present invention, a traffic light recognition method based on yolov5 is adopted, and by improving yolov5, the problems of slow image recognition speed and low accuracy of traditional recognition methods are solved; and the recognition ability of small targets is enhanced, the target recognition accuracy is high, the real-time performance is strong, and the operation is simple. The traffic light recognition method is very suitable for application in unmanned driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0039] Figure 1 This is a flowchart of a preferred embodiment of the traffic light recognition method based on deep learning of the present invention.

[0040] Figure 2 Structural block diagram after adding multi-scale feature extraction layer and feature fusion layer to the yolov5 model.

[0041] Figure 3 This is a schematic diagram of the improved PANet architecture of the yolov5 model.

[0042] Figure 4 This is a curve diagram of the MAP change of the training data set during the training process.

[0043] Figure 5 Schematic diagram of the predicted bounding box and the true bounding box. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] like Figure 1 As shown, a preferred embodiment of the traffic light recognition method based on deep learning includes the following steps:

[0046] Step S1, obtain and annotate a set of traffic light images, and use a convolutional neural network to train the annotated set of traffic light images to obtain a traffic light model that conforms to the real scene. The following is an example of traffic light recognition in the labview software environment.

[0047] First, in the LabVIEW software, build open camera.vi, read the external camera, and implement real-time image reading of yolov5 (face recognition model); then filter and denoise the traffic light pictures collected by the camera to obtain a clearer image. The method of filtering and denoising is: first, convert the color image into a grayscale image through grayscale image conversion, then perform binarization processing, and finally perform Gaussian filtering and denoising processing; select FIR windowedfilter PtByPt VI as the filtering function.

[0048] Afterwards, each traffic light information is divided into ID numbers. For example, ID number "1" can be used to represent a red light in the shape of a right turn, ID number "2" can be used to represent a red light in the shape of a left turn, ID number "3" can be used to represent a red light in the shape of a straight line, ID number "4" can be used to represent a green light in the shape of a right turn, ID number "5" can be used to represent a green light in the shape of a left turn, ID number "6" can be used to represent a green light in the shape of a straight line, ID number "7" can be used to represent a yellow light in the shape of a right turn, ID number "8" can be used to represent a yellow light in the shape of a left turn, and ID number "9" can be used to represent a yellow light in the shape of a straight line. Of course, other ID numbers can also be used to represent traffic lights of the above shapes and colors, and ID numbers representing traffic lights of other shapes can also be added (for example, an ID number can also be added to represent a "U-turn" traffic light). When annotating a set of traffic light images, first mark a bounding box at the position of the traffic light on the traffic light image, and then mark the ID number corresponding to the traffic light information in each bounding box (i.e., the information represented by the color and shape of the traffic light). Through training, a traffic light model is obtained that can recognize the shape and color of traffic lights and output the corresponding ID number.

[0049] The images collected for training are all from actual traffic images, which can improve the accuracy of identifying real traffic lights.

[0050] Step S2: Improve the yolov5 model to increase the small target recognition capability of the yolov5 model. The improvements to the yolov5 model include:

[0051] (1) Figure 2 As shown in the figure, a multi-scale feature extraction layer (SELayer) is added to the last layer of the original feature extraction layer (backbone), and a feature fusion layer is added accordingly; the feature information output by the original feature extraction layer is sent to the multi-scale feature extraction layer and the feature fusion layer respectively, the multi-scale feature extraction layer is used to filter the feature information output by the original feature extraction layer and extract obvious feature information; the feature fusion layer is used to fuse the feature information output by the original feature extraction layer and the feature information output by the multi-scale feature extraction layer. Its purpose is to extract multi-scale spatial information and perform data fusion on all channels of the input feature layer.

[0052] (2) Add a small target detection layer before the target detection layer; the small target detection layer is used to enhance the fine-grained features of small targets through upsampling. Use labview to modify the configuration file of the YOLOv5 model. The original model has only three detection layers. Add a small target detection layer, and use a total of four detection layers for detection, so as to obtain a larger feature map for small target detection. After adding the detection layer, the accuracy of small objects such as traffic lights is improved.

[0053] (3) Figure 3 As shown in the figure, in the PANet architecture of the yolov5 model, the BottleNeckCSP module is replaced with the Ghostbottleneck module, and the order of the submodules of the Ghostbottleneck module is set from front to back to GhostModule 1 module, depthwise module, SEModule module, GhostModule2 module, and Identitymapping module. And the ReLU function of the GhostModule2 module is removed, so that the model size can be reduced, the calculation is reduced, and the test speed can be increased by 20%.

[0054] Step S3, build load_detector.vi in ​​labview, call this VI in the main program flowchart, read the configuration file of the modified yolov5 model and the weight file and data set of the trained traffic light model, so as to obtain the traffic light automatic recognition model, train the traffic light automatic recognition model, make the traffic light automatic recognition model first recognize the traffic light boundary box, and then recognize the ID number corresponding to the traffic light from the traffic light boundary box, the formula for training and recognizing the traffic light boundary box is:

[0055] t x =(G x -P x )

[0056] t y =(G y -P y )

[0057] t w =log(G w / P w )

[0058] t h =log(G h / P h )

[0059] In the formula, G x The horizontal coordinate of the center point of the marked bounding box; G y The vertical coordinate of the center point of the labeled bounding box; G w Indicates the width of the bounding box of the annotation; G h represents the height of the bounding box of the annotation; P x Represents the horizontal coordinate of the center point of the anchor box (i.e., the predicted bounding box); P y Indicates the vertical coordinate of the center point of the anchor box; P w Indicates the width of the anchorbox; P h Indicates the height of the anchor box; t x Represents the lateral offset of the model training output; t y Indicates the vertical offset of the model training output; tw indicates the horizontal scaling of the model training output; th indicates the vertical scaling of the model training output. By not letting the model directly predict the width and height of the actual bounding box, but binding the predicted width and height to the width and height of the anchor box respectively, no matter what the width and height of the model output are at the beginning, they are related to the width and height of the anchor box after conversion. After many times of penalty training, the model knows what shape of bounding box it is responsible for predicting.

[0060] After predicting the bounding box, the traffic light, shape, and color can be identified from the bounding box according to the weight file of the traffic light model, and the corresponding ID number can be output.

[0061] like Figure 4 As shown in the figure, it is the MAP change curve of the training data set during the training process. Figure 4 It can be seen that the MAP of the optimized architecture training data set can reach 80%.

[0062] Step S4, make the yolov5 model read the camera image in real time. Specifically: build IMAQdx ConfigureGrab.vi, configure the camera to read the image in a suitable format to obtain the current frame of the image. After obtaining the current frame of the image, filtering and denoising can be performed to obtain a clear image. The method of filtering and denoising is the same as in step S1.

[0063] Step S5: Use the trained traffic light automatic recognition model to identify the ground truth box of the traffic light from the image captured by the camera, and output the recognition result. Build load_detector.vi in ​​Labview, call this VI in the main program flowchart, and read the modified configuration file and the trained traffic light file. The process of training the traffic light data set is shown in the attached figure. Figure 2 , the MAP of the optimized architecture training data set can reach 80%. The recognition process is:

[0064] First, divide the traffic light image into S×S (e.g., 10×10) grids, and use the improved yolov5 model, the trained traffic light dataset, and the traffic light weight file to identify each grid in turn and predict the bounding box;

[0065] Then calculate the confidence of the predicted bounding box. The calculation formula for the confidence of the predicted bounding box is:

[0066]

[0067] Among them, D conf is the confidence of the predicted bounding box; P(o) is the probability of the detected object. If the center of the object falls within the grid, P(o) is 1, otherwise it is 0; is the intersection-over-union ratio of the predicted bounding box and the true bounding box.

[0068] Then, based on the confidence of the predicted bounding box and non-maximum suppression, the predicted bounding box with the highest confidence for each traffic light is selected. The working principle of non-maximum suppression is as follows:

[0069] For a predicted bounding box B, the model calculates the predicted probability of each category. Let the largest predicted probability be p, and the category corresponding to this probability is the predicted category of B. We also call p the confidence of the predicted bounding box B. On the same image, we sort the predicted bounding boxes of the predicted category non-background from high to low according to the confidence, and obtain a list L. Select the predicted bounding box B1 with the highest confidence from L as the benchmark, and remove all non-benchmark predicted bounding boxes whose IoU ratio with B1 is greater than a certain threshold from L. The threshold here is a pre-set hyperparameter. At this time, L retains the predicted bounding box with the highest confidence and removes other similar predicted bounding boxes. Next, select the predicted bounding box B2 with the second highest confidence from L as the benchmark, and remove all non-benchmark predicted bounding boxes whose IoU ratio with B2 is greater than a certain threshold from L. Repeat this process until all predicted bounding boxes in L have been used as benchmarks. At this time, the IoU ratio of any pair of predicted bounding boxes in L is less than the threshold. Finally, output all predicted bounding boxes in list L. Through non-maximum suppression, only one predicted bounding box with the highest confidence is retained for each traffic light.

[0070] Finally, the real bounding box of the traffic light is identified based on the predicted bounding box filtered by non-maximum suppression, and the recognition result is output. The output recognition result includes the number of bounding boxes identified in the image, the location information of the four corner points of each bounding box, the ID number corresponding to the traffic light information in each bounding box, and the confidence of each bounding box. Figure 5 As shown, the true bounding box can be drawn directly on the image, and the calculation formula of the true bounding box is:

[0071] b x =σ(t x )+c x

[0072] b y =σ(t y )+c y

[0073]

[0074]

[0075] Among them, b x Represents the horizontal coordinate of the center point of the real bounding box; b y represents the ordinate of the center point of the true bounding box; σ() represents the sigmoid function; c x Indicates a multiple of the grid width; b w Indicates the width of the true bounding box; b h Indicates the width of the real bounding box. The coordinates of the four corner points of the real bounding box can be calculated through the center point coordinates, width and height of the real bounding box.

[0076] In this embodiment, a traffic light recognition method based on yolov5 is adopted, and by improving yolov5, the bounding box of the traffic light can be accurately recognized, solving the problem of slow image recognition speed and low accuracy of the traditional recognition method. In addition, the recognition ability of small targets is enhanced, the target recognition accuracy is high, the real-time performance is strong, the operation is simple, and the technology of identifying traffic lights is broadened. The traffic light recognition method is very suitable for application in unmanned driving technology.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A traffic light recognition method based on deep learning, characterized in that: The following steps are involved: Step S1, obtaining and annotating a set of traffic light images, and obtaining a traffic light model capable of recognizing the shape and color of traffic lights through training; Step S2, improving the yolov5 model to increase the small target recognition capability of the yolov5 model; Improvements to the yolov5 model include: A multi-scale feature extraction layer and a feature fusion layer are added after the original feature extraction layer, and a small target detection layer is added before the target detection layer. The feature information output by the original feature extraction layer is sent to the multi-scale feature extraction layer and the feature fusion layer respectively. The multi-scale feature extraction layer is used to filter the feature information output by the original feature extraction layer and extract obvious feature information; the feature fusion layer is used to fuse the feature information output by the original feature extraction layer and the feature information output by the multi-scale feature extraction layer; the small target detection layer is used to enhance the fine-grained features of small targets by upsampling; In the PANet architecture of the yolov5 model, the BottleNeckCSP module is replaced with the Ghostbottleneck module, and the order of the submodules of the Ghostbottleneck module is set from front to back to GhostModule 1 module, depthwise module, SEModule module, GhostModule2 module, and Identity mapping module; and Remove the ReLU function of the GhostModule2 module; Step S3, adding the data set and weight file in the trained traffic light model to the improved yolov5 model to obtain a traffic light automatic recognition model, and training the traffic light automatic recognition model so that the traffic light automatic recognition model first recognizes the traffic light boundary box, and then recognizes the ID number corresponding to the traffic light from the traffic light boundary box; Step S4, enabling the yolov5 model to read the image of the camera in real time; Step S5: using the trained traffic light automatic recognition model to identify the real boundary box of the traffic light from the image captured by the camera, and output the recognition result.

2. The traffic light recognition method based on deep learning according to claim 1, characterized in that: In step S1, traffic lights are assigned ID numbers according to their shapes and colors. When annotating a traffic light picture set, a bounding box is firstly marked at the position of the traffic light on the traffic light picture, and then the ID number corresponding to the shape and color of the traffic light in each bounding box is marked.

3. The traffic light recognition method based on deep learning according to claim 1, characterized in that: In step S4, after reading the image captured by the camera, filtering and denoising are performed. The filtering and denoising method is as follows: First, the color image is converted into a grayscale image through grayscale image conversion, then binarization is performed, and finally Gaussian filtering is performed for denoising; the filter function selects FIR windowed filter PtByPt VI.

4. The traffic light recognition method based on deep learning as claimed in claim 1, characterized in that: In step S3, the formula for model training is: t x =(G x -P x ) t y =(G y -P y ) t w =log(G w / P w ) t h =log(G h / P h ) In the formula, G x The horizontal coordinate of the center point of the marked bounding box; G y The vertical coordinate of the center point of the labeled bounding box; G w Indicates the width of the bounding box of the annotation; G h represents the height of the bounding box of the annotation; P x represents the horizontal coordinate of the center point of the predicted bounding box; P y Represents the ordinate of the center point of the predicted bounding box; P w Indicates the width of the predicted bounding box; P h Represents the height of the predicted bounding box; t x Represents the lateral offset of the model training output; t y Indicates the vertical offset of the model training output; tw indicates the horizontal scaling of the model training output; th represents the vertical scaling of the model training output.

5. The traffic light recognition method based on deep learning as claimed in claim 1, characterized in that: In step S5, the output recognition result includes the number of bounding boxes recognized in the image, the position information of the four corner points of each bounding box, the ID number corresponding to the traffic light information in each bounding box, and the confidence of each bounding box.

6. The traffic light recognition method based on deep learning according to claim 1, characterized in that: In step S5, the method for identifying the traffic light from the image captured by the camera is: First, divide the traffic light image into SxS grids, then identify each grid in turn and predict the bounding box; then calculate the confidence of the bounding box, and filter out the predicted bounding box with the highest confidence for each traffic light based on the confidence of the predicted bounding box and non-maximum suppression; finally, identify the real bounding box of the traffic light and output the recognition result; The calculation formula of the true bounding box is as follows: b x =σ(t x )+c x b y =σ(t y )+c y Where b x Represents the horizontal coordinate of the center point of the real bounding box; b t represents the ordinate of the center point of the true bounding box; σ() represents the sigmoid function; c x Indicates a multiple of the grid width; b w represents the width of the real bounding box; b h Indicates the width of the ground-truth bounding box.

7. The traffic light recognition method based on deep learning as claimed in claim 1, characterized in that: The confidence of the predicted bounding box is calculated as: Among them, D conf is the confidence of the predicted bounding box; P(o) is the probability of the detected object. If the center of the object falls within the grid, P(o) is 1, otherwise it is 0; is the intersection-over-union ratio of the predicted bounding box and the true bounding box.