Traffic light recognition network training and testing method based on visual ranging

Abstract

The present invention provides a method for training a traffic light recognition network based on visual ranging, comprising the following steps: Step 1, annotating the collected images to create a traffic light dataset; Step 2, using the dataset established in Step 1 to train a traffic light recognition network model; Step 3, optimizing the model parameters using a gradient descent method using a loss function, adjusting the hyperparameters of the trained network model using a trainval sub-dataset, and evaluating the final network model using a test sub-dataset; when the loss tends to be flat during training and the network performance has not improved for three consecutive training cycles, the last trained model is selected as the optimal model to complete the training. The present invention also provides a method for testing a traffic light recognition network based on visual ranging. The present invention solves the problems of low traffic light recognition accuracy, large computational complexity, and poor real-time performance that exist in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent transportation and relates to a traffic light recognition network training method and a testing method based on visual ranging. Background Art

[0002] With the rapid development of autonomous driving technology, self-driving cars urgently need to improve their ability to recognize traffic lights. Existing traffic light recognition methods primarily rely on image detection. Due to the small size of traffic lights and low detection accuracy, the recognition process indiscriminately searches through image information, resulting in high computational complexity and poor real-time performance. Current technical solutions primarily focus on deep learning algorithms such as Faster-RCNN, the YOLO series, and SSD networks. However, traditional network algorithms perform an indiscriminate search across the entire image, increasing the computational complexity and failing to meet the real-time performance requirements of autonomous driving systems.

[0003] "Traffic light recognition method, device, equipment and storage medium based on deep learning" (patent application number: 202010930268.0) improves the accuracy of the algorithm in detecting traffic lights to a certain extent by improving the YOLOv3 network structure and introducing the DIOU loss module, but fails to utilize the prior information of traffic lights and has a large amount of calculation.

[0004] "A Traffic Light Recognition Method Based on Image Processing and Deep Learning" (Patent Application No.: 202010255239.9) combines traditional image processing methods with deep convolutional networks for traffic light recognition. This method integrates traditional image detection methods with deep learning methods, uses classification to identify traffic light states, and applies parallel processing technology to effectively shorten detection time. However, this method only uses image information as algorithm input, which can easily lead to false detections at complex traffic intersections.

[0005] The "Real-Time Traffic Light Recognition Method" (Patent Application No.: 201910354808.2, Authorization Announcement No.: CN109949594B) proposes a traffic light recognition method based on an annotated map. In this method, traffic lights and lanes are jointly encoded, and the traffic light status is obtained by obtaining the traffic light ID. This method requires the creation of a complex annotated map and is not applicable to unannotated map areas. Therefore, this method has significant limitations. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides a traffic light recognition network training method and testing method based on visual ranging, which solves the problems of low traffic light recognition accuracy, large computational complexity, and poor real-time performance in the prior art.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is a traffic light recognition network training method based on visual ranging, comprising the following steps:

[0008] Step 1: Label the collected images and create a traffic light dataset: the traffic light dataset consists of a traffic light image, normalized ROI information, and true label information as a basic data unit; the dataset is divided into three sub-datasets: train, trainval, and test in an 8:1:1 ratio. The train and trainval datasets are used for training and performance verification of the training process; the test dataset is used for the final evaluation of network performance;

[0009] Step 2: Use the data set created in step 1 to train the traffic light recognition network model:

[0010] Step 21: First, establish a traffic light recognition network model, which includes three parts: backbone network, feature fusion network and feature decoder;

[0011] Step 22: Send the traffic light images captured by the camera in the training set into the backbone network of the traffic light recognition network model to extract image features and obtain feature maps; then send the extracted feature maps into the feature fusion network to obtain detection head feature maps TC1, TC2, and TC3;

[0012] Step 23: Map the ROI area information normalized in step 1 to the detection head feature map obtained in step 22 to generate the ROI feature map: Assume that the height and width of the detection head feature map are h and h respectively. ti 、w ti , then the position information of the ROI area in the corresponding detection head feature map is The feature map of the ROI area in the detection head feature map mapping area is intercepted as the ROI feature map;

[0013] Among them, P w 、P h are the image width and height, X c is the x-axis coordinate of the center point of the ROI area, X c The horizontal coordinate X of the center point of the image is 中心 As the sampling center, the Gaussian probability density function is used in [0,P w ] range of sampling; Y c Y is the y-axis coordinate of the center point of the ROI area, c The value of Y cmax to Y cmin Random sampling is obtained within the range; the upper and lower boundaries of the y-axis coordinate value of the center point of the ROI area are respectively recorded as Y cmax , Y cmin, where Y cmax =H 板1 +αh1,Y cmin =H 板1 -αh1, where α is the adaptive coefficient, H 板 is the height of the center point of the signal light board, f is the focal length of the camera, H 板1 H 板 The projection in the image coordinate system, h1 is the projection of the vertical length h of the signal light board in the image coordinate system; S c is the horizontal distance between the vehicle and the traffic light, R is the preset threshold, and e is a natural constant;

[0014] Step 24: Compare the obtained ROI feature map with the true label obtained in step 1, and select the ROI feature map that best matches the true label as the initial prediction box feature map;

[0015] Step 25: The feature decoder classifies and fine-tunes the filtered initial prediction frame feature map to obtain the final traffic light prediction frame information;

[0016] Step 26: Calculate the loss function between the traffic light prediction box and the true label through the Loss function, determine the evolution direction of the network parameters, and optimize the traffic light recognition network model;

[0017] Step 3: Use the loss function and gradient descent method to optimize the model parameters, use the trainval sub-dataset to adjust the network model hyperparameters of the trained network model, and use the test sub-dataset to evaluate the final network model; when the loss tends to be flat during training and the network performance has not improved for three consecutive training cycles, select the last trained model as the optimal model to complete the training process.

[0018] Furthermore, the ROI area information in step 1 is extracted and obtained by a ROI extraction system, which includes a GPS module, an electronic map module, a distance measurement module and a ROI extraction module; wherein,

[0019] GPS module obtains vehicle real-time location information (X 1c , Y 1c ) and record the timestamp as Tc; at time Tc, the electronic map module extracts the signal light position coordinates (X L , Y L The ranging module calculates the horizontal distance Sc between the vehicle and the traffic light at time Tc using the following formula;

[0020]

[0021] The ranging module determines whether the distance Sc between the vehicle and the traffic light exceeds the preset threshold R: If Sc is greater than R, the ROI extraction system is not activated, and the ranging module enters the next moment loop to continue detecting the relationship between Sc and R. When Sc is less than or equal to R, it indicates that the vehicle has entered the traffic light recognition area, and the ROI extraction system is activated;

[0022] The ROI extraction module is used to determine the ROI area information of the traffic light and normalize the ROI area feature vector. The normalized ROI area feature vector is expressed as: Among them, P w 、P h are the image width and height respectively.

[0023] Furthermore, the backbone network in step 2 includes three CRC modules and three CRT modules alternating in sequence, each CRC module includes a CONV module, a Residual module and a CBAM module, and each CRT module includes a CONV module, a Residual module and a Transform module;

[0024] Among them, the CONV module is a convolution downsampling module, which downsamples the input feature map by a factor of two; the Residual module is a residual network structure, which reduces the gradient vanishing problem caused by network deepening through jump connections of internal residual blocks; the Transform module is an attention network model; the CBAM module includes two parts: channel attention and spatial attention. The CBAM module infers the attention weights along the spatial and channel dimensions in sequence, and can extract target information from image information.

[0025] Furthermore, the method for the feature fusion network in step 2 to obtain the detection head feature map is:

[0026] First, select the output of the third CRT module of the backbone network, denoted as T3; send T3 to the CONV module for 1×1 convolution, and then send the output to the upsample module. Each upsample module will upsample the feature map once. After two upsamplings, the feature map is concatenated with the feature map output by the second CRT module of the backbone network using the Concat method. The output is further extracted by the CBAM module, and the final output is denoted as T2. Concat is a feature map concatenation method commonly used in deep learning networks.

[0027] Send T2 to the CONV module, use a 1×1 convolution kernel to adjust the number of channels, and then send it to the upsample module for two upsampling. The upsampling result is concatenated with the output of the first CRT module of the backbone network using the Concat method for feature splicing, and the output is further extracted by the CBAM module. The final output result is recorded as T1;

[0028] Select the first CRC module of the backbone network, denoted as C1, send C1 to the CONV module, use a 3×3 convolution kernel to perform two convolution downsamplings, then use the Concat method to concatenate the downsampled output with the output of the second CRC module of the backbone network, and then use the Transform module to extract effective features, and the output feature map is denoted as C2; send C2 to the CONV module, perform two downsamplings, concatenate the downsampled result with the output of the third CRC module of the backbone network using the Concat method, and then use the Transform module to extract features, and the output feature map is denoted as C3;

[0029] After obtaining T1, T2, T3, C1, C2, and C3, T1 is input into the upsample module, and after upsampling, it is concatenated with C1 using the Concat method to output a feature map as the detection head, recorded as TC1; T2 is input into the upsample module, and after upsampling, it is concatenated with C2 using the Concat method to output a feature map as the detection head, recorded as TC2; T3 is input into the upsample module, and after upsampling, it is concatenated with C3 using the Concat method to output a feature map as the detection head, recorded as TC3.

[0030] Furthermore, the step 24 is specifically as follows:

[0031] Step 241: Apply softmax processing to the ROI feature map to divide the ROI feature map into a foreground image and a background image, and select the ROI feature map belonging to the foreground image;

[0032] Step 242: Combine each filtered foreground image ROI feature map with the true label of the corresponding image one by one to calculate the CIOU value;

[0033] Step 243: Each foreground image ROI feature map retains a best matching true label and stores it in array form, denoted as R Best , where R Best Each item includes the following {I roi , I gt , V ciou}, where I roi Represents ROI feature map related information, I gt represents the best true label matching the ROI feature map, Vciou Represents the CIOU value between the foreground image ROI feature map and the best matching true label;

[0034] Step 244, R Best Array Press V ciou Sort by value in descending order, then by V ciou Get R values ​​one by one from large to small Best Elements of the array; let GT be the best matching array between the real label and the ROI region feature map, the initial condition initializes GT to empty, and then judge the obtained R Best I of array element gt Is it already included in the elements of GT? If not, it is considered that the I of the array element roi , I gt For the best matching relationship, store the array element in GT; if R Best I of array element gt If it already exists in the element of GT, it means that the best matching ROI feature map of the real label has been found, and the GT array is not updated; when the length of GT is equal to the number of image labels, or R Best When the array is completely traversed, the process is stopped and the I of the element in GT is finally saved. roi That is the best matching ROI feature map of the real label, and the I of the element in GT roi As the initial prediction box feature map.

[0035] Furthermore, the Loss function consists of three parts, including the target box loss function, confidence loss and classification loss; wherein, the confidence loss and classification loss use the cross entropy loss function form; the target box loss function formula is as follows:

[0036]

[0037] Among them, NCIoU is the target box loss function, CIoU is the target box loss function commonly used in the field of target detection, e is a natural constant; width factor Among them, w p is the prediction box width, w g is the width of the ground truth box of the label; height factor Among them, h p is the prediction box height, h g is the height of the ground-truth box of the label, Represents the area ratio of the true label box to the predicted box.

[0038] Furthermore, the step 25 is specifically as follows:

[0039] The filtered initial prediction box feature map is sent to the fully connected layer of the feature decoder for classification to obtain the prediction box category and confidence. Then, the regression module of the feature decoder is used to correct the size of the initial prediction box feature map to obtain the final prediction box size information. The category and confidence of the prediction box are combined with the prediction box size information to form the traffic light prediction box information [class, conf, box], where class represents the prediction box category, conf represents the confidence of the prediction box category, and box represents the prediction box size information. The regression module adopts the bounding box regression module in the Faster RCNN network.

[0040] The present invention also provides a traffic signal light recognition network testing method based on visual ranging, comprising the following steps:

[0041] Step S1: The test vehicle activates the camera and obtains the distance between the test vehicle and the traffic light ahead in real time through the ROI extraction system. If the distance is less than a threshold, the traffic light detection module is activated; if the distance is greater than the threshold, monitoring continues;

[0042] Step S2: The image captured by the camera is marked with a timestamp as the network input. At the same time, the ROI information extracted at the corresponding timestamp is fed into the trained model, and the predicted box information [class, conf, box] is output; where class represents the predicted box category, conf represents the confidence of the predicted box category, and box represents the predicted box size information;

[0043] Step S3: further filter the prediction boxes outputted in step S2 and retain the most reasonable prediction box combination;

[0044] Step S4: Calculate the average Conf value of each retained prediction box combination among the retained prediction box combinations, and retain the prediction box combination with the highest average Conf as the final prediction box output.

[0045] Furthermore, the step S3 is specifically as follows:

[0046] Step S31: filter all prediction frame information based on conf: traverse all prediction frames, and when conf>T c When , the prediction box is retained, otherwise the prediction box is directly removed; T c Filter thresholds for categories;

[0047] Step S32: Group the retained prediction frames by category, with the prediction frames of the same category grouped together;

[0048] Step S33: Perform maximum suppression operation on the grouped prediction boxes of each category, and retain the output results after maximum suppression of each category; subtract the relative maximum value of the corresponding category from the box of the retained prediction box to restore the original position information of each prediction box;

[0049] Step S34: Screen reasonable prediction frame combinations.

[0050] Furthermore, the step S34 is specifically as follows:

[0051] Step S341: Classify and filter the predicted box categories:

[0052] For the same frame image, the traffic lights are grouped according to the fixed combination relationship between the true labels of each light. There are 7 combinations in total: 1) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped green light; 2) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 3) left-turn arrow-shaped green light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 4) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light; 5) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 6) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 7) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light. Whether the yellow light appears when turning left, going straight or right, it is a reasonable traffic light state;

[0053] The grouped prediction boxes are divided into three categories: 1) Left turn category: left turn arrow-shaped red light, left turn arrow-shaped green light, left turn arrow-shaped yellow light; 2) Straight go category: straight go arrow-shaped red light, straight go arrow-shaped green light, straight go arrow-shaped yellow light, circular red light, circular green light, circular yellow light; 3) Right turn category: right turn arrow-shaped red light, right turn arrow-shaped green light, right turn arrow-shaped yellow light;

[0054] Within each major category, the prediction boxes are screened using the NMS method and then sorted in descending order by conf size. Then, the prediction boxes retained in each major category are traversed and combined one by one, and the prediction boxes that do not meet the above 7 combination forms are eliminated. If the combined prediction box target only contains part of the combination target, it is also considered a reasonable prediction and the prediction box is retained.

[0055] Step S342: traverse the reserved prediction box combinations in sequence, and filter the prediction boxes in each prediction box combination according to their geometric position relationship:

[0056] The prediction box information of each combination is as follows: The prediction box information of the left turn class is as follows [conf l ,x 1l ,y 1l ,x2l ,y 2l ,],conf l Represents the confidence of the left-turn prediction box category, x 1l Indicates the horizontal coordinate value of the upper left corner of the left-turn prediction box, y 1l Indicates the ordinate of the upper left corner of the left-turn prediction box, x 2l Indicates the horizontal coordinate value of the lower right corner of the left-turn prediction box, y 2l Indicates the vertical coordinate of the lower right corner of the left-turn prediction box; the right-turn prediction box information is as follows [conf r ,x 1r ,y 1r ,x 2r ,y 2r ], conf r Represents the confidence level of the right-turn prediction box category, x 1r Indicates the horizontal coordinate value of the upper left corner of the right-turn prediction box, y 1r Indicates the vertical coordinate of the upper left corner of the right-turn prediction box, x 2r Indicates the horizontal coordinate value of the lower right corner of the right-turn prediction box, y 2r Indicates the vertical coordinate of the lower right corner of the right turn prediction box; the straight line prediction box information is as follows [conf s ,x 1s ,y 1s ,x 2s ,y 2s ], conf s Indicates the confidence of the straight line prediction box category, x 1s Indicates the horizontal coordinate value of the upper left corner of the straight line prediction box, y 1s Indicates the vertical coordinate of the upper left corner of the straight line prediction box, x 2s Indicates the horizontal coordinate value of the lower right corner of the straight line prediction box, y 2s Represents the vertical coordinate of the lower right corner of the straight-ahead prediction box; for a traffic light combination at a traffic intersection, the left-turn signal light, the straight-ahead signal light, and the right-turn signal light are equally spaced in the horizontal direction, that is, the horizontal distance between the left-turn prediction box and the straight-ahead prediction box is Horizontal distance between the right-turn prediction box and the straight-ahead prediction box When |X ls -X rs When |≤δ, the prediction box combination is retained, otherwise the prediction box is eliminated, where δ is the distance similarity threshold, and the default value is set to max(0.4X ls , 0.4X rs ), max() means taking the maximum value of the two.

[0057] The beneficial effects of the present invention are:

[0058] 1. A network model for traffic light recognition is proposed. The CRC module and CRT module are introduced into the backbone network of this model. The feature extraction capability of the backbone network is improved through the CBAM module and the Transform module. A jumping feature pyramid structure is adopted in the feature fusion network to achieve the full fusion of deep features and shallow features. Feature maps of different sizes in the backbone network are directly involved in the feature fusion process, so that the final feature map contains richer image information.

[0059] 2. A ROI region information extraction method integrating the visual ranging principle is proposed to narrow the image search space and further improve the accuracy and real-time performance of traffic light recognition.

[0060] 3. A traffic light recognition network training method integrating visual odometry is proposed, and a matching method between ROI area and true label is proposed. The loss function of the training process is improved, so that the model training process can converge faster.

[0061] 4. By combining and classifying the prediction box categories and considering the positional relationship between the targets of a single image signal light, a reasonable and effective prediction box screening method is proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 4 is a flow chart of a traffic light recognition network training method according to an embodiment of the present invention.

[0064] Figure 2 Schematic diagram of a ROI extraction system according to an embodiment of the present invention.

[0065] Figure 3 This is a flow chart of ROI region extraction according to an embodiment of the present invention.

[0066] Figure 4 This is a geometric relationship diagram of the ROI area extracted based on the distance measurement principle in an embodiment of the present invention.

[0067] Figure 5 2 is a schematic diagram of the structure of a traffic light recognition network model according to an embodiment of the present invention.

[0068] Figure 6 It is a structural diagram of the CRC module and the CRT module of the traffic light recognition network model according to an embodiment of the present invention.

[0069] Figure 7 2 is a schematic diagram of the detection head merging of the traffic light recognition network model according to an embodiment of the present invention.

[0070] Figure 8 This is a flow chart of traffic light prediction frame screening according to an embodiment of the present invention.

[0071] Figure 9 This is a flow chart of a traffic light recognition network testing method according to an embodiment of the present invention.

[0072] Figure 10 This is a comparison chart of the traffic light recognition network detection effect of an embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] The present invention provides a traffic light recognition network training method based on visual ranging, such as Figure 1 The specific steps are as follows:

[0075] Step 1: Annotate the collected images and create a traffic light dataset.

[0076] A camera is installed at the center of the front of the experimental vehicle, and the optical axis of the camera is always kept parallel to the ground. Then, the signal light image is captured, and the ROI area information of the signal light image is obtained through the ROI extraction system.

[0077] Furthermore, this step mainly involves a ROI (Region of Interest) extraction system: using the principle of visual ranging to determine the area where the traffic light is located.

[0078] like Figure 2 As shown, the ROI extraction system includes a GPS module, an electronic map module, a distance measurement module and a ROI extraction module.

[0079] like Figure 3 As shown, the GPS module obtains the real-time location information of the vehicle (X 1c , Y 1c ) and record the timestamp as Tc; at time Tc, the electronic map module extracts the signal light position coordinates (X L , Y L); The ranging module uses the following formula to calculate the horizontal distance Sc between the vehicle and the traffic light at time Tc.

[0080]

[0081] Furthermore, the ranging module determines whether the distance Sc between the vehicle and the traffic light exceeds the preset threshold R: if Sc is greater than R, the ROI extraction system is not started, and the ranging module enters the next moment loop to continue detecting the relationship between Sc and R. When Sc is less than or equal to R, it means that the vehicle has entered the traffic light recognition area, and the ROI extraction system is started. The default value of R is 100m.

[0082] The ROI extraction module determines the location area of ​​the traffic light based on the horizontal distance Sc between the vehicle and the traffic light. The specific workflow is as follows:

[0083] First, the camera collects image information at time Tc; in this embodiment, the camera is a LOGI 1080P network camera.

[0084] Depend on Figure 4 It can be seen that according to the principle of visual imaging, let point A be the highest point of the signal light board, point B be the lowest point of the signal light board, C be the intersection of the projection of the center of the signal light board in the Y direction (vertical direction) and the optical axis of the camera, O be the optical center of the camera, h be the vertical length of the signal light board, w be the horizontal length of the signal light board; let the camera installation height be H, the height of the highest point A of the signal light board be H 最高 , H 最高 and h can be determined by the corresponding national standards, and the height of the center point of the signal light board is H 板 =H 最高 -0.5h. H 板1 , A1, B1, C1, h1, w1 are H 板 , projection of A, B, C, h, w in the image coordinate system.

[0085] Triangle OC1A1 is similar to triangle OCA, and triangle OC1B1 is similar to triangle OCB, so we can get:

[0086]

[0087]

[0088] From the above formula, we can get:

[0089]

[0090] From the above formula, we can see that triangle OB1A1 is similar to triangle OBA, so we can get

[0091]

[0092] Depend on Figure 4 It can be seen that OC is the horizontal distance between the camera and the traffic light, that is, OC = Sc, let OC1 be the focal length of the camera, recorded as f, obtained from the internal parameters of the camera; further we can get Similarly

[0093] Furthermore, the ROI information is determined. The specific process is as follows: In order to eliminate the influence of pixel errors caused by factors such as image distortion, the upper and lower value boundaries of the y-axis coordinate value (vertical direction) of the center point of the ROI area are set, which are recorded as Y cmax , Y cmin , where Y cmax =H 板1 +αh1,Y cmin =H 板1 -αh1, set the y-axis coordinate of the center point of the ROI area to Y c , Y c The value of Y cmax to Y cmin Random sampling is obtained within the range, where α is the adaptive coefficient, The farther the distance Sc between the traffic light board and the vehicle, the larger the adaptive coefficient, thereby improving the problem of missed detection of targets in the ROI area caused by the increase in pixel error of long-distance images and improving the distance adaptive ability of the ROI area. In the lateral direction, there is no hard constraint on the position of the traffic light, but because the traffic light board is often located directly in front of the driver's vision, the probability of the traffic light appearing in the center of the image is greater. Let P w 、P h are the image width and height respectively, and the x-axis coordinate of the center point of the ROI area is X c , X c The horizontal coordinate X of the center point of the image is 中心 As the sampling center, the Gaussian probability density function is used in [0,P w ] range. Unlike traditional detection targets, traffic lights are circular or arrow-shaped with a height-to-width ratio of about 1. In order to ensure that the ROI area matches the target frame as much as possible, 9 ROI areas with a height-to-width ratio of 1 but different sizes are set at each center point. The height and width information of the 9 ROI areas are obtained by the following method: the distance between the vehicle and the traffic light is respectively The size of the sampled signal light target is denoted as S i , i is 1-9, S i That is the height and width information of the ROI area. In order to ensure that the ROI area contains all the traffic light targets as much as possible in a small number of cases, let X c The number of sampling points is N xc , Y c The number of sampling points is N yc, N xc 、N yc The following conditions should be met:

[0094]

[0095] T1, T2, and T3 are sampling point thresholds, and their default values ​​are set to 5, 10, and 200 respectively.

[0096] At the same time, the X c 、Y c The ROI area is represented by the vector [X c Y c S i S i ] indicates that the ROI region feature vector is normalized, and the normalized ROI region feature vector is expressed as:

[0097] The collected images are annotated using labelme software to create label files. The traffic light area is specified using a rectangular frame and the actual label category of the traffic light is marked. The label categories are commonly used motor vehicle signal light categories, including round green light (Round Green, denoted as RG), round yellow light (Round Yollow, denoted as RY), round red light (Round Red, denoted as RR), straight arrow green light (Straight Arrow Green, denoted as SAG), straight arrow yellow light (Straight Arrow Yellow, denoted as SAY), straight arrow red light (Straight Arrow Red, denoted as SAR), left turn arrow green light (Left Turn Arrow Green, denoted as LTAG), left turn arrow yellow light (Left Turn Arrow Yellow, denoted as LTAY), left turn arrow red light (Left Turn Arrow Red, denoted as LTAR), right turn arrow green light (Right Turn Arrow Green, denoted as RTAG), right turn arrow yellow light (Right Turn Arrow Yellow, denoted as RTAY), and right turn arrow red light (Right Turn Arrow Red, denoted as RTAR).

[0098] In some implementations, to ensure the richness of the data, operations such as rotation and cropping are performed on the image to expand the data set.

[0099] Finally, the traffic light dataset was synthesized by combining images, ROI information, and ground-truth label information as a basic data unit. This dataset was then divided into three sub-datasets (train, trainval, and test) in an 8:1:1 ratio. These sub-datasets were used for training and testing the traffic light recognition network, respectively. The train and trainval datasets were used for training and performance verification during the training process, while the test dataset was used for the final evaluation of network performance.

[0100] Step 2: Use the data set created in step 1 to train the traffic light recognition network model. The specific steps are as follows:

[0101] Step 21, first, establish a traffic light recognition network model, which mainly includes three parts: backbone network, feature fusion network and feature decoder. The backbone network is used to extract image features; the feature fusion network is used to fuse the extracted image features to obtain the detection head, which is convenient for subsequent traffic light detection based on the fused features; the feature decoder decodes the detection head to obtain the prediction box information. Figure 5 As shown, the traffic light recognition network model includes the following steps:

[0102] Step 211: Extract image features using the backbone network.

[0103] Specifically, the backbone network includes three CRC modules and three CRT modules that are alternated in sequence. Figure 6 As shown, each CRC module includes a CONV module, a Residual module and a CBAM module, and each CRT module includes a CONV module, a Residual module and a Transform module.

[0104] Among them, the CONV module is a convolution downsampling module, which performs double-rate downsampling on the input feature map.

[0105] Among them, the Residual module is a residual network structure, which can reduce the gradient disappearance problem caused by network deepening through the jump connection of internal residual blocks.

[0106] Among them, the Transform module is an excellent attention network model in the current existing technology. The Transform module can improve the network's ability to extract effective features while avoiding network overfitting.

[0107] The CBAM module consists of two parts: channel attention and spatial attention. It sequentially infers attention weights along the spatial and channel dimensions, enabling better extraction of target information from image data. CBAM is a lightweight, universal module that can be applied to CNN architectures in a plug-and-play manner.

[0108] Step 212: Utilize the feature fusion network to obtain a detection head for traffic light detection.

[0109] Specifically, if Figure 5 As shown in the figure, the output of the third CRT module in the backbone grid is first selected, denoted as T3. T3 is fed into the CONV module for a 1×1 convolution, and the output is then fed into the upsample module. Each upsample module upsamples the feature map once. After two upsampling cycles, the output is concatenated with the feature map from the second CRT module using the Concat method. This output is then passed through the CBAM module for further feature extraction, resulting in the final output being denoted as T2. Concat is a commonly used feature map concatenation method in deep learning networks.

[0110] T2 is sent to the CONV module, and the number of channels is adjusted using a 1×1 convolution kernel. Then it is sent to the upsample module for two upsamplings. The upsampling result is concatenated with the output of the first CRT module using the Concat method for feature splicing, and the output is further extracted through the CBAM module. The final output result is recorded as T1.

[0111] Next, we select the first CRC module of the backbone network, denoted as C1, and feed C1 into the CONV module. We use a 3×3 convolution kernel for two convolution downsampling operations. The downsampled output is then concatenated with the output of the second CRC module using the Concat method. The Transform module is then used to extract valid features, and the output feature map is denoted as C2. C2 is then fed into the CONV module for two downsamples. The downsampled result is concatenated with the output of the third CRC module using the Concat method. The Transform module is then used for feature extraction, and the output feature map is denoted as C3.

[0112] After obtaining T1, T2, T3, C1, C2, and C3, if Figure 7 As shown, T1 is input into the upsample module, and after upsampling, it is concatenated with C1 using the Concat method to output a feature map as the detection head, which is recorded as TC1; T2 is input into the upsample module, and after upsampling, it is concatenated with C2 using the Concat method to output a feature map as the detection head, which is recorded as TC2; T3 is input into the upsample module, and after upsampling, it is concatenated with C3 using the Concat method to output a feature map as the detection head, which is recorded as TC3;

[0113] In step 22, the image frames captured by the camera are sent to the backbone network of the traffic light recognition network model described in step 211 to extract image features, and then the extracted feature maps are sent to the feature fusion network described in step 212 to further optimize the feature maps and obtain the detection head feature maps TC1, TC2, and TC3.

[0114] Step 23: Map the ROI region information normalized in step 1 to the detection head feature map obtained in step 22 to generate an ROI feature map. Specifically, let the height and width of the detection head feature map TCi be h and h respectively. ti 、w ti , then the position information of the ROI area in the corresponding TCi feature map is The feature map of the ROI area in the detection head feature map mapping area is intercepted as the ROI feature map.

[0115] In step 24, the obtained ROI feature map is compared with the true label, and the ROI feature map that best matches the true label is selected as the initial prediction box feature map.

[0116] In step 241, the ROI feature map is first processed using softmax to divide the ROI feature map into a foreground image and a background image, and the ROI feature map belonging to the foreground image is screened out. Softmax is an existing binary classification method that can be used to identify whether the ROI feature map contains a target. When the ROI feature map does not contain a traffic light, the ROI feature map is classified as a background image. When the ROI feature map contains a traffic light, the ROI feature map is classified as a foreground image.

[0117] In step 242, each filtered foreground image ROI feature map is combined with the true label of the corresponding image one by one to calculate the CIOU value.

[0118] Among them, CIOU is an existing advanced IOU algorithm, which can consider factors such as the center point distance, IOU, and aspect ratio similarity between the ROI feature map and the true label, and can more comprehensively evaluate the matching relationship between the ROI feature map information and the label information. The larger the CIOU, the more matching the ROI feature map and the label.

[0119] Step 243: Each foreground image ROI feature map retains a best true label that matches it and stores it in an array, denoted as R Best , where R Best Each item includes the following {I roi , I gt , V ciou}, where I roi Contains information related to the ROI feature map, I gt Contains the best true label that matches the ROI feature map, Vciou Represents the CIOU value between the foreground image ROI feature map and its best matching true label.

[0120] Step 244, R Best Array Press V ciou Sort by value in descending order. Then press V ciou Get R values ​​one by one from large to small Best Array elements. Let GT be the best matching array between the real label and the ROI region feature map. Initially, GT is initialized to empty, and then the obtained R is determined. Best I of array element gt Is it already included in the GT element? If not, it is considered that the I roi , I gt For the best matching relationship, store the element in GT; if R Best I of array element gt If it already exists in the GT element, it means that the best matching ROI feature map for the real label has been found and the GT array is not updated. Best When the array is completely traversed, the process is stopped and the I of the element in GT is finally saved. roi That is the best matching ROI feature map of the real label, and the I of the element in GT roi As the initial prediction box feature map.

[0121] Step 25: Classify and fine-tune the filtered initial prediction box feature map to obtain the final traffic light prediction box information. Specifically, the filtered initial prediction box feature map is sent to the fully connected layer of the feature decoder for classification to obtain the prediction box category and confidence. The initial prediction box feature map size is then corrected through the regression module of the feature decoder to obtain the final prediction box size information. The prediction box category and confidence and prediction box size information are combined to form the traffic light prediction box information [class, conf, box], where class represents the prediction box category, conf represents the confidence of the prediction box category, and box represents the prediction box size information. Preferably, the regression module adopts the bounding box regression module in the Faster RCNN network.

[0122] In step 26, the loss function between the traffic light prediction box and the true label is calculated through the Loss function to determine the evolution direction of the network parameters and continuously optimize the network model.

[0123] Specifically, the Loss function refers to the form of the YOLOv5 model loss function, in which the loss function is composed of three parts, including the target box loss function, the confidence loss and the classification loss. The confidence loss and the classification loss are the same as the loss function of the YOLOv5 model, using the cross entropy loss function form. The target box loss function is improved on the basis of CIOU. The CIOU loss function takes into account the intersection-over-union (IOU) between the predicted box and the true label box, the ratio of the coordinate distance of the center point of the predicted box and the true label box to the diagonal length of the minimum circumscribed rectangle of the predicted box and the true label box, and the ratio of the height-to-width ratio of the predicted box to the height-to-width ratio of the true label box. However, the CIOU algorithm does not consider the direct similarity relationship between the height and width of the predicted box and the true label box, so the present invention introduces independent height and width terms. In addition, under the premise that the height and width similarity relationship between the predicted box and the label true box is consistent, the predicted box surrounding the label true box is more easily accepted in the subjective detection sense than the label true box surrounding the predicted box. To this end, the present invention introduces an area term, and considers the above situation through the area ratio of the label true box to the predicted box. In particular, the height term and the width term are both in square form, while the area term is in quadratic form, so that the area term contributes less to the loss, while the height term and the width term contribute more to the loss. Because the situation where different predicted boxes and the label true boxes have the same height and width similarity relationship rarely occurs, the area term loss is used as a secondary loss term.

[0124] The target box loss function formula used in this invention is as follows:

[0125]

[0126] Where NCIoU is the target box loss function used in this paper, CIoU is the target box loss function commonly used in the field of target detection, and e is a natural constant. w is the width factor, where w p is the prediction box width, w g is the width of the ground-truth box of the label. where h p is the prediction box height, h g is the height of the ground-truth box, θ h is the height factor; Represents the area ratio of the true label box to the predicted box.

[0127] In step 3, we optimize the model parameters using the gradient descent method using the loss function. We adjust the hyperparameters of the trained network model using the trainval dataset, and evaluate the final network model using the test dataset. If the loss level reaches a plateau during training and network performance shows no improvement after three consecutive training cycles, the final trained model is selected as the optimal model, completing the training process.

[0128] On the other hand, the present invention provides a traffic light recognition network testing method based on visual ranging, such as Figure 9 The specific steps are as follows:

[0129] In step S1, the test vehicle starts the camera and obtains the distance between the vehicle and the traffic light in front of the road in real time through the ROI extraction system. If the distance is less than the threshold, the traffic light detection module is started. If it is greater than the threshold, monitoring continues.

[0130] In step S2, the image frames captured by the camera are marked with timestamps as network input, and the ROI information extracted from the corresponding timestamps is fed into the optimal model obtained in step 3, and the predicted box information [class, conf, box] is output.

[0131] Step S3: In order to remove redundant prediction box information, the prediction box output in step S2 needs to be further screened, such as Figure 8 The specific steps are as follows:

[0132] Step S31: All prediction frame information is filtered based on conf. Specifically, all prediction frames are traversed. When conf>T c When , the prediction box is retained, otherwise the prediction box is directly removed. c is the category screening threshold, T c The default value is 0.01.

[0133] In step S32, the retained prediction boxes are grouped by category. Prediction boxes of the same category are grouped together. The camera image captured in step 1 may contain one or more of the aforementioned traffic light categories, and grouping is performed based on the traffic light category of each prediction box. Specifically, when the prediction box information contains class="round green light," the box of the prediction box information remains unchanged. When the prediction box information contains class="round red light," a relative maximum value V is added to each box item in the prediction box information to ensure complete separation of the categories and no positional interference. Similarly, when the prediction box information contains class="round yellow light," a value of 2*V is added to each box item in the prediction box information.

[0134] By analogy, when the prediction box information contains class="straight arrow-shaped yellow light", 3*V is added to each box item of the prediction box information. When the prediction box information contains class="straight arrow-shaped red light", 4*V is added to each box item of the prediction box information. When the prediction box information contains class="straight arrow-shaped green light", 5*V is added to each box item of the prediction box information. The default value of V is 4096.

[0135] When the prediction box information contains class="left turn arrow shaped yellow light", 6*V is added to each box item of the prediction box information. When the prediction box information contains class="left turn arrow shaped red light", 7*V is added to each box item of the prediction box information. When the prediction box information contains class="left turn arrow shaped green light", 8*V is added to each box item of the prediction box information. The default value of V is 4096.

[0136] When the prediction box information contains class="right turn arrow shaped yellow light", 9*V is added to each box item of the prediction box information. When the prediction box information contains class="right turn arrow shaped red light", 10*V is added to each box item of the prediction box information. When the prediction box information contains class="right turn arrow shaped green light", 11*V is added to each box item of the prediction box information. The default value of V is 4096.

[0137] In step S33, a maximum suppression (NMS) operation is performed on each of the grouped prediction boxes, and the output of the maximum suppression for each category is retained. NMS is a commonly used prediction box screening method in object detection algorithms. NMS removes redundant prediction boxes by using the intersection-over-union ratio between the prediction boxes.

[0138] The boxes of the retained prediction boxes are subtracted from the relative maximum values ​​of the corresponding categories to restore the original position information of each prediction box.

[0139] Step S34: Screening a reasonable prediction frame combination. The specific process is as follows:

[0140] Step S341: Classify and filter the predicted box categories. Specifically, for the same frame image, there is a fixed combination relationship between the true labels of each traffic light. According to the new national standard, there are 7 common combinations of traffic light states: 1) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped green light; 2) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 3) left-turn arrow-shaped green light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 4) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light; 5) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 6) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 7) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light; In particular, whether a yellow light appears for turning left, going straight or turning right, it is considered to be a reasonable traffic light state.

[0141] The grouped prediction boxes are divided into three categories: 1) Left turn category: left turn arrow-shaped red light, left turn arrow-shaped green light, left turn arrow-shaped yellow light; 2) Straight go category: straight go arrow-shaped red light, straight go arrow-shaped green light, straight go arrow-shaped yellow light, circular red light, circular green light, circular yellow light; 3) Right turn category: right turn arrow-shaped red light, right turn arrow-shaped green light, right turn arrow-shaped yellow light;

[0142] First, within each broad category, prediction boxes are filtered using the NMS method and then sorted in descending order by conf size. Next, the remaining prediction boxes from each broad category are individually combined, and prediction boxes that do not meet the above seven combination forms are eliminated. In particular, if the combined prediction box target only contains part of the combined target, it is considered a reasonable prediction and is retained.

[0143] Step S342, traverse the reserved prediction box combinations in turn, and filter the prediction boxes in each prediction box combination according to the geometric position relationship. Specifically, the prediction box information of each combination is represented as follows: The left turn prediction box information is represented as follows [conf l ,x 1l ,y 1l ,x 2l ,y 2l ,],conf l Represents the confidence of the left-turn prediction box category, x 1l Indicates the horizontal coordinate value of the upper left corner of the left-turn prediction box, y 1l Indicates the ordinate of the upper left corner of the left-turn prediction box, x 2l Indicates the horizontal coordinate value of the lower right corner of the left-turn prediction box, y 2l Indicates the vertical coordinate of the lower right corner of the left-turn prediction box; the right-turn prediction box information is as follows [conf r ,x 1r ,y 1r ,x 2r ,y 2r ], conf r Represents the confidence level of the right-turn prediction box category, x 1r Indicates the horizontal coordinate value of the upper left corner of the right-turn prediction box, y 1r Indicates the vertical coordinate of the upper left corner of the right-turn prediction box, x 2r Indicates the horizontal coordinate value of the lower right corner of the right-turn prediction box, y 2r Indicates the vertical coordinate of the lower right corner of the right turn prediction box; the straight line prediction box information is as follows [conf s ,x 1s ,y 1s ,x 2s ,y 2s ], conf s Indicates the confidence of the straight line prediction box category, x 1s Indicates the horizontal coordinate value of the upper left corner of the straight line prediction box, y1s Indicates the vertical coordinate of the upper left corner of the straight line prediction box, x 2s Indicates the horizontal coordinate value of the lower right corner of the straight line prediction box, y 2s Indicates the vertical coordinate of the lower right corner of the straight-ahead prediction box; for a traffic light combination at a traffic intersection, the left-turn signal light, the straight-ahead signal light, and the right-turn signal light should be approximately equidistantly distributed in the horizontal direction. That is, the horizontal distance between the left-turn prediction box and the straight-ahead prediction box is Horizontal distance between the right-turn prediction box and the straight-ahead prediction box When |X ls -X rs When |≤δ, the prediction box combination is retained, otherwise the prediction box is eliminated, where δ is the distance similarity threshold, and the default value is set to max(0.4X ls , 0.4X rs ), max() means taking the maximum value of the two.

[0144] In step S4, among the retained prediction box combinations, the average Conf value of each retained prediction box combination is calculated, and the prediction box combination with the highest average Conf is retained as the final prediction box output.

[0145] In particular, in the above screening process, if there is no prediction box information that meets the conditions, a null value is directly returned without subsequent screening.

[0146] In some embodiments, in order to ensure that the system has good detection performance under various road conditions, the test process tests the detection effect under difficult detection conditions such as good lighting conditions, foggy weather, rainy and snowy weather.

[0147] In order to test the effect of the method of the present invention on the task of signal light recognition, the method of the present invention is compared with the YOLOv3 method in the prior art in a real traffic environment. Figure 10 As shown, compared with the YOLOv3 algorithm, the detection accuracy of the method described in the present invention is higher. In the YOLOv3 algorithm test effect diagram, the traffic light surrounded by the red frame has a false detection (the true value of the red frame I is SAG, and the false detection is LTAG; the true value of the red frame II is RTAY, and the false detection is SAR), while the method described in the present invention overcomes the above problems. In addition, by comparing the YOLOv3 algorithm and the method described in the present invention, it is found that the model complexity of the method described in this article is relatively low, which significantly reduces the search space of the traffic light target and ensures the real-time performance of the algorithm. Under the conditions of Ubuntu18.04 system and NVIDIA 3060 graphics card, the single-frame image detection time of the method described in this article is 8ms, while the single-frame image detection time of the YOLOv3 model is 15ms. It can be seen that the use of the method described in the present invention shortens the image search time and can achieve better traffic light recognition effect.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A traffic light recognition network training method based on visual ranging, characterized in that: The following steps are involved: Step 1: Label the collected images and create a traffic light dataset: the traffic light dataset consists of a traffic light image, normalized ROI information, and true label information as a basic data unit; the dataset is divided into three sub-datasets: train, trainval, and test in an 8:1:1 ratio. The train and trainval datasets are used for training and performance verification of the training process; the test dataset is used for the final evaluation of network performance; Step 2: Use the data set created in step 1 to train the traffic light recognition network model: Step 21: First, establish a traffic light recognition network model, which includes three parts: backbone network, feature fusion network and feature decoder; Step 22: Send the traffic light images captured by the camera in the training set to the backbone network of the traffic light recognition network model to extract image features and obtain a feature map; The extracted feature maps are then fed into the feature fusion network to obtain detection head feature maps TC1, TC2, and TC3; Step 23: Map the ROI area information normalized in step 1 to the detection head feature map obtained in step 22 to generate the ROI feature map: Assume that the height and width of the detection head feature map are h and h respectively. ti 、w ti , then the position information of the ROI area in the corresponding detection head feature map is The feature map of the ROI area in the detection head feature map mapping area is intercepted as the ROI feature map; Among them, P w 、P h are the image width and height, X c is the x-axis coordinate of the center point of the ROI area, X c The horizontal coordinate X of the center point of the image is 中心 As the sampling center, the Gaussian probability density function is used in [0,P w ] range of sampling; Y c Y is the y-axis coordinate of the center point of the ROI area, c The value of Y cmax to Y cmin Random sampling is obtained within the range; the upper and lower boundaries of the y-axis coordinate value of the center point of the ROI area are respectively recorded as Y cmax , Y cmin , where Y cmax =H 板1 +αh1,Y cmin =H 板1 -αh1, where α is the adaptive coefficient, H 板 is the height of the center point of the signal light board, f is the focal length of the camera, H 板1 H 板 The projection in the image coordinate system, h1 is the projection of the vertical length h of the signal light board in the image coordinate system; S c is the horizontal distance between the vehicle and the traffic light, R is the preset threshold, and e is a natural constant; Step 24: Compare the obtained ROI feature map with the true label obtained in step 1, and select the ROI feature map that best matches the true label as the initial prediction box feature map; Step 25: The feature decoder classifies and fine-tunes the filtered initial prediction frame feature map to obtain the final traffic light prediction frame information; Step 26: Calculate the loss function between the traffic light prediction box and the true label through the Loss function, determine the evolution direction of the network parameters, and optimize the traffic light recognition network model; Step 3: Use the loss function and gradient descent method to optimize the model parameters, use the trainval sub-dataset to adjust the network model hyperparameters of the trained network model, and use the test sub-dataset to evaluate the final network model; when the loss tends to be flat during training and the network performance has not improved for three consecutive training cycles, select the last trained model as the optimal model to complete the training process.

2. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: The ROI area information in step 1 is obtained by extracting the ROI extraction system, which includes a GPS module, an electronic map module, a distance measurement module and a ROI extraction module; wherein, GPS module obtains vehicle real-time location information (X 1c , Y 1c ) and record the timestamp as Tc; at time Tc, the electronic map module extracts the signal light position coordinates (X L , Y L The ranging module calculates the horizontal distance Sc between the vehicle and the traffic light at time Tc using the following formula; The ranging module determines whether the distance Sc between the vehicle and the traffic light exceeds the preset threshold R: If Sc is greater than R, the ROI extraction system is not activated, and the ranging module enters the next moment loop to continue detecting the relationship between Sc and R. When Sc is less than or equal to R, it indicates that the vehicle has entered the traffic light recognition area, and the ROI extraction system is activated; The ROI extraction module is used to determine the ROI area information of the traffic light and normalize the ROI area feature vector. The normalized ROI area feature vector is expressed as: Among them, P w 、P h are the image width and height respectively.

3. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: In step 2, the backbone network includes three CRC modules and three CRT modules that are alternated in sequence, each CRC module includes a CONV module, a Residual module, and a CBAM module, and each CRT module includes a CONV module, a Residual module, and a Transform module; Among them, the CONV module is a convolution downsampling module, which downsamples the input feature map by a factor of two; the Residual module is a residual network structure, which reduces the gradient vanishing problem caused by network deepening through jump connections of internal residual blocks; the Transform module is an attention network model; the CBAM module includes two parts: channel attention and spatial attention. The CBAM module infers the attention weights along the spatial and channel dimensions in sequence, and can extract target information from image information.

4. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: The method for the feature fusion network to obtain the detection head feature map in step 2 is: First, select the output of the third CRT module of the backbone network, denoted as T3; send T3 to the CONV module for 1×1 convolution, and then send the output to the upsample module. Each upsample module will upsample the feature map once. After two upsamplings, the feature map is concatenated with the feature map output by the second CRT module of the backbone network using the Concat method. The output is further extracted by the CBAM module, and the final output is denoted as T2. Concat is a feature map concatenation method commonly used in deep learning networks. Send T2 to the CONV module, use a 1×1 convolution kernel to adjust the number of channels, and then send it to the upsample module for two upsampling. The upsampling result is concatenated with the output of the first CRT module of the backbone network using the Concat method for feature splicing, and the output is further extracted by the CBAM module. The final output result is recorded as T1; Select the first CRC module of the backbone network, denoted as C1, send C1 to the CONV module, use a 3×3 convolution kernel to perform two convolution downsamplings, then use the Concat method to concatenate the downsampled output with the output of the second CRC module of the backbone network, and then use the Transform module to extract effective features, and the output feature map is denoted as C2; send C2 to the CONV module, perform two downsamplings, concatenate the downsampled result with the output of the third CRC module of the backbone network using the Concat method, and then use the Transform module to extract features, and the output feature map is denoted as C3; After obtaining T1, T2, T3, C1, C2, and C3, T1 is input into the upsample module, and after upsampling, it is concatenated with C1 using the Concat method to output a feature map as the detection head, recorded as TC1; T2 is input into the upsample module, and after upsampling, it is concatenated with C2 using the Concat method to output a feature map as the detection head, recorded as TC2; T3 is input into the upsample module, and after upsampling, it is concatenated with C3 using the Concat method to output a feature map as the detection head, recorded as TC3.

5. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: The step 24 is specifically as follows: Step 241: Apply softmax processing to the ROI feature map to divide the ROI feature map into a foreground image and a background image, and select the ROI feature map belonging to the foreground image; Step 242: Combine each filtered foreground image ROI feature map with the true label of the corresponding image one by one to calculate the CIOU value; Step 243: Each foreground image ROI feature map retains a best matching true label and stores it in array form, denoted as R Best , where R Best Each item includes the following {I roi , I gt , V ciou }, where I roi Represents ROI feature map related information, I gt represents the best true label matching the ROI feature map, V ciou Represents the CIOU value between the foreground image ROI feature map and the best matching true label; Step 244, R Best Array Press V ciou Sort by value in descending order, then by V ciou Get R values ​​one by one from large to small Best Elements of the array; let GT be the best matching array between the real label and the ROI region feature map, the initial condition initializes GT to empty, and then judge the obtained R Best I of array element gt Is it already included in the elements of GT? If not, it is considered that the I of the array element roi , I gt For the best matching relationship, store the array element in GT; if R Best I of array element gt If it already exists in the element of GT, it means that the best matching ROI feature map of the real label has been found, and the GT array is not updated; when the length of GT is equal to the number of image labels, or R Best When the array is completely traversed, the process is stopped and the I of the element in GT is finally saved. roi That is the best matching ROI feature map of the real label, and the I of the element in GT roi As the initial prediction box feature map.

6. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: The Loss function consists of three parts, including the target box loss function, confidence loss, and classification loss. Among them, the confidence loss and classification loss use the cross entropy loss function form. The target box loss function formula is as follows: Among them, NCIoU is the target box loss function, CIoU is the target box loss function commonly used in the field of target detection, e is a natural constant; width factor Among them, w p is the prediction box width, w g is the width of the ground truth box of the label; height factor Among them, h p is the prediction box height, h g is the height of the ground-truth box of the label, Represents the area ratio of the true label box to the predicted box.

7. The method for training a traffic light recognition network based on visual ranging according to claim 1, characterized in that: The step 25 is specifically as follows: The filtered initial prediction box feature map is sent to the fully connected layer of the feature decoder for classification to obtain the prediction box category and confidence. Then, the regression module of the feature decoder is used to correct the size of the initial prediction box feature map to obtain the final prediction box size information. The category and confidence of the prediction box are combined with the prediction box size information to form the traffic light prediction box information [class, conf, box], where class represents the prediction box category, conf represents the confidence of the prediction box category, and box represents the prediction box size information. The regression module adopts the bounding box regression module in the Faster RCNN network.

8. A traffic light recognition network testing method based on visual ranging, characterized in that: The following steps are involved: Step S1: The test vehicle activates the camera and obtains the distance between the test vehicle and the traffic light ahead in real time through the ROI extraction system. If the distance is less than a threshold, the traffic light detection module is activated; if the distance is greater than the threshold, monitoring continues; Step S2: The image captured by the camera is marked with a timestamp as a network input, and the ROI information extracted from the corresponding timestamp is fed into the trained model obtained by the method of any one of claims 1 to 7, and the prediction box information [class, conf, box] is output; where class represents the prediction box category, conf represents the confidence of the prediction box category, and box represents the prediction box size information; Step S3: further filter the prediction boxes outputted in step S2 and retain the most reasonable prediction box combination; Step S4: Calculate the average Conf value of each retained prediction box combination among the retained prediction box combinations, and retain the prediction box combination with the highest average Conf as the final prediction box output.

9. The method for testing a traffic light recognition network based on visual ranging according to claim 8, characterized in that: The step S3 is specifically as follows: Step S31: filter all prediction frame information based on conf: traverse all prediction frames, and when conf>T c When , the prediction box is retained, otherwise the prediction box is directly removed; T c Filter thresholds for categories; Step S32: Group the retained prediction frames by category, with the prediction frames of the same category grouped together; Step S33: Perform maximum suppression operation on the grouped prediction boxes of each category, and retain the output results after maximum suppression of each category; subtract the relative maximum value of the corresponding category from the box of the retained prediction box to restore the original position information of each prediction box; Step S34: Screen reasonable prediction frame combinations.

10. A traffic light recognition network testing method based on visual ranging according to claim 9, characterized in that: The step S34 is specifically as follows: Step S341: Classify and filter the predicted box categories: For the same frame image, the traffic lights are grouped according to the fixed combination relationship between the true labels of each light. There are 7 combinations in total: 1) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped green light; 2) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 3) left-turn arrow-shaped green light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 4) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light; 5) left-turn arrow-shaped green light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped green light; 6) left-turn arrow-shaped red light, straight-ahead arrow-shaped red light or circular red light, right-turn arrow-shaped red light; 7) left-turn arrow-shaped red light, straight-ahead arrow-shaped green light or circular green light, right-turn arrow-shaped red light. Whether the yellow light appears when turning left, going straight or right, it is a reasonable traffic light state; The grouped prediction boxes are divided into three categories: 1) Left turn category: left turn arrow-shaped red light, left turn arrow-shaped green light, left turn arrow-shaped yellow light; 2) Straight go category: straight go arrow-shaped red light, straight go arrow-shaped green light, straight go arrow-shaped yellow light, circular red light, circular green light, circular yellow light; 3) Right turn category: right turn arrow-shaped red light, right turn arrow-shaped green light, right turn arrow-shaped yellow light; Within each major category, the prediction boxes are screened using the NMS method and then sorted in descending order by conf size. Then, the prediction boxes retained in each major category are traversed and combined one by one, and the prediction boxes that do not meet the above 7 combination forms are eliminated. If the combined prediction box target only contains part of the combination target, it is also considered a reasonable prediction and the prediction box is retained. Step S342: traverse the reserved prediction box combinations in sequence, and filter the prediction boxes in each prediction box combination according to their geometric position relationship: The prediction box information of each combination is as follows: The prediction box information of the left turn class is as follows [conf l ,x 1l ,y 1l ,x 2l ,y 2l ,],conf l Represents the confidence of the left-turn prediction box category, x 1l Indicates the horizontal coordinate value of the upper left corner of the left-turn prediction box, y 1l Indicates the ordinate of the upper left corner of the left-turn prediction box, x 2l Indicates the horizontal coordinate value of the lower right corner of the left-turn prediction box, y 2l Indicates the vertical coordinate of the lower right corner of the left-turn prediction box; the right-turn prediction box information is as follows [conf r ,x 1r ,y 1r ,x 2r ,y 2r ], conf r Represents the confidence level of the right-turn prediction box category, x 1r Indicates the horizontal coordinate value of the upper left corner of the right-turn prediction box, y 1r Indicates the vertical coordinate of the upper left corner of the right-turn prediction box, x 2r Indicates the horizontal coordinate value of the lower right corner of the right-turn prediction box, y 2r Indicates the vertical coordinate of the lower right corner of the right turn prediction box; the straight line prediction box information is as follows [conf s ,x 1s ,y 1s ,x 2s ,y 2s ], conf s Indicates the confidence of the straight line prediction box category, x 1s Indicates the horizontal coordinate value of the upper left corner of the straight line prediction box, y 1s Indicates the vertical coordinate of the upper left corner of the straight line prediction box, x 2s Indicates the horizontal coordinate value of the lower right corner of the straight line prediction box, y 2s Represents the vertical coordinate of the lower right corner of the straight-ahead prediction box; for a traffic light combination at a traffic intersection, the left-turn signal light, the straight-ahead signal light, and the right-turn signal light are equally spaced in the horizontal direction, that is, the horizontal distance between the left-turn prediction box and the straight-ahead prediction box is Horizontal distance between the right-turn prediction box and the straight-ahead prediction box When |X ls -X rs When |≤δ, the prediction box combination is retained, otherwise the prediction box is eliminated, where δ is the distance similarity threshold, and the default value is set to max(0.4X ls , 0.4X rs ), max() means taking the maximum value of the two.

Images