A lightweight automobile tail light language real-time recognition method

Abstract

The application discloses a kind of light-weighted car tail lamp language real-time identification method, comprising the following steps: design MCA-YOLOv5s network obtains the information and confidence of vehicle tail detection frame;Obtain the tracking sequence of vehicle tail image by Bytetrack module;Design the identification tail lamp language based on the three-dimensional space-time convolutional neural network TSA-X3d network of block attention and mix shuffle attention.The application designs MCA-YOLOv5s network to detect the vehicle tail in traffic scene, obtains the tracking sequence of vehicle tail by Bytetrack network, finally designs TSA-X3d network to extract the space-time feature of tracking sequence to obtain tail lamp language.The overall model of the application is small, and the hardware requirement is low.After training the system model offline, it is convenient to deploy on the device with limited resources, greatly improves the safety of driving, realizes the accurate and efficient of language identification.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, specifically to a lightweight method for real-time recognition of automotive taillight signals. Background Technology

[0002] With the development of autonomous driving and intelligent connected vehicle technologies, intelligent vehicles have become a hot research topic in the field of vehicle engineering. When intelligent vehicles are driving in complex traffic scenarios, they need to collect as much information about their surroundings as possible to help them drive safely. Taillights are a very important feature of vehicles ahead, containing extremely rich semantic information. When the brake lights illuminate, it indicates that the vehicle ahead is slowing down, alerting following vehicles to take appropriate measures, which is beneficial to the safe driving of intelligent vehicles.

[0003] Most existing methods for vehicle taillight status recognition rely on edge extraction or color thresholding. However, vehicle taillights are small targets, making edge extraction difficult and hindering the extraction of complete edge information, thus hindering taillight recognition. Color thresholding, on the other hand, is easily affected by ambient light and weather conditions, resulting in poor stability. Currently, some solutions incorporate deep learning into vehicle taillight status recognition, such as a method for identifying and determining the status of taillights of vehicles ahead (patent application number: 201710443283.0). This method uses two YOLO neural networks to detect the taillight position, acquires the HSV value of the taillight at regular frame intervals, and compares the difference between two adjacent HSV values ​​with a threshold to determine the taillight signal. While this method can identify taillight signals to some extent, it is not suitable for recognizing signals from multiple vehicles and ignores temporal information. For example, a vehicle taillight state recognition method based on action-state joint learning (patent application number: 202110519911.5) uses a CNN-LSTM model to identify the action features of brake lights and turn signals based on vehicle detection and tracking. It extracts the brightness features of the high-mounted brake light through semantic segmentation, combines the brightness and action features to form higher-order features, and finally uses a linear chain conditional random field model to infer the taillight signal by analyzing the higher-order features. While this method utilizes temporal information, it cannot focus on the rear of the vehicle and does not require feature extraction from the vehicle's sides. The model is overly complex and fails to meet real-time requirements.

[0004] Both of the above patents use the YOLO series network for vehicle detection. Among the YOLO series networks, the YOLOv5 network is widely used due to its high accuracy and speed. It includes five versions. Taking YOLOv5s as an example, the structure is shown below:

[0005] Input layer → First convolutional layer → Second convolutional layer → First C3 structure → Third convolutional layer → Second C3 structure → Fourth convolutional layer → Third C3 structure → Fifth convolutional layer → Fourth C3 structure → Spatial pyramid pooling structure → Sixth convolutional layer → Channel fusion of the upsampled feature map and the feature map generated by the third C3 structure → Fifth C3 structure → Seventh convolutional layer → Channel fusion of the upsampled feature map and the feature map generated by the second C3 structure → Sixth C3 structure → Eighth convolutional layer → Channel fusion of the feature map generated by the eighth convolutional layer and the feature map generated by the seventh convolutional layer → Seventh C3 structure → Ninth convolutional layer → Channel fusion of the feature map generated by the ninth convolutional layer and the feature map generated by the sixth convolutional layer → Eighth C3 structure.

[0006] The C3 structure consists of three convolutional layers. The first convolutional layer to the spatial pyramid pooling structure is the backbone network, the sixth convolutional layer to the eighth C3 structure is the bottleneck network, and the last three C3 structures generate three prediction maps through ordinary convolution, which is part of the prediction network. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, this invention designs a lightweight real-time taillight signal recognition method for automobiles. This method not only achieves a good balance between accuracy and real-time performance but is also easily deployed on resource-constrained edge terminals, helping to determine the driving intentions of vehicles ahead and ensuring the safe operation of intelligent vehicles.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: A lightweight method for real-time recognition of automotive taillight signals, comprising the following steps:

[0009] A. Design an MCA-YOLOv5s network to acquire information and confidence levels of the vehicle's rear detection bounding box.

[0010] A1. Labeling the rear of vehicles in traffic scenes: Obtain vehicle rear detection datasets through public datasets or dashcams, and label the rear of vehicles in traffic scenes in YOLO format. In YOLO format, each image corresponds to a text file, and each line of the text file corresponds to information about the rear of a vehicle. Each line includes 5 numbers: the first number represents the target category, the second number represents the ratio of the horizontal coordinate of the vehicle rear image center to the image width, the third number represents the ratio of the vertical coordinate of the vehicle rear image center to the image height, the fourth number represents the ratio of the width of the vehicle rear rectangle to the image width, and the fifth number represents the ratio of the height of the vehicle rear rectangle to the image height. Adjacent numbers are separated by spaces.

[0011] A2. Establishing the MCA-YOLOv5s network: The YOLOv5s network is improved. The YOLOv5s network is a single-stage object detection network, including a backbone network, a bottleneck network, and a prediction network. The backbone network consists of a feature extraction network and a spatial pyramid pooling structure; the bottleneck network consists of a feature pyramid network and a C3 structure; and the prediction network consists of ordinary convolutional layers. The C3 structure consists of three convolutional layers. The backbone network in the YOLOv5s network is replaced with 11 Cneck modules. Finally, all C3 structures are replaced with depthwise separable convolutions to obtain the final MCA-YOLOv5s network. The MCA-YOLOv5s network is a YOLOv5s network based on coordinate attention and inverse residual structures; the Cneck modules are inverse residual structure modules.

[0012] A3. Training the vehicle rear detection dataset: Divide the vehicle rear detection dataset into a training set and a validation set according to an a:b ratio, and input them into the MCA-YOLOv5s network for training. The training set is used to train the MCA-YOLOv5s network, and the validation set is used to provide feedback on the training results. The weight file that performs best on the validation set is saved to obtain the vehicle rear detection box information and confidence score.

[0013] A4. Reduce network parameter count using depthwise separable convolutional structures: Divide the depthwise separable convolution in step A2 into channel-wise convolution and pointwise convolution. In channel-wise convolution, each channel of the concatenated feature map in the MCA-YOLOv5s network is convolved with only one kernel to obtain a new feature map. Pointwise convolution weights and combines the new feature maps obtained from channel-wise convolution in the depth direction to generate the final feature map.

[0014] A5. Use the cneck module structure to reduce the number of network parameters: The cneck module in step A2 first increases the dimensionality of the input feature map through a 1×1 convolution, then performs a channel-wise depthwise convolution operation on the increased dimensionality feature map through a 3×3 convolution, uses the coordinate attention module to obtain the channel information and position information of the input feature map, then reduces the dimensionality of the feature map through a 1×1 convolution, and finally performs residual skip connections to alleviate the gradient vanishing phenomenon that occurs as the network deepens.

[0015] A6. Enhance network feature extraction capabilities using a coordinate attention module structure: Perform global average pooling on the coordinate attention module from step A5 in both the X and Y directions to preserve the long-range dependencies between the channel attention in both directions. Then, concatenate the information from the two directions and perform a convolution operation to allow the information from the two directions to interact. Based on this, decompose the feature map and perform convolution operations separately, simultaneously focusing on the X and Y direction information of the feature map to obtain its positional information. The relevant formulas are as follows:

[0016]

[0017]

[0018]

[0019] f=δ(F1([z h ,z w ]))

[0020] g h =σ(F h (f h ))

[0021] g w =σ(F w (f w ))

[0022]

[0023] In the formula, z c For global average pooling, and The outputs of the c-th channel with height h and width w are respectively, f is the feature after convolution transformation, and g is the output of the c-th channel with height h and width w respectively. h g represents the features of the feature map after convolution along the X direction. w The feature map is the result of a convolution transformation along the Y direction, y c (i,j) represents the output of the coordinate attention module after passing through the value of the c-th channel of the input feature map in the i-th row and j-th column. H and W are the width and height of the feature map, respectively, and σ and δ are the activation functions. c (i,j) represents the value of the c-th channel of the input feature map in the i-th row and j-th column, x c (h,i) represents the value of the c-th channel of the input feature map in the h-th row and i-th column, x c (j, w) represents the value of the c-th channel of the input feature map in the j-th row and w-th column, z h and z w The outputs of average pooling of the feature maps in the h and w directions are F1 and F2, respectively. h F w All are 1×1 convolutions. and These are the features of the c-th channel feature map after convolution transformation along the X and Y directions, respectively.

[0024] A7. Obtain information and confidence of vehicle rear detection boxes: Use the trained MCA-YOLOv5s network to detect traffic scenes from the vehicle's perspective and obtain information and confidence of vehicle rear detection boxes.

[0025] B. Obtain the tracking sequence of vehicle rear images using the Bytetrack module.

[0026] The vehicle rear detection bounding box information and confidence score obtained in step A are input into the Bytetrack module to obtain the tracking sequence of the vehicle rear image. The Bytetrack module takes as input the predicted bounding box obtained after Kalman filtering of the vehicle rear image location in the previous frame and the detection bounding box in the current frame, and classifies the detection bounding boxes into high-scoring and low-scoring boxes according to a threshold. The calculation method of the Bytetrack module includes the following steps:

[0027] B1. The high-resolution detection box and the predicted box obtained after Kalman filtering of the vehicle rear image in the previous frame are matched by appearance feature matching or IoU matching. Finally, the successfully matched trajectory, the unmatched trajectory and the unmatched detection box are obtained. The unmatched detection box is used to generate a new tracking box. The IoU matching is the overlap matching.

[0028] B2. Perform IoU matching on unmatched trajectories and low-scoring detection boxes. Trajectories that still do not match after two matching attempts are retained for 30 frames and then deleted.

[0029] C. Design a 3D spatiotemporal convolutional neural network, TSA-X3d network, based on block attention and shuffling attention, to recognize taillight signals.

[0030] C1. Using publicly available datasets or dashcam image sequences of vehicle rear-end sequences, crop the images of vehicle rear-ends in traffic scenes at a certain frame rate and save them separately according to different vehicles and taillight signals. Specifically, based on the brake light and turn signal states, vehicle taillight signals are categorized into the following 8 types: OOO, BOO, OOR, OLO, BOR, BLO, OLR, and BLR. Here, OOO indicates the vehicle is stationary or moving straight; BOO indicates the vehicle is braking; OOR indicates the vehicle is preparing to turn right or is currently turning right; OLO indicates the vehicle is preparing to turn left or is currently turning left; BOR indicates the vehicle is preparing to turn right or is currently turning right and braking; BLO indicates the vehicle is preparing to turn left or is currently turning left and braking; OLR indicates the vehicle is in an emergency; and BLR indicates the vehicle is in an emergency and braking.

[0031] The first part of the C2, TSA-X3d network is the ResNetBasicStem, the residual network backbone, which consists of temporal and spatial convolutional layers. This is followed by four ResStages, each containing 3, 5, 11, and 7 ResBlocks, respectively. Each ResBlock contains 3 convolutional layers, 1 ReLU activation function, and 1 residual connection. The second convolutional layer in each ResBlock uses both spatial and temporal convolutions, both of which are depthwise separable. Between the two convolutions, a batch normalization layer, a Swish activation function, and an SA attention module are used. After the temporal convolution, a TSE attention module is used. Finally, the output is passed through a ConvPool convolutional pooling layer, an FCLayer fully connected layer, and an AdaptiveAvgPool3d adaptive average pooling layer.

[0032] C3. Divide the vehicle rear sequence dataset into training and validation sets according to the c:d ratio and input them into the TSA-X3d network for training. Save the weight file with the highest accuracy on the validation set for light signal recognition.

[0033] C4. The TSE attention module from step C2 first compresses the input feature map using an average pooling layer with a kernel size of 7. Then, it uses a 1×1 convolution to reduce the dimensionality of the compressed feature map before inputting it into the ReLU activation function (linear rectified function). Next, it uses a 1×1 convolution to increase the dimensionality of the ReLU activation function's output before inputting it into the Sigmoid activation function (S-shaped growth curve activation function). Finally, it uses nearest neighbor interpolation to restore the output of the Sigmoid activation function to the dimension of the input tensor and performs the following dot product operation with the feature map input to the TSE attention module:

[0034] S = F near (σ(W2·ReLU(W1·F ap (X))))

[0035]

[0036] In the formula, S is the output of the TSE attention module, S is the output after the convolution operation, X is the input feature map, σ is the sigmoid activation function, W1 and W2 are 1×1 convolutions, and F is the output of the TSE attention module. near It is the nearest neighbor interpolation method, ReLU is the activation function, and F... ap It is average pooling. It is a dot product operation.

[0037] C5. Divide the channel features of the SA attention module in step C2 into g groups. Each group's channel features are divided into two parts. The first part is compressed to 1×1×1 using an average pooling layer. The second part is processed by a group normalization layer without changing the input dimension. Then, both parts are processed by linear functions for feature extraction. Finally, after passing through a sigmoid activation function, the features are multiplied by the original feature values ​​to obtain the importance of different channels and spaces. After obtaining the weighted grouped features, the SA attention module merges the two parts by concatenation. Finally, the features from different groups are merged to restore the input dimension and then a "shuffling" operation is performed to enhance the information interaction between different groups and improve the network's ability to extract temporal and spatial information.

[0038] X′ k1 =σ(F c1 (F A (X k1 )))·X k1 =σ(W1S1+b1)·X k1

[0039] X′ k2 =σ(F c2 (F GN (X k2 )))·X k2 =σ(W2S2+b2)·X k2

[0040] In the formula, X' k1 X' is the output feature of the first part. k2 The output features of the second part are defined by σ, the activation function, S1 (features after average pooling), and S2 (features after normalization). W1 and b1 represent the weights and biases of the linear function in the first part, respectively, while W2 and b2 represent the weights and biases of the linear function in the second part. X k1 X is the input feature map for the first part. k2 For the input feature map of the second part, F A For global average pooling, F GN For group normalization, F c1 F is a linear function of the first part. c2 It is a linear function of the second part.

[0041] C6. Use TensorRT to accelerate the MCA-YOLOv5s network and perform quantization processing on the TSA-X3d network, converting the saved weight file from FP32 to FP16. TensorRT is a C++ inference framework running on NVIDIA graphics processors.

[0042] Furthermore, in step A3, the value of a:b is 7:3, 4:1, or 9:1; and in step C3, the value of c:d is 7:3, 4:1, or 9:1.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention designs an MCA-YOLOv5s network to detect the rear of vehicles in traffic scenarios, uses a Bytetrack network to obtain the tracking sequence of the vehicle's rear, and finally designs a TSA-X3d network to extract spatiotemporal features from the tracking sequence to obtain the taillight signals. The overall model of this invention is relatively small and has low hardware requirements. After offline training of the system model, it can be easily deployed on devices with limited resources, greatly improving driving safety.

[0045] 2. This invention uses TensorRT to accelerate the MCA-YOLOv5s network, reducing the vehicle tail detection time per frame to 1-3ms. It also performs quantization processing on the TSA-X3d network, converting the saved weight file from FP32 to FP16, increasing the number of segmented segments processed per second to 3.33 times the original, reducing the model size to 26.65% of the original, and achieving an overall model inference speed of over 25fps, thus realizing accurate and efficient light signal recognition. Attached Figure Description

[0046] Figure 1 This is the overall framework diagram.

[0047] Figure 2 This is a structural diagram of MCA-YOLOv5s.

[0048] Figure 3 This is the cneck module structure.

[0049] Figure 4 This is a coordinate attention module structure.

[0050] Figure 5 This is a schematic diagram of Bytetrack.

[0051] Figure 6 This is a structural diagram of TSA-X3d.

[0052] Figure 7 This is the TSE attention module structure.

[0053] Figure 8 This is the SA attention module structure.

[0054] Figure 9 The result of taillight signal recognition for Scene 1.

[0055] Figure 10The result is the taillight signal recognition result for Scene 2.

[0056] Figure 11 The result is the recognition result of the taillight signal in scene three.

[0057] Figure 12 The result is the recognition result of the taillight signal in the scene. Detailed Implementation

[0058] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand the advantages of the present invention from the content disclosed in this specification. The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. To provide a deep understanding of the present invention, many specific details will be included in the following description. Furthermore, to avoid confusion or obscuring the focus of the present invention, some specific details will be omitted in the description.

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be described in further detail below with reference to the accompanying drawings. The overall framework diagram is as follows: Figure 1 As shown, it includes the following steps:

[0060] A. Design an MCA-YOLOv5s network to acquire information and confidence levels of the vehicle's rear detection bounding box.

[0061] A1. Labeling the rear of vehicles in traffic scenes: Obtain vehicle rear detection datasets through public datasets or dashcams, and label the rear of vehicles in traffic scenes in YOLO format. In YOLO format, each image corresponds to a text file, and each line of the text file corresponds to information about the rear of a vehicle. Each line includes 5 numbers: the first number represents the target category, the second number represents the ratio of the horizontal coordinate of the vehicle rear image center to the image width, the third number represents the ratio of the vertical coordinate of the vehicle rear image center to the image height, the fourth number represents the ratio of the width of the vehicle rear rectangle to the image width, and the fifth number represents the ratio of the height of the vehicle rear rectangle to the image height. Adjacent numbers are separated by spaces.

[0062] A2. Establishing the MCA-YOLOv5s network: The YOLOv5s network is improved. The YOLOv5s network is a single-stage object detection network, including a backbone network, a bottleneck network, and a prediction network. The backbone network consists of a feature extraction network and a spatial pyramid pooling structure; the bottleneck network consists of a feature pyramid network and a C3 structure; and the prediction network consists of ordinary convolutional layers. The C3 structure consists of three convolutional layers. The backbone network in the YOLOv5s network is replaced with 11 Cneck modules. Finally, all C3 structures are replaced with depthwise separable convolutions to obtain the final MCA-YOLOv5s network. The MCA-YOLOv5s network is a YOLOv5s network based on coordinate attention and inverse residual structures; the Cneck modules are inverse residual structure modules.

[0063] A3. Training the vehicle rear detection dataset: The vehicle rear detection dataset is divided into training and validation sets in an a:b ratio and input into the MCA-YOLOv5s network for training. The training set is used to train the MCA-YOLOv5s network, and the validation set is used to provide feedback on the training results. The weight file that performs best on the validation set is saved to obtain vehicle rear detection box information and confidence scores. The number of iterations is 300, the batch size is 16, the initial learning rate is 0.01, the optimizer is SGD (Stochastic Gradient Descent), and the momentum factor is 0.937. Training platform configuration: Ubuntu 18.04 LTS system, Intel Xeon processor E5-2620 v4@2.10GHz×32, 32GB RAM, NVIDIA Titan graphics card, Python 3.8 programming language.

[0064] A4. Reduce network parameter count using depthwise separable convolutional structures: Divide the depthwise separable convolution in step A2 into channel-wise convolution and pointwise convolution. In channel-wise convolution, each channel of the concatenated feature map in the MCA-YOLOv5s network is convolved with only one kernel to obtain a new feature map. Pointwise convolution weights and combines the new feature maps obtained from channel-wise convolution in the depth direction to generate the final feature map.

[0065] A5. Use the cneck module structure to reduce the number of network parameters: The cneck module in step A2 first increases the dimensionality of the input feature map through a 1×1 convolution, then performs a channel-wise depthwise convolution operation on the increased dimensionality feature map through a 3×3 convolution, uses the coordinate attention module to obtain the channel information and position information of the input feature map, then reduces the dimensionality of the feature map through a 1×1 convolution, and finally performs residual skip connections to alleviate the gradient vanishing phenomenon that occurs as the network deepens.

[0066] A6. Enhance network feature extraction capabilities using a coordinate attention module structure: Perform global average pooling on the coordinate attention module from step A5 in both the X and Y directions to preserve the long-range dependencies between the channel attention in both directions. Then, concatenate the information from the two directions and perform a convolution operation to allow the information from the two directions to interact. Based on this, decompose the feature map and perform convolution operations separately, simultaneously focusing on the X and Y direction information of the feature map to obtain its positional information. The relevant formulas are as follows:

[0067]

[0068]

[0069]

[0070] f=δ(F1([z h ,z w ]))

[0071] g h =σ(F h (f h ))

[0072] g w =σ(F w (f w ))

[0073]

[0074] In the formula, z c For global average pooling, and The outputs of the c-th channel with height h and width w are respectively, f is the feature after convolution transformation, and g is the output of the c-th channel with height h and width w respectively. h g represents the features of the feature map after convolution along the X direction. w The feature map is the result of a convolution transformation along the Y direction, y c (i,j) represents the output of the coordinate attention module after passing through the value of the c-th channel of the input feature map in the i-th row and j-th column. H and W are the width and height of the feature map, respectively, and σ and δ are the activation functions. c (i,j) represents the value of the c-th channel of the input feature map in the i-th row and j-th column, x c (h,i) represents the value of the c-th channel of the input feature map in the h-th row and i-th column, x c (j, w) represents the value of the c-th channel of the input feature map in the j-th row and w-th column, z h and z w The outputs of average pooling of the feature maps in the h and w directions are F1 and F2, respectively. h F w All are 1×1 convolutions. and These are the features of the c-th channel feature map after convolution transformation along the X and Y directions, respectively.

[0075] A7. Obtain information and confidence of vehicle rear detection boxes: Use the trained MCA-YOLOv5s network to detect traffic scenes from the vehicle's perspective and obtain information and confidence of vehicle rear detection boxes.

[0076] B. Obtain the tracking sequence of vehicle rear images using the Bytetrack module.

[0077] The vehicle rear detection bounding box information and confidence score obtained in step A are input into the Bytetrack module to obtain a tracking sequence for the vehicle rear image. Combined with... Figure 5 The Bytetrack module takes as input the predicted bounding box obtained by Kalman filtering the location of the vehicle's rear image in the previous frame and the detection bounding box in the current frame. The detection bounding boxes are then categorized into high-scoring and low-scoring boxes based on a threshold. The calculation method of the Bytetrack module includes the following steps:

[0078] B1. The high-resolution detection box and the predicted box obtained after Kalman filtering of the vehicle rear image in the previous frame are matched by appearance feature matching or IoU matching. Finally, the successfully matched trajectory, the unmatched trajectory and the unmatched detection box are obtained. The unmatched detection box is used to generate a new tracking box. The IoU matching is the overlap matching.

[0079] B2. Perform IoU matching on unmatched trajectories and low-scoring detection boxes. Trajectories that still do not match after two matching attempts are retained for 30 frames and then deleted.

[0080] C. Design a 3D spatiotemporal convolutional neural network, TSA-X3d network, based on block attention and shuffling attention, to recognize taillight signals.

[0081] C1. Using publicly available datasets or dashcam image sequences of vehicle rear-end sequences, crop the images of vehicle rear-ends in traffic scenes at a certain frame rate and save them separately according to different vehicles and taillight signals. Specifically, based on the brake light and turn signal states, vehicle taillight signals are categorized into the following 8 types: OOO, BOO, OOR, OLO, BOR, BLO, OLR, and BLR. Here, OOO indicates the vehicle is stationary or moving straight; BOO indicates the vehicle is braking; OOR indicates the vehicle is preparing to turn right or is currently turning right; OLO indicates the vehicle is preparing to turn left or is currently turning left; BOR indicates the vehicle is preparing to turn right or is currently turning right and braking; BLO indicates the vehicle is preparing to turn left or is currently turning left and braking; OLR indicates the vehicle is in an emergency; and BLR indicates the vehicle is in an emergency and braking.

[0082] The first part of the C2, TSA-X3d network is the ResNetBasicStem, the residual network backbone, which consists of temporal and spatial convolutional layers. This is followed by four ResStages, each containing 3, 5, 11, and 7 ResBlocks, respectively. Each ResBlock contains 3 convolutional layers, 1 ReLU activation function, and 1 residual connection. The second convolutional layer in each ResBlock uses both spatial and temporal convolutions, both of which are depthwise separable. Between the two convolutions, a batch normalization layer, a Swish activation function, and an SA attention module are used. After the temporal convolution, a TSE attention module is used. Finally, the output is passed through a ConvPool convolutional pooling layer, an FCLayer fully connected layer, and an AdaptiveAvgPool3d adaptive average pooling layer.

[0083] C3. The vehicle rear sequence dataset is divided into training and validation sets according to a c:d ratio and input into the TSA-X3d network for training. The weight file with the highest accuracy on the validation set is saved for light signal recognition. For each image input to the model, eight image processing methods are applied: adaptive histogram equalization, rotation and translation, saturation adjustment, brightness adjustment, random shifting of each channel, blurring, image quality reduction, and random grayscale coefficients. A sliding window with a stride of 1 is used, inputting every 10 frames into the network for training. To accelerate network convergence, the mean and variance of the training set are calculated, and the images are standardized to a resolution of 224×224. The optimizer is Adam (Adaptive Moment Estimation), with an initial learning rate of 0.0001, weight decay of 0.0001, and cosine annealing to adjust the learning rate. The loss function is cross-entropy loss, and the number of iterations is 50.

[0084] C4. The TSE attention module from step C2 first compresses the input feature map using an average pooling layer with a kernel size of 7. Then, it uses a 1×1 convolution to reduce the dimensionality of the compressed feature map before inputting it into the ReLU activation function (linear rectified function). Next, it uses a 1×1 convolution to increase the dimensionality of the ReLU activation function's output before inputting it into the Sigmoid activation function (S-shaped growth curve activation function). Finally, it uses nearest neighbor interpolation to restore the output of the Sigmoid activation function to the dimension of the input tensor and performs the following dot product operation with the feature map input to the TSE attention module:

[0085] S = F near (σ(W2·ReLU(W1·F ap (X))))

[0086]

[0087] In the formula, S is the output of the TSE attention module, S is the output after the convolution operation, X is the input feature map, σ is the sigmoid activation function, W1 and W2 are 1×1 convolutions, and F is the output of the TSE attention module. near It is the nearest neighbor interpolation method, ReLU is the activation function, and F... ap It is average pooling. It is a dot product operation.

[0088] C5. Divide the channel features of the SA attention module in step C2 into g groups. Each group's channel features are divided into two parts. The first part is compressed to 1×1×1 using an average pooling layer. The second part is processed by a group normalization layer without changing the input dimension. Then, both parts are processed by linear functions for feature extraction. Finally, after passing through a sigmoid activation function, the features are multiplied by the original feature values ​​to obtain the importance of different channels and spaces. After obtaining the weighted grouped features, the SA attention module merges the two parts by concatenation. Finally, the features from different groups are merged to restore the input dimension and then a "shuffling" operation is performed to enhance the information interaction between different groups and improve the network's ability to extract temporal and spatial information.

[0089] X′ k1 =σ(F c1 (F A (X k1 )))·X k1 =σ(W1S1+b1)·X k1

[0090] X′ k2 =σ(F c2 (F GN (X k2 )))·X k2 =σ(W2S2+b2)·X k2

[0091] In the formula, X' k1 X' is the output feature of the first part. k2 The output features of the second part are defined by σ, the activation function, S1 (features after average pooling), and S2 (features after normalization). W1 and b1 represent the weights and biases of the linear function in the first part, respectively, while W2 and b2 represent the weights and biases of the linear function in the second part. X k1 X is the input feature map for the first part. k2 For the input feature map of the second part, F A For global average pooling, F GN For group normalization, F c1 F is a linear function of the first part. c2It is a linear function of the second part.

[0092] C6. Use TensorRT to accelerate the MCA-YOLOv5s network and perform quantization processing on the TSA-X3d network, converting the saved weight file from FP32 to FP16. TensorRT is a C++ inference framework that can run on various NVIDIA graphics processors.

[0093] C7. This invention selects certain scenarios to test taillight signaling, and the test results are as follows: Figures 9-12 As shown, the effectiveness and practicality of the present invention are demonstrated.

[0094] Furthermore, in step A3, the value of a:b is 7:3, 4:1, or 9:1; and in step C3, the value of c:d is 7:3, 4:1, or 9:1.

[0095] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A lightweight method for real-time recognition of automotive taillight signals, characterized in that: Includes the following steps: A. Design an MCA-YOLOv5s network to acquire information and confidence levels of the vehicle's rear detection bounding box. A1. Labeling the rear of vehicles in traffic scenes: Obtain vehicle rear detection datasets through public datasets or dashcams, and label the rear of vehicles in traffic scenes in YOLO format. In YOLO format, each image corresponds to a text file, and each line of the text file corresponds to information about the rear of a vehicle. Each line includes 5 numbers: the first number represents the target category, the second number represents the ratio of the horizontal coordinate of the center of the vehicle rear image to the image width, the third number represents the ratio of the vertical coordinate of the center of the vehicle rear image to the image height, the fourth number represents the ratio of the width of the vehicle rear rectangle to the image width, and the fifth number represents the ratio of the height of the vehicle rear rectangle to the image height. Adjacent numbers are separated by spaces. A2. Establishing the MCA-YOLOv5s network: The YOLOv5s network is improved. The YOLOv5s network is a single-stage object detection network, including a backbone network, a bottleneck network, and a prediction network. The backbone network consists of a feature extraction network and a spatial pyramid pooling structure; the bottleneck network consists of a feature pyramid network and a C3 structure; and the prediction network consists of ordinary convolutional layers. The C3 structure consists of three convolutional layers. The backbone network in the YOLOv5s network is replaced with 11 cneck modules. Finally, all C3 structures are replaced with depthwise separable convolutions to obtain the MCA-YOLOv5s network. The MCA-YOLOv5s network is a YOLOv5s network based on coordinate attention and inverse residual structures; the cneck module is an inverse residual structure module. A3. Training the vehicle rear detection dataset: Divide the vehicle rear detection dataset into a training set and a validation set according to the ratio a:b and input them into the MCA-YOLOv5s network for training; the training set is used to train the MCA-YOLOv5s network, and the validation set is used to provide feedback on the training results. The weight file that performs best on the validation set is saved to obtain the vehicle rear detection box information and confidence score. A4. Reduce network parameter count by using depthwise separable convolutional structures: Divide the depthwise separable convolution in step A2 into channel-wise convolution and pointwise convolution; In channel-wise convolution, one channel of the concatenated feature map in the MCA-YOLOv5s network is convolved with only one convolutional kernel to obtain a new feature map; Pointwise convolution weights and combines the new feature maps obtained by channel-wise convolution in the depth direction to generate the final feature map. A5. Use the cneck module structure to reduce the number of network parameters: The cneck module in step A2 first increases the dimension of the input feature map through a 1×1 convolution, then performs a channel-wise depthwise convolution operation on the increased dimension feature map through a 3×3 convolution, uses the coordinate attention module to obtain the channel information and position information of the input feature map, then reduces the dimension of the feature map through a 1×1 convolution, and finally performs residual skip connections to alleviate the gradient vanishing phenomenon that occurs as the network deepens. A6. Enhance network feature extraction capabilities using a coordinate attention module structure: Perform global average pooling on the coordinate attention module in step A5 in both the X and Y directions to preserve the long-range dependencies of channel attention in both directions. Then, concatenate the information from the two directions and perform a convolution operation to allow the information from the two directions to interact. Based on this, decompose the feature map and perform convolution operations on each separately, simultaneously focusing on the X and Y direction information of the feature map to obtain the positional information of the feature map. The relevant formulas are as follows: f=δ(F1([z h ,z w ])) g h =σ(F h (f h )) g w =σ(F w (f w )) In the formula, zc represents global average pooling. and The outputs of the c-th channel with height h and width w are respectively, f is the feature after convolution transformation, and g is the output of the c-th channel with height h and width w respectively. h g represents the features of the feature map after convolution along the X direction. w The feature map is the result of a convolution transformation along the Y direction, y c (i, j) represents the output of the coordinate attention module after passing through the value of the c-th channel of the input feature map in the i-th row and j-th column. H and W are the width and height of the feature map, respectively, and σ and δ are the activation functions. c (i, j) represents the value of the c-th channel of the input feature map in the i-th row and j-th column, x c (h, i) represents the value of the c-th channel of the input feature map in the h-th row and i-th column, x c (j, w) represents the value of the c-th channel of the input feature map in the j-th row and w-th column, z h and z w The outputs of average pooling of the feature maps in the h and W directions are F1 and F2, respectively. h F w All are 1×1 convolutions. and These are the features of the c-th channel feature map after convolution transformation along the X and Y directions, respectively; A7. Obtain information and confidence of vehicle rear detection boxes: Use the trained MCA-YOLOv5s network to detect the traffic scene in the vehicle's view and obtain information and confidence of vehicle rear detection boxes. B. Obtain the tracking sequence of vehicle rear images using the Bytetrack module. The vehicle rear detection bounding box information and confidence score obtained in step A are input into the Bytetrack module to obtain the tracking sequence of the vehicle rear image; the Bytetrack module is input with the predicted bounding box obtained after Kalman filtering of the vehicle rear image position in the previous frame and the detection bounding box in the current frame, and the detection bounding boxes are divided into high-scoring detection bounding boxes and low-scoring detection bounding boxes according to a threshold; the calculation method of the Bytetrack module includes the following steps: B1. The high-resolution detection box and the predicted box obtained after Kalman filtering of the vehicle rear image in the previous frame are matched by appearance feature matching or IoU matching. Finally, the successfully matched trajectory, the unmatched trajectory and the unmatched detection box are obtained. The unmatched detection box is used to generate a new tracking box. The IoU matching is the overlap matching. B2. Perform IoU matching on unmatched trajectories and low-scoring detection boxes. Trajectories that still do not match after two matching attempts are retained for 30 frames and then deleted. C. Design a 3D spatiotemporal convolutional neural network, TSA-X3d network, based on block attention and shuffling attention, to recognize taillight signals. C1. Using publicly available datasets or dashcam sequences of vehicle rear-end images, crop the images of vehicle rear ends in traffic scenes at a certain frame rate and save them separately according to different vehicles and different taillight signals. Among them, the taillight signals are divided into the following 8 categories according to the brake light and turn signal status: OOO, BOO, OOR, OLO, BOR, BLO, OLR, BLR. Wherein, OOO indicates that the vehicle is stationary or moving straight, BOO indicates that the vehicle is braking, OOR indicates that the vehicle is preparing to turn right or is turning right, OLO indicates that the vehicle is preparing to turn left or is turning left, BOR indicates that the vehicle is preparing to turn right or is turning right and braking, BLO indicates that the vehicle is preparing to turn left or is turning left and braking, OLR indicates that the vehicle is in an emergency, and BLR indicates that the vehicle is in an emergency and braking. The first part of the C2, TSA-X3d network is the ResNetBasicStem, which is the residual network backbone, consisting of temporal and spatial convolutional layers. This is followed by four ResStages, each containing 3, 5, 11, and 7 ResBlocks, respectively. Each ResBlock contains 3 convolutional layers, 1 ReLU activation function, and 1 residual connection. The second convolutional layer of each ResBlock uses both spatial and temporal convolutions, both of which are depthwise separable. Between the two convolutions, a batch normalization layer, a Swish activation function, and an SA attention module are used. After the temporal convolution, a TSE attention module is used. Finally, the network outputs the final result through a ConvPool convolutional pooling layer, an FCLayer fully connected layer, and an AdaptiveAvgPool3d adaptive average pooling layer. C3. Divide the vehicle rear sequence dataset into training and validation sets according to the c:d ratio and input them into the TSA-X3d network for training. Save the weight file with the highest accuracy on the validation set for light signal recognition. C4. The TSE attention module from step C2 first compresses the input feature map using an average pooling layer with a kernel of 7. Then, it uses a 1×1 convolution to reduce the dimensionality of the compressed feature map and inputs it into the ReLU activation function (linear rectified function). Next, it uses a 1×1 convolution to increase the dimensionality of the output of the ReLU activation function and inputs it into the Sigmoid activation function (S-shaped growth curve activation function). Finally, it uses nearest neighbor interpolation to restore the output of the Sigmoid activation function to the dimension of the input tensor and performs the following dot product operation with the feature map input to the TSE attention module: S=F near (σ(W2·ReLU(W1·F ap (X)))) In the formula, S is the output of the TSE attention module, S is the output after the convolution operation, X is the input feature map, σ is the sigmoid activation function, W1 and W2 are 1×1 convolutions, and F is the output of the TSE attention module. near It is the nearest neighbor interpolation method, ReLU is the activation function, and F... ap It is average pooling. It is a dot product operation; C5. Divide the channel features of the SA attention module in step C2 into g groups. The channel features of each group are divided into two parts on average. The first part is compressed into a 1×1×1 feature map by an average pooling layer. The second part is passed through a group normalization layer without changing the dimension of the input. Then, the two parts are processed by linear functions for feature extraction. Finally, after passing through the Sigmoid activation function, the features are multiplied by the original feature values ​​to obtain the importance of different channels and spaces. After obtaining the weighted grouped features, the SA attention module merges the two parts by concatenation. Finally, the features of different groups are merged to restore the dimension of the input and then "shuffled" to enhance the information interaction between different groups and enhance the network's ability to extract temporal and spatial information. X′ k1 =σ(F c1 (F A (X k1 )))·X k1 =σ(W1S1+b1)·X k1 X′ k2 =σ(F c2 (F GN (X k2 )))·X k2 =σ(W2S2+b2)·X k2 In the formula, X′ k1 X′ is the output feature of the first part. k2 The output features of the second part are defined by σ, the activation function, S1 (features after average pooling), and S2 (features after normalization). W1 and b1 represent the weights and biases of the linear function in the first part, respectively, while W2 and b2 represent the weights and biases of the linear function in the second part. X k1 X is the input feature map for the first part. k2 For the input feature map of the second part, F A For global average pooling, F GN For group normalization, F c1 F is a linear function of the first part. c2 It is a linear function of the second part; C6. Use TensorRT to accelerate the MCA-YOLOv5s network and perform quantization processing on the TSA-X3d network, converting the saved weight file from FP32 to FP16; TensorRT is a C++ inference framework running on NVIDIA graphics processors.

2. The lightweight real-time recognition method for automotive taillight signals according to claim 1, characterized in that: In step A3, the value of a:b is 7:3, 4:1, or 9:1; in step C3, the value of c:d is 7:3, 4:1, or 9:1.

Images