Deep learning preceding vehicle intention recognition method based on taillight prior visual knowledge optimization

Abstract

The invention discloses a deep learning preceding vehicle intention recognition method for taillight prior visual knowledge optimization. The method comprises the following steps: (1) designing a structure of SP-Conv and point-by-point convolution PwConv in an SAIR module; (2) in the SAIR-D module, the step length of SP-Conv is set to be a numerical value larger than 1 to achieve down-sampling, and meanwhile, space pooling CASP residual error connection of column attention is added to the SAIR-D on the basis of the SAIR module; and (3) in the DFT-MVT module, a deformable linear self-attention unit DFAT is put forward to replace original multi-head self-attention, the DFAT unit improves the attention ability to tail lamps of different shapes through the deformable self-attention, and the speed is increased by fusing a focusing linear self-attention calculation method. According to the method, the taillight state recognition precision is improved on the premise of keeping low delay of the model, and the driving intention of the front vehicle can be better assisted to be judged.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile driving assistance, and in particular relates to a deep learning preceding vehicle intention recognition method optimized by priori visual knowledge of taillights. Background Art

[0002] In intelligent driving, accurate recognition of the intention of the vehicle ahead is of great reference significance for autonomous driving route planning. Among them, the status of the vehicle's taillights, such as brake lights, high-mounted brake lights, and turn signals, is a key signal for judging the driving intention of the vehicle ahead. Real-time and accurate recognition of these signals can assist in determining the driving intention of the vehicle ahead and formulating a refined autonomous driving strategy. The current mainstream vehicle taillight status recognition method uses traditional image processing or deep learning technology.

[0003] The vehicle taillight state recognition method based on traditional image processing technology mainly relies on hard-coded manual feature extraction, such as D. Nava et al. in the literature 1: D. Nava, G. Panzani, and SMSavaresi. A collision warning oriented brake lights detection and classification algorithm based on a monocamera sensor [C] / / IEEE Intelligent Transportation Systems Conference. 2019: pp. 319–324, that is, "Collision warning oriented brake lights detection and classification algorithm based on monocular camera sensor" [C] / / IEEE Intelligent Transportation Systems Conference. 2019: pp. 319–324, which locates the position of the lights on the front vehicle target by coordinates and classifies them according to brightness to achieve front vehicle brake detection. However, this type of method is easily disturbed by complex detection environments and has poor robustness.

[0004] In recent years, as deep learning technology continues to mature, it is gradually becoming mainstream in autonomous driving visual perception applications. Qiaohong Li et al. used YOLOv3 to directly detect vehicle taillights and classify their states in the literature 2: Li Q, Garg S, Nie J, et al. A highly efficient vehicle taillight detection approach based on deep learning [J]. IEEE transactions on intelligent transportation systems, 2020, 22 (7): 4716-4726. However, due to the small area and diverse forms of vehicle taillights, the detection performance of this method will drop significantly in urban roads with a large number of vehicles and complex scenes. Therefore, most of the current deep learning taillight state detection methods mostly perform secondary classification on vehicle targets to improve performance. These methods may combine the results of deep network calculation with traditional image processing results, such as reference 3: Song W, Liu S, Zhang T, et al. Action-state joint learning-based vehicle taillight recognition indiverse actual traffic scenes [J]. IEEE transactions on intelligent transportation systems, 2022, 23(10): 18088-18099, i.e., "Vehicle taillight recognition in diverse actual traffic scenes based on action-state joint learning" [J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10): 18088-18099 and reference 4: Jeon HJ, Nguyen VD, Duong TT, et al. A deep learning framework for robust and real-time taillight detection under various road conditions [J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11): 20061-20072.IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11): 20061-20072, or based on pure deep learning technology, such as Reference 5: Frossard D, Kee E, Urtasun R. Deepsignals: Predicting intent of drivers through visual signals [C] / / International Conference on Robotics and Automation. 2019: 9697-9703, i.e., Deep signals: Predicting intent of drivers through visual signals [C] / / International Conference on Robotics and Automation. 2019: 9697-9703 and Reference 6: Klemp M, Wagner R, Rosch K, et al. Vehicle Intention Classification Using Visual Clues [C] / / IEEE International Conference on Robotics and Automation.2024:16395-16401, namely "Vehicle Intent Classification Based on Visual Cues" [C] / / IEEE International Conference on Robotics and Automation.2024:16395-16401, shows good development potential. .

[0005] In summary, most of the existing deep learning-based taillight status recognition methods adopt general machine learning frameworks and modules, and lack targeted optimization for taillight recognition applications. Summary of the invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention provides a deep learning preceding vehicle intention recognition method optimized by prior visual knowledge of taillights. Based on the MobileVit architecture, the present invention introduces prior visual knowledge such as the position and shape of the preceding vehicle's taillights, designs a spatial attention optimized convolution module SAIR and a downsampling convolution module SAIR-D, and a deformable linear Transformer module DFT-MVT, and through the serial stacking of the above modules, enables the network to more effectively and autonomously focus on taillights of different shapes in the vehicle target image, thereby improving the accuracy of taillight status recognition; while maintaining low latency of the model, the recognition accuracy is improved, which can better assist in judging the driving intention of the preceding vehicle.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for front vehicle intention recognition based on deep learning optimized by prior visual knowledge of taillights comprises the following steps:

[0009] (1) In the SAIR module, a structure of SP-Conv plus pointwise convolution PwConv is designed. In each SP-Conv unit, first, through channel dimension splitting, the input features are divided into two parts; then, one part undergoes two operations. One is to generate spatial attention weights through average pooling AvgPool, depthwise convolution DwConv, and softmax function, and the other is to extract features through DwConv. Then, the two are multiplied element-wise, and after passing through DwConv again, updated features are obtained; finally, the updated features are concatenated with the original other part of the features along the channel dimension to generate output features; the SP-Conv unit performs convolution operations only on a part of the features; at the same time, spatial attention is added to the convolution operation part; the calculation formula is:

[0010] SP-Conv = Concat((1 - r)X, DwConv(X′))

[0011] X′ = Softmax(DwConv(AvgPool(rX))) ⊙ DwConv(rX)

[0012] Among them, r is the channel splitting ratio of the input features, Concat represents the concatenation operation, X is the input features, and ⊙ represents element-wise multiplication;

[0013] (2) In the SAIR-D module, the stride of SP-Conv is set to a value greater than 1 to achieve downsampling. At the same time, SAIR-D adds a column attention spatial pooling CASP residual connection on the basis of the SAIR module to reduce information loss during the downsampling process;

[0014] (3) In the DFT-MVT module, a deformable linear self-attention unit DFAT is proposed to replace the original multi-head self-attention. The DFAT unit improves the attention ability to taillights of different shapes through deformable self-attention and integrates the focused linear self-attention calculation method to reduce the attention calculation overhead.

[0015] Furthermore, in the above (2), the working process of CASP is as follows:

[0016] First, for the input feature map X with size H×W×C, a linear column window of H×1×C is used to slide on the feature map, and the original features are divided into W window feature maps;

[0017] Secondly, perform a fast Fourier transform FFT on each window feature map. For the w-th window feature map X w , 0 ≤ w < W, the transformation result at the k-th frequency point is denoted as where K represents the number of FFT transformation frequency points;

[0018] Again, combine all After concatenation, it passes through a PwConv, a batch normalization layer, a SiLU nonlinear layer, and then a PwConv and a Sigmoid nonlinear layer network module to calculate X w The attention weights are:

[0019]

[0020] Finally, using ω(X w ) as the weight of each eigenvalue in the corresponding column, guiding the average pooling process of X:

[0021] CASP(X)=AvgPool(ω(X)⊙PwConv(X)).

[0022] Furthermore, the working process of (3) is as follows:

[0023] First, the query Q is obtained by linearly mapping the input feature map X, that is, For the linear mapping weight, and through an Offset network and a Weight network, calculate the weighted offset of Q

[0024]

[0025] Among them, grid(X) is the predefined reference grid point on X. In the Offset network, Q passes through a DwConv layer, a GeLU nonlinear layer, a PwConv layer, and a Tanh nonlinear layer in sequence to obtain a learnable offset:

[0026] Offset(Q)=Tanh(PwConv(GeLU(DwConv(Q))))

[0027] In the Weight network, Q passes through a DwConv, a GeLU nonlinear layer, a PwConv layer, and a Sigmoid nonlinear layer to obtain a learnable weight:

[0028] Weight(Q)=Sigmoid(PwConv(GeLU(DwConv(Q))))

[0029] Secondly, for Q and Perform bilinear interpolation and use linear mapping weights and Generate K and V, that is Where h(·; ·) represents bilinear interpolation;

[0030] Finally, perform focused linear self-attention calculation on Q, K, and V to obtain the updated features:

[0031]

[0032] in is the focusing function, p is the focusing ratio, and d represents the dimension of K.

[0033] The beneficial effects of the present invention are mainly manifested in: by introducing prior visual knowledge such as the position and shape of the taillights of the preceding vehicle, designing the spatial attention optimized convolution module SAIR and the downsampling convolution module SAIR-D, and the deformable linear Transformer module DFT-MVT, and through the serial stacking of the above modules, the network can more effectively and autonomously focus on taillights of different shapes in the vehicle target image, thereby improving the accuracy of taillight status recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method for recognizing the intention of a preceding vehicle through deep learning optimized by prior visual knowledge of taillights described in the present invention.

[0035] Figure 2 This is the SAIR structure diagram of the convolution module.

[0036] Figure 3 This is the structure diagram of the SP-Conv module.

[0037] Figure 4 This is the structure diagram of the downsampling convolution module SAIR-D.

[0038] Figure 5 This is the self-attention process diagram of DFAT. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings.

[0040] Reference Figure 1 to Figure 5 A deep learning method for identifying the intention of a preceding vehicle based on taillight prior visual knowledge optimization is proposed. The invention is based on the MobileVit architecture, introduces prior visual knowledge such as the position and shape of the preceding vehicle taillight, designs a spatial attention optimized convolution module SAIR and a downsampling convolution module SAIR-D, and a deformable linear Transformer module DFT-MVT, and through the serial stacking of the above modules, enables the network to more effectively and autonomously focus on taillights of different shapes in the vehicle target image. The method comprises the following steps:

[0041] (1) In the SAIR module, a structure of SP-Conv plus pointwise convolution PwConv is designed. In each SP-Conv unit, first, through channel dimension splitting, the input features are divided into two parts. Then, one part undergoes two operations. One is to generate spatial attention weights through average pooling AvgPool, depthwise convolution DwConv, and softmax function, and the other is to extract features through DwConv, and the two are multiplied element-wise, and then updated features are obtained through DwConv. Finally, the updated features are concatenated with the original other part of the features along the channel dimension to generate output features. The SP-Conv unit improves the computational efficiency by performing convolution operations only on a part of the features. At the same time, spatial attention is added to the convolution operation part to improve the feature extraction effect. Therefore, the SP-Conv unit can better explore the spatial correlation between taillights while reducing complexity, and its calculation formula is:

[0042] SP-Conv = Concat((1 - r)X, DwConv(X′))

[0043] X′ = Softmax(DwConv(AvgPool(rX))) ⊙ DwConv(rX)

[0044] Where r is the channel splitting ratio of the input features, Concat represents the concatenation operation, X is the input features, and ⊙ represents element-wise multiplication.

[0045] (2) In the SAIR-D module, the stride of SP-Conv is set to a value greater than 1 to achieve downsampling. At the same time, SAIR-D adds a column attention spatial pooling (ASP) residual connection on the basis of the SAIR module to reduce information loss during the downsampling process. The working process of CASP is as follows:

[0046] First, for the input feature map X of size H×W×C, a linear column window of size H×1×C is used to slide on the feature map, and the original features are divided into W window feature maps.

[0047] Second, perform a fast Fourier transform (FFT) on each window feature map. For the w-th (0 ≤ w < W) window feature map X w , the transformation result at the k-th (0 ≤ k < K) frequency point is denoted as Where K represents the number of FFT transformation frequency points;

[0048] Third, jointly use all After concatenation, through a network module of a PwConv, a batch normalization layer, a SiLU non-linear layer, and then through a PwConv and a Sigmoid non-linear layer, calculate X wThe attention weights are:

[0049]

[0050] Finally, using ω(X w ) as the weight of each eigenvalue in the corresponding column, guiding the average pooling process of X:

[0051] CASP(X)=AvgPool(ω(X)⊙PwConv(X))

[0052] (3) In the DFT-MVT module, a deformable linear self-attention unit DFAT is proposed to replace the original multi-head self-attention. The DFAT unit improves the ability to pay attention to taillights of different shapes through deformable self-attention, and integrates the focused linear self-attention calculation method to reduce the attention calculation overhead. The working process is as follows:

[0053] First, the query Q is obtained by linearly mapping the input feature map X, that is, It is a linear mapping weight, and passes through an Offset network and a Weight network to calculate the weighted offset of Q

[0054]

[0055] Where grid(X) is the predefined reference grid point on X. In the Offset network, Q passes through a DwConv layer, a GeLU nonlinear layer, a PwConv layer, and a Tanh nonlinear layer to obtain a learnable offset:

[0056] Offset(Q)=Tanh(PwConv(GeLU(DwConv(Q))))

[0057] In the Weight network, Q passes through a DwConv, a GeLU nonlinear layer, a PwConv layer, and a Sigmoid nonlinear layer to obtain a learnable weight:

[0058] Weight(Q)=Sigmoid(PwConv(GeLU(DwConv(Q))))

[0059] Secondly, for Q and Perform bilinear interpolation and use linear mapping weights and Generate K and V, that is Where h(·; ·) represents bilinear interpolation;

[0060] Finally, perform focused linear self-attention calculation on Q, K, and V to obtain the updated features:

[0061]

[0062] in is the focusing function, p is the focusing ratio, and d represents the dimension of K.

[0063] In this embodiment, by designing the spatial attention optimized convolution module SAIR and the downsampling convolution module SAIR-D, and the deformable linear Transformer module DFT-MVT, the network can more effectively and autonomously focus on the taillight targets in the vehicle image. Among them, the SP-Conv unit in the SAIR module not only has a stronger spatial receptive field, but also has the advantages of light weight and low computing resources. The CASP unit in the SAIR-D module uses the prior knowledge of the taillight position to generate a column attention enhanced pooling downsampling process, which improves the perception of different taillight states. DFT-MVT generates a network through a position offset grid and weights, so that the attention network can more effectively focus on taillight targets of different shapes. Through the serial stacking of the above modules, the accuracy of taillight status recognition is improved while maintaining low model latency, which can better assist in judging the driving intention of the preceding vehicle.

[0064] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A deep learning method for identifying the intention of a preceding vehicle based on taillight prior visual knowledge optimization, characterized in that: The method comprises the following steps: (1) The SAIR module designs a structure of SP-Conv plus point-wise convolution PwConv. In each SP-Conv unit, the input features are first divided into two parts through channel dimension segmentation. Then, one part undergoes two operations. One is to generate spatial attention weights through average pooling AvgPool, depth-wise convolution DwConv and normalized exponential function Softmax. The other is to extract features through DwConv and multiply the two element-wise, and then obtain updated features through DwConv. Finally, the updated features are concatenated with the original features along the channel dimension to generate output features. The SP-Conv unit performs convolution operations on only part of the features. At the same time, spatial attention is added to the convolution operation part. The calculation formula is: SP-Conv=Concat((1-r)X,DwConv(X′)) X′=Softmax(DwConv(AvgPool(rX)))⊙DwConv(rX) Among them, r is the ratio of input feature segmentation by channel, Concat represents the connection operation, X is the input feature, and ⊙ represents element-by-element multiplication; (2) In the SAIR-D module, the step size of SP-Conv is set to a value greater than 1 to achieve downsampling. At the same time, SAIR-D adds a column-attention spatial pooling CASP residual connection based on the SAIR module to reduce information loss during the downsampling process; (3) In the DFT-MVT module, a deformable linear self-attention unit DFAT is proposed to replace the original multi-head self-attention. The DFAT unit improves the ability to pay attention to taillights of different shapes through deformable self-attention, and integrates the focused linear self-attention calculation method to reduce the attention calculation overhead.

2. The deep learning method for identifying the intention of the preceding vehicle based on taillight prior visual knowledge optimization as claimed in claim 1, characterized in that: In (2), the working process of CASP is as follows: First, for an input feature map X of size H×W×C, a linear column window of size H×1×C is used to slide on the feature map to split the original feature into W window feature maps; Secondly, perform a fast Fourier transform (FFT) on each window feature map. For the w-th window feature map X w , where 0 ≤ w < W, the transformation result at the k-th frequency point is denoted as where K represents the number of FFT transform frequency points; Again, combine all After concatenation, it passes through a PwConv, a batch normalization layer, a SiLU nonlinear layer, and then a PwConv and a Sigmoid nonlinear layer network module to calculate X w The attention weights are: Finally, using ω(X w ) as the weight of each eigenvalue in the corresponding column, guiding the average pooling process of X: CASP(X)=AvgPool(ω(X)⊙PwConv(X)).

3. The deep learning preceding vehicle intention recognition method optimized by taillight prior visual knowledge as claimed in claim 1 or 2, characterized in that: The working process of (3) is as follows: First, the query Q is obtained by linearly mapping the input feature map X, that is, It is a linear mapping weight, and passes through an Offset network and a Weight network to calculate the weighted offset of Q Among them, grid(X) is the predefined reference grid point on X. In the Offset network, Q passes through a DwConv layer, a GeLU nonlinear layer, a PwConv layer, and a Tanh nonlinear layer in sequence to obtain a learnable offset: Offset(Q)=Tanh(PwConv(GeLU(DwConv(Q))))In the Weight network, Q passes through a DwConv, a GeLU nonlinear layer, a PwConv layer, and a Sigmoid nonlinear layer to obtain a learnable weight: Weight(Q)=Sigmoid(PwConv(GeLU(DwConv(Q)))) Secondly, for Q and Perform bilinear interpolation and use linear mapping weights and Generate K and V, that is Where h(·; ·) represents bilinear interpolation; Finally, focus linear self-attention calculation is performed on Q, K, and V to obtain the updated features: in is the focusing function, p is the focusing ratio, and d represents the dimension of K.

Images