A lane detection method using self-attention mechanism and variable convolution

Abstract

This invention discloses a lane detection method that uses a self-attention mechanism and variable convolution for feature extraction optimization. The method comprises 1. a main process for initial image feature extraction; 2. feature fusion calculation; 3. a variable convolution calculation stage; 4. a lane position prediction stage; and 5. an image segmentation feature extraction stage. This method uses a ResNet convolutional network, fully connected layer calculations, and image gridding modeling to detect and identify lanes in raw images. By combining the self-attention mechanism, variable convolution methods, and multi-scale feature fusion, the method enhances the network's ability to extract global information. It also eliminates prior information during loss calculation, thereby reducing false detections in complex scenarios caused by model overfitting.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and relates to an image lane line detection method, specifically a lane detection method using a self-attention mechanism and variable convolution. Background Art

[0002] Lane detection is widely used in today's autonomous driving technology. Within the deep learning landscape, modern lane detection methods primarily treat lane detection as a pixel segmentation problem. Traditional CNN models' disregard for context and global information leads to reduced detection efficiency under occlusion and strong light conditions. While the SCNN method emphasizes global information extraction, its cyclical computational process increases computational complexity. The UFSA method treats lane detection as a row-based selection problem using global features. It divides the image into a grid, where an image of (H, W) size is divided into grids of (h, w) size, where h and w are much smaller than H and W. UFSA transforms the pixel segmentation process into a grid classification process, focusing solely on the classification of each grid. This significantly reduces the number of parameters, improves computational speed, and significantly reduces computational cost. However, its F1-score performance on the complex culane dataset still leaves room for improvement. First, the image gridding method used by UFSA to improve computing speed cannot achieve detailed classification of distant images with fewer pixels. Second, its use of the ResNet convolutional network as the backbone for feature extraction cannot extract more global features because the receptive field of the convolution kernel is small and confined to a local rectangular range. Finally, the polynomial loss function used by the UFSA authors when training the network includes a method to limit the curvature of lane lines. As a result, the lane lines generated by the detection are relatively straight and do not perform well in curved conditions. Summary of the Invention

[0003] The purpose of this invention is to propose a lane detection method using a self-attention mechanism and variable convolution to solve the problem that current lane line detection technology is difficult to make accurate detection in scenarios that require more global information, such as road curves, strong light, and occlusion.

[0004] The present invention provides a lane detection method using a self-attention mechanism and variable convolution, comprising the following steps:

[0005] S1, obtain original image data and construct a dataset;

[0006] S2, data set preprocessing, the original image data is subjected to occlusion, brightness adjustment, contrast adjustment, and random noise generation to obtain preprocessed image data;

[0007] S3. Use the preprocessed image data as input for feature extraction

[0008] S3-1, extract the three feature maps T1~T3 generated by the last three convolution modules through ResNet18 composed of 4 convolutional stack modules;

[0009] S3-2, perform self-attention extraction on the output feature map T3 to obtain feature map T4;

[0010] S4, feature fusion, due to the different number of channels and sizes of T1~T4, the size of the feature maps T1~T4 needs to be transformed before fusion. First, the convolution operation of T1~T4 is performed to unify them to a specific number of channels C. f =64, and then from T4 to T1, a feature map T that combines different convolutional layers and self-attention features is obtained by upsampling and adding one by one. c ;

[0011] S5, the feature map T that combines different convolutional layers and self-attention features c Perform variable convolution to obtain a new feature map T o

[0012] S6, Lane Line Prediction

[0013] T o As input, the calculation layer of two layers of full connection and ReLU combination outputs the probability of each grid belonging to each lane line. The probability matrix M composed of them has a size of (lanes, H, W), where lanes is the number of channels of the number of lane lines, H and W are the grid sizes, and each channel represents the classification information of a lane line in each grid. The probability matrix M is calculated with the label of the dataset for cross entropy loss and back propagated. fc is the fully connected layer

[0014] M=fc(relu(fc(T o ))).

[0015] As an advantage, the three characteristic graphs T produced in step S3-1 are i , size is (C i ×H i ×W i ),i∈[1,3], where C is the number of channels, H and W are the length and width.

[0016] As a preference, in step S3-2, the self-attention extraction method is: self-attention information is extracted only from a feature map T3 outputted by ResNet at the end, and first unified to a specific number of channels C through a pre-processing convolution layer. A , after passing through three different convolutional layers, query_conv, key_conv and val_conv, the feature T is obtained q ,T k and Tv , the dimensions are (C A ×H×W), and both are merged in two or three dimensions, and then T q Swap one and two dimensions to get the sizes (HW×C A ),(C A ×HW),(C A ×HW). Then the attention vector T is obtained through the following formula a

[0017] T a =(softmax(T q ×T k ))×T v

[0018] Output results through T ai With T i Weighted output:

[0019] T ai =final_conv(α*T a +T3)

[0020] Among them, final_conv is the last layer of convolution for feature filtering, and the final feature map output is added as T4 to the subsequent input

[0021] Preferably, in step S4, the feature fusion method is:

[0022] S4-1, upsampling to T by linear interpolation i-1 Same length and width dimensions;

[0023] S4-2, the same size T i and T i-1 Add the corresponding positions to get the new T i-1 ;

[0024] S4-3, i=i-1;

[0025] S4-4, repeat S4-1 to S4-3 to obtain the feature map T c .

[0026] Preferably, step S5 includes the following sub-steps:

[0027] S5-1, first of all, T c Through two computing units consisting of three layers of convolution: three layers of convolution, T is extracted cls and T pts , the former is the feature map to which variable convolution is applied, and the latter is used to generate a vector specifying the variable convolution position;

[0028] S5-2, Tpts Then generate T through two layers of convolution offset , the size is (N×H×W), where N is the number of calculation positions contained in the variable convolution receptive field, T offset The value of is the value of the convolution calculation point deviating from the convolution position. A positive number is a rightward shift, and a negative number is a leftward shift.

[0029] T offset =conv(T pts )

[0030] S5-3, T offset The purpose of calculating the loss with the label of the dataset is to make T offset The value of falls on the position of the lane line in the image, that is, in order to make the position of the variable convolution convoluted be the lane line position, it can be called "strip convolution";

[0031] S5-4, T offset Application in T cls Perform variable convolution operation on T offset The convolution position determined by the value is used for convolution calculation to obtain the new feature map T o ,

[0032] T o =deformConv(T cls ,T offset ).

[0033] The present invention has the following characteristics and beneficial effects:

[0034] By combining self-attention, variable convolution, and multi-scale feature fusion, the network's ability to extract global information is enhanced. Prior information is eliminated during loss calculation, reducing false detections in complex scenarios caused by model overfitting. This method demonstrates improved detection performance in complex scenarios such as curves, strong light, shadows, and occlusions in the lane detection dataset culane. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the overall implementation scheme of the present invention;

[0036] Figure 2 This is a flow chart of variable convolution processing in an embodiment of the present invention.

[0037] Figure 3 、 4 This is an example diagram of the implementation of the present invention DETAILED DESCRIPTION

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the embodiments will be further introduced below in conjunction with the accompanying drawings.

[0039] The main innovations of this implementation focus on modifying the UFSA's feature extraction, training methods, and loss calculation to create a lane detection model with improved detection performance. This model utilizes a self-attention mechanism, feature pyramids (FPN), and variable convolution to further extract features from image data. A new computational process is added to calculate the loss of the variable convolution results, ensuring that the convolution position corresponds to the lane line location.

[0040] First, the UFSA architecture uses only ResNet34 as the backbone network for feature extraction, while the two fully connected layers used after feature extraction are used for classification. Considering the small receptive field of convolutional networks and their inability to fully extract global features, the present invention adds a self-attention mechanism network and a variable convolutional network to improve the receptive field of feature extraction and enhance the proportion of global information in the final feature map. In addition, based on the fact that UFSA performs slightly poorly on far-end curves, the present invention adds an FPN structure to enhance the network's feature extraction of subtle lanes.

[0041] Secondly, the UFSA architecture, driven by faster computation speed, lacks in-depth feature extraction. This approach uses a method that limits lane curvature as prior information to guide loss calculation, negatively impacting scenarios like subtle curves and occlusions. This paper eliminates this prior information-driven loss calculation and instead uses the deeper feature extraction mentioned above to enable the network to independently learn to handle complex scenarios.

[0042] Finally, we add loss calculation for the result points of the variable convolution output to ensure that the convolution position is the lane line position, making the convolution receptive field "strip-shaped" and allowing the convolution process to pay more attention to the lane line position information.

[0043] Combining the above methods, this paper proposes a new network structure. Compared with UFSA, it greatly improves the feature extraction capability and the final lane line detection capability while ensuring the available computing speed. The following is a detailed description of the network structure. Please refer to the attached Figure 1 .

[0044] 1 Backbone feature extraction

[0045] The 1-1 main process accepts the original image data as input, and then inputs the original image data into the ResNet18

[0046] 1-2ResNet18 performs preliminary feature extraction: ResNet consists of 4 similar convolutional stacking modules, and the feature maps generated by the last three modules are extracted for subsequent use. Therefore, this step obtains 3 feature maps T i , size is (Ci ×H i ×W i ),i∈[1,3], where C is the number of channels, H and W are the length and width.

[0047] 1-3 output feature map self-attention extraction: In order to maintain the original calculation speed, only the last output feature map T3 of ResNet is used for self-attention information extraction. First, it is unified to a specific number of channels C through the preprocessing convolution layer. A , after passing through three different convolutional layers, query_conv, key_conv and val_conv, the feature T is obtained q ,T k and T v . Dimensions are (C A ×H×W) and perform two-dimensional and three-dimensional merging, and then T q Swap one and two dimensions to get the sizes (HW×C A ),(C A ×HW),(C A ×HW). Then the attention vector T is obtained through the following formula a

[0048] T a =(softmax(T q ×T k ))×T v

[0049] Output results through T ai With T i Weighted output:

[0050] T ai =final_conv(α*T a +T3)

[0051] Among them, final_conv is the last layer of convolution for feature filtering. The final feature map output is added as T4 to the subsequent input

[0052] 2. Feature fusion: Since the number of channels and sizes of T1 to T4 are different, they need to be resized before fusion. First, convolution operations are performed on T1 to T4 to unify them to a specific number of channels C. f =64, from T4 to T1, for T i Do the following:

[0053] 2-1 Upsampling to T by linear interpolation i-1 Same length and width dimensions

[0054] 2-2 Put the same size T i and Ti-1 Add the corresponding positions to get the new T i-1

[0055] 2-3i=i-1

[0056] 2-4 Repeat 2-1 to 2-3 to obtain a feature map T that combines different convolutional layers and self-attention features. c

[0057] 3 Variable convolution method.

[0058] 3-1 First, T c Through two identical computing units: three-layer convolution, T is extracted cls and T pts The former is the feature map to which variable convolution is applied, and the latter is used to generate a vector that specifies the position of the variable convolution.

[0059] 3-2 T pts Then generate T through two layers of convolution offset , the size is (N×H×W), where N is the number of calculation positions contained in the variable convolution receptive field, T offset The value is the deviation of the convolution calculation point from the convolution position. A positive number is a rightward offset, and a negative number is a leftward offset.

[0060] T offset =conv(T pts )

[0061] 3-3 T offset The purpose of calculating the loss with the label of the dataset is to make T offset The value of falls on the position of the lane line in the image, that is, in order to make the position of the variable convolution convoluted be the lane line position, it can be called "strip convolution".

[0062] 3-4 T offset Application in T cls Perform variable convolution operation on T offset The convolution position determined by the value is used for convolution calculation to obtain the new feature map T o .

[0063] T o =deformConv(T cls ,T offset )

[0064] 4. Lane line prediction stage. oAs input, the calculation layer of two layers of full connection and ReLU combination outputs the probability of each grid belonging to each lane line, which is composed of a probability matrix M with a size of (lanes, H, W). Among them, lanes is the number of channels of the number of lane lines, and H and W are the grid sizes. Each channel represents the classification information of a lane line in each grid. The probability matrix M is calculated with the label of the dataset for cross entropy loss and back propagated. fc is the fully connected layer

[0065] M=fc(relu(fc(T o )))

[0066] Image segmentation loss calculation: In the process of 1-2 of the present invention, 3 feature maps T are obtained i , each of which is fed into a computational unit consisting of convolution, batchnormalization, and ReLU activation, and repeated three times for feature extraction. The resulting three feature maps are stacked and then subjected to four feature fusion calculations in the aforementioned computational unit. Finally, a single convolution operation is performed to output the segmented data of the original image. The output data is then compared with the dataset's labels using a cross-entropy loss function, followed by backpropagation. This allows the backbone network to learn the ability to extract features for the segmented image, enhancing its understanding of the original image data.

[0067] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A lane detection method using self-attention mechanism and variable convolution, characterized in that: The steps include: S1, obtain original image data and construct a dataset; S2, data set preprocessing, the original image data is subjected to occlusion, brightness adjustment, contrast adjustment, and random noise generation to obtain preprocessed image data; S3. Use the preprocessed image data as input for feature extraction S3-1, extract the three feature maps T1~T3 generated by the last three convolution modules through ResNet18 composed of 4 convolutional stack modules; S3-2, perform self-attention extraction on the output feature map T3 to obtain feature map T4; S4, feature fusion, before the fusion of feature maps T1~T4, the size needs to be transformed. First, the convolution operation of T1~T4 is performed to unify them to a specific number of channels C f =64; then, from T4 to T1, a feature map T that combines different convolutional layers and self-attention features is obtained by upsampling and adding one by one. c ; S5, the feature map T that combines different convolutional layers and self-attention features c Perform variable convolution to obtain a new feature map T o : The following sub-steps are included: S5-1, first of all, T c Through two computing units consisting of three layers of convolution: three layers of convolution, T is extracted cls and T pts , the former is the feature map to which variable convolution is applied, and the latter is used to generate a vector specifying the variable convolution position; S5-2, T pts Then generate T through two layers of convolution offset , the size is (N×H×W), where N is the number of calculation positions contained in the variable convolution receptive field, T offset The value of is the value of the convolution calculation point deviating from the convolution position. A positive number is a rightward shift, and a negative number is a leftward shift. T offset =conv(T pts ) S5-3, T offset Calculate the loss with the label of the dataset; S5-4, T offset Application in T cls Perform variable convolution operation on T offset The convolution position determined by the value is used for convolution calculation to obtain the new feature map T o , T o =deformConv(T cls ,T offset ); S6, Lane Line Prediction T o As input, the calculation layer of two layers of full connection and ReLU combination outputs the probability of each grid belonging to each lane line. The probability matrix M composed of them has a size of (lanes, H, W), where lanes is the number of channels of the number of lane lines, H and W are the grid sizes, and each channel represents the classification information of a lane line in each grid. The probability matrix M is calculated with the label of the dataset for cross entropy loss and back propagated. fc is the fully connected layer M=fc(relu(fc(T o )))。 2. The lane detection method using self-attention mechanism and variable convolution according to claim 1, characterized in that: The three feature maps T produced in step S3-1 i , size is (C i ×H i ×W i ), i∈[1,3] feature map, where C is the number of channels, H and W are the length and width.

3. The lane detection method using self-attention mechanism and variable convolution according to claim 2, characterized in that: In step S3-2, the self-attention extraction method is as follows: self-attention information is extracted only from the last feature map T3 output by ResNet, and first unified to a specific number of channels C through the preprocessing convolution layer. A , after passing through three different convolutional layers, query_conv, key_conv and val_conv, the feature T is obtained q ,T k and T v , the dimensions are (C A ×H×W), and both are merged in two or three dimensions, and then T q Swap one and two dimensions to get the sizes (HW×C A ),(C A ×HW),(C A ×HW), and then the attention vector T is obtained by the following formula a : T a =(softmax(T q ×T k ))×T v Output results through T ai With T i Weighted output: T ai =final_conv(α*T a +T3) Among them, final_conv is the last layer of convolution for feature filtering, and the final feature map output obtained is added as T4 to the subsequent input.

4. The lane detection method using self-attention mechanism and variable convolution according to claim 1, characterized in that In step S4, the feature fusion method is: S4-1, upsampling to T by linear interpolation i-1 Same length and width dimensions; S4-2, the same size T i and T i-1 Add the corresponding positions to get the new T i-1 ; S4-3, i=i-1; S4-4, repeat S4-1 to S4-3 to obtain the feature map T c .

Images