A lane line detection method based on deformable attention mechanism

Abstract

The application discloses a lane line detection method based on a deformable attention mechanism, comprising the following steps: (1) constructing a main convolution network based on a residual structure; (2) adding a pooling layer and a full connection layer to the top layer of the main convolution network to form a first deep model and pre-training; (3) extracting the main convolution network in the first deep model, using the main convolution network to generate a feature map, obtaining different scale feature maps based on a feature pyramid network, and adding a feature flip fusion layer, a deformable attention convolution network, a pooling layer and a convolution layer to the feature pyramid network to form a second deep model; (4) training the second deep model using a picture lane line detection data set; and (5) analyzing a to-be-detected picture by using the second deep model, predicting a key point sequence corresponding to each lane line in the picture, and forming a lane line detection picture. The application can promote the model to have better attention distribution on a complex deformation object, improve the robustness of the model, and make the lane line detection more accurate.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision and image processing, and particularly relates to a lane line detection method based on a deformable attention mechanism. BACKGROUND

[0002] With the discovery and in-depth study of deep learning theory, multiple fields and tasks under automatic driving have had rapid breakthroughs and significant improvements. Among them, lane line detection has become one of the most urgent computer vision tasks and a current popular research direction due to the high requirements of automatic driving on its accuracy and real-time performance. The lane line detection task requires a computer vision system to predict the specific position and type of lane lines on an arbitrary size picture. The current mainstream lane line detection solutions all adopt a convolutional network architecture, which can be specifically divided into three categories: segmentation-based methods, anchor-based methods, and key parameter-based methods.

[0003] The segmentation-based method is currently more famous for the work of Zheng et al. in 2021, which was published in the AAAI Conference on Artificial Intelligence conference, and is entitled "Resa: Recurrent feature-shift aggregator for lane detection". The work introduces that the RESA method is the frontier solution in this field, which aggregates global features through a real-time feature aggregation module.

[0004] The anchor-based method is currently more famous for the work of Zheng et al. in 2022, which was published in the IEEE / CVF conference on computer vision and pattern recognition conference, and is entitled "Clrnet: Cross layer refinement network for lane detection". The work introduces that the CLRNet is the frontier solution in this method field, which makes lane line features interact with multi-scale global features through a RoIGather module and a cascading idea.

[0005] However, considering the overall structure of the lane from the perspective of the front camera in the driving scene, the road has approximately equidistant lane lines, which are distributed more symmetrically. This attribute is suitable for modeling, for example, the existence of a left lane line is likely to indicate the corresponding existence of a right lane line, and the structure of the left lane can help describe the right lane. However, because the camera image will have a slight misalignment, a more precise lane line detection model needs to be designed to solve this complex deformation problem. SUMMARY

[0006] The application discloses a lane line detection method based on a deformable attention mechanism, which can promote the detection model to have better attention distribution on objects with relatively complex deformation, improve the robustness of the trained model, and make the lane line detection more accurate.

[0007] A lane line detection method based on a deformable attention mechanism comprises the following steps:

[0008] (1) constructing a main convolutional network based on a residual structure;

[0009] (2) adding an image pooling layer and a fully connected layer network to the top layer of the main convolutional network to form a first deep model, and pre-training the first deep model on a large data set;

[0010] (3) extracting the main convolutional network in the pre-trained first deep model, using the main convolutional network to generate feature maps, obtaining feature maps of different scales based on a feature pyramid network, and inputting the feature maps of each scale into a detection network based on a cascading idea to form a second deep model;

[0011] The detection network comprises a feature flipping fusion layer, a deformable attention convolutional network, a pooling layer and a convolutional layer.

[0012] (4) training the second deep model using a data set for lane line detection of pictures until the model converges;

[0013] (5) using the trained second deep model to analyze a to-be-detected picture, predicting a key point sequence corresponding to each lane line in the picture, and forming a lane line detection map output.

[0014] The application first constructs a main convolutional neural network based on a residual structure, then pre-trains a deep learning network comprising a pooling-fully connected layer as an initialization of the main convolutional network, then adds a feature flipping fusion layer and a deformable attention convolutional detection network, and finally trains the network on a training data set (a lane line detection task) to predict a key point sequence of a lane line in an input picture. The method can promote the deep network to learn a relatively symmetrical distribution of lane lines, and simultaneously optimize the detection of lane lines with relatively complex deformation caused by camera shooting, so that a robust lane line detection model is more easily generated.

[0015] In step (2), the large data set is an image recognition data set ImageNet.

[0016] The image pooling layer is added to the top layer of the main convolutional network, so that the image is changed from a three-dimensional feature map into a feature vector; then the fully connected network is used to deform the feature vector into an N-dimensional vector, corresponding to N image categories of the large data set.

[0017] The specific pre-training process is as follows:

[0018] The first deep model is trained on GPU, each GPU calculates 32 images at a time, and 8 GPUs are trained in parallel;

[0019] The SGDM optimization algorithm is used, the initial learning rate is 0.256, and the learning rate is reduced by 10% after every 30 cycles, a total of 90 cycles, the Momentum parameter is set to 0.9, until the model converges.

[0020] In step (3), the feature map of each scale is input into the detection network based on the cascade idea, the detection network first performs feature flipping fusion and deformable attention processing on the input feature map to obtain a new feature map, and then uses average pooling to obtain a preliminary feature vector, the dimension size of which is (C, ), wherein C is the channel number of the corresponding feature map, represents the width size of the corresponding feature map; finally, a 1-dimensional convolution layer is used to obtain the final fused feature vector, and the size of the 1-dimensional convolution kernel is set to 3.

[0021] The definition of deformable attention is as follows:

[0022]

[0023] Wherein y(p) is the value corresponding to the output feature map position p; K represents the number of positions sampled during convolution, w k represents the weight size of convolution at the corresponding position; p k is the convolution offset coordinate corresponding to position p, Δp k is the offset learned by the deformable attention network, x(p+p k +Δp k is the value of the corresponding position of the input feature map after offset; Δm k is the weight learned by the deformable attention network;

[0024] Assuming that the convolution kernel size of the deformable attention convolution network is S, the specific value of the offset coordinate p k is selected from the following dynamic set:

[0025]

[0026] The convolution layer in the detection network is initialized with a normal distribution with a mean of 0 and a variance of 0.001.

[0027] The specific process of step (4) is as follows:

[0028] (4-1) Input the picture of the training set into the second deep model, and generate multiple scale feature maps based on the feature pyramid network;

[0029] (4-2) input the feature map of each scale into the corresponding detection network based on the cascade idea, and the detection network processes the input feature map to obtain a feature vector;

[0030] (4-3) the generated feature vector is respectively sent into the 1D convolution layer of the classification and regression task, to generate the predicted probability, starting point, angle, lane length and coordinate offset; the Focal loss, Smooth-L1 loss and Line IoU loss loss functions are used to calculate the gradient of the parameters on the network, and the Adam optimization algorithm is used to update the parameter value; the initial learning rate is set to 10 -3 ;

[0031] (4-4) repeat steps (4-1) to (4-3) until the model converges.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] 1. Based on the understanding of the receptive field of the convolutional network, the present application adopts a new deformable attention, which improves the attention of the neural network to different shaped objects, so that it is easier to learn the connotation correlation between the input and the output.

[0034] 2. The present application designs a flip fusion layer to fuse the feature map, which is beneficial to utilize the symmetry of the foreground lane line image.

[0035] 3. The present application has strong applicability, and can be directly and effectively deployed in most existing target detection technologies by adding deformable attention to the ordinary convolutional network, without affecting other method processes. DETAILED DESCRIPTION

[0036] Figure 1 The present application is a lane line detection method based on a deformable attention mechanism, and the flowchart is shown in the figure.

[0037] Figure 2 The present application is a model structure diagram.

[0038] Figure 3 The present application is a deformable attention convolutional layer added in the flip feature fusion layer. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings and examples, and it should be pointed out that the following examples are intended to facilitate the understanding of the present application, and do not have any limiting effect on it.

[0040] As Figure 1 shown, a lane line detection method based on a deformable attention mechanism comprises the following steps:

[0041] S01, a residual structure based backbone convolutional neural network is constructed.

[0042] A residual structure based backbone convolutional neural network is constructed to extract the initial feature map. In this embodiment, the network depth, network width, image resolution, convolution span and other parameters are kept unchanged according to the design of ResNet-34.

[0043] S02, the top layer of the constructed backbone convolutional network is added with a pooling layer and a fully connected layer network to form a first deep model, and pre-training is performed on a large data set.

[0044] The specific process of step S02 is as follows:

[0045] S021, an image pooling layer is added to the top layer of the backbone convolutional network, so that the image is changed from a three-dimensional feature map to a feature vector, and then a fully connected network is used to deform it into a 1000-dimensional vector corresponding to the 1000 image categories of ImageNet;

[0046] S022, the constructed first deep model is trained on a GPU, 32 images are calculated at a time for each GPU, and 8 GPUs are used for parallel training;

[0047] S023, the SGDM optimization algorithm is used, the initial learning rate is 0.256, and the learning rate is reduced to 10% after every 30 cycles, a total of 90 cycles, the Momentum parameter is set to 0.9, until the model converges.

[0048] In this embodiment, in order to alleviate the problem of too small amount of lane line detection data, a large data set is used, which is a large-scale image recognition data set ImageNet, which can accelerate the convergence speed and training quality of lane line detection.

[0049] ImageNet is the first super large-scale image recognition data set, which was published in the paper "Imagenet: A large-scale hierarchical image database" by Jia Deng et al. of Stanford University at Conference on Computer Vision and Pattern Recognition 2009. The data set searches for pictures of 1000 categories of objects on the Internet, and the image size is maintained at about 256*256, and each category has more than 1000 training pictures. The training set has 1281167 pictures, and the verification set has 50000 pictures.

[0050] The backbone convolutional network is followed by an image pooling layer to reduce the feature map to a feature vector, which is then converted to a prediction vector of length 1000 using a fully connected layer. The pre-training is completed on ImageNet through the same training mode as ResNet-34.

[0051] S03, the extracted parameters are used for the initialization of the backbone convolutional network, and a feature flipping fusion layer and a deformable attention convolutional network are added.

[0052] Specifically, the feature maps in the pre-trained first deep model are extracted, and feature maps of different scales are obtained based on the feature pyramid network. A feature flipping fusion layer, a deformable attention convolutional network, a pooling layer and a convolutional layer are added after the feature maps to form a second deep model. As shown in Figure 2 , it is a schematic diagram of the model structure of the whole application.

[0053] In this step, the feature maps generated by the backbone convolutional network in the previous step are used. For the backbone convolutional network, the size of the output feature map is unchanged. For the layers with unchanged output size, they are classified as a stage. The features output by each stage are extracted, and the sizes of the feature maps output by the stages are and times the size of the original image. By adding convolutional layers and up-sampling layers, the feature maps of adjacent sizes are fused by adding corresponding elements to generate new feature maps.

[0054] For each feature map, a detection module based on a convolutional layer and a pooling layer is added after it. The module first flips and fuses the input feature map horizontally. Since there is a certain distortion in the foreground camera shooting, a deformable attention is added through a convolutional layer. The definition of the deformable attention is as follows:

[0055]

[0056] where y(p) is the value corresponding to the position p of the output feature map; K represents the number of positions sampled during convolution, for example, if the convolution kernel size is 3*3, then K is equal to 9, w k represents the weight size of the corresponding position convolution; p k is the convolution offset coordinate corresponding to position p, Δp k is the offset learned by the deformable attention network, x(p+p k + Δp k ) is the value of the corresponding position of the input feature map after offset; Δm k is the attention weight learned by the deformable attention network, and the weight of some unwanted sampling points can be learned to be 0. Assuming that the convolution kernel size of the deformable attention network is S, the offset coordinate p kThe specific numerical value is selected from the following dynamic set:

[0057]

[0058] As shown in Figure 3 , it is a schematic diagram of the deformable attention convolutional layer added in the flip feature fusion layer of the present application. For an input feature map, assuming that the original is a 3*3 convolutional operation, in order to learn the offset, define another 3*3 convolutional layer, and the length and width of the output offset dimension are the length and width of the original feature map, and the channel number is 3K, wherein the first 2K channels correspond to the learned offset, and the remaining K channels pass through the sigmoid layer to obtain the attention weight Am of each position. k .

[0059] Through deformable attention, the convolutional receptive field is no longer originally square, but through the learned offset and attention weight, the receptive field is close to the actual shape of the object, so that the subsequent convolution area can always cover around the object shape.

[0060] In addition, the detection network has a total of 3, and 3 feature maps of different scales correspond to one detection network with the same structure. The detection network adopts the idea of cascade, and a set of prediction values obtained by the previous detection network is used as the anchor of the next detection network, that is, further prediction and improvement are made on the basis of the previous prediction.

[0061] S04, using the picture lane line detection dataset to train the second deep model until the model converges.

[0062] The specific process of step S04 is as follows:

[0063] S041, input the picture of the training set into the second deep model, and after calculation, a plurality of scale feature maps are generated, and the size of each scale feature map is and times the size of the original input picture.

[0064] S042, input each scale feature map into the corresponding detection network based on the idea of cascade, and the detection network first performs feature flip fusion and deformable attention processing to obtain a new feature map, and then uses average pooling to obtain a preliminary feature vector, and the dimension size of the preliminary feature vector is (C, ), wherein C is the channel number of the corresponding feature map, represents the width size of the corresponding feature map, and finally a 1-dimensional convolutional layer is used to obtain a final fused feature vector, and the size of the 1-dimensional convolutional kernel is set to 3.

[0065] S043, the generated feature vectors are respectively sent into the 1-dimensional convolution layer of the classification and regression task, to generate the predicted probability, starting point, angle, lane length and coordinate offset, the gradient of the parameters on the network is calculated using the Focal loss, Smooth-L1 loss and Line IoU loss loss function, and the parameter value is updated using the Adam optimization algorithm; the initial learning rate is set to 10 -3 .

[0066] S044, repeat the above steps until the model converges.

[0067] In this embodiment, the training data set uses the CULane data set, which is published in the paper "Spatial as deep: Spatial cnn for traffic scene understanding" by Xingang Pan et al. on Thirty-Second AAAI Conference on Artificial Intelligence. The training set of this data set has 88880 images, the verification set has 9675 images, and the test set has 34680 images. The test set is divided into normal category and 8 challenging lane line categories. The image size is 1640*590, and the size is changed to 800*320 when input.

[0068] S05, using the trained second deep model to analyze the to-be-tested picture, predicting the key point sequence corresponding to each lane line in the image, and forming a lane line detection graph output.

[0069] The above embodiments describe the technical solutions and advantages of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement within the scope of the present application should be included in the protection scope of the present application.

Claims

1. A lane line detection method based on a deformable attention mechanism, characterized in that, Includes the following steps: (1) Construct a backbone convolutional network based on residual structure; (2) Add image pooling layers and fully connected layers to the top layer of the backbone convolutional network to form the first deep model, and pre-train it on a large dataset; (3) Extract the backbone convolutional network from the pre-trained first deep model, use it to generate feature maps, obtain feature maps of different scales based on the feature pyramid network, and input the feature maps of each scale into the detection network based on the cascade idea to jointly form the second deep model. The detection network includes a feature flipping fusion layer, a deformable attention convolutional network, pooling layers, and convolutional layers. The feature maps at each scale are input into a detection network based on the cascaded approach. The detection network first performs feature flipping fusion and deformable attention processing on the input feature maps to obtain new feature maps, and then uses average pooling to obtain preliminary feature vectors with dimensions (C, ...). ), where C is the number of channels in the corresponding feature map, This represents the width of the corresponding feature map; finally, a 1D convolutional layer is used to obtain the final fused feature vector, with the 1D convolutional kernel size set to 3. The definition of deformable attention is as follows: Where y(p) is the numerical value corresponding to position p in the output feature map; K represents the number of sampling positions during convolution, and w k p represents the weights of the convolution at the corresponding position. k It is the convolution offset coordinate corresponding to position p, Δp k It is the offset learned by the deformable attention network, x(p+p k +Δp k ) is the numerical value at the corresponding position after the input feature map is offset; Δm k These are the weights learned by the deformable attention network; Assume the kernel size of the deformable attention convolutional network is S, and the offset coordinate is p. k The specific values ​​are selected from the following dynamic set: The convolutional layers in the detection network are initialized with a normal distribution with a mean of 0 and a variance of 0.

001. (4) Train a second deep model using the image lane line detection dataset until the model converges; (5) The trained second-depth model is used to analyze the test image and predict the key point sequence corresponding to each lane line in the image to form a lane line detection map output.

2. The lane line detection method based on deformable attention mechanism according to claim 1, characterized in that, In step (2), the large dataset used is the ImageNet image recognition dataset.

3. The lane line detection method based on deformable attention mechanism according to claim 2, characterized in that, In step (2), an image pooling layer is added to the top layer of the backbone convolutional network, so that the image is transformed from a three-dimensional feature map into a feature vector; then a fully connected network is used to transform the feature vector into an N-dimensional vector, corresponding to the N image types in the large dataset.

4. The lane detection method based on deformable attention mechanism according to claim 1, characterized in that, In step (2), the specific pre-training process is as follows: The first deep learning model was trained on GPUs, with each GPU computing 32 images at a time, and 8 GPUs trained in parallel. The SGDM optimization algorithm was used with an initial learning rate of 0.256, which was reduced to 10% after every 30 iterations for a total of 90 iterations. The Momentum parameter was set to 0.9 until the model converged.

5. The lane line detection method based on deformable attention mechanism according to claim 1, characterized in that, The specific process of step (4) is as follows: (4-1) Input the images from the training set into the second deep model, and generate feature maps of multiple scales after passing through the feature pyramid network; (4-2) Input the feature map of each scale into the corresponding detection network based on the cascade idea. The detection network processes the input feature map to obtain the feature vector. (4-3) The generated feature vectors are fed into the 1D convolutional layers of the classification and regression tasks, respectively, to generate the predicted probabilities, starting points, angles, lane lengths, and coordinate offsets; the gradients of the parameters on the network are calculated using Focalloss, Smooth-L1 loss, and Line IoUloss loss functions, and the parameter values ​​are updated using the Adam optimization algorithm; the initial learning rate is set to 10. -3 ; (4-4) Repeat steps (4-1) to (4-3) above until the model converges.

Images