Enhanced single-stage lane line detection method for automatic driving

Abstract

The invention relates to the technical field of target detection of deep learning, in particular to an enhanced single-stage type lane line detection method for automatic driving, which comprises the following steps: firstly, extracting universal features through a pre-trained backbone network, then improving receptive field information corresponding to an original image on a feature map through a Dilated Encoder module, and finally obtaining a target detection result; according to the method, the position on a corresponding feature map is found according to set line-anchor position information, features are extracted, the extracted features are better fused with global lane line feature information by introducing a feature fusion module of a self-attention mechanism, and finally, a lane line detection result is output by three full-connection prediction branches. And in the model training stage, the line-anchor is finely adjusted through the gradient information returned by the loss. According to the method, a new network structure is provided, a single-stage network structure design is adopted, the network structure and a feature fusion module are improved, and a multi-head self-attention mechanism is combined, so that the method can better utilize global lane line information, the lane line detection precision can be well improved, the lane line is modeled as a line-anchor, and the lane line detection efficiency is improved. A learnable line-anchor strategy is adopted, and meanwhile, an improved objective function can enable network convergence to be faster.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning target detection, and in particular to an enhanced single-stage lane line detection method for autonomous driving. Background Art

[0002] Deep learning technology has made a series of relevant progress in the field of autonomous driving. Lane line detection is a crucial subsystem in the autonomous driving system. Accurate and fast lane line detection is crucial to achieving autonomous driving.

[0003] At present, lane detection methods can be mainly divided into segmentation-based methods, line-anchor-based methods, row classification-based methods and parameter-based methods. In the segmentation-based method, the lane line is predicted based on pixels, by judging whether the image pixel belongs to the lane line or background information. The line-anchor-based prediction method uses the anchor mechanism, but unlike the rectangular anchor used in traditional target detection, the line-anchor is used to fit the shape of the lane line in the lane detection task. The row classification-based method predicts the position information of the lane line in each row. The parameter-based method directly models the parameters of the lane line.

[0004] However, existing detection methods have the problems of low detection accuracy and consume a lot of computing resources, making it difficult to meet detection needs in complex scenarios. Summary of the invention

[0005] In order to improve the detection capability of complex scenes, the present invention adopts a single-stage network structure design, improves the network structure and feature fusion module, and combines the multi-head self-attention mechanism, so that the method can better utilize the global lane line information. At the same time, the objective function is improved to enable the network to converge faster.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An enhanced single-level lane detection method for autonomous driving comprises the following steps:

[0008] First, common features are extracted through the pre-trained backbone network, and then the receptive field information of the original image on the feature map is improved through the Dilated Encoder module (residual connection expansion module). According to the set line-anchor position information, the position on the corresponding feature map is found and the features are extracted. These extracted features are better integrated with the global lane line feature information through the feature fusion module that introduces the self-attention mechanism. Finally, the lane line detection results are output by three fully connected prediction branches.

[0009] Further, more specifically, the method comprises the following steps:

[0010] Step 0: Initialization of the detection method: including network initialization and image preprocessing;

[0011] Step 1: The input is image data, and the feature extraction is performed on the preprocessed image, which is a feature map after 32 times downsampling;

[0012] Step 2: The features extracted by the backbone network are further encoded by the Dilated Encoder to extract target information of different receptive fields. The Dilated Encoder module is composed of 4 series-connected dilated convolutions, with expansion rates of 2, 4, 6, and 8 times, respectively, so that the series-connected network can also obtain feature information of receptive fields of different scales.

[0013] Step 3: According to the line-anchor information pre-set on the original image, its position information on the 32-fold downsampled feature map is obtained and the corresponding features are extracted, that is, the features of each line-anchor. Then, the anchor feature aggregation module based on the attention mechanism is used to fully learn the global lane line information;

[0014] Step 4: Predict through the linear layer and output the angle, length and starting point coordinate information of the lane line;

[0015] Step 5: Calculate the loss: By calculating the loss between the predicted target and the true value, the lane line starting point coordinate loss, lane line length loss and angle loss are lost.

[0016] Furthermore, in the step Step0, the backbone network adopts pre-trained ResNet weights.

[0017] Furthermore, the image preprocessing in Step 0 includes image scaling, random horizontal flipping, and random affine transformation.

[0018] Furthermore, in the step Step 1, ResNet50 is used as the backbone network.

[0019] Furthermore, the calculation formula in Step 5 is as follows:

[0020] L all =λ1L start_xy +λ2L length +λ3L score +λ3L angle (1-1)

[0021] where λ1L start_xy and L lengthThey are the loss of x-coordinate and y-coordinate of the lane line starting point, the lane line length loss, and the loss is calculated using smooth L1. score is the lane line confidence loss, calculated using Focal loss, L angle The angle loss proposed in this invention is calculated as shown in (1-2)

[0022]

[0023] N proposal It represents the number of points that make up a completed lane line, and α and β represent the angles between the predicted and real line segments, respectively.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention proposes a new network structure, adopts a single-stage network structure design, improves the network structure and feature fusion module, and combines the multi-head self-attention mechanism, so that the method can better utilize the global lane line information, which can greatly improve the accuracy of lane line detection. At the same time, the improved objective function can make the network converge faster.

[0026] 2. The present invention improves the loss function and adds angle regression loss. The method can achieve rapid convergence in 5000 iterations. And under the same number of training rounds, the convergence effect of the method is better.

[0027] 3. This method introduces a learnable line-anchor strategy. During the training process, the shape of the line-anchor is continuously fine-tuned, and a smaller number of line-anchors is used while achieving better detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0029] Figure 1 This is a flow chart of algorithm training of the present invention.

[0030] Figure 2 This is a network structure diagram of the single-stage lane line detection method of the present invention.

[0031] Figure 3 The detection results of the baseline network and the algorithm are shown in Figure 2.

[0032] Figure 4 This is a comparison chart of the loss curves. DETAILED DESCRIPTION

[0033] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0034] The lane detection method proposed in this invention is a line-anchor based method. The network part adopts an end-to-end single-stage detection architecture. The overall structure diagram is as follows: Figure 2 As shown. The detection system accepts RGB image data, first extracts common features through a pre-trained backbone network. The backbone network adopts the ResNet series structure, and then passes through the Dilated Encoder module to enhance the receptive field information of the original image on the feature map. According to the set line-anchor position information, the position on the corresponding feature map is found and features are extracted. These extracted features are better integrated with the global lane line feature information by introducing a feature fusion module with a self-attention mechanism. Finally, the lane line detection results are output by three fully connected prediction branches. The specific detection method is as follows Figure 1 As shown, the following steps are included:

[0035] Step 0: Initialization of the detection method. This includes network initialization and image preprocessing. The backbone network uses pre-trained ResNet weights, and the image preprocessing part includes image scaling, random horizontal flipping, and random affine transformation.

[0036] Step 1: The input is image data. This paper uses ResNet50 as the backbone network to extract features from the preprocessed image and obtain the feature map after 32 times downsampling.

[0037] Step 2: The features extracted by the backbone network are further encoded by the Dilated Encoder to extract target information of different receptive fields. The Dilated Encoder module is composed of 4 series-connected dilated convolutions, with expansion rates of 2, 4, 6, and 8 times, respectively, so that the series-connected network can also obtain feature information of receptive fields of different scales.

[0038] Step 3: According to the line-anchor information pre-set on the original image, its position information on the 32-fold downsampled feature map is obtained and the corresponding features are extracted, that is, the features of each line-anchor. Then, the anchor feature aggregation module based on the attention mechanism is used to fully learn the global lane line information.

[0039] Step 4: Predict through the linear layer and output the angle, length and starting point coordinate information of the lane line.

[0040] Step 5: Calculate the loss. By calculating the loss between the predicted target and the true value, the loss of the lane line starting point coordinates, lane line length and angle is lost. The calculation formula is shown in (1-1).

[0041] L all =λ1L start_xy +λ2L length +λ3L score +λ3L angle (1-1)

[0042] where λ1L start_xy and L length They are the loss of x-coordinate and y-coordinate of the lane line starting point and the lane line length loss, respectively. Smooth L1 is used to calculate the loss. score L is the lane line confidence loss, calculated using Focal loss. angle This is the angle loss proposed in this paper, and the calculation formula is shown in (1-2).

[0043]

[0044] N proposal It represents the number of points that make up a completed lane line, and α and β represent the angles between the predicted and real line segments, respectively.

[0045] Example: Lane Detection Experiment

[0046] 1) Comparison of test results

[0047] This experiment is trained and tested on the Tusimple lane line dataset. Tusimple is a lane line detection dataset proposed in 2017, including complex weather scenes and different traffic conditions. The training set mainly includes 3626 video clips and 3626 annotated frames, and the test set includes 2782 video clips. Each clip contains 20 frames of image data per second, and the size of each frame is 1280*720.

[0048] like Figure 3 As shown, the left column is the original image, the middle column is the detection result of the baseline network, and the right column is the detection result of this method. By comparison, it can be seen that in the first picture, the method can detect the complete four lane lines, and the detection effect of the curved part of the second lane line is better. In the third picture, although the third lane line appears intermittent and blurred, the method of the present invention also effectively detects the lane line. The test data and actual detection results on Tusimple have well proved that the method of the present invention has good detection performance.

[0049] like Figure 4 As shown in the figure, the left figure is the loss curve of this method, and the right figure is the loss curve of the baseline method. By improving the loss function and adding angle regression loss, this method can achieve fast convergence in 5000 iterations. And under the same number of training rounds, the convergence effect of this method is better.

[0050] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An enhanced single-stage lane detection method for autonomous driving, characterized in that: The following steps are involved: First, common features are extracted through the pre-trained backbone network, and then the receptive field information of the original image on the feature map is improved through the Dilated Encoder module. The position of the corresponding line-anchor on the feature map is found and features are extracted according to the set line-anchor position information. These extracted features are better integrated with the global lane line feature information through the feature fusion module that introduces the self-attention mechanism. Finally, the lane line detection results are output by three fully connected prediction branches. During the model training stage, the line-anchor is fine-tuned through the gradient information returned by the loss.

2. The enhanced single-level lane detection method for autonomous driving according to claim 1, characterized in that: The enhanced single-level lane line detection method for autonomous driving more specifically includes the following steps: Step 0: Initialization of the detection method: including network initialization and image preprocessing; Step 1: The input is image data, and the feature extraction is performed on the preprocessed image, which is a feature map after 32 times downsampling; Step 2: The features extracted by the backbone network are further encoded by the Dilated Encoder to extract target information of different receptive fields. The Dilated Encoder module is composed of 4 series-connected dilated convolutions, with expansion rates of 2, 4, 6, and 8 times, respectively, so that the series-connected network can also obtain feature information of receptive fields of different scales. Step 3: According to the line-anchor information pre-set on the original image, its position information on the 32-fold downsampled feature map is obtained and the corresponding features are extracted, that is, the features of each line-anchor. Then, the anchor feature aggregation module based on the attention mechanism is used to fully learn the global lane line information; Step 4: Predict through the linear layer and output the angle, length and starting point coordinate information of the lane line; Step 5: Calculate the loss: By calculating the loss between the predicted target and the true value, the lane line starting point coordinate loss, lane line length loss and angle loss are lost.

3. The enhanced single-level lane detection method for autonomous driving according to claim 2, characterized in that: In the step Step 0, the backbone network uses pre-trained ResNet weights.

4. The enhanced single-level lane detection method for autonomous driving according to claim 2, characterized in that: The image preprocessing in Step 0 includes image scaling, random horizontal flipping, and random affine transformation.

5. The enhanced single-level lane detection method for autonomous driving according to claim 2, characterized in that: In the step Step 1, ResNet50 is used as the backbone network.

6. The enhanced single-level lane detection method for autonomous driving according to claim 2, characterized in that: The calculation formula in Step 5 is as follows: L all =λ1L start_xy +λ2L length +λ3L score +λ3L angle (1-1) where λ1L start_xy and L length They are the loss of x-coordinate and y-coordinate of the lane line starting point, the lane line length loss, and the loss is calculated using smooth L1. score is the lane line confidence loss, calculated using Focal loss, L angle The angle loss proposed in this invention is calculated as shown in (1-2) N proposal It represents the number of points that make up a completed lane line, and α and β represent the angles between the predicted and real line segments, respectively.

Images