Fast lane line detection method based on feature fusion

Abstract

In order to improve the performance of real-time lane line detection in a complex scene, the invention provides a simple and lightweight fast lane line detection method based on feature fusion. According to the method, semantic information contained in the deep-layer feature map and position information contained in the shallow-layer feature map can be fused. In the feature extraction process, expansion convolution is used, the receptive field of the model is increased, and more global information in the image is extracted. And meanwhile, a channel attention mechanism is used in a feature extraction process, and a channel containing lane line structure information is endowed with a relatively large weight, so that the detection accuracy is further improved. And finally, an experiment is carried out on a challenging lane line detection data set CULane, and the experiment result shows the effectiveness of the method provided in the aspects of precision and speed.

Description

Technical Field

[0001] The present invention relates to a feature fusion problem in the field of fast lane line detection, and in particular to a feature fusion method based on a feature pyramid. Background Art

[0002] Lane detection is a key step in driver assistance systems and self-driving cars. It can provide decision-making information for self-driving vehicles to ensure that the vehicle can drive safely on the road. Lane detection in the real world is a challenging task because it is often affected by different weather, lighting conditions and occlusions. At the same time, the computing resources for lane detection in practical application scenarios are limited, so it is very important to develop a lightweight, fast and accurate lane detection system.

[0003] At present, the ultra fast lane detection algorithm based on line-by-line classification has made good research progress in detection rate. However, the detection accuracy of the algorithm needs to be improved. The algorithm has two major networks: the backbone network and the auxiliary network. The auxiliary network is only valid during training, and only the backbone network is used for prediction. The feature fusion stage of the algorithm is relatively simple, and it lacks the extraction of spatial context position information in the image, which leads to slightly insufficient detection accuracy of the model. For the above reasons, the balance between detection accuracy and speed in the real-time lane detection system is still facing huge technical challenges, which is also the main bottleneck of the real-time lane detection system. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a lane line detection method based on feature fusion to improve the detection accuracy of a fast lane line detection system.

[0005] The following technical measures are adopted to solve the above technical problems: A lane line detection method based on feature fusion mainly includes the following steps:

[0006] (1) Feature fusion: The deep feature map of the backbone is upsampled and then fused with the shallow feature map, so as to fuse the semantic information contained in the deep network with the position information contained in the shallow network.

[0007] (2) Expanding the receptive field: Applying dilated convolution on the fused feature map and setting different dilation rates of the dilated convolution can expand the receptive field of the model and better utilize the information of different scales and global features in the image.

[0008] (3) Channel attention: The channel attention mechanism is applied to the fused feature map, focusing on the input image containing lane line information to improve the network's attention to the lane lines in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Lane line representation based on line-by-line classification.

[0010] Figure 2 This is the lane line detection model diagram of the present invention.

[0011] Figure 3 Schematic diagram of dilated convolution.

[0012] Figure 4 Schematic diagram of the channel attention mechanism.

[0013] Figure 5 This is a comparison chart of the detection accuracy of the method of the present invention and other lane line detection algorithms on the CULane dataset.

[0014] Figure 6 This is a comparison chart of the detection speed of the present invention and other lane line detection algorithms. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings. It is necessary to point out here that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Persons skilled in the art in this field may make some non-essential improvements and adjustments to the present invention based on the above content of the present invention, which still fall within the scope of protection of the present invention.

[0016] Figure 1 For the lane line representation method based on line-by-line classification, the input image is divided into a grid with a predefined number of line anchors (h), and each line anchor is divided into a predefined number of grid cells (w). Assuming that the maximum number of lanes is a predefined number C, the location of the lane line can be determined by a grid of size h×w. At the end of each line anchor, a cell is appended to indicate whether a lane line exists in the line anchor.

[0017] Figure 2 In particular, a lane line detection method based on feature fusion is involved, which can be divided into the following steps:

[0018] (1) Feature fusion: The deep feature map of the backbone is upsampled and then fused with the shallow feature map, so as to fuse the semantic information contained in the deep network with the position information contained in the shallow network.

[0019] (2) Expanding the receptive field: Applying dilated convolution to the feature map fused in (1) and setting different dilation rates of dilated convolution to expand the receptive field of the model can better utilize the information of different scales and global features in the image.

[0020] (3) Channel attention: The channel attention mechanism is applied to the feature map fused in (1), focusing on the input image containing lane line information to improve the network's attention to the lane lines in the image.

[0021] Specifically, in step (1), the present invention first upsamples the feature map C4 in the auxiliary branch, and gradually restores it to an image with the same resolution as C3 and C2. During the restoration process, the feature map with the same resolution as the original feature map is merged with the original feature map.

[0022] Specifically, for C4, the number of channels is first compressed by 1×1 convolution, and the spatial size is expanded to the same as C3 by 2x upsampling. For C3, its number of channels is compressed by 1×1 convolution. Next, the C3 image after the compressed number of channels is concatenated with the C4 upsampled image. Therefore, the fused image of C3 and C4 with the same spatial size as C3 is obtained. Finally, the fused image of C3 and C4 is upsampled by 2x and concatenated with C2 after the number of channels is compressed.

[0023] In step (2), the present invention applies dilated convolution to the fused feature map. The schematic diagram of dilated convolution is as follows: Figure 3 As shown in the figure, the dilated convolution expands the receptive field of the model, allowing the model to focus on global information and enhance the model's ability to extract spatial context information.

[0024] In step (3), the process diagram of the channel attention mechanism is as follows: Figure 4 As shown in the figure, the input feature map F is first subjected to global maximum pooling and global average pooling operations in parallel, and the feature map size is changed from C×H×W to C×1×1. Then, it passes through the multi-layer perceptron MLP module. During this process, the number of channels is compressed to 1 / r times the original number, and then expanded to the original number of channels. Two output features are obtained through the ReLU activation function. The output features are summed at the element level, and then subjected to the sigmoid activation operation to generate the final channel attention weight. Channel attention can be expressed by the following formula:

[0025]

[0026] Where σ(·) is the sigmoid function, W0 and W1 are the weights of the multi-layer perceptron MLP, represents a feature map with a size of C×1×1 after a global average pooling operation, Represents a feature map with a size of C×1×1 after the global maximum pooling operation.

[0027] To better illustrate the effectiveness of the present invention, the present invention is used to conduct experiments on the challenging classic lane detection dataset CULane, and then compared with many current mainstream methods. The detection accuracy is compared. Figure 5 Table 1 shows the comparison of detection accuracy and speed between the proposed method and other methods, indicating that the proposed algorithm improves detection accuracy without reducing detection speed compared with the ultra fast lane detection algorithm. The CULane dataset is evaluated using F1-measure:

[0028]

[0029] TP represents the predicted part with IoU>0.5, FP represents the predicted part with IoU<0.5, TN represents the part where there is no lane line and the predicted result does not have the lane line, and FN represents the part where there is a lane line but the predicted result does not have the lane line.

[0030] Figure 6 The detection time of other methods of the present invention is compared, and experiments show that the detection time of other methods is higher than that of the present invention.

[0031] Table 1 Comparison results of the present invention and other methods on the CULane dataset

[0032]

[0033] Note: The bold data in each column is the highest detection accuracy in that scene.

Claims

1. A fast lane detection method based on feature fusion, characterized in that The following steps are involved: (1) Feature fusion: The deep feature map of the backbone is upsampled and then fused with the shallow feature map, so that the semantic information contained in the deep network is fused with the position information contained in the shallow network; (2) Expanding the receptive field: Applying dilated convolution to the feature map fused in (1) and setting different dilation rates of dilated convolution to expand the receptive field of the model can better utilize the information of different scales and global features in the image; (3) Channel attention: The channel attention mechanism is applied to the feature map fused in (1), focusing on the lane line information. The input image with information is used to increase the network's attention to the lane lines in the image.

2. The lane line detection method based on feature fusion according to claim 1 is characterized in that The strategy for fusing deep features with shallow features described in step (1) uses ResNet as a feature extraction network, extracts feature maps of the second, third, and fourth blocks of ResNet respectively, and upsamples the feature map of the fourth block of the deepest layer. And gradually restore it to the same resolution as the third and second blocks, and then fuse the feature maps of the same size after restoration and before restoration.

3. The lane line detection method based on feature fusion according to claim 1 is characterized in that In the stage of expanding the receptive field described in step (2), when the convolution kernel of the dilated convolution is the same as that of the ordinary convolution, that is, when the number of parameters is the same, the dilated convolution has a larger receptive field, and it can also ensure that the size of the output feature map remains unchanged; On the one hand, dilated convolution expands the receptive field, making it easier to detect large targets. On the other hand, it improves image resolution and can more accurately locate target objects.

4. The lane line detection method based on feature fusion according to claim 1 is characterized in that The channel attention stage described in step (3) compresses the feature map in the spatial dimension to obtain a one-dimensional vector before performing operations. When compressing the spatial dimension, two operations, maximum pooling and average pooling, are used to take the most prominent feature and the average of all features respectively. These two operations can aggregate different aspects of spatial information in the feature map. Channel attention allows the model to pay more attention to the channel feature map containing lane line elements.

Images