Night lane line detection method and device based on decomposition convolution

Abstract

The invention relates to the technical field of computer vision, lane line detection and automatic driving, provides a night lane line detection method and device based on decomposition convolution, and aims to improve the accuracy and stability of lane line detection under a night light condition. The method comprises the following steps: firstly, acquiring image data of various vehicles on a night traffic road, marking the data, and establishing a reference data set CULane-Night for night lane line detection; based on the data set, the invention provides a lane line detection model MDCR-Net, and the model comprises a specially designed encoder module (MDCRM) and an attention module (SPA). The MDCRM adopts a multi-scale decomposable convolution residual module, can effectively extract features of lane lines in a complex scene at night, and realizes lightweight design by reducing model parameters and calculation amount. The SPA module comprehensively considers lane information in the horizontal direction and the vertical direction through a strip-shaped pooling technology, so that the recognition capability of the model on the sheltered area is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, lane line detection and autonomous driving, and in particular to a method and device for nighttime lane line detection based on decomposed convolution. Background Art

[0002] Advanced Driver Assistance System (ADAS) is a key technology in autonomous driving. Thanks to deep learning, it can help drivers in various driving scenarios. For ADAS, lane detection is an important guarantee for stable vehicle driving. In night scenes, due to extreme changes in lighting conditions, severe occlusions and harsh ground environments, lane detection tasks that rely on visual cues such as cameras face huge challenges.

[0003] Existing lane detection methods can be divided into traditional methods based on image processing and deep methods based on deep learning. During night driving, lane visibility is reduced due to factors such as poor lighting, obstacle occlusion, and ground contamination, making the night lane detection task extremely challenging. Traditional methods rely on manual annotation to extract lane features, and then achieve lane detection by performing operations such as night image enhancement, curve fitting models, and lane marking grouping. However, such methods have the problem of weak generalization ability and are usually only applicable to lane detection in specific scenarios, and are difficult to cope with complex and changeable night environments. With the rapid development of deep learning technology, methods based on deep learning have gradually replaced traditional methods. Deep learning methods have good feature representation and learning capabilities, and can learn more complex and abstract features from a large amount of data, thereby improving the generalization ability of lane detection.

[0004] In recent years, with the development of deep learning theory and algorithms, the accuracy and speed of night lane detection algorithms have been improved. However, most algorithms have large parameters, complex network structures, large model sizes, high computing power requirements for devices, and difficult to deploy models to embedded and mobile devices. Therefore, it is urgent to improve the existing lane detection network and propose a lane detection method for low-light environments based on deep learning to solve the above problems. Summary of the invention

[0005] The present invention proposes a nighttime lane line detection method and device based on decomposed convolution, which solves the problems in the prior art of large algorithm parameters, complex network structure, large model size, high requirements on the computing power of the equipment, and difficulty in deploying the model to embedded and mobile devices.

[0006] The technical solution of the present invention is as follows: A method and device for detecting lane lines at night based on decomposed convolution, comprising the following steps: S1, collecting various nighttime traffic road data, and marking each lane line in these image data to establish a benchmark data set (CULane-Night) for nighttime lane line detection;

[0007] S2, preprocess the (CULane-Night) data set collected in S1, and divide it into a training set and a test set in proportion;

[0008] S3, build a lane detection model (MDCR-Net) suitable for night scenes;

[0009] S4, using the training set in S2 to train the constructed nighttime lane line detection network based on deep learning to obtain the trained model weights;

[0010] S5, collecting image data containing lane lines on a real road, and inputting it into the nighttime lane line detection model constructed in S3, and using the model weights obtained in S4 to detect lane line instances on the real road;

[0011] S6, deploy the model trained in S4 to the embedded device and conduct field testing.

[0012] In S1, multiple different types of cars are used as collection sources to collect image data of various types of traffic roads. The image data covers various road sections and urban and rural roads at night under different weather conditions.

[0013] In S3, the MDCR-Net model includes an encoder module, a SPA attention module, and a decoder module;

[0014] Among them, the encoder module is used to extract lane line features of image data and generate feature maps;

[0015] The SPA attention module can highlight the lane lines in the image using the features extracted by the encoder module;

[0016] The decoder module is used to process the processing results of the encoder module and the SPA attention module to obtain the lane line data in the input image data.

[0017] In this process, the design of the encoder module is the key. The encoder module of this method consists of a multi-scale decomposed convolution residual module (MDCRM), a feature flipping module (FFM) and a dual-branch downsampling module. Compared with the traditional encoder module, this structure is more complex and more powerful;

[0018] Among them, the multi-scale decomposition convolution residual module (MDCRM) is used to extract multi-scale features in the image, and captures local and global lane information by decomposing convolution. The multi-scale decomposition convolution residual module (MDCRM) is the core of the encoder module, which enables the model to detect lane lines more accurately in night environments. Compared with the traditional residual convolution module, MDCRM can more effectively utilize the information of the convolution kernel, thereby improving the efficiency of feature extraction;

[0019] The feature flipping module (FFM) is used to further enhance the representation ability of the feature map and introduce the location information of the symmetric lanes. By introducing the flipping operation on the feature map, the model can have better adaptability to different image transformations, thereby improving the robustness and stability of detection.

[0020] The dual-branch downsampling module is used to reduce the size of the feature map in the encoder, thereby reducing the computational complexity of subsequent processing. This module can simultaneously retain the high-level semantic information and low-level detail information of the feature map, providing better input for subsequent lane line detection tasks; In general, the encoder module of this method combines a variety of advanced technologies to effectively extract the features of night lane images, thereby achieving accurate and efficient lane line detection.

[0021] The process of highlighting the lane lines in the image by the SPA attention module using the features extracted by the encoder module includes integrating the semantic features of different levels extracted by the encoder module, combining multi-scale information, including shallow and deep features, to obtain richer information representation. The comprehensive use of these features helps to enhance the expressiveness of lane line features and improve the accuracy of detection. The SPA attention module converts the comprehensive feature information into a confidence score for the existence of lane lines through pooling operations in both horizontal and vertical directions. The pooling operations in the horizontal and vertical directions help capture the distribution and direction of lane lines in the image, thereby achieving accurate prediction of occluded positions; in this way, even in the presence of occlusion, the model can accurately identify the position and direction of lane lines.

[0022] The process of the decoder module processing the processing results of the encoder module and the attention module and obtaining each lane line in the input image data includes:

[0023] A dual-branch upsampling module is used in the decoder module, one of which uses bilinear interpolation upsampling to restore coarse-grained features, and the other uses deconvolution upsampling to repair detail loss. The advantage of this structure is that it can more comprehensively consider the different levels of image features, thereby better retaining the detail information of the image. The branch that restores coarse-grained features is used to help capture the overall semantic information, while the branch that repairs detail loss focuses on retaining detail features. Through the collaborative work of the two branches, the dual-branch structure balances the needs of coarse and precise upsampling to a certain extent, maintaining the inference speed while improving the accuracy of image segmentation, achieving a better balance. The feature map extracted in the encoder module is upsampled and restored to the original image size, and then the output result is obtained after a softmax operation. Then the lane position confidence score extracted by the attention module is applied to the output result to obtain the final detection result.

[0024] In S4, the model training process includes the following steps:

[0025] S4-1, input the training set in S2 into the encoder module of the network to obtain the lane feature map, and then send the obtained lane feature map to the decoder module to obtain the output result, and calculate the segmentation loss. The calculation formula is: Lseg = LCE (Spred, Sgt), where Spred is the predicted value of the model, Sgt is the true label, and LCE is the binary cross entropy loss function;

[0026] S4-2, input the lane feature map obtained in S4-1 into the attention module, obtain the lane line position confidence weight matrix, map it back to the original image and calculate the SPA attention module loss. The calculation formula is:

[0027]

[0028]

[0029] In the formula

[0030] Where ppred is the lane line prediction probability map obtained by softmax operation of the segmentation result Spred finally output by the decoder, that is, ppred = φ(Spred); and are the lane line position weight maps output by the SPA attention module in the horizontal and vertical directions respectively; LMsE is the mean square error loss, and γ is the regularization coefficient in the SPA attention module loss, which has an important impact on the performance of the model;

[0031] S4-3, repeat S4-1 and S4-2, optimize and adjust various parameters, and finally obtain an optimal solution. The optimal solution can make the output result as close to the true label as possible. The optimal solution is used as the final weight of the model.

[0032] In S6, model deployment includes the following steps:

[0033] First, we use a trained PyTorch model and convert its weight file to the ONNX (OpenNeural Network Exchange) format. The main purpose of this step is to achieve cross-platform model exchange and interoperability. As an open standard, ONNX enables model conversion and sharing between different deep learning frameworks, thereby improving the flexibility and portability of the model.

[0034] Next, the model is further optimized using TensorRT (NVIDIA TensorRT), an engine and runtime library that optimizes deep learning inference for specific hardware platforms and application requirements. The optimization process includes layer fusion, precision adjustment, memory optimization, and parallel acceleration, aiming to increase inference speed and reduce power consumption.

[0035] Multi-layer fusion combines multiple neural network layers into a single computational operation, thereby reducing computational and memory overhead; precision adjustment further improves the inference speed by reducing the computational precision of the model (for example, using FP16 or INT8); in terms of memory optimization, TensorRT will minimize the model's memory footprint to adapt to embedded devices with limited resources; and parallel acceleration utilizes the parallel computing capabilities of the GPU to further improve the model's inference efficiency;

[0036] This series of optimization measures makes the model more suitable for embedded devices and actual application scenarios, further improving the performance and applicability of the model; by combining the cross-platform characteristics of ONNX and the high-performance optimization of TensorRT, deep learning models can be quickly and effectively deployed on various hardware platforms, thereby achieving a wider range of applications and better performance.

[0037] The beneficial effects of the present invention are:

[0038] 1. The MDCRM in the present invention adopts a multi-scale decomposable convolution residual module, which can effectively extract the features of lane lines in complex scenes at night, and at the same time achieve lightweight design by reducing model parameters and calculation amount;

[0039] 2. The SPA module in the present invention comprehensively considers lane information in the horizontal and vertical directions through strip pooling technology, thereby improving the model's ability to recognize occluded areas;

[0040] 3. The nighttime lane line detection method proposed in the present invention overcomes the problem of low lane line detection accuracy of traditional methods under nighttime light conditions, and provides an effective solution for improving nighttime lane line detection technology;

[0041] 4. The present invention effectively deploys the designed nighttime lane line detection model on embedded devices to promote the operation of the model at the lowest cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0043] Figure 1 A flowchart of a nighttime lane line detection method proposed in an embodiment of the present invention;

[0044] Figure 2 A grid structure diagram of the lane detection model proposed in an embodiment of the present invention;

[0045] Figure 3 (a) is a network structure diagram of a multi-scale decomposition convolutional residual module in an encoder proposed in an embodiment of the present invention;

[0046] Figure 3 (b) is a network structure diagram of a feature flipping module in an encoder provided by an embodiment of the present invention;

[0047] Figure 3 (c) is a network structure diagram of a dual-branch downsampling module in an encoder provided by an embodiment of the present invention;

[0048] Figure 4 This is a network structure diagram of the attention module proposed in an embodiment of the present invention;

[0049] Figure 5 This is a network structure diagram of a dual-branch upsampling module in a decoder proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 The present invention provides a technical solution: a method and device for nighttime lane line detection based on decomposed convolution, comprising the following steps:

[0052] S1, collect various nighttime traffic road data and mark each lane line in these image data to establish a benchmark dataset for nighttime lane line detection (CULane-Night);

[0053] S2, preprocess the (CULane-Night) data set collected in S1, and divide it into a training set and a test set in proportion;

[0054] S3, build a lane detection model (MDCR-Net) suitable for night scenes;

[0055] S4, using the training set in S2 to train the constructed nighttime lane line detection network based on deep learning to obtain the trained model weights;

[0056] S5, collecting image data containing lane lines on a real road, and inputting it into the nighttime lane line detection model constructed in S3, and using the model weights obtained in S4 to detect lane line instances on the real road;

[0057] S6, deploy the model trained in S4 to the embedded device and conduct field testing.

[0058] As a preferred implementation scheme of the present invention, in S1 of the present invention, multiple different types of cars are used as collection sources to collect image data of various types of traffic roads. The image data covers various road sections and urban and rural roads at night under different weather conditions. In a specific embodiment, the CULane-Night night data set has a total of 8,500 data, including 6,090 training sets and 2,410 test sets, with a resolution of 1,640*590.

[0059] As a preferred embodiment of the present invention, in S4, Figure 2 As shown in the figure, the training process of the lane detection model includes the following steps:

[0060] In the first step, the training set in S2 is input into the encoder module of the network to obtain the lane feature map. In this process, the encoder module consists of a multi-scale decomposition convolution residual module (MDCRM), a feature flipping module (FFM) and a dual-branch downsampling module. Compared with the traditional encoder module, this structure is more complex and more powerful.

[0061] First, refer to Figure 3(a) The multi-scale decomposed convolution residual module (MDCRM) is used to extract multi-scale features in the image. It captures local and global lane information by decomposing convolution. The multi-scale decomposed convolution residual module (MDCRM) is the core of the encoder module, which enables the model to detect lane lines more accurately in night environments. Compared with the traditional residual convolution module, MDCRM can more effectively utilize the information of the convolution kernel, thereby improving the efficiency of feature extraction;

[0062] Secondly, refer to Figure 3 (b) The feature flipping module (FFM) is used to further enhance the representation ability of the feature map and introduce the location information of the symmetric lanes. By introducing the flipping operation on the feature map, the model can have better adaptability to different image transformations, thereby improving the robustness and stability of the detection.

[0063] Finally, refer to Figure 3 (c), the dual-branch downsampling module is used to reduce the size of the feature map in the encoder, thereby reducing the computational complexity of subsequent processing. This module can simultaneously retain the high-level semantic information and low-level detail information of the feature map, providing better input for subsequent lane line detection tasks; In general, the encoder module of this method combines a variety of advanced technologies to effectively extract the features of night lane images, thereby achieving accurate and efficient lane line detection.

[0064] In the second step, the lane feature map obtained in the first step is input into the SPA attention module. The SPA attention module converts the comprehensive feature information into a confidence score for the existence of lane lines through horizontal and vertical pooling operations. The horizontal and vertical pooling operations help capture the distribution and direction of lane lines in the image, thereby achieving accurate prediction of occlusion positions; in this way, even in the presence of occlusion, the model can accurately identify the position and direction of lane lines;

[0065] Reference Figure 4 , obtain the lane line position confidence weight matrix, map it back to the original image and calculate the attention module loss. The calculation formula is: After mapping back to the original image, calculate the SPA attention module loss. The calculation formula is:

[0066]

[0067]

[0068] In the formula

[0069] Where ppred is the lane line prediction probability map obtained by softmax operation of the segmentation result Spred finally output by the decoder, that is, ppred = φ(Spred); and are the lane line position weight maps output by the SPA attention module in the horizontal and vertical directions respectively; LMsE is the mean square error loss, and γ is the regularization coefficient in the SPA attention module loss, which has an important impact on the performance of the model;

[0070] The third step is to input the lane feature map obtained in the first step into the decoder to obtain the output result. Figure 5 , calculate the segmentation loss, the formula is Lseg = LCE (Spred, Sgt), where Spred is the predicted value of the model, Sgt is the true label, and LCE is the binary cross entropy loss function;

[0071] The fourth step is to repeat the first and second steps, optimize and adjust various parameters, and finally obtain an optimal solution. The optimal solution can make the output result as close to the true label as possible. The optimal solution is used as the final weight of the model.

[0072] Table 1 is a performance comparison between the method of the present invention and the semantic segmentation method on the CULane-Night dataset. Compared with these models, the model of the present invention performs well on the CULane-Night dataset, surpassing the previous segmentation-based methods. Compared with the best result, the IoU and F1-measure indicators are improved by 10.03% and 7.96% respectively, which once again confirms that the model has more excellent performance in night scenes.

[0073] Table 1

[0074]

[0075]

[0076] Table 2 is a performance comparison of the method of the present invention and the lane line detection method on the CULane-Night dataset. Compared with these models, the model of the present invention still shows superior performance on the CULane-Night dataset. As shown in Table 3, the model provided by the present invention still performs best in the iou and f1-measure indicators, reaching 66.07% and 77.72% respectively, and has a small number of parameters (only 5.79M). It can be seen that the model provided by the present invention can still accurately detect lane lines in night scenes and has higher efficiency.

[0077] Table 2

[0078]

[0079]

[0080] Model deployment in S6 includes the following steps:

[0081] First, the applicant uses a trained PyTorch model and converts its weight file into ONNX (Open Neural Network Exchange) format. The main purpose of this step is to achieve cross-platform model exchange and interoperability; as an open standard, ONNX enables model conversion and sharing between different deep learning frameworks, thereby improving the flexibility and portability of the model;

[0082] Next, the applicant further optimized the model using TensorRT (NVIDIA TensorRT); TensorRT is an engine and runtime library that optimizes deep learning reasoning for specific hardware platforms and application requirements; the optimization process includes layer fusion, precision adjustment, memory optimization, and parallel acceleration, aiming to increase reasoning speed and reduce power consumption;

[0083] Multi-layer fusion combines multiple neural network layers into a single computational operation, thereby reducing computational and memory overhead; precision adjustment further improves the inference speed by reducing the computational precision of the model (for example, using FP16 or INT8); in terms of memory optimization, TensorRT will minimize the model's memory footprint to adapt to embedded devices with limited resources; and parallel acceleration utilizes the parallel computing capabilities of the GPU to further improve the model's inference efficiency;

[0084] This series of optimization measures makes the model more suitable for embedded devices and actual application scenarios, further improving the performance and applicability of the model; by combining the cross-platform characteristics of ONNX and the high-performance optimization of TensorRT, the deep learning model can be quickly and effectively deployed on various hardware platforms, thereby achieving wider applications and better performance. As shown in Table 3, the frame rate comparison between the method of the present invention and other methods on the embedded devices Jetson Nano B01 and Jetson NX is shown, which shows that the model of the present invention has obvious advantages over other models in terms of deployment;

[0085] Table 3

[0086]

[0087] In summary, compared with the existing open source lane line detection algorithm based on deep learning, the present invention has certain advantages.

[0088] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A nighttime lane detection method and device based on decomposed convolution, characterized in that: The method comprises the following steps: S1, collecting various nighttime traffic road data, and marking each lane line in the image data to establish a benchmark data set (CULane-Night) for nighttime lane line detection; S2, preprocess the (CULane-Night) data set collected in S1, and divide it into a training set and a test set in proportion; S3, build a lane detection model (MDCR-Net) suitable for night scenes; S4, using the training set in S2 to train the constructed nighttime lane line detection network based on deep learning to obtain the trained model weights; S5, collecting image data containing lane lines on a real road, and inputting it into the nighttime lane line detection model constructed in S3, and using the model weights obtained in S4 to detect lane line instances on the real road; S6, deploy the model trained in S4 to the embedded device and conduct field testing.

2. The method and device for nighttime lane line detection based on decomposition convolution according to claim 1, characterized in that: In S1, multiple different types of cars are used as collection sources to collect image data of various types of traffic roads. The image data covers various road sections and urban and rural roads at night under different weather conditions.

3. The method and device for nighttime lane line detection based on decomposition convolution according to claim 1, characterized in that: In S3, the MDCR-Net model includes an encoder module, a SPA attention module, and a decoder module; Among them, the encoder module is used to extract lane line features of image data and generate feature maps; The SPA attention module can highlight the lane lines in the image using the features extracted by the encoder module; The decoder module is used to process the processing results of the encoder module and the SPA attention module to obtain the lane line data in the input image data.

4. The method and device for nighttime lane line detection based on decomposition convolution according to claim 3, characterized in that: The encoder module consists of a multi-scale decomposed convolutional residual module (MDCRM), a feature flipping module (FFM), and a dual-branch downsampling module; Among them, the Multi-scale Decomposed Convolutional Residual Module (MDCRM) is used to extract multi-scale features in the image and capture local and global lane information by decomposing convolutions; The feature flipping module (FFM) is used to further enhance the representation capability of the feature map and introduce the location information of the symmetric lanes by introducing a flipping operation on the feature map; A dual-branch downsampling module is used to reduce the size of feature maps in the encoder.

5. The method and device for nighttime lane line detection based on decomposition convolution according to claim 3, characterized in that: The process of the SPA attention module highlighting the lane lines in the image using the features extracted by the encoder module includes integrating the semantic features of different levels extracted by the encoder module and combining multi-scale information. The SPA attention module converts the comprehensive feature information into a confidence score for the existence of lane lines through pooling operations in both horizontal and vertical directions.

6. The method and device for nighttime lane line detection based on decomposition convolution according to claim 3, characterized in that: The process of the decoder module processing the processing results of the encoder module and the attention module and obtaining each lane line in the input image data includes: A dual-branch upsampling module is used in the decoder module, one of which uses bilinear interpolation upsampling to restore coarse-grained features, and the other uses deconvolution upsampling to repair detail loss. The branch that restores coarse-grained features is used to help capture overall semantic information, while the branch that repairs detail loss focuses on retaining detail features. The feature map extracted in the encoder module is upsampled and restored to the original image size, and then the output result is obtained after a softmax operation. Then, the lane position confidence score extracted by the attention module is applied to the output result to obtain the final detection result.

7. The method and device for nighttime lane line detection based on decomposition convolution according to claim 3, characterized in that: In S4, the model training process includes the following steps: S4-1, input the training set in S2 into the encoder module of the network to obtain the lane feature map, and then send the obtained lane feature map to the decoder module to obtain the output result, and calculate the segmentation loss. The calculation formula is: Lseg = LCE (Spred, Sgt), where Spred is the predicted value of the model, Sgt is the true label, and LCE is the binary cross entropy loss function; S4-2, input the lane feature map obtained in S4-1 into the attention module, obtain the lane line position confidence weight matrix, map it back to the original image and calculate the SPA attention module loss. The calculation formula is: In the formula Where ppred is the lane line prediction probability map obtained by softmax operation of the segmentation result Spred finally output by the decoder, that is, ppred = φ(Spred); and are the lane line position weight maps output by the SPA attention module in the horizontal and vertical directions respectively; LMsE is the mean square error loss, and γ is the regularization coefficient in the SPA attention module loss; S4-3, repeat S4-1 and S4-2, optimize and adjust various parameters, and finally obtain an optimal solution. The optimal solution can make the output result as close to the true label as possible. The optimal solution is used as the final weight of the model.

8. The method and device for nighttime lane line detection based on decomposition convolution according to claim 1, characterized in that: In S6, model deployment includes the following steps: First, take the trained PyTorch model and convert its weight file to ONNX (Open NeuralNetwork Exchange) format; Next, TensorRT (NVIDIA TensorRT) is used to further optimize the model. The optimization process includes layer fusion, precision adjustment, memory optimization, and parallel acceleration.

Images