A lane line detection method and system fusing a multi-branch structure and an attention mechanism

Abstract

The application discloses a lane line detection method and system fusing a multi-branch structure and an attention mechanism, first adopts a multi-branch structure in an image coding part to reserve high-resolution feature information; simultaneously uses a CBAM module to fuse an attention mechanism to enhance feature extraction capability in a channel and a spatial dimension so as to reserve more detail information in a transmission process, is favorable to sufficient fusion of features and effectively improves lane line detection precision. In an image coding process, an activation function selects swish instead of ReLU to reduce the influence of network gradient disappearance caused by an increase in network layers; simultaneously adopts a skip connection structure to connect feature maps of the image coding process and the image decoding process in an image decoding part, thereby realizing sufficient utilization of multi-layer feature information, promoting cross-layer feature fusion and being favorable to acceleration of network convergence.

Description

Technical Field

[0001] This invention relates to a lane detection method that integrates a multi-branch structure and an attention mechanism, belonging to the field of target detection technology. Background Technology

[0002] With the continuous improvement of living standards and the development of science and technology, autonomous driving technology has attracted widespread attention from researchers. Accurate lane line identification is a crucial step in realizing autonomous driving. Lane line detection involves processing road images acquired by visual sensors to extract lane lines and road boundary information. This enables vehicles to automatically drive in designated lanes or provides drivers with crucial information for judgment, achieving driving safety and preventing accidents. Therefore, in-depth research into vehicle detection methods in intelligent connected transportation scenarios and the real-time, accurate execution of lane line detection tasks has significant engineering application value.

[0003] Lane detection using deep learning is a current research hotspot. The basic idea is to build a deep neural network to detect lane lines, and then train the network with a large amount of data to automatically extract lane line features and thus achieve lane line detection. For example, Ko Y, Lee Y, Azam S, et al. Key pointsestimation and point instance segmentation approach for lane detection[J].IEEE Transactions on Intelligent Transportation Systems, 2021. proposed the PINet algorithm based on key points, which uses fewer parameters and can detect lane lines in any direction and of any number; Su J, Chen C, Zhang K, et al. Structure Guided Lane Detection[J].arXiv preprint arXiv:2105.05403, 2021. proposed the SGNet algorithm, which makes full use of the structural information related to lane lines, and can accurately extract lane lines and classify and locate them; Hou Y, Ma Z, Liu C, et al. Learning lightweight lanedetection cnns by self attention distillation[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2019:1013-1021. proposed the SAD algorithm, which can enhance the feature representation ability of convolutional neural networks by using semantic segmentation and knowledge distillation.

[0004] In existing deep learning-based lane detection methods, relying solely on single-branch neural network structures is insufficient to achieve high-precision lane detection. Image data is prone to losing high-resolution feature information during continuous downsampling and learning, leading to insufficient feature fusion and ultimately inaccurate detection results. Furthermore, in deep learning, neural network learning converges slowly with increasing network depth, and may even exhibit gradient vanishing issues. Summary of the Invention

[0005] The technical problem to be solved by this invention is that in existing deep learning-based lane detection methods, it is difficult to achieve high-precision lane detection by relying solely on a single-branch neural network structure.

[0006] To address the aforementioned technical problems, this invention provides a lane detection method that integrates a multi-branch structure and an attention mechanism, comprising the following steps:

[0007] 1) Select the TuSimple dataset as the training set, and perform image preprocessing on the lane line data in the training set to obtain image b;

[0008] 2) Reduce the size of the processed dataset image b to 48×48 to obtain image c;

[0009] 3) Image c is encoded. During the image encoding process, convolution operation is used to extract lane line features from image c, and attention mechanism is used to enhance the feature extraction capability. The extracted features are used to obtain feature map d. A multi-branch structure is used to retain high-resolution feature information to obtain feature map e.

[0010] 4) The extracted feature map e is sent to the image decoding module for decoding;

[0011] 5) In the image decoding module, the lane line features in image c are restored by deconvolution operation. At the same time, a skip connection structure is used to connect the feature maps of the image encoding process and the image decoding process to achieve cross-layer feature fusion and obtain feature map f.

[0012] 6) Output the final lane line result image g by passing the feature map f through the softmax function.

[0013] A lane detection system integrating a multi-branch structure and an attention mechanism includes the following functional modules;

[0014] Preprocessing module: Select the TuSimple dataset as the training set, perform image preprocessing on the lane line data in the training set to obtain image b;

[0015] Image reduction module: Reduces the processed dataset image b to 48×48 to obtain image c;

[0016] Image encoding module: Image c is encoded. During the image encoding process, convolution operation is used to extract lane line features from image c, and an attention mechanism is used to enhance the feature extraction capability. The extracted features are used to obtain feature map d. A multi-branch structure is used to retain high-resolution feature information to obtain feature map e.

[0017] Image decoding module: The extracted feature map e is sent to the image decoding module for decoding;

[0018] Feature map module: In the image decoding module, lane line features in image c are restored through deconvolution operation. At the same time, a skip connection structure is used to connect the feature maps of the image encoding process and the image decoding process to achieve cross-layer feature fusion and obtain feature map f.

[0019] Lane line result module: Outputs the final lane line result map g by passing the feature map f through the softmax function.

[0020] A computer-readable storage medium is provided for storing the lane detection system and method that integrates the multi-branch structure and attention mechanism described above.

[0021] The beneficial effects achieved by this invention are as follows: To address the problem of insufficient feature fusion and low detection accuracy caused by the loss of high-resolution feature information during lane detection, this invention utilizes a multi-branch structure and an attention mechanism to enhance feature extraction capabilities. During the forward propagation of image data, the multi-branch network structure continuously performs downsampling operations to learn low-resolution feature information while retaining high-resolution feature information for feature fusion. The use of a multi-branch network structure in the lane detection method can compensate for the high-resolution feature loss problem inherent in single-branch structures. Simultaneously, by incorporating an attention mechanism through a CBAM module, attention is focused on lane line information while irrelevant information is ignored, further enhancing feature extraction capabilities in both channel and spatial dimensions. This preserves more detailed information during transmission, facilitating thorough feature fusion. Therefore, while meeting real-time requirements, it can effectively improve lane detection accuracy.

[0022] Secondly, addressing the issue of slow convergence and even gradient vanishing caused by increased network depth in neural networks, this invention employs the swish function and skip connection structures to accelerate network convergence. In image encoding, the invention uses the swish activation function instead of ReLU to reduce the impact of gradient vanishing caused by increased network layers. Simultaneously, in the image decoding section, skip connection structures are used to connect feature maps at symmetrical positions in each network path of the parallel network structure, achieving cross-layer feature fusion. This allows the network to incorporate contextual information during forward propagation and to perform lossless propagation during loss propagation, effectively accelerating network convergence. Therefore, this invention achieves efficient and accurate lane detection, further ensuring the safe, reliable, and stable operation of intelligent connected transportation systems. Attached Figure Description

[0023] Figure 1 This is a flowchart of the lane line detection process of the present invention;

[0024] Figure 2 The network architecture diagram for the lane line detection method is shown below.

[0025] Figure 3 This is a schematic diagram of the CBAM module structure;

[0026] Figure 4 A schematic diagram of the feature extraction network structure that integrates the attention mechanism;

[0027] Figure 5(a) is a schematic diagram of the basic structure of a single-branch network; Figure 5(b) is a schematic diagram of the basic structure of a multi-branch network.

[0028] Figure 6 A schematic diagram of the improved multi-branch structure;

[0029] Figure 7 This is a schematic diagram of a jump connection structure. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0031] The lane detection method proposed in this invention is trained and tested using the TuSimple dataset. First, a video stream of the road ahead is acquired using an onboard camera. Then, the video stream is converted into consecutive frame images. Next, the region of interest is dynamically extracted. Finally, the proposed lane detection method is used for processing, achieving efficient and accurate lane detection.

[0032] The specific steps of the lane detection method of the present invention, which integrates multi-branch structure and attention mechanism, are as follows: Figure 1 As shown, it includes the following steps:

[0033] 1) Select the TuSimple dataset as the training set, and perform image preprocessing on the lane line data in the training set. The specific image preprocessing process is as follows:

[0034] First, the input image 'a' is converted to grayscale. Then, the grayscale image is filtered to remove noise and converted into a threshold segmentation image.

[0035] Then perform Canny edge detection;

[0036] Finally, the region of interest is dynamically extracted and image enhancement processing is performed.

[0037] After preprocessing, image b is obtained;

[0038] 2) Reduce the size of the processed dataset image b to 48×48 to obtain image c;

[0039] 3) Image c is encoded. During the image encoding process, convolution operation is used to extract lane line features from image c, and attention mechanism is used to enhance the feature extraction capability. The extracted features are used to obtain feature map d. A multi-branch structure is used to retain high-resolution feature information to obtain feature map e.

[0040] 4) The extracted feature map e is sent to the image decoding module for decoding;

[0041] 5) In the image decoding module, the lane line features in image c are restored by deconvolution operation. At the same time, a skip connection structure is used to connect the feature maps of the image encoding process and the image decoding process to achieve cross-layer feature fusion and obtain feature map f.

[0042] 6) Output the final lane line result image g by passing the feature map f through the softmax function.

[0043] The network architecture of the lane line detection method proposed in this invention is as follows: Figure 2 As shown, to meet the real-time requirements of lane line detection, swish is first chosen as the activation function to reduce the impact of gradient vanishing caused by the increase in network layers. Secondly, this invention proposes a feature extraction network structure incorporating an attention mechanism, which enhances the ability to extract lane line features. Then, this invention improves the commonly used multi-branch structure by pruning redundant network structures, reducing computational resource consumption and effectively learning richer feature information. Finally, this invention further improves the network structure through a skip connection structure, connecting feature maps from the image encoding process with those from the image decoding process, allowing the network to fully integrate feature information during forward propagation. This achieves efficient and accurate lane line detection, further ensuring the safe, reliable, and stable operation of intelligent connected transportation systems.

[0044] In step 3), to address the problem of insufficient feature fusion and low detection accuracy caused by high-resolution feature loss, the specific steps are as follows:

[0045] Step 1: In the image encoding process, the swish function is used instead of ReLU to reduce the impact of gradient vanishing caused by the increase in the number of network layers;

[0046] In deep learning, neural networks use ReLU activation function and normalization layers to speed up network convergence. However, the traditional ReLU activation function loses some feature information due to its characteristic of taking 0 for negative values. Therefore, the lane detection method proposed in this invention uses swish instead of ReLU as the activation function during image encoding to reduce the impact of gradient vanishing caused by the increase in the number of network layers.

[0047] The calculation method of the swish function is shown in formula (1):

[0048] swish(x) = x·Sigmod(β·x)

[0049]

[0050] Where: x is the input of the activation function, β is a trainable parameter, and Sigmoid is the activation function in the neural network. Its advantage is that the output range is limited, so the data is less likely to diverge during transmission.

[0051] The swish function is unbounded at the upper limit but bounded at the lower limit, smooth, and non-monotonic. Compared with the ReLU activation function, swish can better improve the accuracy of the network and is also beneficial to network convergence.

[0052] Step 2: The CBAM module is integrated with the attention mechanism module to effectively enhance the feature extraction capabilities in both channel and spatial dimensions, thereby preserving more detailed information during transmission.

[0053] CBAM is a lightweight, general-purpose module that can be seamlessly integrated into CNN architectures. It combines channel and spatial attention mechanisms, leveraging the differences in the relationships between channels and space to learn feature maps of networks with different weights. This outperforms traditional networks like SENet (Squeeze-and-Excitation Networks), which only extract weights in the channel dimension. The CBAM module structure is as follows: Figure 3 As shown, the original input feature map F is processed by calculating attention weights based on its spatial and channel dimensions. These attention weights are then applied to the original feature map, resulting in a RefinedFeature. This not only reduces the number of parameters and saves computation, but the entire module can also be integrated into existing neural networks. If the input is an H×W×C feature map F, after spatial compression, global average pooling and max pooling are used to obtain two 1×1×C channel descriptions. These are then processed through a two-layer shared network structure, using the Sigmoid activation function to obtain the channel attention weights M. C The original features F and M C Multiplying them yields F′. The input and output of the spatial attention module are similar to those of the channel attention module. The channel attention-processed F′ is used as the input to the spatial attention module. First, it undergoes global max pooling and global average pooling, resulting in two H×W×1 feature maps. Then, dimensionality reduction through convolution and activation using the Sigmoid function yield the spatial attention weights M. S , connect F′ and M S Multiplying them together yields F", which is a weighted fusion feature map network in both channel and spatial dimensions.

[0054] This invention proposes a feature extraction network structure that integrates an attention mechanism, such as... Figure 4 As shown, the specific steps are as follows:

[0055] First, lane line features in image c are extracted using a shallow network to obtain feature map c_a, which is then sent to the channel branch and the spatial branch.

[0056] Secondly, global average pooling is performed along the channel dimension to prevent overfitting, and lane line feature information is extracted through a fully connected layer to obtain the channel attention map c_b; at the same time, multi-layer convolutional operations are used to encode the spatial branch to obtain the spatial attention feature map c_c.

[0057] Then, the channel attention map c_b and the spatial attention map c_c are added together and fed into a 3×3 convolutional layer to obtain the feature map c_d;

[0058] In addition, the feature map c_d is processed by the Sigmod function and a feature weighting operation is performed to obtain the feature map c_e;

[0059] Finally, feature map c_a is multiplied with feature map c_e to obtain feature map c_f, and feature map c_f is input into the max pooling layer for information filtering to obtain feature map d.

[0060] The lane line detection method proposed in this invention uses the CBAM module to enhance the feature extraction capability in the channel and spatial dimensions. In addition, by filtering the transmitted features and transmitting more detailed information, it is more conducive to the full fusion of features, the reduction of training loss, and accurate localization and classification, thus effectively improving the detection accuracy of lane lines.

[0061] Step 3: Use a multi-branch structure in the image coding part to preserve high-resolution feature information;

[0062] Traditional lane detection methods often focus only on the relationships between contextual features, rarely considering the relationships between high- and low-resolution features during forward propagation. In many applications, a single-branch neural network is insufficient to provide high-precision lane detection. Figure 5(a) illustrates a common single-branch network structure. In this structure, image data is prone to losing high-resolution feature information during continuous downsampling, leading to insufficient feature fusion and inadequate detection accuracy. Figure 5(b) illustrates a common multi-branch network structure. In this structure, deeper branches input low-resolution images to effectively extract global contextual features with minimal computational overhead; shallower branches input high-resolution images to extract spatial detail information. The computational results from multiple branches are fused to form the final semantic segmentation result.

[0063] However, the commonly used multi-branch network basic structures also have certain drawbacks. Some model parameters are invalid and do not participate in subsequent calculations, resulting in redundancy and wasting computational resources. The multi-branch network basic architecture proposed in this invention is as follows: Figure 6 As shown, during the forward propagation of image data, low-resolution feature information is continuously learned through downsampling operations, while high-resolution feature information is also preserved. Furthermore, Figure 6 By pruning redundant network structures, we can ensure that richer features are learned effectively with minimal computational resource consumption.

[0064] This invention employs the following in the image encoding section: Figure 6 The structure and specific steps are as follows:

[0065] First, the feature map d obtained by the feature extraction network incorporating an attention mechanism is used as the input to the multi-branch structure;

[0066] Secondly, feature map d is forward-propagated to obtain feature map d_a; at the same time, during the propagation of feature map d, downsampling operation is used to learn the feature information of low-resolution feature map d_b.

[0067] Finally, the feature information of feature map d_a and feature map d_b are fused to obtain feature map e.

[0068] To further enhance the feature extraction capability of the network and ensure that feature information can be accurately transmitted without losing its details, multiple convolution operations are performed in the image encoding part to fully extract the feature information of the image. This allows all input features to be extracted and transmitted by a multi-branch structure, improving the reusability of features.

[0069] In step 5), to address the problem that increased network depth leads to slow network learning convergence or even gradient vanishing, a skip connection structure is used for network learning, which is beneficial for lane line feature fusion and accelerates network convergence.

[0070] In deep learning-based lane detection methods, as the network depth increases, convergence slows down due to the increasing number of network layers. Therefore, this invention leverages the advantages of residual network skip connection operations, such as... Figure 7 As shown, the skip connection structure is used to connect the feature maps at symmetrical positions in each network path in the parallel network structure to achieve cross-layer feature fusion. This allows the network to fuse contextual information when propagating forward and to perform lossless propagation when propagating in the loss direction, thus accelerating network convergence.

[0071] The lane detection method proposed in this invention uses a skip connection structure for network learning in the image decoding part, and further combines this with deconvolution operations. The specific steps are as follows: First, the feature map e obtained from the image encoding process is used as the input to the image decoding part; second, deconvolution is used to reconstruct the features of image e, obtaining feature map e_a; third, a normalization layer is used to normalize feature map e_a, obtaining feature map e_b; finally, a skip connection structure is used to connect feature map d from the image encoding process with feature map e_b from the image decoding process, obtaining feature map f.

[0072] In deep neural networks, inconsistent data distribution across layers makes convergence and training extremely difficult. Transforming each layer's data to have a mean of 0 and a variance of 1 ensures a uniform data distribution, facilitating convergence. Normalization accelerates convergence. Furthermore, large activation function outputs result in small gradients, slowing the learning rate. Normalization prevents vanishing and exploding gradients by minimizing the output size. It also prevents overfitting by ensuring the network doesn't learn in the same direction. Finally, skip connections connect feature maps from the image encoding and decoding processes, allowing for network learning during decoding and improving performance.

[0073] A lane detection system integrating a multi-branch structure and an attention mechanism includes the following functional modules;

[0074] Preprocessing module: Select the TuSimple dataset as the training set, perform image preprocessing on the lane line data in the training set to obtain image b;

[0075] Image reduction module: Reduces the processed dataset image b to 48×48 to obtain image c;

[0076] Image encoding module: Image c is encoded. During the image encoding process, convolution operation is used to extract lane line features from image c, and an attention mechanism is used to enhance the feature extraction capability. The extracted features are used to obtain feature map d. A multi-branch structure is used to retain high-resolution feature information to obtain feature map e.

[0077] Image decoding module: The extracted feature map e is sent to the image decoding module for decoding;

[0078] Feature map module: In the image decoding module, lane line features in image c are restored through deconvolution operation. At the same time, a skip connection structure is used to connect the feature maps of the image encoding process and the image decoding process to achieve cross-layer feature fusion and obtain feature map f.

[0079] Lane line result module: Outputs the final lane line result map g by passing the feature map f through the softmax function.

[0080] A computer-readable storage medium is provided for storing the lane detection system and method that integrates the multi-branch structure and attention mechanism described above.

[0081] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A lane line detection method fusing a multi-branch structure and an attention mechanism, characterized in that, Comprise the following steps: 1) Select TuSimple dataset as training set, image preprocessing is carried out on lane line data in training set, and image b is obtained; 2) The processed dataset image b is reduced to 48x48, and image c is obtained; 3) Image c is encoded, lane line features of image c are extracted by using convolution operation in image encoding process, and the feature extraction capability is enhanced by using attention mechanism, feature map d is extracted, and feature map e is obtained by using multi-branch structure to reserve high resolution feature information; In step 3), CBAM module is adopted to integrate attention mechanism module, and the specific steps are as follows: Lane line features in image c are extracted by shallow network, feature map c_a is obtained, and is sent to channel branch and spatial branch; Global average pooling is carried out along the channel dimension to prevent overfitting, and lane line feature information is extracted through full connection layer, and channel attention map c_b is obtained; At the same time, the spatial branch is encoded by using multi-layer convolution operation, and the spatial attention feature map c_c is obtained; The channel attention map c_b and the spatial attention map c_c are added and sent to the 3x3 convolution layer to obtain the feature map c_d; The feature map c_d is processed by Sigmod function and feature weighting operation to obtain the feature map c_e; The feature map c_a and the feature map c_e are multiplied to obtain the feature map c_f, and the feature map c_f is input into the maximum pooling layer for information screening to obtain the feature map d; In step 3), in the image encoding process, the multi-branch structure is adopted to reserve high resolution feature information, and the specific steps are as follows: The feature map d obtained by the feature extraction network integrated with attention mechanism is used as the input of the multi-branch structure; The feature map d is forward transmitted to obtain the feature map d_a; At the same time, in the process of transmitting the feature map d, the low resolution feature map d_b is learned by using down sampling operation; The feature information of the feature map d_a and the feature map d_b is fused to obtain the feature map e; 4) The extracted feature map e is sent to the image decoding module for decoding; 5) In the image decoding module, the lane line features in image c are restored by inverse convolution operation, and the feature maps of image encoding process and image decoding process are connected by using jump connection structure to realize cross layer feature fusion, and the feature map f is obtained; 6) The feature map f is output by softmax function to obtain the final lane line result graph g.

2. The lane line detection method of claim 1, wherein: In step 1), the specific process of image preprocessing is as follows: Firstly, the input image a is grayed, and then the gray image is filtered and denoised and converted into a threshold segmentation image; Then, Canny edge detection is carried out; Finally, the region of interest is dynamically extracted and image enhancement processing is carried out; After preprocessing, image b is obtained.

3. The lane line detection method of claim 1, wherein: In step 3), in the image encoding process, the activation function uses swish function, The calculation method of swish function is shown in formula (1): ； (1) ； where: is an input to an activation function, is a trainable parameter, is an activation function in a neural network.

4. The lane line detection method of claim 1, wherein: In step 5), the specific steps are as follows: The feature map e obtained by image encoding process is used as the input of image decoding part; Inverse convolution operation is used to restore the features of image e to obtain feature map e_a; The feature map e_a is normalized by a normalization layer to obtain a feature map e_b; The feature map d of the image encoding process and the feature map e_b of the image decoding process are connected using a skip connection structure to obtain a feature map f.

5. A lane line detection system fusing multi-branch structure and attention mechanism, characterized in that: It comprises the following functional modules: A preprocessing module: the TuSimple dataset is selected as the training set, and the lane line data in the training set is image preprocessed to obtain an image b; An image reduction module: the processed image b is reduced to 48x48 to obtain an image c; An image encoding module: the image c is image encoded, lane line features are extracted from the image c using convolution operation during the image encoding process, and the feature extraction capability is enhanced using an attention mechanism, and features are extracted to obtain a feature map d, and a multi-branch structure is used to retain high-resolution feature information to obtain a feature map e; An image decoding module: the extracted feature map e is sent to the image decoding module for decoding; A feature map module: in the image decoding module, the lane line features in the image c are restored through inverse convolution operation, and the feature maps of the image encoding process and the image decoding process are connected using a skip connection structure to realize cross-layer feature fusion, and a feature map f is obtained; A lane line result module: the feature map f is output through a softmax function to obtain a final lane line result graph g; In the image encoding module, a CBAM module is used to integrate an attention mechanism module, and the specific steps are as follows: Lane line features in the image c are extracted through a shallow network to obtain a feature map c_a, which is sent to a channel branch and a spatial branch; Global average pooling is performed along the channel dimension to prevent overfitting, and a full connection layer is used to extract lane line feature information to obtain a channel attention map c_b; at the same time, a multi-layer convolution operation is used in the spatial branch to obtain a spatial attention feature map c_c; The channel attention map c_b and the spatial attention map c_c are added and sent to a 3x3 convolution layer to obtain a feature map c_d; The feature map c_d is processed by a Sigmod function and a feature weighting operation to obtain a feature map c_e; The feature map c_a and the feature map c_e are multiplied to obtain a feature map c_f, and the feature map c_f is input into a max pooling layer for information screening to obtain a feature map d; In the image encoding module, a multi-branch structure is used to retain high-resolution feature information, and the specific steps are as follows: The feature map d obtained by the feature extraction network integrated with the attention mechanism is used as the input of the multi-branch structure; The feature map d is forward transmitted to obtain a feature map d_a; at the same time, during the transmission of the feature map d, a down-sampling operation is used to learn the feature information of a low-resolution feature map d_b; The feature information of the feature map d_a and the feature map d_b is fused to obtain a feature map e.

6. A computer-readable storage medium for storing the lane line detection method with the fusion of the multi-branch structure and the attention mechanism according to any one of claims 1-4.

Images