A lane line detection method based on feature adaptive reinforcement and context semantic fusion

Abstract

The application discloses a lane line detection method based on feature adaptive reinforcement and context semantic fusion. The method comprises the following steps: acquiring a lane line data set and performing data enhancement on the data set; inputting into a backbone network for feature extraction; inputting the feature map extracted through the backbone network into an Mamba-based aggregated feature pyramid module for context semantic fusion; using a nested U-shaped hollow convolution module NUACM to perform feature adaptive reinforcement on a shallow feature map; using a guided attention module GAM to perform feature adaptive reinforcement on a deep feature map; then inputting the reinforced features into a detection head for training, and obtaining a final lane line prediction map by continuously iterating the position of a prior box. The method combines the Mamba-based aggregated feature pyramid, the nested U-shaped hollow convolution module NUACM and the guided attention module GAM to perform context semantic fusion and adaptive reinforcement on the feature map extracted by the backbone network, thereby effectively improving the lane line detection precision.

Description

Technical Field

[0001] This invention relates to a lane line detection method based on feature adaptive enhancement and contextual semantic fusion, belonging to the field of computer vision technology. Background Technology

[0002] With the continuous expansion of traffic volume, obtaining high-precision lane line information to assist traffic management has become an urgent need in the field of intelligent transportation. In actual urban road conditions, the dense interplay of traffic participants and the ever-changing, complex environment poses a significant challenge to the safety and efficiency of traffic systems. Traditional detection algorithms often exhibit low accuracy when dealing with blurred or occluded lane lines. Thanks to the rapid development of deep learning, models can automatically learn deep features from massive amounts of data, effectively overcoming this bottleneck. Deep learning methods have significantly improved the processing performance of multi-scale and blurred lane lines, and also enhanced the accuracy of identifying dynamic targets and irregular regions, gradually replacing traditional algorithms and becoming the mainstream approach in this field.

[0003] Deep learning-based lane detection methods have made some progress in terms of accuracy and adaptability, but the following problems still exist. First, accurate lane prediction highly depends on global contextual coherence and cross-layer semantic alignment. Second, lane features contain multi-scale edge information, which requires deep fusion and accurate capture by the model. Finally, current deep networks often lack dynamic region guidance mechanisms based on global context in deep feature processing, resulting in low detection accuracy. Summary of the Invention

[0004] The purpose of this invention is to solve the above-mentioned problems in existing methods, and to disclose a lane line detection method based on feature adaptive enhancement and contextual semantic fusion.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: S1: Obtain the lane line dataset, perform data augmentation on the dataset, and then input the images into the backbone network to obtain a five-layer feature map, denoted as F. i , where i represents the feature level, 1≤i≤5; S2: Contextual semantic fusion is performed using a Mamba-based aggregated feature pyramid. This module first preprocesses the three feature maps F3, F4, and F5 with 3×3 convolutions. Then, it flattens them into a one-dimensional sequence along four directions: horizontal forward, horizontal backward, vertical forward, and vertical backward. The SSM discretization of the state space equation enables selective memorization of long-range key features and background filtering. Subsequently, a feature fusion strategy combining top-down and bottom-up approaches is adopted, which both transmits global semantic information to the shallow feature maps and injects high-resolution local information into the deep feature maps. Finally, the output is feature maps H3, H4, and H5 enhanced by the Mamba-based aggregated feature pyramid. S3: The nested U-shaped dilated convolution module NUACM is used to enhance the shallow feature map H3 after it has been enhanced by the Mamba-based aggregated feature pyramid. H3 expands the receptive field by using depth-separable dilated convolutions with dilation rates r of 1, 2, 4 and 8 respectively. The nested U-shaped structure is used to concatenate the features of r=1 and r=8 and the features of r=2 and r=4 in the channel dimension at the same time, so that the model can effectively learn local features while maintaining high resolution. After 1×1 convolution fusion, it is residually connected with the original features, and the final output is the feature map E3. S4: The Guided Attention Module (GAM) is used to enhance the deep feature maps H4 and H5 after Mamba-based aggregated feature pyramid enhancement. H4 and H5 respectively extract global vectors v through global pooling, and then predict the Gaussian attention center and diffusion range through linear mapping. A two-dimensional Gaussian attention mask is generated through Gaussian distribution and the mask matrix is ​​obtained. Finally, it is weighted and fused with the original features. S5: Input the three-layer feature map, enhanced by the nested U-shaped dilated convolutional module NUACM and the guided attention module GAM, into the detection head H. a , where a represents the feature level, 1≤a≤3; S6: Continuously update the prior box P through iteration. b Where b represents the number of optimizations, 1≤b≤3. Finally, the lane line prediction map is output through confidence filtering and NMS deduplication.

[0006] A further technical solution involves using the Mamba-based aggregated feature pyramid to perform contextual semantic fusion on multi-level feature maps. This module first preprocesses the F3, F4, and F5 feature maps using 3×3 convolutions, then flattens them into a one-dimensional sequence along four directions: horizontal forward, horizontal backward, vertical forward, and vertical backward. It then uses the SSM discretized state-space equation to achieve selective memorization and background filtering of long-range key features. Afterward, it restores the feature maps to a two-dimensional space and fuses them. Subsequently, a feature fusion strategy combining top-down and bottom-up approaches is adopted, which both transmits global semantic information to shallow feature maps and injects high-resolution local information into deep feature maps. To avoid irrelevant interference during feature extraction in shallow feature networks, this module designs a top-down semantic fusion path. This path enables deep feature maps to... The network can pass global features to shallow feature maps; after the deep feature F5 extracts the state space features, it becomes M5, which is then fused with feature map F4 through upsampling; after F4 extracts the state space features, it becomes M4, which is then fused with feature map F3 through upsampling, completing the top-down transmission of global features and effectively filtering background noise; traditional feature pyramids usually perform simple feature stitching after all levels are processed, which easily leads to the loss of edge information in global features; therefore, this module designs a bottom-up fusion strategy after the top-down path, injecting the high-resolution localization information of the shallow layer into the deep global features through downsampling, and finally outputting feature maps H3, H4, and H5 enhanced by the Mamba-based aggregated feature pyramid.

[0007] A further technical solution involves the nested U-shaped dilated convolution module NUACM operating on the shallow feature map H3, which has been enhanced by a Mamba-based aggregated feature pyramid. First, a 1×1 convolution module is used for channel dimension reduction and alignment. To expand the receptive field while maintaining computational lightweightness, this module employs a depthwise separable dilated convolution with an expansion rate of r. The feature map H3 sequentially passes through concatenated modules with expansion rates r of 1, 2, 4, and 8, extracting features by gradually expanding the receptive field. To avoid the problem of discontinuous sampling points caused by dilated convolution, this module designs a nested U-shaped structure, concatenating features with expansion rates r=1 and r=8 along the channel dimension, and concatenating features with expansion rates r=2 and r=4 along the channel dimension. Subsequently, a 1×1 convolution is used for channel dimension reduction and cross-channel information fusion. Finally, a global residual connection is performed with the original feature map, outputting the feature map E3. This allows the model to fully learn local information without reducing resolution.

[0008] A further technical solution involves the Guided Attention (GAM) module acting on the deep feature map H5 enhanced by the Mamba-based aggregated feature pyramid. First, global average pooling is used to compress the spatial dimension, extracting a vector v containing global features. Then, two layers of linear mapping, activation functions, and normalization operations are used to obtain the relative coordinates of the image. Based on the prediction parameters, the module dynamically calculates a two-dimensional Gaussian attention mask using a Gaussian distribution on the normalized spatial grid coordinates. This mask has the highest weight at the center point. Finally, the generated mask matrix is ​​multiplied element-wise with the original feature map.

[0009] The beneficial effects of this invention are as follows: This invention discloses a lane detection method based on feature adaptive enhancement and contextual semantic fusion. First, this invention designs a Mamba-based aggregated feature pyramid, integrating the ability to model long sequences with linear complexity and a bidirectional path semantic fusion structure. This significantly enhances the ability to connect long-distance contexts of lane lines and injects high-resolution geometric boundaries from the bottom layer into the deep network, allowing the model to learn global features more effectively. Second, this invention extracts multi-scale features of lane lines by constructing a nested U-shaped dilated convolution module (NUACM). Combined with depthwise separable convolution, this expands the receptive field while effectively preventing the fragmentation of thin lane line features, enabling more accurate capture of local edge details. Finally, the guided attention module (GAM) employs a two-dimensional Gaussian guided attention strategy, further improving the model's accuracy in extremely complex environments. The overall method, through multi-module collaboration, achieves accurate perception of local details and extraction of global semantic information in complex traffic scenarios, enabling high-precision lane detection and providing a reliable environmental perception foundation for autonomous driving path planning and safety decisions. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating the overall network process according to an embodiment of the present invention. Figure 2 This is a diagram illustrating the overall network architecture of an embodiment of the present invention. Figure 3 This is a diagram of the Mamba-based aggregation feature pyramid structure according to an embodiment of the present invention; Figure 4 This is a structural diagram of the nested U-shaped dilated convolutional module NUACM according to an embodiment of the present invention; Figure 5 This is a structural diagram of the Guided Attention Module (GAM) according to an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0012] This invention provides a lane line detection method based on feature adaptive enhancement and contextual semantic fusion, such as... Figure 1 As shown, it includes the following steps: S1: Obtain the lane line dataset, perform data augmentation on the dataset, and then input the images into the backbone network to obtain a five-layer feature map, denoted as F. i Where i represents the feature level, 1≤i≤5; for example Figure 2 As shown; S2: Construct an aggregated feature pyramid based on Mamba, the structure of which can be found in the diagram below. Figure 3 ; S2-1: The Mamba-based aggregated feature pyramid performs contextual semantic fusion. This module first preprocesses the F3, F4, and F5 feature maps using 3×3 convolutions, then flattens them into a one-dimensional sequence along four directions: horizontal forward, horizontal reverse, vertical forward, and vertical reverse. It then uses SSM to discretize the state-space equation to achieve selective memorization and background filtering of long-range key features. Finally, it restores the feature maps to a two-dimensional space and fuses them, where x... t Represents the direction sequence, h t Let A represent the hidden state, B and C be the state transition matrix, and D be the feedforward matrix. t The output features are represented by the following formula: .

[0013] S2-2: Subsequently, a feature fusion strategy combining top-down and bottom-up approaches is adopted, which both transmits global semantic information to shallow feature maps and injects high-resolution local information into deep feature maps. To avoid irrelevant interference during feature extraction in shallow feature networks, this module designs a top-down semantic fusion path, which enables deep networks to transmit global features to shallow feature maps. After deep feature F5 extracts state space features, it becomes M5, which is then fused with feature map F4 through upsampling. After F4 extracts state space features, it becomes M4, which is then fused with feature map F3 through upsampling, completing the top-down transmission of global features and effectively filtering background noise. i The enhanced features are represented by LayerNorm, the linear layer is represented by SSM, and the discretized state-space equation is represented by O. iThis represents the final output feature map. Interpolation represents linear interpolation, calculated using the following formula: .

[0014] S2-3: Traditional feature pyramids typically perform simple feature concatenation after processing all levels, which can easily lead to the loss of edge information in the global features. Therefore, this module designs a bottom-up fusion strategy after the top-down path, injecting the high-resolution localization information from the shallow layers into the deep global features through downsampling, resulting in the final feature maps H3, H4, and H5. MaxPool() represents max pooling, i.e., a downsampling operation, calculated as follows: .

[0015] S3: Construct a nested U-shaped dilated convolutional module NUACM, the structure of which is shown in the diagram. Figure 4 ; S3-1: For the shallow feature map H3 after Mamba-based aggregated feature pyramid enhancement, the channel dimension is first reduced and aligned using a 1×1 convolution module. S3-2: To expand the receptive field while maintaining computational lightweightness, this module employs a depthwise separable dilated convolution with an expansion rate of r. Features H3 are sequentially passed through concatenated modules with expansion rates r of 1, 2, 4, and 8, extracting features by progressively expanding the receptive field. To avoid the sampling point discontinuity problem caused by dilated convolution, this module designs a nested U-shaped structure, concatenating features with expansion rates r=1 and r=8 along the channel dimension, and concatenating features with expansion rates r=2 and r=4 along the channel dimension. Subsequently, 1×1 convolutions are used for channel dimensionality reduction and cross-channel information fusion. Finally, a global residual connection is performed with the original feature map, outputting feature map E3. This allows the model to fully learn local information without reducing resolution. Here, α represents the concatenated depthwise separable dilated convolution modules with expansion rates of 1, 2, 4, and 8, Concat represents channel concatenation, and Conv... 1×1 () represents a 1×1 convolution, D i The feature map after processing by the depthwise separable dilated convolution module is represented by the following formula:

[0016] S4: Construct the Guided Attention Module (GAM), its structure diagram is shown below. Figure 5 ; S4-1: For the deep feature maps H4 and H5 enhanced by Mamba's aggregated feature pyramid, the spatial dimension is first compressed using adaptive global average pooling to extract a vector v containing global features. Then, the relative coordinates of the image are obtained through two layers of linear mapping, activation function, and normalization operations; where H represents the height of the image, W represents the width of the image, (c x ,c y (s) represents the coordinates of the center point of the region of interest in the network's adaptive prediction. x ,s y The numbers () represent the diffusion range along the horizontal and vertical directions, ReLU represents the activation function, and BN represents the normalization function. The calculation formula is as follows:

[0017] S4-2: Based on the above prediction parameters, the module dynamically calculates a two-dimensional Gaussian attention mask and obtains a mask matrix on the normalized spatial grid coordinates using a Gaussian distribution. The mask has the highest weight at the center point. Finally, the generated mask matrix is ​​multiplied element-wise with the original feature map. Here, M(x,y) represents the mask matrix, x represents the horizontal coordinate, and y represents the vertical coordinate. The calculation formula is shown below:

[0018] S5: Input the three-layer feature map, enhanced by the nested U-shaped dilated convolutional module NUACM and the guided attention module GAM, into the detection head H. a , where a represents the feature level, 1≤a≤3; S6: Continuously update the prior box P through iteration. b Where b represents the number of optimizations, 1≤b≤3. Finally, the lane line prediction map is output through confidence filtering and NMS deduplication.

[0019] To verify the effectiveness of the above embodiments, the performance of this invention was compared with other advanced methods on two datasets, CULane and TuSimple. The experimental results on the CULane dataset used ten metrics: F1... 50 Normal, Crowd, Dazzle, Shadow, Noline, Arrow, Curve, Cross, Night. Of these 10 metrics, only Cross is considered a suitable candidate for F1. 50 For the Normal, Crowd, Dazzle, Shadow, Noline, Arrow, Curve, and Night metrics, higher values ​​indicate better performance. Three metrics were selected for the TuSimple dataset: F1, Accuracy, and FP. Except for FP, higher values ​​for F1 and Accuracy generally indicate better performance.

[0020] As shown in Table 1, the experimental results demonstrate that the proposed method outperforms existing methods in multiple metrics on the CULane dataset. The method proposed in this embodiment achieves the highest F1 score on the CULane dataset. 50 The performance index is 80.76%, which is 0.63% higher than the state-of-the-art CLRNet. It also improves the performance by 0.72%, 0.77%, 0.2%, and 0.85% in the four challenge categories of Crowd, Noline, Arrow, and Curve, respectively, demonstrating the effectiveness of the method in this embodiment.

[0021] Table 1. Comparison of test accuracy on the CULane dataset.

[0022] As shown in Table 2, the experimental results of this embodiment outperform existing methods in F1 score on the TuSimple dataset. The method of this embodiment surpasses the previous state-of-the-art level by 0.17% in F1 score, proving the effectiveness of the method of this embodiment.

[0023] Table 2 Comparison of detection accuracy on the TuSimple dataset

Claims

1. A lane line detection method based on feature adaptive reinforcement and context semantic fusion, characterized in that The steps include the following: S1: obtain a lane line dataset, perform a data augmentation operation on the dataset, and then input the picture into a backbone network to obtain a five-level feature map, denoted as F i where i represents the level of the feature, 1≤i≤5. S2: Contextual semantic fusion is performed using a Mamba-based aggregated feature pyramid. This module first preprocesses the three feature maps F3, F4, and F5 with 3×3 convolutions. Then, it flattens them into a one-dimensional sequence along four directions: horizontal forward, horizontal backward, vertical forward, and vertical backward. The SSM discretization of the state space equation enables selective memorization of long-range key features and background filtering. Subsequently, a feature fusion strategy combining top-down and bottom-up approaches is adopted, which both transmits global semantic information to the shallow feature maps and injects high-resolution local information into the deep feature maps. Finally, the output is feature maps H3, H4, and H5 enhanced by the Mamba-based aggregated feature pyramid. S3: The nested U-shaped dilated convolution module NUACM is used to enhance the shallow feature map H3 after it has been enhanced by the Mamba-based aggregated feature pyramid. H3 expands the receptive field by using depth-separable dilated convolutions with dilation rates r of 1, 2, 4 and 8 respectively. The nested U-shaped structure is used to concatenate the features of r=1 and r=8 and the features of r=2 and r=4 in the channel dimension at the same time, so that the model can effectively learn local features while maintaining high resolution. After 1×1 convolution fusion, it is residually connected with the original features, and the final output is the feature map E3. S4: The Guided Attention Module (GAM) is used to enhance the deep feature maps H4 and H5 after Mamba-based aggregated feature pyramid enhancement. H4 and H5 respectively extract global vectors v through global pooling, and then predict the Gaussian attention center and diffusion range through linear mapping. A two-dimensional Gaussian attention mask is generated through Gaussian distribution and the mask matrix is ​​obtained. Finally, it is weighted and fused with the original features. S5: input the three-layer feature map enhanced by the nested U-shaped cavity convolution module NUACM and the guided attention GAM module into the detection head H a wherein a represents the level of the feature, 1≤a≤3; S6: continuously update the prior box P by iteration b , where b represents the number of optimizations, 1≤b≤3, and finally the lane line prediction map is output through confidence screening and NMS deduplication.

2. The lane line detection method based on feature adaptive reinforcement and context semantic fusion according to claim 1, wherein, The Mamba-based aggregated feature pyramid performs contextual semantic fusion on multi-level feature maps. This module first preprocesses the F3, F4, and F5 feature maps using 3×3 convolutions, then flattens them into a one-dimensional sequence along four directions: horizontal forward, horizontal backward, vertical forward, and vertical backward. Selective memorization and background filtering of long-range key features are achieved through SSM discretization of the state space equation. The sequences are then restored to two-dimensional space and fused. A feature fusion strategy combining top-down and bottom-up approaches is then employed, transmitting global semantic information to shallow feature maps while injecting high-resolution local information into deep feature maps. To avoid irrelevant interference during feature extraction in shallow feature networks, this module designs a top-down semantic fusion path, enabling deep networks to incorporate all semantic information. The global features are passed to the shallow feature map; after the deep feature F5 extracts the state space features, it becomes M5, which is then fused with the feature map F4 through upsampling; after the F4 extracts the state space features, it becomes M4, which is then fused with the feature map F3 through upsampling, thus completing the top-down transfer of global features and effectively filtering background noise. Traditional feature pyramids usually perform simple feature stitching after all levels are processed, which can easily lead to the loss of edge information in global features. Therefore, this module designs a bottom-up fusion strategy after the top-down path, injecting the high-resolution positioning information of the shallow layer into the deep global features through downsampling, and finally outputting feature maps H3, H4, and H5 after being enhanced by the Mamba-based aggregated feature pyramid.

3. The lane line detection method based on feature adaptive reinforcement and context semantic fusion according to claim 1, characterized in that, The nested U-shaped dilated convolution module NUACM operates on the shallow feature map H3 after enhancement by the Mamba-based aggregated feature pyramid. First, it performs channel dimension reduction and alignment using a 1×1 convolution module. To expand the receptive field while maintaining computational lightweightness, this module employs a depthwise separable dilated convolution with an expansion rate of r. The feature map H3 sequentially passes through concatenated modules with expansion rates r of 1, 2, 4, and 8, extracting features by gradually expanding the receptive field. To avoid the discontinuous sampling point problem caused by dilated convolution, this module designs a nested U-shaped structure, concatenating features with expansion rates r=1 and r=8 along the channel dimension, and concatenating features with expansion rates r=2 and r=4 along the channel dimension. Subsequently, 1×1 convolution is used for channel dimensionality reduction and cross-channel information fusion. Finally, a global residual connection is performed with the original feature map, outputting feature map E3. This allows the model to fully learn local information without reducing resolution.

4. The lane line detection method based on feature adaptive reinforcement and context semantic fusion according to claim 1, characterized in that, The Guided Attention Module (GAM) operates on the deep feature map H5 enhanced by the Mamba-based aggregated feature pyramid. First, it compresses the spatial dimension through global average pooling to extract a vector v containing global features. Then, it obtains the relative coordinates of the image through two layers of linear mapping, activation function, and normalization operation. Based on the prediction parameters, the module dynamically calculates a two-dimensional Gaussian attention mask on the normalized spatial grid coordinates using a Gaussian distribution. The mask has the highest weight at the center point. Finally, the generated mask matrix is ​​multiplied element-wise with the original feature map.

Images