Lane line detection method based on time sequence curvature and multi-scale context

Abstract

The invention relates to the field of automatic driving, and particularly discloses a lane line detection method based on time sequence curvature and multi-scale context. The method comprises the following specific implementation steps of: giving a data set image and preprocessing the data set image; lane features of an image are extracted through a backbone network, a multi-scale feature map is generated, and the multi-scale feature map is continuously processed by a double-branch detection framework. According to the framework, two parallel branches of time sequence modeling and attention enhancement are fused, firstly, a feature map is sent into a multi-scale attention branch, multi-scale channels and spatial features are fused, and semantic information is enriched; and meanwhile, the time sequence optimization branch captures prior knowledge by using time sequence information, extracts context information and models curvature change. And finally, carrying out weighted fusion on the output features of the two branches and the original trunk features, processing and refining lane prediction through a loss function, and outputting a final result. According to the method, the problems that visual clues of lane lines are lacked in a complex scene and a long-distance dependency relationship is difficult to model in a curve scene are effectively solved, and the robustness and accuracy of the model are enhanced.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and deep learning, and in particular to a lane detection method based on temporal curvature and multi-scale context. Background Technology

[0002] In autonomous driving systems, lane lines are crucial for vehicle path planning. Lane line detection, as a core technology of intelligent transportation and advanced driver assistance systems (ADAS), can identify lane boundaries in real time, providing critical information for lane keeping assist (LKA) and lane departure warning (LDW) systems, and effectively reducing accident rates.

[0003] With the development of CNNs, deep learning-based methods have become the mainstream technology in lane detection. However, they still face many limitations in complex environments. For example, due to the inherent properties of lane lines being long, thin, and continuous, the number of background pixels in an image is significantly greater than the number of lane line pixels. When lane lines are occluded by other objects, visibility decreases, and background noise severely interferes with lane feature extraction, creating a detection challenge without visual cues. Furthermore, curves are critical nodes on roads, and their detection accuracy is fundamental to safety control in autonomous driving systems. Due to changes in lane curvature, long-distance curves in images are prone to geometric deformation, making it difficult for models to capture long-distance dependencies. Simultaneously, curve scenes are often accompanied by environmental interference such as shadows and occlusions, leading to the loss of local features during detection and affecting detection efficiency. Therefore, exploring new lane detection methods to enhance adaptability to dynamic environments remains an unavoidable challenge in lane detection. Summary of the Invention

[0004] The purpose of this invention is to address the challenges of lane detection in scenarios such as occlusion and curve detection by providing a lane detection method based on temporal curvature and multi-scale context. A dual-branch lane detection framework integrating temporal modeling and multi-scale attention enhancement is proposed to improve the model's robustness to occluded scenarios and curved lanes. A lightweight feature fusion mechanism is designed to integrate different attention mechanisms, focusing on key lane features across multiple spatial and channel dimensions, and incorporating multi-scale contextual information to improve lane detection performance. A separate branch is introduced to process image temporal information, modeling the temporal changes in lane curvature, and fully utilizing information from preceding and following images to enhance the robustness of lane detection on curves.

[0005] To achieve the aforementioned objectives, the following technical solution is adopted:

[0006] A lane detection method based on temporal curvature and multi-scale context includes the following steps:

[0007] Step 1: Image Preprocessing of the Dataset. Normalization and centering operations are performed to make the image data more suitable for subsequent model processing. Common data augmentation techniques, including horizontal flipping, random affine transformation, and color dithering, are used to improve the model's generalization ability. Images are adjusted to a fixed size suitable for the model to ensure consistent model input and stabilize model training.

[0008] Step 2: Generate multi-scale feature maps. The ResNet18 / 34 / 101 residual network was selected as the backbone network for the entire experiment. The image was processed through the backbone network to extract lane features at each level, generating multi-scale feature maps.

[0009] Step 3: Design and Implementation of the Dual-Branch Lane Detection Framework. The two branches represent the multi-scale attention branch and the temporal optimization branch, which process feature maps in parallel. The multi-scale attention branch aggregates channel features and spatial features, cascading channel and spatial attention to capture multi-scale lane information. Simultaneously, the temporal optimization branch captures lane context information through inter-image dependencies, models long-distance dependencies in curved scenes, and solves the occlusion blur problem.

[0010] Step 4: Feature Fusion. The output features of the two branches are weighted and fused with the features output by the original backbone network.

[0011] Step 5: Refine lane predictions and output the final lane lines. A composite loss function is used to jointly optimize lane classification, regression, direction estimation, and time alignment, further refining lane features to obtain the final lane predictions.

[0012] Step 6: Visualize the lane line detection results.

[0013] Based on the above-mentioned technical solution, a further implementation detail of a lane detection method based on temporal curvature and multi-scale context is as follows: In step one:

[0014] The input image size for the backbone network is fixed at 320×800 to ensure model compatibility and improve convergence and generalization ability. The input image resizing operation can be represented as:

[0015] (1)

[0016] Where Wtarget and Htarget are the target width and height, the Resize function uses an interpolation algorithm to perform scaling.

[0017] In step two:

[0018] ResNet extracts multi-level lane line features through residual blocks and skip connections, and then fuses them through a feature pyramid network. A basic residual block can be represented as:

[0019] (2)

[0020] Where x and y represent the input features and output features, respectively, and Wi represents the weights.

[0021] In step three:

[0022] The multi-scale attention branch introduces a multi-scale attention mechanism to process features at different scales based on lane information at different levels. Specifically, efficient channel attention (ECA) acquires channel features between different levels and suppresses background noise; the convolutional attention mechanism (CBAM) focuses on the features of the last output layer of the backbone network to acquire high-level features containing global context information, and then concatenates the channel features Fc and spatial features Fs to generate the fused feature F. The specific calculation expressions for channel attention are shown in (3) and (4), the specific implementation of spatial attention satisfies equations (5) and (6), and the fused feature F is shown in equation (7):

[0023] , (3)

[0024] (4)

[0025] Among them, feature map Channel attention map , It is a collection of channel descriptors. This represents the sigmoid activation function.

[0026] (5)

[0027] (6)

[0028] Among them, spatial attention mask It is generated by applying convolutional layers to the average pooling and max pooling feature maps connected along the channel dimension. This represents a convolution with a kernel size of 3×3. This indicates channel cascading.

[0029] (7)

[0030] In the formula, γ represents the hyperparameter.

[0031] The temporal optimization branch utilizes temporal information to obtain lane prior knowledge, extract lane context information, and model long-distance lane dependencies. Specifically, given a series of consecutive frames... Extract the feature vector corresponding to the predicted i of each lane at time t. The time series that forms the feature vector This sequence is then input into a bidirectional gated recurrent unit (Bi-GRU) network. The calculation formula is expressed as:

[0032] , ,

[0033] (8)

[0034] Wherein, hidden representation It contains rich temporal context information for the prediction of the i-th lane at the current frame t, encoding information from the past and future. Furthermore, to further enhance modeling capabilities, The process involves sequentially passing through self-attention blocks, residual connections, layer normalization, and a lightweight multilayer perceptron (MLP), as detailed in equation (9):

[0035] (9)

[0036] In step four:

[0037] The specific calculation expression for feature fusion is as follows:

[0038] (10)

[0039] In the formula, These represent the output features of the temporal optimization branch, the multi-scale attention branch, and the backbone network, respectively.

[0040] In step five:

[0041] The refined feature Fre is input into the classification and regression heads, and lane presence and geometry predictions are generated by calculating lane classification, regression, direction estimation, and temporal smoothness loss. The specific formulas for the composite loss function and temporal smoothness loss are expressed by (11) and (12):

[0042] (11)

[0043] (12)

[0044] The loss weights in the composite loss function are λ cls =1.0, λ reg =0.5, λ dir =0.3, λ temp =0.2. In equation (12), This represents the time-enhanced feature vector of the i-th lane in frame t.

[0045] This invention focuses on lane line detection in complex environments, and has the following beneficial effects:

[0046] In the multi-scale attention branch, an attention mechanism, cascaded channels, and spatial features are introduced, enabling the model to automatically and selectively focus on lane features, suppress background noise, and enhance the overall structural information of lane lines, thereby improving the model's performance and generalization ability. Specifically, addressing the real-time requirements of autonomous driving, the lightweight design of the attention mechanism allows lane line detection to reduce redundant computation while maintaining accuracy. In the temporal optimization branch, a bidirectional gated recurrent unit establishes inter-image dependencies, effectively utilizing image continuity and temporal information to acquire lane prior knowledge and extract lane context information. This solves the problem of difficulty in modeling long-distance dependencies in curved scenarios, effectively improving the model's robustness against interference. Furthermore, to enhance the consistency of lane predictions between adjacent frames, a temporal smoothing loss is introduced into the final prediction features. This penalizes differences in lane line parameters between adjacent frames, utilizes historical frame information to complete missing features in the current frame, guides the prediction results to align with historical trends, and forces the model to learn continuity. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of a lane detection method based on temporal curvature and multi-scale context in this invention;

[0048] Figure 2 This is the overall network architecture diagram in this invention, and the specific implementation details of the dual-branch architecture are included.

[0049] Figure 3 This is an ablation study diagram of the composite loss weight in this invention;

[0050] Figure 4 This is a visualization of the results of comparing the actual ground conditions with the original model in this invention; Detailed Implementation

[0051] To enhance understanding of the present invention, it will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that these embodiments are for illustrative purposes only and are not intended to limit the invention. The accompanying drawings show only the parts relevant to the present invention for clarity.

[0052] like Figure 1 As shown in Figures 2, 3, and 4, this embodiment proposes a lane detection method based on temporal curvature and multi-scale context. The specific implementation steps are as follows:

[0053] Step 1: Process the lane detection dataset images through normalization and centering operations. Then, preprocess the images using common data augmentation techniques, including horizontal flipping, random affine transformation, and color dithering, to improve the model's generalization ability. Finally, fix the input image size of the backbone network at 320×800 to ensure model compatibility and improve convergence and generalization. The input image resizing operation can be represented as:

[0054] (1)

[0055] Where Wtarget and Htarget are the target width and height, the Resize function uses an interpolation algorithm to perform scaling.

[0056] Step 2: ResNet18 / 34 / 101 residual networks were selected as the backbone networks for the entire experiment. Lane line features at each level were extracted from the images using ResNet residual blocks and skip connections, and then fused using a feature pyramid network to generate multi-scale feature maps. In the ResNet network, a basic residual block can be represented as:

[0057] (2)

[0058] Where x and y represent the input features and output features, respectively, and Wi represents the weights.

[0059] Step 3: The two branches process the feature maps after processing by the backbone network in parallel. The multi-scale attention branch cascades channel and spatial attention, extracting lane features at different scales based on lane information between different layers of the backbone network. Specifically, efficient channel attention (ECA) acquires channel features between different layers and suppresses background noise; the convolutional attention mechanism (CBAM) focuses on the features of the last output layer of the backbone network, acquires high-level features containing global context information, and then cascades channel features Fc and spatial features Fs to generate fused features F. In the entire branch processing process, the specific calculation expressions of channel attention are shown in (3) and (4), the specific implementation of spatial attention satisfies equations (5) and (6), and the fused feature F is shown in equation (7):

[0060] , (3)

[0061] (4)

[0062] Among them, feature map Channel attention map , It is a collection of channel descriptors. This represents the sigmoid activation function.

[0063] (5)

[0064] (6)

[0065] Among them, spatial attention mask It is generated by applying convolutional layers to the average pooling and max pooling feature maps connected along the channel dimension. This represents a convolution with a kernel size of 3×3. This indicates channel cascading.

[0066] (7)

[0067] In the formula, γ represents the hyperparameter.

[0068] Meanwhile, the temporal optimization branch utilizes temporal information to acquire lane prior knowledge, captures lane context information through inter-image dependencies, models long-distance dependencies in curved scenes, and solves the occlusion blur problem. Further detailed implementation involves, given a series of consecutive frames... Extract the feature vector corresponding to the lane prediction i at time t. The time series that forms the feature vector This sequence is then input into a bidirectional gated recurrent unit (Bi-GRU) network. For detailed extensions, see calculation expression (8):

[0069] , ,

[0070] (8)

[0071] Wherein, hidden representation It contains rich temporal context information for the prediction of the i-th lane at the current frame t, encoding information from the past and future. Furthermore, to further enhance modeling capabilities, The process involves sequentially passing through self-attention blocks, residual connections, layer normalization, and a lightweight multilayer perceptron (MLP), as detailed in equation (9):

[0072] (9)

[0073] Step 4: The output features of the two branches and the original backbone network output features are weighted and concatenated for fusion. The detailed calculation expression is as follows:

[0074] (10)

[0075] In the formula, These represent the output features of the temporal optimization branch, the multi-scale attention branch, and the backbone network, respectively.

[0076] Step 5: The refined fusion feature Fre is input into the classification and regression heads. A composite loss function is used to calculate lane classification, regression, direction estimation, and temporal smoothness loss to generate the final prediction of lane presence and geometry, thereby further refining the lane features.

[0077] (11)

[0078] (12)

[0079] through Figure 3 Analysis of ablation experiment results showed that the loss weights in the composite loss function were set to λ respectively. cls =1.0, λ reg =0.5, λ dir =0.3, λ temp =0.2. In equation (12), This represents the time-enhanced feature vector of the i-th lane in frame t.

[0080] Step 6: Visualize the final lane detection results. Figure 4 The invention demonstrates the differences between the visualization results of the present invention and the original model under four complex environments: curves, extreme lighting, occlusion, and nighttime, proving the effectiveness and robustness of the present invention in lane line detection under complex environments.

[0081] To verify the effectiveness of this method in lane detection under complex environments, experiments were conducted on the CULane public lane detection benchmark dataset, with 80 training iterations performed on CULane. Throughout the optimization process, AdamW was used as the optimizer, combined with cosine decaying learning rate scheduling, where the initial learning rate was set to 1e−3. All experiments were performed using PyTorch. The experimental results on the dataset are listed in Table 1; only the results with ResNet18 as the backbone network are listed here: Table 1 Experimental results on CULane Network Model F1@50 Normal Crowd Dazzle Shadow Noline Arrow Cross Night Curve Original model 78.90 93.50 77.80 71.60 78.80 52.10 90.20 1365 74.40 74.40 Improved Model 81.03 95.07 79.32 75.16 84.04 55.40 91.25 1013 76.41 76.94

[0082] As can be seen from the table, the improved model has significantly improved accuracy in both the nine types of scenarios and the overall detection, demonstrating the model's ability to handle the lack of visual cues in challenging scenarios and confirming the model's robustness in lane line detection in complex environments.

[0083] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the invention. The scope of protection of the present invention is defined by the claims, and any improvements or modifications made based on the claims fall within the scope of protection of the present invention.

Claims

1. A lane detection method based on temporal curvature and multi-scale context, characterized in that, The method includes the following steps: Step 1: Image preprocessing of the dataset; through normalization, centering and other operations, the image data is made more suitable for subsequent model processing. Common data augmentation techniques, including horizontal flipping, random affine transformation and color jitter, are used to improve the model's generalization ability. The images are adjusted to a fixed size that fits the model to ensure consistent model input and stable model training. Step 2: Generate multi-scale feature maps; Select ResNet18 / 34 / 101 residual network as the backbone network of the entire experiment. The image is processed by the backbone network to extract lane features at each level and generate multi-scale feature maps. Step 3: Design and implementation of the dual-branch lane detection framework; the two branches represent the multi-scale attention branch and the temporal optimization branch, which process feature maps in parallel. The multi-scale attention branch aggregates channel features and spatial features, cascades channel and spatial attention, and captures multi-scale lane information. At the same time, the temporal optimization branch captures lane context information through the inter-image dependency relationship, models long-distance dependency in the curve scene, and solves the occlusion blur problem. Step 4: Feature Fusion; The output features of the two branches and the output features of the original backbone network are weighted and fused. Step 5: Refine lane predictions and output the final lane lines; A composite loss function is used to jointly optimize lane classification, regression, direction estimation, and time alignment to further refine lane features and obtain the final lane predictions. Step 6: Visualize the lane line detection results.

2. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, In step one, the fixed size of the input image is set to 320×800 to ensure model compatibility and improve the model's convergence and generalization ability; the input image resizing operation can be represented as: (1) Among them, W target and H target The target width and height are used, and the Resize function uses an interpolation algorithm to perform scaling.

3. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, In step two, ResNet extracts multi-level lane line features through residual blocks and skip-connection structures, and fuses them through a feature pyramid network; a basic residual block can be represented as: (2) Where x and y represent the input and output features, respectively, and W i Indicates the weight.

4. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, In step three, the multi-scale attention branch introduces a multi-scale attention mechanism to process features at different scales based on lane information at different levels. Specifically, efficient channel attention (ECA) acquires channel features between different levels and suppresses background noise; convolutional attention mechanism (CBAM) focuses on the features of the last output layer of the backbone network to acquire high-level features containing global context information, and then cascades the channel features F. c and spatial features F s The fusion feature F is generated; the specific calculation expression of channel attention is expressed as (3)(4), the specific implementation of spatial attention satisfies equation (5)(6), and the fusion feature F is shown in equation (7): , , (3) (4) Among them, feature map Channel attention map , It is a collection of channel descriptors. This represents the sigmoid activation function; (5) (6) Among them, spatial attention mask It is generated by applying convolutional layers to the average pooling and max pooling feature maps connected along the channel dimension. This represents a convolution with a kernel size of 3×3. Indicates channel cascading; (7) In the formula, γ represents the hyperparameter.

5. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, In step three, the temporal optimization branch uses temporal information to obtain lane prior knowledge, extracts lane context information, and models long-distance lane dependencies. The specific implementation involves providing a series of consecutive frames. Extract the feature vector corresponding to the predicted i of each lane at time t. The time series that forms the feature vector This sequence is then input into a bidirectional gated cyclic unit (Bi-GRU) network; The calculation formula is expressed as follows: ， ， . (8) Wherein, hidden representation It contains rich temporal context information for the prediction of the i-th lane at the current frame t, and encodes information from the past and future. (9) In addition, to further enhance modeling capabilities, The process involves a self-attention block, residual connection, layer normalization, and a lightweight multilayer perceptron (MLP) in sequence, as shown in Equation (9).

6. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, The specific calculation expression for feature fusion in step four is as follows: (10) In the formula, These represent the output features of the temporal optimization branch, the multi-scale attention branch, and the backbone network, respectively.

7. The lane line detection method for complex environments based on deep learning according to claim 1, characterized in that, In step five, the refined feature F re The input classification and regression heads are used to calculate lane classification, regression, direction estimation, and time smoothness loss to produce the final prediction of lane presence and geometry; the composite loss function and the specific formula for time smoothness loss are expressed by (11)(12): (11) (12) The loss weights in the composite loss function are λ cls =1.0, λ reg =0.5, λ dir =0.3, λ temp =0.2; In equation (12), This represents the time-enhanced feature vector of the i-th lane in frame t.

Images