Lane line detection method based on timing aggregation

Abstract

The application discloses a lane line detection method based on time sequence aggregation and relates to the technical field of automatic driving image detection. The method comprises the following steps: constructing a time sequence lane line detection network, wherein the time sequence lane line detection network comprises a multi-frame input construction module, a shared backbone feature extraction network, a time sequence aggregation module, a feature pyramid fusion network and a lane prior refinement detection head; training and optimizing the time sequence lane line detection network; inputting a current frame and two historical images before the current frame into the optimized time sequence lane line detection network, detecting the current frame and the two historical images by using the time sequence lane line detection network, and outputting a lane line detection result. The method can effectively improve the performance and stability of lane line detection in a complex scene while keeping a low additional calculation overhead.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving image detection technology, and in particular to a lane line detection method based on temporal aggregation. Background Technology

[0002] In recent years, single-frame lane detection methods have made significant progress in terms of detection accuracy, structural representation, and real-time performance. Existing research mainly focuses on adaptability to complex scenes, global semantic modeling, and efficient detection. Zou Liang et al. addressed the problems of blurred, broken, and occluded lane lines in harsh environments such as low light, backlight, heavy fog, rain, and snow by proposing a lane detection method based on a masked twin network and power-law adjustment loss, which enhances feature discrimination ability and environmental adaptability without increasing inference overhead. Yue Yongheng et al. improved lane feature extraction and cross-scale feature fusion capabilities in complex scenes by introducing a multi-scale spatial attention mechanism and a path aggregation network. Su et al. proposed GSENet, which improves lane detection from the perspective of global semantic enhancement to improve detection performance under occlusion, curves, and abnormal lighting conditions.

[0003] To ensure both detection speed and structural representation capabilities, researchers have proposed methods based on anchor points, priors, and structured representations. Qin et al. proposed a hybrid anchor-driven ordinal classification framework, transforming lane line detection into a coordinate classification problem on anchor points, thus improving detection speed while enhancing localization capabilities in complex scenes. Lucas et al. proposed LaneATT, combining the anchor point mechanism with global attention, achieving a good balance between accuracy and real-time performance. Liu et al. proposed CondLaneNet, employing a top-down approach to first detect lane instances and then predict lane shapes, thereby enhancing the ability to model complex topological structures.

[0004] Zheng et al. proposed CLRNet, which utilizes a cross-layer refinement mechanism to fuse high-level semantic information and low-level detailed features, and further improves lane localization accuracy by combining global context aggregation. Subsequently, Honda proposed CLRerNet, which improved the ranking quality of detection results from the perspective of confidence learning. Overall, single-frame lane detection methods have made significant progress in feature representation, structural modeling, and detection efficiency, but most of these methods still rely on the current frame image itself. When there are occlusions, low light at night, strong light reflections, or missing lane lines in the scene, relying solely on single-frame information still has certain limitations.

[0005] Compared with single-frame methods, adjacent frames in a video have natural temporal continuity. When the lane lines in the current frame are affected by occlusion, strong light, shadow or local missingness, the historical frames often retain relatively complete structural information. Therefore, using temporal context to assist the detection of the current frame has gradually become an important research direction in lane line detection. Zhang et al. systematically focused on this problem earlier, proposed the VIL-100 dataset, and gave the video instance lane line detection baseline MMA-Net, which provided a foundation for subsequent video lane line detection research. Wang et al.

[10] proposed TGC-Net, which introduced temporal consistency and geometric consistency into video instance lane line detection at the same time to enhance the continuity and stability of cross-frame results. Jin et al. proposed RVLD, which enhances the detection results of the next frame by recursively passing the current frame state; subsequently, they proposed OMR, which introduced the occlusion-aware memory optimization module to combine the current frame features, the previous moment output and historical memory to improve the detection performance in occluded scenarios. Zhou et al. proposed LaneTCA, which further modeled short-term and long-term information from the perspective of temporal context aggregation to enhance the lane prediction ability of the current frame.

[0006] In addition, Wang et al. proposed HomoFusion, which utilizes temporal complementary information between adjacent frames to improve the recognition effect of road markings under occlusion, shadow and glare conditions.

[0007] Overall, existing temporal methods have demonstrated that historical frame information significantly improves the stability and robustness of lane detection in complex scenes. However, many methods lean towards specialized video detection frameworks, resulting in relatively complex structures. In contrast, incorporating historical frame information at a lower cost to temporally enhance current frame features while retaining the core structure of a high-performance single-frame detection framework remains a valuable research approach. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to provide a lane line detection method that can effectively improve the performance and stability of lane line detection in complex scenarios while maintaining low additional computational overhead.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a lane detection method based on temporal aggregation, the method comprising the following steps:

[0010] A temporal lane detection network is constructed, which includes a multi-frame input construction module, a shared backbone feature extraction network, a temporal aggregation module, a feature pyramid fusion network, and a lane prior refinement detection head.

[0011] Training and optimization of the temporal lane detection network;

[0012] The current frame and its two previous historical frames are input into the optimized temporal lane detection network, which then performs lane detection and outputs the lane detection results.

[0013] A further technical solution is that the multi-frame input construction module adopts a multi-frame input organization method:

[0014] The current frame and its two previous historical frames are combined to construct a time-series sample:

[0015] ;

[0016] in To monitor the center frame, and For historical auxiliary frames;

[0017] During the sample construction process, historical frames are retrieved from the same video sequence as the current frame in chronological order. When the sequence is at the beginning and there are fewer than two historical frames, the nearest valid frame is used to copy and complete the sequence.

[0018] A further technical solution involves employing a multi-frame enhancement strategy in the shared backbone feature extraction network, including the following steps:

[0019] Three frames within the same time-series sample share the same set of random augmentation parameters. Let the random augmentation transformation be denoted as... ,in The geometric transformation parameters obtained from sampling represent the enhancement form of three frames in a time-series sample. for:

[0020] .

[0021] A further technical solution is that the time-series aggregation module includes a current frame projection branch, a historical frame projection branch, a gated weight generation branch, and an output mapping branch. The output of the current frame projection branch and the output of the historical frame projection branch are connected to the input of the output mapping branch via the gated weight generation branch.

[0022] A further technical solution is that the processing method of the time-series aggregation module includes the following steps:

[0023] Let the highest-level feature of the current frame be The highest-level feature of historical frames is ,in ;

[0024] First, the features of the current frame and the features of the historical frames are each subjected to a 1x1 convolutional projection to obtain the representation in the unified latent space:

[0025] ;

[0026] ;

[0027] in, Indicates the projection branch of the current frame. This represents the historical frame projection branch, whose function is to compress and reassemble channel information so that cross-frame comparisons can be performed within a unified feature space.

[0028] Next, to describe the correlation and differences between the current frame and historical frames, the projection features of the current frame, the projection features of historical frames, and the absolute differences between the two are concatenated into a joint representation:

[0029] ;

[0030] in This indicates splicing along the channel dimension;

[0031] The gated branch then generates a position-by-position, channel-by-channel weighted graph based on the joint features:

[0032] ;

[0033] in Indicates a gating network. The activation function is sigmoid; the gated branch consists of two consecutive 1x1 convolutional layers with a ReLU nonlinear mapping inserted in between; the output... The value range is [0,1], which is used to control the degree of retention of historical features in different spatial locations and channels;

[0034] After obtaining the gate weights, the module weights the historical frame projection features and restores them to the original feature channel space through the output mapping branch, forming historical residual information:

[0035] ;

[0036] in, Indicates the output mapping branch, This represents element-wise multiplication;

[0037] For two frames of historical input, the module calculates respectively. and The final enhanced highest-level feature representation of the current frame is as follows:

[0038] ;

[0039] The features of the current frame are always fully preserved as the main branch, while historical frames only provide incremental information in the gating-allowed positions and channels.

[0040] The beneficial effects of the above technical solution are as follows: the method enhances the representation of the current frame by utilizing the high-level semantic features of the current frame and the two frames preceding it, thereby improving the detection robustness in complex scenes. Simultaneously, to ensure the spatial correspondence of multi-frame inputs during the training phase, a temporal consistency data processing strategy is further designed, enabling the current frame and historical frames to share the same set of deterministic enhancement parameters. Furthermore, support for historical frame inputs is extended during the tensor and normalization stages, thus constructing a complete multi-frame training and inference process. The proposed method can effectively improve the performance and stability of lane detection in complex scenes while maintaining low additional computational overhead. Attached Figure Description

[0041] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0042] Figure 1 This is a flowchart of the method described in the embodiments of the present invention;

[0043] Figure 2 This is a block diagram illustrating the principle of the temporal lane detection network in the method described in this embodiment of the invention.

[0044] Figure 3 This is a block diagram illustrating the principle of the temporal aggregation module in the temporal lane detection network described in this embodiment of the invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0046] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0047] like Figure 1 As shown in the figure, this invention discloses a lane detection method based on temporal aggregation, the method comprising the following steps:

[0048] A temporal lane detection network is constructed, comprising a multi-frame input construction module, a shared backbone feature extraction network, a temporal aggregation module, a feature pyramid fusion network, and a lane prior refinement detection head, as shown in the specific structure below. Figure 2 As shown;

[0049] Training and optimization of the temporal lane detection network;

[0050] The current frame and its two previous historical frames are input into the optimized temporal lane detection network, which then performs lane detection and outputs the lane detection results.

[0051] The above steps will be explained in detail below with reference to specific content:

[0052] The multi-frame input building block adopts a multi-frame input organization method:

[0053] Regarding the input format, since future frames cannot be obtained in advance in real-world scenarios, this application adopts a temporal modeling method that utilizes only historical information, that is, constructing a temporal sample from the current frame and its two preceding historical frames:

[0054] (1)

[0055] in To monitor the center frame, and For historical auxiliary frames.

[0056] During sample construction, historical frames are retrieved chronologically from the same video sequence as the current frame. When the sequence is at the beginning and there are fewer than two historical frames, the most recent valid frame is used to pad the sequence, ensuring a constant input dimension for the network. This approach preserves the original data organization and avoids training instability caused by variations in the number of input frames.

[0057] Time series aggregation module:

[0058] Lane lines are a type of target with a slender geometric structure and strong continuity. In scenes with occlusion, backlighting, shadows, or at night, local areas in the current frame may lack sufficiently clear texture and edge information. Relying solely on the current frame for judgment can easily lead to lane interruptions, lane deviations, or false detections. In contrast, historical frames typically retain continuous structural information related to the current scene. Therefore, it is necessary to utilize cross-frame context to enhance the high-level semantic representation of the current frame.

[0059] Based on this approach, this application designs a temporal aggregation module to selectively fuse the highest-level features. This module employs a design approach of current frame dominance, historical frame assistance, gating selection, and residual back-addition. It introduces valid information from historical frames at locations where the current frame requires additional contextual support, and maintains the dominance of current frame features as much as possible at locations where the current frame already has sufficient discriminative basis. The temporal aggregation module consists of a current frame projection branch, a historical frame projection branch, a gating weight generation branch, and an output mapping branch, as follows: Figure 3 As shown.

[0060] Let the highest-level feature of the current frame be The highest-level feature of historical frames is ,in First, the features of the current frame and the features of historical frames are each subjected to a 1x1 convolutional projection to obtain representations in the unified latent space:

[0061] (2)

[0062] (3)

[0063] in, Indicates the projection branch of the current frame. The historical frame projection branch is used to compress and reassemble channel information, enabling cross-frame comparisons to be performed within a unified feature space. Then, to describe the correlation and differences between the current frame and historical frames, the projection features of the current frame, the projection features of the historical frames, and their absolute differences are concatenated into a joint representation.

[0064] (4)

[0065] in This indicates that the data is spliced ​​along the channel dimension. Through joint representation, the semantic information of each frame is preserved, and explicit change clues are provided, which helps the gating branch to determine whether historical information should be introduced.

[0066] The gated branch then generates a position-by-position, channel-by-channel weighted graph based on the joint features:

[0067] (5)

[0068] in Indicates a gating network. The activation function is sigmoid. The gated branch consists of two consecutive 1x1 convolutional layers with a ReLU nonlinear mapping inserted in between. The output... The value range is [0,1], which is used to control the degree of retention of historical features in different spatial locations and channels.

[0069] After obtaining the gate weights, the module weights the historical frame projection features and restores them to the original feature channel space through the output mapping branch, forming historical residual information:

[0070] (6)

[0071] in, Indicates the output mapping branch, This represents element-wise multiplication. Through this process, historical frames are not directly superimposed onto the current frame as whole features, but are first gated and then used for enhancement as residual information.

[0072] For two frames of historical input, the module calculates respectively. and The final enhanced highest-level feature representation of the current frame is as follows:

[0073] (7)

[0074] This demonstrates that the features of the current frame are always fully preserved as the main branch, while historical frames only provide incremental information at gated locations and channels. This residual-based temporal enhancement ensures the current frame remains the center of perception while avoiding positional drift and background noise accumulation caused by excessive historical information.

[0075] Multi-frame enhancement strategy for shared backbone feature extraction networks:

[0076] In multi-frame input scenarios, performing random geometric augmentation independently on the current frame and historical frames would disrupt the spatial correspondence in the original video sequence, causing lane structures that should be continuous to become artificially misaligned after augmentation, thus affecting the learning effect of the temporal module. To address this, this application employs a consistent multi-frame augmentation strategy in the data preprocessing stage, where three frames within the same temporal sample share the same set of random augmentation parameters.

[0077] Let the random augmentation transformation be denoted as ,in Let represent the geometric transformation parameters obtained from sampling. Then, the enhancement form for three frames of an image in a time series sample is:

[0078] (8)

[0079] In implementation, geometric operations such as scaling, flipping, and affine transformations use completely consistent parameter settings across the three frames to ensure that the enhanced frames maintain a reasonable spatial correspondence. The same consistent processing strategy is also used for transformations that affect the appearance distribution, such as brightness and color, to reduce unnecessary cross-frame differences during training.

[0080] Experimental Design:

[0081] Data sets and evaluation metrics:

[0082] The CULane dataset is one of the most widely used public datasets in the field of lane detection. It covers a variety of real driving environments such as urban roads and highways, and can comprehensively reflect the detection performance of the model under conditions such as occlusion, strong light, shadow, night, curves and intersections.

[0083] In terms of evaluation metrics, the CULane dataset primarily uses F1 scores, including F1@50, F1@75, and mean F1. It measures accuracy. and recall rate The calculation is performed using the following formula:

[0084] (9)

[0085] (10)

[0086] (11)

[0087] in It is a true positive. It was a false positive. False negatives are considered false negatives. During the calculation, each lane line is treated as a narrow, elongated region of fixed width. In the evaluation, the predicted lane lines and the labeled lane lines are first rasterized, and pairwise IoUs are calculated and matched one-to-one. For F1@50, a true positive is recorded when the IoU of the matched pair is greater than 0.5; otherwise, it is recorded as a false positive or missed detection. For F1@75, a true positive is recorded when the IoU of the matched pair is greater than 0.75; otherwise, it is recorded as a false positive or missed detection. The meanF1 is the average of the F1 scores obtained for different IoU values. Its calculation formula is as follows:

[0088] (12)

[0089] Furthermore, F1@50 is used as the default evaluation metric for different scenarios. For Cross scenarios, due to their more open road structures and the frequent interruptions or discontinuities in lane markings at intersections, the evaluation focus is not on the complete fit of lane markings, but rather on the model's ability to suppress false detections. Therefore, only FP is typically calculated, with a smaller value indicating better performance.

[0090] Tusimple dataset:

[0091] The TuSimple dataset primarily consists of highway scenes. Compared to the CULane dataset, which emphasizes complex urban scenes and extreme interference conditions, it mainly features good or moderate weather conditions and has relatively lower scene complexity. It uses Accuracy, F1, FP, and FN as evaluation metrics.

[0092] Experimental environment:

[0093] The experimental environment for this application was: Ubuntu system, Nvidia GeForce RTX4090 graphics card with 24GB of video memory, training based on the PyTorch framework, with an initial learning rate of 0.0006 for the CULane dataset and 0.001 for the Tusimple dataset, 15 training epochs for the CULane dataset and 70 training epochs for the Tusimple dataset.

[0094] Experimental results:

[0095] The experimental results on the CULane dataset are shown in Table 1:

[0096] Table 1: Experimental results on the CULane dataset

[0097]

[0098] As can be seen, the method proposed in this application achieves better results than other methods. The F1@50 score is improved by 0.33 percentage points compared to CLRNet, and the F1@75 score is improved by 0.41 percentage points compared to CLRNet. This indicates that the temporal information provided by historical frames helps to enhance the continuity of the lane structure in the current frame, so that the prediction results can still maintain good alignment ability in the case of local missing parts and blurred boundaries.

[0099] Meanwhile, the performance improved by 0.52 percentage points in "congested" scenarios, 0.94 percentage points in "glare" scenarios, 0.43 percentage points in "wireless" scenarios, 0.15 percentage points in "arrow" scenarios, and 0.53 percentage points in "night" scenarios. In complex scenarios, problems such as severe occlusion, incomplete lane lines, or strong interference from road markings are common. Therefore, relying solely on the current frame for judgment can easily lead to missed detections or unstable localization. The method proposed in this application, by introducing historical frames to supplement the features of the current frame, can alleviate these problems to some extent, thus demonstrating a more significant gain in these challenging scenarios.

[0100] However, the mean F1 score decreased by 0.04 percentage points compared to CLRNet. This is mainly because the mean F1 score reflects the overall average performance, while the gains of the proposed method are more concentrated in complex scenes such as congestion, glare, and nighttime, with relatively limited gains in ordinary scenes. At the same time, temporal information may also introduce slight perturbations in some simple scenes, causing the overall average score to remain basically unchanged or even decrease slightly.

[0101] The experimental results on the Tusimple dataset are shown in Table 2.

[0102] Table 2: Experimental results on the Tusimple dataset

[0103]

[0104] Experimental results on the TuSimple dataset show that the proposed method performs similarly to CLRNet, with comparable evaluation metrics. This is primarily due to the specific characteristics of the TuSimple dataset. The TuSimple dataset mainly features highway scenes with relatively regular lane structures, clear road markings, and minimal occlusion, strong light interference, and complex backgrounds. In such scenes, the current frame typically provides sufficient lane information, and single-frame detection methods can achieve high accuracy; therefore, the additional information provided by historical frames is limited.

[0105] On the other hand, the overall metrics of the TuSimple dataset are already at a high level, and the differences between models are small, so the experimental results are also quite similar.

[0106] This application addresses the issues of lane line interruption and unstable localization that arise in single-frame detection under conditions such as occlusion, shadows, low-light conditions at night, and missing lane markings. It proposes a temporal aggregation-based lane line detection method and designs a temporal aggregation module and a multi-frame enhancement strategy. By incorporating historical frame information, the continuity and robustness of the current frame's detection results are improved. Experimental results show that the proposed method has better detection performance in complex scenes, and its overall performance outperforms mainstream methods.

[0107] Nevertheless, this method still has certain limitations. It only utilizes the first two historical frames, resulting in limited use of temporal information. When severe occlusion occurs in multiple consecutive frames or the scene changes rapidly, the supplementary role of historical information is affected. Furthermore, the existing temporal aggregation module does not incorporate motion modeling and geometric alignment mechanisms. Therefore, when there are significant changes in vehicle posture or obvious differences in perspective between adjacent frames, the correspondence between historical information and the current frame remains unstable. Future research could focus on strengthening temporal modeling, improving cross-frame alignment mechanisms, and incorporating lane geometric constraints to continuously improve the accuracy and stability of lane detection in complex scenarios.

[0108] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0109] The relevant references are as follows:

[0110] [4]Qin, Zequn, Pengyi Zhang, and Xi Li. "Ultra fast deep lanedetection with hybrid anchor driven ordinal classification." IEEEtransactions on pattern analysis and machine intelligence 46.5 (2022): 2555-2568.

[0111] [5]Tabelini, Lucas, et al. "Keep your eyes on the lane: Real-timeattention-guided lane detection." Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2021.

[0112] [6]Liu, Lizhe, et al. "Condlanenet: a top-to-down lane detectionframework based on conditional convolution." Proceedings of the IEEE / CVFinternational conference on computer vision. 2021.

[0113] [7]Zheng, Tu, et al. "Clrnet: Cross layer refinement network for lanedetection." Proceedings of the IEEE / CVF conference on computer vision andpattern recognition. 2022.

[0114]

[15] Su, Jinming, et al. "Structure guided lane detection." arXivpreprint arXiv:2105.05403 (2021)。

Claims

1. A lane detection method based on temporal aggregation, characterized in that... The method includes the following steps: A temporal lane detection network is constructed, which includes a multi-frame input construction module, a shared backbone feature extraction network, a temporal aggregation module, a feature pyramid fusion network, and a lane prior refinement detection head. Training and optimization of the temporal lane detection network; The current frame and its two previous historical frames are input into the optimized temporal lane detection network, which then performs lane detection and outputs the lane detection results.

2. The lane detection method based on temporal aggregation as described in claim 1, characterized in that, The multi-frame input construction module adopts a multi-frame input organization method: The current frame and its two previous historical frames are combined to construct a time-series sample: ； in To monitor the center frame, and For historical auxiliary frames; During the sample construction process, historical frames are retrieved from the same video sequence as the current frame in chronological order. When the sequence is at the beginning and there are fewer than two historical frames, the nearest valid frame is used to copy and complete the sequence.

3. The lane detection method based on temporal aggregation as described in claim 2, characterized in that, The shared backbone feature extraction network employs a multi-frame enhancement strategy, including the following steps: Three frames within the same time-series sample share the same set of random augmentation parameters. Let the random augmentation transformation be denoted as... ,in The geometric transformation parameters obtained from sampling represent the enhancement form of three frames in a time-series sample. for: 。 4. The lane detection method based on temporal aggregation as described in claim 2, characterized in that, The time-series aggregation module includes a current frame projection branch, a historical frame projection branch, a gated weight generation branch, and an output mapping branch. The output of the current frame projection branch and the output of the historical frame projection branch are connected to the input of the output mapping branch via the gated weight generation branch.

5. The lane detection method based on temporal aggregation as described in claim 4, characterized in that, The processing method of the time-series aggregation module includes the following steps: Let the highest-level feature of the current frame be The highest-level feature of historical frames is ,in ; First, the features of the current frame and the features of the historical frames are each subjected to a 1x1 convolutional projection to obtain the representation in the unified latent space: ； ； in, Indicates the projection branch of the current frame. This represents the historical frame projection branch, whose function is to compress and reassemble channel information so that cross-frame comparisons can be performed within a unified feature space. Next, to describe the correlation and differences between the current frame and historical frames, the projection features of the current frame, the projection features of historical frames, and the absolute differences between the two are concatenated into a joint representation: ； in This indicates splicing along the channel dimension; The gated branch then generates a position-by-position, channel-by-channel weighted graph based on the joint features: ； in Indicates a gating network. The activation function is sigmoid; the gated branch consists of two consecutive 1x1 convolutional layers with a ReLU nonlinear mapping inserted in between; the output... The value range is [0,1], which is used to control the degree of retention of historical features in different spatial locations and channels; After obtaining the gate weights, the module weights the historical frame projection features and restores them to the original feature channel space through the output mapping branch, forming historical residual information: ； in, Indicates the output mapping branch, This represents element-wise multiplication; For two frames of historical input, the module calculates respectively. and The final enhanced highest-level feature representation of the current frame is as follows: ； The features of the current frame are always fully preserved as the main branch, while historical frames only provide incremental information in the gating-allowed positions and channels.

Images