Space-time adaptive video lane line detection method and system based on differential memory and wavelet guidance

Abstract

The invention discloses a space-time adaptive video lane line detection method based on differential memory and wavelet guidance, and mainly solves the problems of difficulty in collaborative modeling of space-time characteristics, easy loss of shallow geometric details and insufficient utilization of deep time sequence information in a dynamic scene in the prior art. According to the implementation scheme, the method comprises the following steps: acquiring marked video sequence data from a public video lane line detection data set, preprocessing the marked video sequence data, and dividing the preprocessed video sequence data into a training set and a test set; constructing a video lane line detection network comprising a feature extraction unit, a differential memory time sequence alignment unit, a wavelet-guided direction perception feature enhancement unit, a space-time adaptive fusion unit and a feature decoding unit; iteratively training the video lane line detection network through back propagation by using the training set; and inputting the test set into the trained video lane line detection network, and outputting a lane line. According to the method, the detection precision and robustness in challenging environments such as shielding, strong light and motion blur are remarkably improved, and the method can be used for realizing accurate extraction and stable tracking of lane lines in a dynamic driving scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and specifically relates to a spatiotemporal adaptive video lane line detection method and system, which can be used to achieve accurate extraction and stable tracking of lane lines in dynamic driving scenarios. Background Technology

[0002] With the rapid development of autonomous driving technology and advanced driver assistance systems, environmental perception, as a fundamental module for vehicle decision-making and control, directly impacts driving safety in terms of accuracy and stability. Among numerous perception tasks, lane detection is a crucial link in realizing functions such as lane keeping, path planning, and lateral vehicle control. As an important component of road traffic geometry, lane lines provide the most direct lateral position reference for vehicles, and their detection accuracy has a decisive impact on the reliability of the overall system. Traditional single-frame image detection methods rely on spatial information and cannot cope with temporal challenges in dynamic scenes, such as when lane lines are briefly occluded. Therefore, video lane detection technology has emerged, its core being the use of the spatiotemporal correlation between consecutive frames to compensate for the deficiencies of single-frame information with motion information. However, video lane detection still faces severe technical bottlenecks in practical applications. First, existing video detection algorithms often struggle to achieve efficient collaborative modeling in the spatiotemporal dimension. When dealing with dynamic displacement, these methods typically lack explicit perception of inter-frame motion states, leading to the introduction of historical information that can easily generate noise interference and cause detection errors when vehicle speed changes abruptly or when a vehicle cuts in. Secondly, lane lines, as artificial constructs with strong geometric priors, primarily possess features concentrated in shallow, high-frequency edge information. Existing deep convolutional models, due to downsampling operations, easily lose these crucial shallow geometric details during the extraction of deep semantic features, especially when dealing with heavily worn or distant small lane lines, where edge responses often become extremely weak. Furthermore, existing methods remain insufficient in representing deep temporal correlations. Lacking effective memory adjustment mechanisms, models often struggle to distinguish stable background information from instantaneous dynamic interference, limiting temporal compensation capabilities and frequently resulting in discontinuous detection results when faced with long-term occlusion or severe motion blur.

[0003] The patent document with application number CN202410893878 discloses "a video lane line detection method based on multi-source feature fusion". The implementation steps are as follows: (1) Construct a foggy weather lane line detection dataset VIL-hazing67; (2) Construct a phase branch module, extract the phase map features of the current frame of the video through a phase encoder, and refine them into phase-dehazing features and phase-lane line detection features; (3) Construct a color-dehazing branch module, extract the color-dehazing features of the current frame, and concatenate and reduce the dimensions with the phase-dehazing features to decompose them into color-dehazing-dehazing features and color-dehazing-lane line features; (4) Construct a color-lane line branch module, extract and fuse the features of the ordered video sequence of the past n frames, and obtain the color-lane line features through a spatial attention module; (5) Construct a mask self-attention module, use the phase lane line detection features and color-dehazing lane line features to enhance the color lane line features, and output the final lane line detection result. The advantage of this method lies in its effective improvement of lane line detection accuracy and robustness under low visibility conditions such as heavy fog through multi-source feature fusion. However, its shortcomings are that the method only relies on the simple concatenation of the current frame and a limited number of historical frames, lacking the ability to model efficient fusion of inter-frame information. Secondly, its spatial feature enhancement relies solely on traditional spatial attention mechanisms, failing to fully exploit the directional and geometric features of lane lines, thus resulting in insufficient accuracy in lane line detection in complex topological regions such as curves and bifurcations.

[0004] Patent document CN202411580951 discloses "A video lane detection method and system based on instance feature guidance". The implementation steps are as follows: (1) Construct a detection network composed of a backbone network, a neck network and a line anchor feature interaction fusion module, and use a dual-branch structure with shared weights to extract multi-level image features of the current frame and the reference frame respectively; (2) Initialize the line anchor set on the image, and input it together with the multi-level features into the line anchor feature interaction fusion module to gradually refine the line anchor position from the high level to the low level; (3) Integrate the global context information of the current frame and the reference frame in each level through a cross-attention mechanism to enhance the line anchor feature representation; (4) Utilize the classification and regression branches to output the line anchor confidence and position parameters, and obtain the final lane detection result after non-maximum suppression. Although this method only requires a single reference frame to achieve temporal information fusion, it effectively improves the detection accuracy and robustness in difficult scenarios such as occlusion and damage. However, since this method relies on only a single reference frame for temporal modeling and lacks the ability to explicitly capture long-range dynamic changes, it is prone to trajectory breakage under extreme conditions such as continuous lane line occlusion or sudden changes in illumination. At the same time, since this method adopts a fixed-level progressive refinement strategy in the line anchor interaction fusion process, it is difficult to adaptively adjust the fusion weights according to scene changes, which limits the robustness of the model in dynamic scenes. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a spatiotemporally adaptive video lane detection method based on differential memory and wavelet guidance. This method avoids the loss of lane spatial geometric details in traditional attention mechanisms, improves the effective extraction and modeling of motion features between consecutive frames, enhances the effective utilization of long-term dynamic information in lane detection, and improves the detection accuracy of adaptive feature fusion under dynamic occlusion and complex scenes.

[0006] The technical approach to achieve the objective of this invention is as follows: By constructing a differential memory temporal alignment module and introducing an inter-frame differential dynamic perception attention mechanism, the problem of insufficient continuous inter-frame motion feature extraction and modeling capabilities in existing methods is solved, enabling effective utilization of long-term temporal dynamic information; by designing a wavelet-guided direction perception feature enhancement module, wavelet transform is used to decompose lane line features into sub-bands with clear directional information in the frequency domain, and enhancement is applied to different sub-bands to avoid the loss of lane line spatial geometric details in traditional attention mechanisms, thereby improving the representation capability of lane geometric structure; by introducing a spatiotemporal adaptive fusion module, adaptive fusion and dynamic trade-off of geometric details in clear lane line regions and temporal information in occluded regions are achieved, thereby improving detection accuracy in complex scenes and ultimately achieving dynamic balance and collaborative optimization of spatiotemporal features.

[0007] Based on the above ideas, the technical solution of the present invention includes:

[0008] 1. A spatiotemporally adaptive video lane detection method based on differential memory and wavelet guidance, characterized in that,

[0009] include:

[0010] (1) Obtain labeled video sequence data from the publicly available video lane detection dataset, perform preprocessing including image size normalization and data augmentation to obtain the preprocessed video sequence dataset, and then divide it into training set and test set according to the set ratio;

[0011] (2) Construct a video lane detection network including a feature extraction unit, a differential memory temporal alignment unit, a wavelet-guided direction-aware feature enhancement unit, a spatiotemporal adaptive fusion unit, and a feature decoding unit;

[0012] (3) The video lane detection network is iteratively trained using the training set through backpropagation to obtain the trained video lane detection network;

[0013] (4) Input the test set into the trained video lane detection network and output the lane lines.

[0014] Furthermore, the feature extraction unit includes an input layer, a multi-scale feature extraction layer, and an output layer. The input layer takes three consecutive frames of lane line images as input, and the multi-scale feature extraction layer uses ResNet-18 to perform preliminary feature extraction on the input lane line images.

[0015] Furthermore, the differential memory temporal alignment unit includes a kv feature generation layer, a kv feature weight allocation layer, a frame difference generation layer, a differential attention weight generation layer, a motion intensity coefficient extraction layer, and a dynamic perception layer, which are used to achieve adaptive alignment of motion features between multiple frames, suppress noise interference, and enhance motion consistency features in consecutive frames.

[0016] Furthermore, the wavelet-guided direction-aware feature enhancement unit includes a wavelet transform layer, a direction-aware enhancement layer, a wavelet inverse transform layer, and a feature connection layer, which are used to improve the geometric structure representation of lane lines in shallow features and enhance edge details and directional continuity.

[0017] Furthermore, the spatiotemporal adaptive fusion unit is used to adaptively fuse the temporal features with the spatial features output by the three-level cascaded wavelet-guided direction-aware feature enhancement unit.

[0018] Furthermore, the feature decoding unit includes a probability map generation layer, a coefficient map generation layer, and an output layer, which are used to generate the lane line pixel-level probability distribution and parameter representation, and output the final lane line instance detection result.

[0019] 2. A spatiotemporal adaptive video lane detection system based on differential memory and wavelet guidance, characterized in that it comprises:

[0020] The data preprocessing module is used to obtain labeled video sequence data from publicly available video lane detection datasets, perform preprocessing operations on the obtained datasets, and obtain standardized network input data.

[0021] The network construction module is used to build a video lane line detection network. Through multi-level feature processing and heterogeneous feature adaptive fusion, it can accurately identify lane lines in complex dynamic scenes.

[0022] The network training module is used to input the standardized training set into the video lane detection network, and train the model by forward propagation to calculate the prediction results, backpropagation to calculate the gradient, and iteratively updating the network parameters until the loss function converges, so as to finally obtain the trained video lane detection network.

[0023] The detection module is used to input the test set into the trained video lane detection network and output the final lane lines.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] Firstly, this invention uses a differential memory temporal alignment unit to perform key-value mapping, differential feature calculation, and memory adjustment of deep features in consecutive frames. This enables multi-frame feature alignment and temporal information filtering within the feature space, without relying on explicit optical flow estimation. It can obtain more stable multi-frame temporal associations in complex dynamic scenes, thereby significantly improving the temporal continuity of lane detection.

[0026] Secondly, this invention introduces a wavelet-guided direction-aware feature enhancement unit, which combines two-dimensional wavelet transform with direction-aware convolution to explicitly model the directional structure and high-frequency edge details of lane lines. This not only adaptively enhances geometric features in different directions but also avoids the shortcomings of traditional convolutional isotropic receptive fields in representing slender structures. As a result, it has better geometric expression capabilities in curved road sections, alternating solid and dashed lines, weak textures, and uneven lighting scenarios.

[0027] Thirdly, the present invention designs a spatiotemporal adaptive fusion unit, which achieves fine-grained fusion of spatial features and temporal features through spatial region saliency extraction and pixel-level dynamic weight allocation. This design can retain geometric details in structurally clear regions and compensate and reconstruct in uncertain regions such as occlusion and blurring by relying on temporal context. This avoids problems such as detection breaks, trajectory jitter and artifacts caused by rigid spatiotemporal feature coupling due to traditional fixed weight splicing or simple weighting methods, and significantly improves the detection performance of the system in dynamic and complex driving environments. Attached Figure Description

[0028] Figure 1 This is a flowchart of the spatiotemporal adaptive video lane detection method based on differential memory and wavelet guidance of the present invention;

[0029] Figure 2 This is a diagram of the video lane line network framework constructed in the method of this invention;

[0030] Figure 3 yes Figure 2 A schematic diagram of the dynamic sensing layer in the diagram;

[0031] Figure 4 yes Figure 2 A schematic diagram of the spatiotemporal adaptive fusion unit in the diagram;

[0032] Figure 5 This is a block diagram of the spatiotemporal adaptive video lane detection system based on differential memory and wavelet guidance of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0034] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and for the purpose of understanding, and their sequence is not limited.

[0035] Example 1: A spatiotemporal adaptive video lane detection method based on differential memory and wavelet guidance.

[0036] Reference Figure 1 The implementation steps of this embodiment include the following:

[0037] Step 1: Data preparation and preprocessing.

[0038] 1.1) Obtain video sequence data containing consecutive frames and lane line annotation information from the publicly available video lane line detection dataset VIL-100;

[0039] 1.2) Preprocessing the video sequence data:

[0040] 1.2.1) The size of the input video frames is uniformly adjusted to the preset size of 640×384 through image size normalization to meet the fixed dimension requirements of network input;

[0041] 1.2.2) Data augmentation is performed on the normalized frame sequence to perform operations such as horizontal flipping, affine transformation, channel shuffling, and brightness adjustment to enhance the model's generalization ability and robustness.

[0042] 1.3) Divide the preprocessed dataset into non-overlapping training and test sets according to a pre-set ratio of 8:2.

[0043] Step 2: Establish feature extraction units.

[0044] Using the existing ResNet-18 residual network as the feature extraction subnetwork, features are extracted from three consecutive input frames of the image to obtain a multi-scale feature map of the image, which is the shallow feature map of the current frame. and deep features The first deep feature of history Deep features of the second historical frame , as input to subsequent units;

[0045] Step 3: Establish differential memory timing alignment units.

[0046] 3.1) Establish a key-value feature generation layer. Using the deep features of three consecutive frames as input, the layer generates key-value features for the three consecutive frames through two 3×3 convolutions, i.e., the key-value pairs of the current frame. / Historical first frame key-value pair / Historical second frame key-value pair / ;

[0047] 3.2) Establish a key-value feature weight allocation layer, using the key features of two historical frames. , or value characteristics , As input, the two inputs are first concatenated along the channel dimension and then processed sequentially using batch normalization, 1×1 convolution, 3×3 convolution, 1×1 convolution, and Softmax activation function to obtain a weight matrix. This weight matrix is ​​then multiplied by each of the two inputs to obtain the enhanced input features. Finally, the enhanced input features are summed to obtain the fused historical frame key-value features. / This enables adaptive alignment and enhancement of historical information within the current frame;

[0048] 3.3) Establish a frame difference generation layer, using the deep features of three consecutive frames as input, and then using the deep features of the current frame... Deep features of the first historical frame Subtraction, extracting deep features from the first historical frame Deep features of the second historical frame Subtract the two difference features, then concatenate them along the channel dimension to output the difference feature. ;

[0049] 3.4) Establish a differential attention weight generation layer, using differential features. As input, the matrix is ​​sequentially processed through a 3×3 convolution and a sigmoid activation function to obtain the differential attention weight matrix. It is used to characterize local motion changes and structural drift, and measures "which regions have undergone dynamic changes" in a pixel-level manner, guiding the matching and fusion of temporal features in the spatiotemporal domain, making the model pay more attention to dynamic regions;

[0050] 3.5) Establish a motion intensity coefficient extraction layer, also using differential features. As input, the data is sequentially processed through 3×3 convolution, average pooling, dimension rearrangement, linear processing, and sigmoid activation function to obtain the motion intensity coefficients of the current frame. and historical frame motion intensity coefficient This is used to achieve adaptive control of memory strength;

[0051] 3.6) Establish a dynamic perception layer:

[0052] Reference Figure 3 The implementation of this step includes the following:

[0053] With differential attention weight matrix Motion intensity coefficient of the current frame Historical frame motion intensity coefficient Current frame key features / and fusion of historical frame key features / As input;

[0054] Key features of the current frame and fusion of historical frame key features Perform dimensional rearrangement to generate the rearranged key features of the current frame. and fusion of historical frame key features And the key features of the rearranged fused historical frames. Transpose the data to obtain the transposed features of the fused historical frame key. The key features of the current frame after rearrangement With transpose feature Multiply to generate the first weight matrix;

[0055] Combine the first weight matrix with the differential attention weight matrix. Multiplying them yields a second weight matrix, which is used to emphasize the regions where features change.

[0056] The second weight matrix is ​​added to the first weight matrix to form a residual path. The softmax activation function is then applied to the resulting residual path to obtain the inter-frame matching weights. :

[0057] ,

[0058] in, For dimensions.

[0059] Historical frame features Perform dimensional rearrangement to obtain rearranged historical frame value features. The rearranged historical frame value features Inter-frame matching weights Multiply and output the fused level 1 features. :

[0060] ,

[0061] Level 1 features Perform dimensional rearrangement to obtain the rearranged level 1 features. The rearranged first-level features After 3×3 convolution and motion intensity coefficients from historical frames Multiplication yields level 2 features, a process that adaptively scales the contribution of historical memory information.

[0062] Current frame value features With the motion intensity coefficient of the current frame Multiplication yields a level 3 feature, and this motion intensity coefficient of the current frame is used to adjust the degree of preservation of the original information of the current frame.

[0063] The second-level features are added to the third-level features to obtain the fourth-level features, thereby achieving a balanced fusion of historical and current information under motion perception.

[0064] Compare the level 4 features with the value features of the current frame. By concatenating along the channel dimension, a time-series feature with temporal consistency is output. :

[0065] ,

[0066] Where [,] indicates feature concatenation in the channel dimension. This indicates element-wise multiplication.

[0067] 3.7) Connect the above layers to form a differential memory timing alignment unit:

[0068] The key-value feature generation layer is connected to the key-value feature weight allocation layer to form the first branch, which outputs the fused historical frame key-value features; the frame difference generation layer is connected to the differential attention weight generation layer to form the second branch, which outputs the differential attention weights; and the frame difference generation layer is then connected to the motion intensity coefficient extraction layer to form the third branch, which outputs the first motion intensity coefficient and the second motion intensity coefficient.

[0069] The fused historical frame key-value features, differential attention weights, first motion intensity coefficient, and second motion intensity coefficient output from the three branches, along with the current frame key-value features output from the kv feature generation layer, are input into the dynamic perception layer to form a differential memory temporal alignment unit, such as... Figure 2 The upper part is shown in the image.

[0070] Step 4: Establish a wavelet-guided direction-aware feature enhancement unit.

[0071] 4.1) Establish a wavelet transform layer, using the shallow features of the current frame. As input, it is decomposed into four sub-bands through two-dimensional discrete Haar wavelet transform: low-frequency sub-band features Horizontal high-frequency subband characteristics Vertical high-frequency subband characteristics Diagonal high-frequency subband characteristics This wavelet domain decoupling provides a feature separation basis for subsequent targeted enhancement;

[0072] 4.2) Establish a direction-aware enhancement layer to perform targeted enhancement processing on each feature based on the characteristics of different sub-bands:

[0073] Low-frequency subband characteristics Enhanced low-frequency subband features are obtained by performing 3x3 convolution. ;

[0074] Features of horizontal high-frequency subbands Enhanced horizontal high-frequency subband features are obtained through vertical convolution (VDC). Vertical convolution employs dynamic serpentine convolution along the y-axis.

[0075] Vertical high-frequency subband characteristics Enhanced vertical high-frequency subband features are obtained through horizontal convolution (HDC). The horizontal convolution uses dynamic serpentine convolution along the x-axis.

[0076] Diagonal high-frequency subband characteristics Enhanced diagonal high-frequency subband features obtained through deformable convolution. ;

[0077] 4.3) Establish an inverse wavelet transform layer to enhance the low-frequency subband features. Enhanced horizontal high-frequency subband characteristics Enhanced vertical high-frequency subband characteristics Enhanced diagonal high-frequency subband characteristics As input, the input enhancement features are obtained through two-dimensional discrete Haar wavelet inverse transform. ;

[0078] 4.4) Establish a feature connection layer to enhance the input features. Compared with the original input features Adding them together yields the spatial features. ;

[0079] 4.5) The wavelet transform layer, direction-aware enhancement layer, inverse wavelet transform layer, and feature connection layer are sequentially connected to form a wavelet-guided direction-aware feature enhancement unit, such as... Figure 2 The lower half is shown.

[0080] Step 5: Establish a spatiotemporal adaptive fusion unit.

[0081] Reference Figure 4 The implementation of this step includes the following:

[0082] 5.1) Spatial features The spatial features are enhanced by a two-stage cascaded wavelet-guided direction-aware feature enhancement unit. ;

[0083] 5.2) Temporal characteristics The time-series features are obtained after upsampling. and the sampled time-series features and enhanced spatial features Adding them together yields the basic fusion features. ;

[0084] 5.3) Fusion Features After performing flat pooling and max pooling, feature concatenation is performed along the channel dimension to obtain the first-level feature;

[0085] 5.4) After performing a 7×7 convolution on the first-level features, then combine them with the fused features. Secondary features are obtained by concatenating channel-dimensional features;

[0086] 5.5) The secondary features are sequentially processed through 7×7 convolution and a sigmoid activation function to generate temporal weights. and spatial weights The temporal and spatial weights autonomously decide the fusion ratio of spatial and temporal features. That is, in areas where the lane line structure is clear, the geometric details of the current frame are emphasized, while in occluded or blurred areas, the motion trend and continuity of historical frames are emphasized.

[0087] 5.6) Sampled time-series features With time series weights Multiplication will enhance spatial features. Spatial weights Multiply the two products and then add them together to obtain the fused feature. :

[0088] ,

[0089] in This indicates element-wise multiplication.

[0090] Step 6, establish the feature decoding unit:

[0091] The feature decoding unit is used to reconstruct lane lines from the fused features, and its implementation includes the following:

[0092] 6.1) Establish a probabilistic graph generation layer to fuse features. As input, the data is sequentially processed through 3x3 convolution, 3x3 convolution, 1x1 convolution, and Softmax activation function to obtain binary classification probability map features;

[0093] 6.2) Establish a coefficient map generation layer, using the probabilistic map features as input, and generate lane line polynomial coefficient map features through 3x3 convolution, batch normalization, ReLU activation function, position encoding, and deformable convolution;

[0094] 6.3) Establish an output layer, using probability map features and coefficient map features as input, and sequentially pass through filtering, non-maximum suppression algorithm, and coordinate transformation to obtain the final lane line output;

[0095] 6.4) Connect the probability map generation layer to the coefficient map generation layer, and then connect the probability map generation layer and the coefficient map generation layer together to the output layer to form a feature decoding unit.

[0096] Step 7: Construct a video lane line detection network.

[0097] like Figure 2 As shown, a differential memory temporal alignment unit and a 3-level cascaded wavelet-guided direction-aware feature enhancement unit are connected in parallel. A feature extraction unit is then connected to the input of this parallel connection, and a spatiotemporal adaptive fusion unit and a feature decoding unit are sequentially connected to the output of this parallel connection to form a video lane detection network.

[0098] Step 8: Train the video lane detection network.

[0099] 8.1) Establish the total loss function :

[0100] ;

[0101] in, Let be the focus loss function for lane line probability prediction. This represents the logarithmic operation to the base 2. Represents the category weight factor. This indicates adjustable hyperparameters. This represents the model's predicted probability for the correct category. To predict the probability that a given pixel represents a lane line in the model. This is the actual label for that pixel;

[0102] To constrain the segment intersection-over-union ratio (IoU) loss function for lane shape regression, N represents the number of sampling points for the lane lines. Indicates the radius of expansion. This represents the x-coordinate of the predicted lane line point at the i-th position. This represents the x-coordinate of the actual lane line point at position i;

[0103] 8.2) Set the learning rate to 0.0002, the weight decay to 0.0002, and the learning rate decay coefficient to 0.4;

[0104] 8.3) Randomly initialize all weights and biases in the network;

[0105] 8.4) Input the data from the training set into the video lane detection network in batches, and the network obtains the prediction results through forward propagation;

[0106] 8.5) Calculate the difference between the predicted value and the true label, and use the difference to obtain the loss value through the loss function;

[0107] 8.6) Starting from the loss value, use the chain rule to calculate the gradient of the loss function with respect to each parameter in the network in reverse. Use the calculated gradients to update all parameters using the Adam optimizer.

[0108] 8.7) Repeat steps 8.4) to 8.6) until the loss value no longer decreases significantly. Then, the model is considered to have converged. Save the optimal network weight file to obtain the trained video lane detection network.

[0109] Step 9: Input the test set into the trained video lane detection network and output the lane lines.

[0110] Example 2: Spatiotemporal adaptive video lane detection system based on differential memory and wavelet guidance.

[0111] Reference Figure 5 This example includes: data preprocessing module 1, network construction module 2, network training module 3, and detection module 4. The network construction module 2 includes: feature extraction submodule 21, differential memory temporal alignment submodule 22, wavelet-guided direction-aware feature enhancement submodule 23, spatiotemporal adaptive fusion submodule 24, and feature decoding submodule 25.

[0112] The working principle of the entire system is as follows:

[0113] The data preprocessing module 1 is used to obtain labeled video sequence data from the publicly available video lane detection dataset, perform preprocessing operations on it to obtain standardized network input data, and then divide it into training and test sets.

[0114] The network construction module 2 is used to construct a video lane detection network to detect lane lines. The feature extraction submodule 21 is used to extract the deep features of three consecutive frames of images and the shallow features of the current frame image from the input data. The deep features of the three consecutive frames are input to the differential memory temporal alignment submodule 22, and the shallow features of the current frame image are input to the wavelet-guided direction perception feature enhancement submodule 23.

[0115] The differential memory temporal alignment submodule 22 extracts temporal features from the deep features of three consecutive frames of images through historical frame feature alignment and differential attention mechanism, and then inputs the temporal features into the spatiotemporal adaptive fusion submodule 24.

[0116] The wavelet-guided direction-aware feature enhancement submodule 23 enhances the shallow features of the current frame image through wavelet transform and directional convolution, extracts spatial features, and then inputs the spatial features into the spatiotemporal adaptive fusion submodule 24.

[0117] The spatiotemporal adaptive fusion submodule 24 fuses temporal and spatial features with pixel-level dynamic weights through a spatial attention mechanism, extracts fused features, and then inputs the fused features to the feature decoding submodule 25.

[0118] The feature decoding submodule 25 is used to perform a multi-stage decoding process of probabilistic map generation, coefficient map regression and post-processing reconstruction on the fused features to extract lane lines.

[0119] The network training module 3 is used to input the training set into the video lane detection network, and train the model by forward propagation to calculate the prediction results, backpropagation to calculate the gradient, and iteratively updating the network parameters until the loss function converges, so as to finally obtain the trained video lane detection network.

[0120] The detection module 4 is used to input the test set into the trained video lane detection network and output the final lane lines.

[0121] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0122] In this embodiment, the direct coupling or communication connection between the modules can be achieved through indirect coupling or communication connection via interfaces, devices, or modules. The functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0123] The effects of this invention will be further illustrated below with simulation experiments:

[0124] 1. Simulation experimental conditions:

[0125] The hardware platform for the simulation experiment of this invention is a server with an NVIDIA GTX4090Ti graphics card, an AMD EPYC7302 16-Core Processor processor, and 128GB of memory.

[0126] The software platform for the simulation experiment of this invention is: a code execution environment for the PyTorch 1.8.1 deep learning framework is built in the Python 3.6.12 virtual environment of Anaconda.

[0127] The input images used in the simulation experiment of this invention are from the publicly available dataset VIL-100, which consists of 100 videos, each with 100 frames, for a total of 10,000 frames. Of these 100 videos, 97 were collected by a single forward-looking camera installed near the rearview mirror, and the remaining 3 videos were all from the internet and were filmed in hazy weather to increase the complexity and realism of the collected dataset.

[0128] The simulation dataset considers 10 typical scenarios: normal, congested, curved road, damaged road, shadow, road markings, glare, fog, night, and intersection. The dataset is split into training and test sets in an 8:2 ratio, and all 10 scenarios are presented in both sets.

[0129] The simulations used three comparison methods: MMA-Net, RVLD, and OMR, and their sources are as follows:

[0130] The existing technology MMA-Net refers to the work of Zhang, Yujun, et al. in "Vil-100: A new dataset and a baseline model for..." video instance lane detection. Proceedings of the IEEE / CVFinternational conference on computer The paper "Vision. 2021:15681-15690" proposes a video instance lane detection method called Multi-Level Memory Aggregation Network, or MMA-Net for short.

[0131] The existing technology RVLD refers to the work of Jin, Dongkwon et al. in "Recursive video lane detection. Proceedings of..." The IEEE / CVF International Conference on Computer Vision. 2023: 8473–8482 proposed a recursive video lane detection network, abbreviated as RVLD.

[0132] The existing technology OMR refers to the work of Jin, Dongkwon et al. in "OMR: Occlusion-aware memory-based refinement". The paper "For video lane detection. European Conference on ComputerVision.2024: 129-145" proposes a video lane detection network based on occlusion perception and memory enhancement, abbreviated as OMR.

[0133] 2. Simulation content and result analysis:

[0134] Under the above conditions, lane line detection was performed on the VIL-100 dataset using the present invention and three existing detection methods: MMA-Net, RVLD, and OMR. The detection results of the four methods were evaluated by calculating the F1-Score.

[0135] ,

[0136] in, ,

[0137] ,

[0138] The detection results of the four methods are then evaluated by calculating the average intersection-union ratio (mIOU).

[0139]

[0140] Finally, the F1-Score and average crossover ratio (mIoU) of the four methods were statistically compared, and the results are shown in Table 1.

[0141] Table 1. Performance Indicators of Lane Detection of the Invention and Existing Technologies in Simulation Experiments

[0142]

[0143] As can be seen from Table 1, the F1-Score of the method of the present invention is 93.7%, and the average crossover ratio mIoU is 79.7%. Both of these indicators are higher than those of the three existing technologies, indicating that the present invention can achieve better detection performance in video lane line detection tasks.

Claims

1. A spatiotemporal adaptive video lane detection method based on differential memory and wavelet guidance, characterized in that, include: (1) Obtain labeled video sequence data from the publicly available video lane detection dataset, perform preprocessing including image size normalization and data augmentation to obtain the preprocessed video sequence dataset, and then divide it into training set and test set according to the set ratio; (2) Construct a video lane detection network including a feature extraction unit, a differential memory temporal alignment unit, a wavelet-guided direction-aware feature enhancement unit, a spatiotemporal adaptive fusion unit, and a feature decoding unit; (3) The video lane detection network is iteratively trained using the training set through backpropagation to obtain the trained video lane detection network; (4) Input the test set into the trained video lane detection network and output the lane lines.

2. The method according to claim 1, characterized in that, The implementation of (1) includes the following: 1a) Obtain image sequences containing consecutive video frames and their corresponding lane line annotation information from the publicly available video lane line detection dataset VIL-100; 1b) The Resize function in the imgaug library is used to normalize the size of each frame in the acquired image sequence so that it has a uniform input size of 640×384; 1c) Perform data augmentation preprocessing on the normalized image sequence: 1c1) The HorizontalFlip function in the imgaug library is used to horizontally flip the image after normalization, with a 50% probability. At the same time, the x-coordinate of the lane line label is adjusted to ensure spatial consistency between the image and the label. 1c2) Apply the Affine function from the imgaug library to the horizontally flipped image to perform an affine transformation operation, including: The image width is randomly shifted by ±10% in the horizontal direction, the image height is randomly shifted by ±10% in the vertical direction, the random rotation angle is evenly distributed between -10 degrees and +10 degrees, and the random scaling ratio is evenly distributed between 0.8 times and 1.2 times. 1c3) The ChannelShuffle function from the imgaug library is used to randomly rearrange the RGB channel order of the image after affine transformation, thereby enhancing the robustness of the model to color changes. 1c4) The MultiplyAndAddToBrightness function from the imgaug library is used to adjust the image brightness after the channel is shuffled. The brightness multiplication factor is evenly distributed between 0.85 and 1.15, and an additive brightness offset is added evenly distributed between -10 and +10. 1c5) The AddToHueAndSaturation function from the imgaug library is used to adjust the hue and saturation of the image after brightness adjustment. The adjustment values ​​are evenly distributed between -10 and +10 to simulate color changes under different lighting conditions. 1d) Divide the preprocessed video sequence dataset into training and testing sets in an 8:2 ratio.

3. The method according to claim 1, characterized in that, The feature extraction unit in the video lane detection network (2) includes an input layer, a multi-scale feature extraction layer and an output layer. The input layer is the input of three consecutive frames of lane images. The multi-scale feature extraction layer uses ResNet-18 to perform preliminary feature extraction on the input lane images. After feature extraction, the direct output of ResNet-18 is used as the output of the deep feature output layer, and the intermediate feature map of ResNet-18 is used as the output of the shallow feature layer.

4. The method according to claim 1, characterized in that, The differential memory temporal alignment unit in the video lane detection network (2) includes a kv feature generation layer, a kv feature weight allocation layer, a frame difference generation layer, a differential attention weight generation layer, a motion intensity coefficient extraction layer, and a dynamic perception layer. This key-value feature generation layer takes the deep features of three consecutive frames as input and generates key-value features for the three consecutive frames through a 3×3 convolution. This key-value feature weighting layer takes the key or value features of two historical frames as input. First, the two inputs are concatenated in the channel dimension. The concatenated features are then processed by batch normalization, 1×1 convolution, 3×3 convolution, 1×1 convolution, and Softmax activation function to obtain a weight matrix. This weight matrix is ​​then multiplied by the two inputs to obtain the enhanced input features. Finally, the enhanced input features are subjected to matrix addition to obtain the fused historical key-value features. This frame difference generation layer takes the deep features of three consecutive frames as input, subtracts the current frame feature from the feature of the first historical frame, and subtracts the feature of the first historical frame from the feature matrix of the second historical frame to obtain two difference features. These two difference features are then concatenated along the channel dimension to output the difference features. This differential attention weight generation layer takes the differential features as input and passes them sequentially through a 3×3 convolution and a sigmoid activation function to obtain the differential attention weight matrix. This motion intensity coefficient extraction layer also uses the difference features as input, and sequentially goes through 3×3 convolution, average pooling, dimension rearrangement, linear processing, and sigmoid activation function to obtain the motion intensity coefficient of the current frame and the motion intensity coefficient of the historical frames; This dynamic perception layer takes differential features, current frame key-value features, and fused historical frame key-value features as feature inputs. It guides the matching and fusion of features in the spatiotemporal domain through a differential attention weight matrix, and achieves adaptive control of memory features through motion intensity coefficients, ultimately outputting temporal features.

5. The method according to claim 4, characterized in that: The dynamic perception layer guides feature matching and fusion in the spatiotemporal domain through a differential attention weight matrix, which is implemented as follows: The dimensions of the current frame k features and the fused historical frame k features are rearranged separately. The rearranged current frame k features are multiplied with the fused historical frame k features to obtain the first weight matrix. Multiply the first weight matrix by the differential attention weight matrix to output the second weight matrix; The second weight matrix is ​​added to the first weight matrix and then passed through the Softmax activation function to obtain the third weight matrix; The historical frame v features after dimensional rearrangement are multiplied by the third weight matrix to obtain the first-level features, thus achieving feature fusion in the spatiotemporal domain; The dynamic perception layer achieves adaptive control of the memorized features through the motion intensity coefficients of the current frame and the motion intensity coefficients of historical frames, as follows: Level 1 features are obtained by rearranging dimensions, performing 3×3 convolution, and multiplying with the motion intensity coefficients of historical frames. Multiply the current frame v feature by the current frame motion intensity coefficient to obtain the 3-level feature; Adding the level 2 features to the level 3 features yields the level 4 features; The temporal features are obtained by concatenating the level 4 features with the current frame v features in the channel dimension. This process realizes the adaptive weighted control of historical memory features and current frame features.

6. The method according to claim 1, characterized in that, The wavelet-guided direction perception feature enhancement unit in the video lane detection network in (2) includes a wavelet transform layer, a direction perception enhancement layer, an inverse wavelet transform layer, and a feature connection layer. This wavelet transform layer takes the shallow features of the current frame as input and obtains low-frequency features, horizontal high-frequency features, vertical high-frequency features and diagonal high-frequency features through two-dimensional discrete Haar wavelet transform. This orientation-aware enhancement layer takes low-frequency features, horizontal high-frequency features, vertical high-frequency features, and diagonal high-frequency features as inputs, and performs targeted enhancement processing on each feature: low-frequency features are processed by 3x3 convolution, horizontal high-frequency features are processed by vertical convolution, vertical high-frequency features are processed by horizontal convolution, and diagonal high-frequency features are processed by deformable convolution. Among them, the horizontal convolution adopts dynamic serpentine convolution along the x-axis, and the vertical convolution adopts dynamic serpentine convolution along the y-axis. This wavelet inverse transform layer takes the enhanced low-frequency features, horizontal high-frequency features, vertical high-frequency features, and diagonal high-frequency features as inputs, and obtains the enhanced shallow features through discrete wavelet inverse transform. This feature connection layer takes the current frame's shallow features and the enhanced shallow features as inputs, and adds the two inputs to obtain spatial features.

7. The method according to claim 1, characterized in that, The (2) spatiotemporal adaptive fusion unit in the video lane detection network is used to adaptively fuse temporal features with spatial features output by the three-level cascaded wavelet-guided direction-aware feature enhancement unit. Its implementation includes: (2a) The initial features are obtained by adding the temporal features obtained through upsampling to the spatial features; (2b) The initial features are processed in parallel through max pooling and average pooling, and then the features are concatenated along the channel dimension; (2c) The concatenated features are convolved with a 7x7 convolution to obtain level 1 features; (2d) After concatenating the first-level features with the initial features, the temporal weights and spatial weights are obtained by performing a 7x7 convolution and a sigmoid activation function. (2e) Multiply the temporal weights by the upsampled temporal features to obtain the enhanced temporal features; (2f) Multiplying the spatial weights by the spatial features yields the enhanced spatial features; (2g) The enhanced temporal features are added to the enhanced spatial features and then fused through a 1x1 convolution to obtain the fused features.

8. The method according to claim 1, characterized in that, The feature decoding unit in the video lane detection network (2) includes a probability graph generation layer, a coefficient graph generation layer, and an output layer. This probabilistic map generation layer takes the fused features as input and passes them sequentially through 3x3 convolution, 3x3 convolution, 1x1 convolution, and Softmax activation function to obtain binary classification probabilistic map features; This coefficient map generation layer takes the probabilistic map features as input and generates lane line multinomial coefficient map features through 3x3 convolution, batch normalization, ReLU activation function, position encoding, and deformable convolution. The output layer takes probability map features and coefficient map features as input, and then passes them through filtering, non-maximum suppression algorithm, and coordinate transformation to obtain the final lane line output.

9. The method according to claim 1, characterized in that, The iterative training of the video lane detection network in (3) includes the following: (3a) The focus loss function and the line segment intersection-union loss function are combined to form the total loss function of the detection network. : ； in, Let be the focus loss function for lane line probability prediction. Represents the category weight factor. This indicates adjustable hyperparameters. This represents the model's predicted probability for the correct category. To predict the probability that a given pixel represents a lane line in the model. This is the actual label for that pixel; To constrain the segment intersection-over-union ratio (IoU) loss function for lane shape regression, N represents the number of sampling points for the lane lines. Indicates the radius of expansion. This represents the x-coordinate of the predicted lane line point at the i-th position. This represents the x-coordinate of the actual lane line point at position i; (3b) Set the learning rate to 0.0002, the weight decay to 0.0002, and the learning rate decay coefficient to 0.4; (3c) Randomly initialize the network weights and biases; (3d) Input the training set into the video lane detection network and calculate the predicted values ​​through forward propagation; (3e) Calculate the loss value based on the error between the predicted value and the true label; (3f) Starting with the calculated loss value, calculate the gradient of the loss function with respect to all network parameters using the backpropagation algorithm; (3g) Use the calculated gradients to update all parameters using the Adam optimizer; (3h) Repeat steps (3d) to (3g) until the loss value no longer decreases significantly. Then, the model is considered to have converged. Save the optimal network weight file to obtain the trained video lane detection network.

10. A spatiotemporal adaptive video lane detection system based on differential memory and wavelet guidance, characterized in that, include: The data preprocessing module is used to obtain labeled video sequence data from publicly available video lane detection datasets, perform preprocessing operations on the obtained datasets, and obtain standardized network input data. The network construction module is used to build a video lane line detection network. Through multi-level feature processing and heterogeneous feature adaptive fusion, it can accurately identify lane lines in complex dynamic scenes. The network training module is used to input the standardized training set into the video lane detection network, and train the model by forward propagation to calculate the prediction results, backpropagation to calculate the gradient, and iteratively updating the network parameters until the loss function converges, so as to finally obtain the trained video lane detection network. The detection module is used to input the test set into the trained video lane detection network and output the final lane lines.

11. The method according to claim 1, characterized in that, The network construction module includes: The feature extraction submodule is used to extract multi-scale features from the input image, providing basic feature representations for subsequent temporal and spatial modeling. The differential memory temporal alignment submodule is used to extract inter-frame displacement features and construct a dynamic memory bank by combining historical information. Enhance the model's time-series awareness; The wavelet-guided direction-aware feature enhancement submodule is used to enhance the geometric structure perception capability of lane lines in the current frame. The spatiotemporal adaptive fusion submodule is used to adaptively fuse the spatial geometric features of the current frame with the temporal features of historical frames, so as to achieve a dynamic trade-off between spatial details and temporal continuity. The feature decoding submodule is used to reconstruct lane lines from the fused features.

Images