Lane line detection method and device, computer equipment and storage medium
By constructing the LSTR-CoMer model and utilizing multi-scale feature extraction and fusion techniques, the problem of insufficient lane line detection accuracy of the LSTR model in complex environments was solved, achieving high-precision and robust lane line detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2026-04-03
AI Technical Summary
Existing LSTR models have limitations in handling multi-scale and local features, and cannot accurately identify lane lines, especially in complex environments such as changes in lighting, occlusion, and wear.
The LSTR-CoMer model is constructed by adding a patch embedding module, a multi-receptive field feature pyramid module, and a CNN-Transformer bidirectional interactive fusion module to the LSTR model to achieve multi-scale feature extraction and fusion. Combined with the global context information capture capability of the Transformer module, the accuracy and robustness of lane line detection are improved.
It significantly improves the accuracy and robustness of lane line detection, can accurately identify lane lines in complex scenarios, performs excellently, and has strong adaptability and anti-interference capabilities.
Smart Images
Figure CN121789159A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and autonomous driving technology, specifically relating to a lane line detection method, device, computer equipment, and storage medium. Background Technology
[0002] Intelligent driving systems are a technological field that both traditional and new energy vehicle manufacturers are continuously tackling. With the rapid development of autonomous driving technology, these systems provide drivers with comprehensive road information across all scenarios and, combined with LiDAR and visual sensors, ensure vehicle safety. Lane detection technology is a crucial component of autonomous driving systems, used to identify road markings and provide dynamic safety zones for speeding vehicles. Past research has primarily categorized lane detection methods into two main types: traditional manual feature extraction methods and deep learning methods.
[0003] Traditional lane detection methods primarily rely on manual extraction of lane features. These methods can be categorized into feature-based and model-based approaches. The former transforms the input image to grayscale, then uses edge detection algorithms, such as the Canny edge detection algorithm, to extract feature information, followed by Hough transform to detect lane positions. The latter utilizes mathematical models to establish lane line parameter models that conform to the structured road surface. Traditional lane detection methods are simple to implement and perform well on well-lit structured roads, but they lack noise resistance in complex environments, fail to effectively utilize global contextual information when fitting lane lines, and have low detection efficiency.
[0004] With the rapid development of deep learning in image processing, deep learning technology has gained favor among researchers. Deep learning-based lane detection methods are divided into detection-based and segmentation-based methods. The UFLD (Ultra-Fast-Lane-Detection) model is the first to treat lane detection as a row anchor classification and localization problem based on global features, significantly reducing computational costs. It also proposes a structural loss function to fit lane line positions, achieving best-in-class real-time performance compared to other methods based on row anchor classification, which have weaker positional awareness and lower detection accuracy. The LaneATT lane attention converter model uses anchor points for feature pooling and proposes an anchor-based attention mechanism, improving the model's ability to handle lane line occlusion and missing lines. The Generative Adversarial Network (GANet) model differs from previous methods based on predetermined anchor point regression. It directly regresses each keypoint to the starting position of the lane line and globally associates keypoints and lane lines by predicting the offset of each keypoint from its respective lane starting point, improving detection efficiency. SCNN (Spatial CNN) fully utilizes the spatial characteristics of convolution. This method slices the feature map from multiple directions and performs convolution calculations along the slice directions, increasing the information interaction between features. However, this method increases the cost of transferring feature information between adjacent slices and has a slow inference speed. CondLaneNet is a top-down lane instance detection model. It dynamically adjusts the weights of the convolution kernels based on the lane distribution in the input scene, achieving a balance between high accuracy and robustness. Segmentation-based methods can accurately detect the location of lane lines in images, but such methods usually require complex post-processing, making real-time performance difficult to meet requirements. The LSTR (Lane Shape Prediction with Transformers) detection model introduces the Transformer module into the lane line detection task. This method innovatively uses the Transformer module to completely replace the feature extraction and post-processing steps, directly outputting lane line shape parameters in an end-to-end manner, greatly simplifying the processing flow, increasing the interpretability of the model, and achieving an astonishing detection speed of 420 FPS. Although this method achieves the best results in the same period, it also has some shortcomings in the modeling process: First, the Transformer module usually processes input features at the same resolution, which limits its ability to handle multi-scale features; Second, it is not good at capturing local lane features. When lane lines are locally worn, broken or partially occluded, the LSTR model may not be able to respond to these local changes, which ultimately leads to the inability to accurately identify lane lines. Summary of the Invention
[0005] To address the problem that the LSTR model cannot accurately identify lane lines, this invention provides a lane line detection method, apparatus, computer equipment, and storage medium.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] In a first aspect, the present invention provides a lane line detection method, including...
[0008] Acquire the target lane line image;
[0009] An LSTR model is constructed, comprising an input module, multiple feature extraction backbone modules, and a feature recognition module. A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module, forming a multi-scale enhanced feature fusion lane detection model, LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through the multi-receptive field feature pyramid module. The features extracted from the multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. The feature recognition module identifies the features obtained after interactive fusion.
[0010] Train the LSTR-CoMer model to obtain a lane line recognition model;
[0011] The target lane line image is input into the lane line recognition model to identify the lane line location.
[0012] Optionally, the feature extraction backbone module further includes a Transformer module, which consists of an encoder and a decoder. The encoder includes a self-attention module and a feedforward neural network. The self-attention module is used to capture long-range dependencies between various positions in the input feature sequence. The feedforward neural network performs a non-linear transformation on the attention output at each position. The decoder includes a self-attention module, a cross-attention module, and a feedforward neural network. The cross-attention module selectively focuses on relevant features in the encoder output by calculating the attention score between the decoder query sequence and the encoder output.
[0013] Optionally, the feature recognition module includes a lane line shape representation; the lane line shape representation represents the shape parameters of the lane line in polynomial form; the polynomial form includes a cubic curve polynomial under normal conditions, a simple polynomial where the camera optical axis is parallel to the road surface, and a complex polynomial where the camera optical axis has an angle with the road surface;
[0014] The cubic polynomial under normal conditions is expressed by the following formula:
[0015] X = kZ 3 +mZ 2 +nZ+b
[0016] Where k, m, n, and b are preset real coefficients, and k ≠ 0, and (X, Z) represents a point on the road surface;
[0017] The simple polynomial that the camera's optical axis is parallel to the road surface is expressed by the following formula:
[0018]
[0019] Where k′, m′, n′, and b′ are composite parameters of the camera's intrinsic and extrinsic parameters and real coefficients, and (u,v) represents a pixel in the plane;
[0020] The complex polynomial that creates an angle between the camera's optical axis and the road surface is expressed by the following formula:
[0021]
[0022] Where f represents the focal length in pixels, (u′,v′) represents the point position after pitch conversion, and φ represents the angle between the camera's optical axis and the road surface.
[0023] Optionally, the multi-receptive-field feature pyramid module includes a feature pyramid, a linear layer, and a multi-receptive-field depth-separable convolutional layer; the feature pyramid is used to extract visual features at different scales from the lane line feature map, the linear layer performs dimensionality reduction on the visual features, and the multi-receptive-field depth-separable convolutional layer captures multi-scale spatial information, restores and fuses the processed visual features, and generates an enhanced multi-scale feature pyramid.
[0024] Optionally, the multi-receptive-field depth-separable convolutional layer includes multiple depth-separable convolutional layers with different kernel sizes, wherein each depth-separable convolutional layer first performs depth convolution on the input features, processing each channel independently; then it performs pointwise convolution, linearly combining the channels to capture multi-scale spatial information.
[0025] Optionally, training the LSTR-CoMer model to obtain a lane recognition model capable of recognizing lane lines includes:
[0026] Acquire sample data, which includes sample lane line images and sample annotations corresponding to the sample lane line images;
[0027] The sample lane line images are input into the LSTR-CoMer model to obtain the recognition results;
[0028] The LSTR-CoMer model was trained with the goal of minimizing the deviation between the recognition result and the sample label to obtain the lane line recognition model.
[0029] Optionally, inputting the target lane line image into the lane line recognition model to obtain the lane line position includes:
[0030] The target lane line image is input into the lane line recognition model. The original feature map is obtained by the Backbone module, and the compressed feature map is obtained by the patch embedding module.
[0031] The original feature map is processed by a multi-receptive field feature pyramid module to extract features, resulting in multi-scale feature pyramids with different resolutions; the compressed feature map is then processed by the Transformer module to obtain Transformer features.
[0032] The CNN-Transformer bidirectional interactive fusion module enables bidirectional interaction between the multi-scale feature pyramid and Transformer features, and further facilitates feature interaction between different features in multiple feature extraction backbone modules.
[0033] The lane line positions are obtained by identifying the features obtained after interactive fusion.
[0034] In a second aspect, the present invention provides a lane line detection device, comprising:
[0035] The acquisition module is used to acquire images of the target lane lines;
[0036] A construction module is used to build an LSTR model, which includes an input module, multiple feature extraction backbone modules, and a feature recognition module. A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module to form a multi-scale enhanced feature fusion lane detection model, LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through the multi-receptive field feature pyramid module. The features extracted from the multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. The feature recognition module identifies the features obtained after interactive fusion.
[0037] The training module is used to train the LSTR-CoMer model to obtain a lane line recognition model;
[0038] The recognition module is used to input the target lane line image into the lane line recognition model to identify the lane line position.
[0039] Optionally, the feature extraction backbone module further includes a Transformer module, which consists of an encoder and a decoder. The encoder includes a self-attention module and a feedforward neural network. The self-attention module is used to capture long-range dependencies between various positions in the input feature sequence. The feedforward neural network performs a non-linear transformation on the attention output at each position. The decoder includes a self-attention module, a cross-attention module, and a feedforward neural network. The cross-attention module selectively focuses on relevant features in the encoder output by calculating the attention score between the decoder query sequence and the encoder output.
[0040] Optionally, the feature recognition module includes a lane line shape representation; the lane line shape representation represents the shape parameters of the lane line in polynomial form; the polynomial form includes a cubic curve polynomial under normal conditions, a simple polynomial where the camera optical axis is parallel to the road surface, and a complex polynomial where the camera optical axis has an angle with the road surface;
[0041] The cubic polynomial under normal conditions is expressed by the following formula:
[0042] X = kZ 3 +mZ 2 +nZ+b
[0043] Where k, m, n, and b are preset real coefficients, and k ≠ 0, and (X, Z) represents a point on the road surface;
[0044] The simple polynomial that the camera's optical axis is parallel to the road surface is expressed by the following formula:
[0045]
[0046] Where k′, m′, n′, and b′ are composite parameters of the camera's intrinsic and extrinsic parameters and real coefficients, and (u,v) represents a pixel in the plane;
[0047] The complex polynomial that creates an angle between the camera's optical axis and the road surface is expressed by the following formula:
[0048]
[0049] Where f represents the focal length in pixels, (u′,v′) represents the point position after pitch conversion, and φ represents the angle between the camera's optical axis and the road surface.
[0050] Optionally, the multi-receptive-field feature pyramid module includes a feature pyramid, a linear layer, and a multi-receptive-field depth-separable convolutional layer; the feature pyramid is used to extract visual features at different scales, the linear layer performs dimensionality reduction on the visual features, and the multi-receptive-field depth-separable convolutional layer captures multi-scale spatial information, restores and fuses the processed visual features, and generates an enhanced multi-scale feature pyramid.
[0051] Optionally, the multi-receptive-field depth-separable convolutional layer includes multiple depth-separable convolutional layers with different kernel sizes, wherein each depth-separable convolutional layer first performs depth convolution on the input features, processing each channel independently; then it performs pointwise convolution, linearly combining the channels to capture multi-scale spatial information.
[0052] Optionally, the training module is used to acquire sample data, which includes sample lane line images and sample annotations corresponding to the sample lane line images; input the sample lane line images into the LSTR-CoMer model to obtain recognition results; train the LSTR-CoMer model with the goal of minimizing the deviation between the recognition results and the sample annotations to obtain a lane line recognition model.
[0053] Optionally, the recognition module is used to input the target lane line image into the lane line recognition model, and obtain the original feature map through the Backbone module and the compressed feature map through the patch embedding module, respectively.
[0054] The original feature map is processed by a multi-receptive field feature pyramid module to extract features, resulting in multi-scale feature pyramids with different resolutions; the compressed feature map is then processed by the Transformer module to obtain Transformer features.
[0055] The CNN-Transformer bidirectional interactive fusion module enables bidirectional interaction between the multi-scale feature pyramid and Transformer features, and further facilitates feature interaction between different features in multiple feature extraction backbone modules.
[0056] The lane line positions are obtained by identifying the features obtained after interactive fusion.
[0057] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the lane line detection method described above.
[0058] Fourthly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the lane detection method described above.
[0059] The lane line detection method provided by this invention has the following beneficial effects:
[0060] By employing a multi-receptive-field feature pyramid module and a CNN-Transformer bidirectional fusion module to perform multi-scale feature recognition and fusion on lane line images, the accuracy of lane line detection is significantly improved. Multi-scale feature extraction and effective fusion of local and global features enable the model to accurately identify lane lines in complex scenes. Furthermore, the combination of multi-receptive-field features extracted by the multi-receptive-field feature pyramid module and the global contextual information capture capability of the Transformer module allows the model to perform excellently in complex environments such as lighting changes, occlusion, and lane line wear, exhibiting strong robustness and adaptability, further enhancing the accuracy of lane line detection. Attached Figure Description
[0061] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for the present invention will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 This is a schematic flowchart of a lane line detection method provided by the present invention according to an exemplary embodiment.
[0063] Figure 2This is a block diagram of a lane line detection device provided by the present invention according to an exemplary embodiment. Detailed Implementation
[0064] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.
[0065] In lane detection research, acquiring multi-scale features and enhancing the interaction of feature information within the model are two urgent tasks. The former can capture lane line information of various sizes and improve the perception of lane line shape and position, while the latter can simultaneously focus on details and overall structure in the image, thus enabling accurate lane line detection even in complex scenes such as curves, occlusions, and different lighting conditions. Therefore, this invention focuses on solving the above problems and designs a lane line detection method based on LSTR and visual converter with convolutional multi-scale feature interaction Vi t-CoMer.
[0066] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0067] In a first aspect, the present invention provides a lane line detection method, such as... Figure 1 As shown, it includes the following steps:
[0068] S101. Obtain the target lane line image.
[0069] Specifically, the image can be acquired through a vehicle-mounted camera or other image acquisition devices; this invention does not limit this.
[0070] S102. Construct the LSTR-CoMer model.
[0071] Specifically, an LSTR model is first constructed, which includes an input module, multiple feature extraction backbone modules, and a feature recognition module. A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module, forming the multi-scale enhanced feature fusion lane detection model LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through this module. The features extracted from the multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. Finally, the feature recognition module identifies the features obtained after interactive fusion.
[0072] The Patch Embedding module is used to compress the input lane line image to 1 / 16 of the original resolution and obtain the compressed feature map.
[0073] In one embodiment, the feature extraction backbone module further includes a Transformer module, which consists of an encoder and a decoder. The encoder includes a self-attention module and a feedforward neural network. The self-attention module is used to capture long-range dependencies between positions in the input feature sequence. The feedforward neural network performs a non-linear transformation on the attention output at each position. The decoder includes a self-attention module, a cross-attention module, and a feedforward neural network. The cross-attention module selectively focuses on relevant features in the encoder output by calculating the attention score between the decoder query sequence and the encoder output.
[0074] In addition, the feature recognition module includes a lane line shape representation; the lane line shape representation uses a polynomial form to represent the shape parameters of the lane line; the polynomial form includes a cubic curve polynomial under normal conditions, a simple polynomial where the camera optical axis is parallel to the road surface, and a complex polynomial where the camera optical axis is at an angle to the road surface.
[0075] The cubic polynomial under normal conditions is calculated using the following formula:
[0076] X = kZ 3 +mZ 2 +nZ+b
[0077] Where k, m, n, and b are preset real coefficients, and k ≠ 0, and (X, Z) represents a point on the road surface.
[0078] The simple polynomial that the camera's optical axis is parallel to the road surface is calculated using the following formula:
[0079]
[0080] Where k′, m′, n′, and b′ are composite parameters of the camera's intrinsic and extrinsic parameters and real coefficients, and (u,v) represents a pixel in the plane.
[0081] The complex polynomial that creates an angle between the camera's optical axis and the road surface is calculated using the following formula:
[0082]
[0083] Where f represents the focal length in pixels, (u′,v′) represents the point position after pitch conversion, and φ represents the angle between the camera's optical axis and the road surface.
[0084] In another embodiment, the multi-receptive-field feature pyramid module includes a feature pyramid, a linear layer, and a multi-receptive-field depth-separable convolutional layer. The feature pyramid is used to extract visual features at different scales from the lane line feature map, helping the model to better adapt to lane lines of different widths and distances. The linear layer performs dimensionality reduction on the visual features, the multi-receptive-field depth-separable convolutional layer captures multi-scale spatial information, and finally the processed visual features are restored and fused to generate an enhanced multi-scale feature pyramid.
[0085] The multi-receptive-field depth-separable convolutional layer comprises multiple depth-separable convolutional layers with different kernel sizes. Each depth-separable convolutional layer first performs depthwise convolution on the input features, processing each channel independently; then it performs pointwise convolution, linearly combining the channels to capture multi-scale spatial information. This design allows the network to capture spatial information at different scales through convolutional kernels of different sizes, thereby perceiving both small-scale details and large-scale contextual information.
[0086] Furthermore, the flow relationship of the input feature map among the feature pyramid, linear layers, and multi-receptive-field depth-separable convolutional layers satisfies the following equation:
[0087] F = FC(DWConv(FC(C)))
[0088] Where C represents the feature map in the input feature pyramid, FC represents a linear layer, and DWConv represents a multi-receptive-field deep separable convolution. F represents the feature sequence output by the multi-receptive-field feature pyramid module.
[0089] In another embodiment, the CNN-Transformer bidirectional fusion module includes a convolutional neural network feature extraction unit, a location embedding layer, a cross-attention module, and a feedforward neural network. The location embedding layer includes location encoding based on sine and cosine functions to add unique location information to each input feature. The cross-attention module uses self-attention layers and cross-attention layers to calculate attention scores between different feature sequences, promoting the effective fusion of local and global features.
[0090] A convolutional neural network (CNN) feature extraction unit extracts local features from the input image and combines this with a location embedding layer to preserve positional information. A cross-attention module calculates the dependencies between positions in the feature sequence. The CNN features are then fused with the output of the Transformer encoder. A feedforward neural network performs a non-linear transformation on the fused features to generate the final output features. The self-attention layer and cross-attention layer employ a multi-head attention mechanism to enhance the model's ability to capture complex relationships between features. This further strengthens the model's focus on lane line details, thereby obtaining multi-scale features with rich semantic information.
[0091] S103. Train the LSTR-CoMer model to obtain a lane line recognition model.
[0092] Specifically, sample data can be obtained first, including sample lane line images and corresponding sample annotations. The sample lane line images are then input into the LSTR-CoMer model to obtain recognition results. The LSTR-CoMer model is trained with the goal of minimizing the deviation between the recognition results and the sample annotations to obtain the lane line recognition model.
[0093] Among these methods, the TuSimple dataset, which has a high scene focus, can be used as sample data for model training and validation. Data augmentation methods can be used to enhance the diversity of the sample data, and the CULane dataset, which has a high scene complexity, can be used to assist model training to improve detection capabilities in extreme environments.
[0094] For example, the diversity of sample data can be improved by methods such as horizontal flipping, random rotation, horizontal translation, random cropping, and color jittering.
[0095] S104. Input the target lane line image into the lane line recognition model to identify the lane line position.
[0096] Specifically, the target lane line image is input into the lane line recognition model. The Backbone module identifies the original feature map, and the patch embedding module compresses the target lane line image to obtain a compressed feature map. The multi-receptive field feature pyramid module extracts features from the original feature map, resulting in a multi-scale feature pyramid with different resolutions. The compressed feature map is then processed by the Transformer module to obtain Transformer features, which include global features, positional features, multi-scale features, edge and shape features, and robustness features. A CNN-Transformer bidirectional interactive fusion module facilitates bidirectional interaction between the multi-scale feature pyramid and the Transformer features, and further integrates different features from multiple feature extraction backbone modules. The fused features are then used to identify the lane line position.
[0097] Using the above method, multi-scale feature recognition and fusion of lane line images are performed through a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional fusion module, significantly improving the accuracy of lane line detection. Multi-scale feature extraction and effective fusion of local and global features enable the model to accurately identify lane lines in complex scenes. Furthermore, combining the multi-receptive field features extracted by the multi-receptive field feature pyramid module with the global context information capture capability of the Transformer module, the model performs excellently in complex environments such as lighting changes, occlusion, and lane line wear, exhibiting strong robustness and adaptability, further enhancing the accuracy of lane line detection.
[0098] Secondly, the present invention provides a lane line detection device, such as... Figure 2 As shown, it includes:
[0099] The acquisition module 201 is used to acquire the target lane line image.
[0100] Module 202 is used to construct an LSTR model, which includes an input module, multiple feature extraction backbone modules, and a feature recognition module. A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module to form a multi-scale enhanced feature fusion lane detection model, LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through this module. The features extracted from the multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. The feature recognition module identifies the features obtained after interactive fusion.
[0101] Training module 203 is used to train the LSTR-CoMer model to obtain a lane line recognition model.
[0102] The recognition module 204 is used to input the target lane line image into the lane line recognition model to identify the lane line position.
[0103] In addition, the feature extraction backbone module also includes a Transformer module, which consists of an encoder and a decoder. The encoder includes a self-attention module and a feedforward neural network. The self-attention module is used to capture long-range dependencies between various positions in the input feature sequence. The feedforward neural network performs a non-linear transformation on the attention output at each position. The decoder includes a self-attention module, a cross-attention module, and a feedforward neural network. The cross-attention module selectively focuses on relevant features in the encoder output by calculating the attention score between the decoder query sequence and the encoder output.
[0104] Specifically, the shape of the lane line is represented by a polynomial form, which includes a cubic curve polynomial under normal conditions, a simple polynomial where the camera optical axis is parallel to the road surface, and a complex polynomial where the camera optical axis is at an angle to the road surface.
[0105] The cubic polynomial under normal conditions is calculated using the following formula:
[0106] X = kZ 3 +mZ 2 +nZ+b
[0107] Where k, m, n, and b are preset real coefficients, and k ≠ 0, and (X, Z) represents a point on the road surface;
[0108] The simple polynomial that the camera's optical axis is parallel to the road surface is calculated using the following formula:
[0109]
[0110] Where k′, m′, n′, and b′ are composite parameters of the camera's intrinsic and extrinsic parameters and real coefficients, and (u,v) represents a pixel in the plane;
[0111] The complex polynomial that creates an angle between the camera's optical axis and the road surface is calculated using the following formula:
[0112]
[0113] Where f represents the focal length in pixels, and (u′, v′) represents the corresponding point position after pitch conversion.
[0114] Furthermore, the multi-receptive-field feature pyramid module includes a feature pyramid, a linear layer, and a multi-receptive-field depth-separable convolutional layer. The feature pyramid is used to extract visual features at different scales from the lane line feature map. The linear layer performs dimensionality reduction on the visual features. The multi-receptive-field depth-separable convolutional layer captures multi-scale spatial information, restores and fuses the processed visual features, and generates an enhanced multi-scale feature sequence.
[0115] The multi-receptive-field depth-separable convolutional layer includes multiple depth-separable convolutional layers with different kernel sizes. Each depth-separable convolutional layer first performs depth convolution on the input features, processing each channel independently; then it performs pointwise convolution, linearly combining the channels to capture multi-scale spatial information.
[0116] In addition, the training module 203 is used to acquire sample data, which includes sample lane line images and sample annotations corresponding to the sample lane line images; input the sample lane line images into the LSTR-CoMer model to obtain recognition results; train the LSTR-CoMer model with the optimization objective of minimizing the deviation between the recognition results and the sample annotations to obtain a lane line recognition model.
[0117] Optionally, the recognition module 204 is used to input the target lane line image into the lane line recognition model, and obtain the original feature map by the Backbone module, and obtain the compressed feature map by the patch embedding module to compress and recognize the target lane line image.
[0118] Feature extraction is performed on the original feature map using a feature extraction backbone to obtain feature pyramids with different resolutions. The feature pyramids are then enhanced using a multi-receptive-field feature pyramid module. The compressed feature map is then processed by the Transformer module to obtain Transformer features.
[0119] The CNN-Transformer bidirectional interactive fusion module enables bidirectional interaction between the feature pyramid and Transformer features; different features in multiple feature extraction backbone modules then interact with each other.
[0120] The lane line positions are obtained by identifying the features obtained after interactive fusion.
[0121] Using the aforementioned device, multi-scale feature recognition and fusion of lane line images are performed through a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional fusion module, significantly improving the accuracy of lane line detection. Multi-scale feature extraction and effective fusion of local and global features enable the model to accurately identify lane lines in complex scenes. Furthermore, combining the multi-receptive field features extracted by the multi-receptive field feature pyramid module with the global context information capture capability of the Transformer module, the model performs excellently in complex environments such as lighting changes, occlusion, and lane line wear, exhibiting strong robustness and adaptability, further enhancing the accuracy of lane line detection.
[0122] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The lane line detection method provided.
[0123] Fourthly, the present invention also provides a computer device. At the hardware level, this computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above-mentioned functions. Figure 1 The lane line detection method provided.
[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0128] It should be noted that the above-described specific embodiments enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A lane line detection method, characterized in that, include: Acquire the target lane line image; Construct an LSTR model, which includes an input module, multiple feature extraction backbone modules, and a feature recognition module; A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module to form the multi-scale enhanced feature fusion lane detection model LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through the multi-receptive field feature pyramid module. The features extracted from multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. The feature recognition module identifies the features obtained after interactive fusion. Train the LSTR-CoMer model to obtain a lane line recognition model; The target lane line image is input into the lane line recognition model to identify the lane line location.
2. The lane line detection method according to claim 1, characterized in that, The feature extraction backbone module also includes a Transformer module, which consists of an encoder and a decoder. The encoder includes a self-attention module and a feedforward neural network. The self-attention module is used to capture long-range dependencies between various positions in the input feature sequence. The feedforward neural network performs a non-linear transformation on the attention output at each position. The decoder includes a self-attention module, a cross-attention module, and a feedforward neural network. The cross-attention module selectively focuses on relevant features in the encoder output by calculating the attention score between the decoder query sequence and the encoder output.
3. The lane line detection method according to claim 1, characterized in that, The feature recognition module includes a lane line shape representation; the lane line shape representation uses a polynomial form to represent the shape parameters of the lane line; the polynomial form includes a cubic curve polynomial under normal conditions, a simple polynomial where the camera optical axis is parallel to the road surface, and a complex polynomial where the camera optical axis has an angle with the road surface. The cubic polynomial under normal conditions is expressed by the following formula: X=kZ 3 +mZ 2 +nZ+b Where k, m, n, and b are preset real coefficients, and k ≠ 0, (X, Z) represents the coordinates of a point on the road surface; The simple polynomial that the camera's optical axis is parallel to the road surface is expressed by the following formula: Where k′, m′, n′, and b′ are composite parameters of the camera's intrinsic and extrinsic parameters and real coefficients, and (u,v) represents a pixel in the plane; The complex polynomial that creates an angle between the camera's optical axis and the road surface is expressed by the following formula: Where f represents the focal length in pixels, (u′,v′) represents the point position after pitch conversion, and φ represents the angle between the camera's optical axis and the road surface.
4. The lane line detection method according to claim 1, characterized in that, The multi-receptive-field feature pyramid module includes a feature pyramid, a linear layer, and a multi-receptive-field depth-separable convolutional layer. The feature pyramid is used to extract visual features at different scales from the lane line feature map. The linear layer performs dimensionality reduction on the visual features. The multi-receptive-field depth-separable convolutional layer captures multi-scale spatial information, restores and fuses the processed visual features, and generates an enhanced multi-scale feature pyramid.
5. The lane line detection method according to claim 4, characterized in that, The multi-receptive-field depth-separable convolutional layer includes multiple depth-separable convolutional layers with different kernel sizes. Each depth-separable convolutional layer first performs depth convolution on the input features, processing each channel independently; then it performs pointwise convolution, linearly combining the channels to capture multi-scale spatial information.
6. The lane line detection method according to claim 1, characterized in that, The process of training the LSTR-CoMer model to obtain a lane line recognition model capable of recognizing lane lines includes: Acquire sample data, which includes sample lane line images and sample annotations corresponding to the sample lane line images; The sample lane line images are input into the LSTR-CoMer model to obtain the recognition results; The LSTR-CoMer model was trained with the goal of minimizing the deviation between the recognition result and the sample label to obtain the lane line recognition model.
7. The lane line detection method according to claim 1, characterized in that, The step of inputting the target lane line image into the lane line recognition model to obtain the lane line position includes: The target lane line image is input into the lane line recognition model. The original feature map is obtained by the Backbone module, and the compressed feature map is obtained by the patch embedding module. The original feature map is processed by a multi-receptive field feature pyramid module to extract features, resulting in multi-scale feature pyramids with different resolutions; the compressed feature map is then processed by the Transformer module to obtain Transformer features. The CNN-Transformer bidirectional interactive fusion module enables bidirectional interaction between the multi-scale feature pyramid and Transformer features, and further facilitates feature interaction between different features in multiple feature extraction backbone modules. The lane line positions are obtained by identifying the features obtained after interactive fusion.
8. A lane line detection device, characterized in that, include: The acquisition module is used to acquire images of the target lane lines; A construction module is used to build an LSTR model, which includes an input module, multiple feature extraction backbone modules, and a feature recognition module. A patch embedding module is added to the input module of the LSTR model, and a multi-receptive field feature pyramid module and a CNN-Transformer bidirectional interactive fusion module are added to the feature extraction backbone module to form the multi-scale enhanced feature fusion lane detection model LSTR-CoMer. Lane images are acquired in parallel through the patch embedding module and the backbone module in the input module. The multi-receptive field feature pyramid module in the feature extraction backbone module is connected to the backbone module, and a multi-scale feature pyramid of the lane image is extracted through the multi-receptive field feature pyramid module. The features extracted from multiple feature extraction backbone modules are interacted through the CNN-Transformer bidirectional interactive fusion modules at the beginning and end of the feature extraction backbone module. The feature recognition module identifies the features obtained after interactive fusion. The training module is used to train the LSTR-CoMer model to obtain a lane line recognition model; The recognition module is used to input the target lane line image into the lane line recognition model to identify the lane line position.
9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the lane line detection method according to any one of claims 1 to 7.
10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the lane detection method according to any one of claims 1 to 7.