Lane line detection method and system
By segmenting and feature extraction of the current road surface image in lane line detection, integrating global and local features, and performing instance segmentation and clustering, the problem that lane line detection in the prior art is difficult to take into account real-time and accuracy, and efficient and accurate lane line detection is achieved.
Patent Information
- Application Number
- CN202411762360.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-11
AI Technical Summary
The prior art is difficult to take into account the real-time and accuracy of detection in lane line detection. In particular, convolutional neural networks cannot effectively extract the global characteristics of lane lines because the local receptive field cannot effectively extract the global characteristics of lane lines. The calculation overhead of deep learning networks when building the global connection of lane lines.
By segmenting the acquired current road surface image, global and local features are extracted, and instance segmentation and clustering are performed after fusing to obtain lane line detection results. This method reduces the amount of data extracted by global feature through image segmentation, takes into account both local and global information, avoids background interference, and distinguishes lane lines through clustering.
It improves the accuracy and efficiency of lane line detection, meets the dual requirements of real-time and accuracy, reduces calculation overhead, and avoids background interference.
Smart Images
Figure CN119810777A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a lane line detection method and system. Background Art
[0002] In recent years, with the rapid development of the new energy vehicle industry, the development and research of autonomous driving technology, as one of the core competitiveness of automobiles, has attracted extensive attention from domestic and foreign automobile companies and universities. Among them, the basis of autonomous driving is accurate environmental perception, and lane line detection, as an important part of vehicle perception, can provide data guarantee for downstream tasks such as path planning, lane keeping, and behavioral decision-making. Therefore, obtaining reliable, stable, and accurate lane line perception results is of great significance to promoting the development of autonomous driving technology.
[0003] In the prior art, the mainstream lane line detection methods are divided into two types, one is lane line recognition based on image processing, and the other is lane line detection based on deep learning theory. Among them, the method based on image processing for detection has shortcomings that are difficult to make up for in practical applications, such as poor detection accuracy and poor generalization. The lane line detection method based on deep learning is limited by the characteristics of the network. When detecting lane lines, it is impossible to consider the real-time and accuracy of the detection at the same time. For example, when using a convolutional neural network to detect lane lines, because the convolutional neural network only has a local receptive field, it is impossible to build a connection between the closer and farther parts of the same lane line, and thus the feature extraction effect of a long object such as a lane line is not good. When using a deep learning network method with a global receptive field to build a connection between the closer and farther parts of the lane line, a large amount of computing processing is required, resulting in too much computing overhead and the problem of being unable to meet the real-time requirements of lane line detection. Summary of the invention
[0004] In order to solve the above technical problems, the present invention discloses a lane line detection method and system, which are used to meet the real-time requirements and accuracy requirements of lane line detection.
[0005] In order to achieve the above object, the present invention discloses a lane line detection method, comprising:
[0006] Segmenting the acquired current road surface image to acquire a first road surface image and a second road surface image;
[0007] Extracting global features of lane lines corresponding to the first road surface image and local features of lane lines corresponding to the second road surface image;
[0008] Fusion of the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image;
[0009] Performing instance segmentation on the lane feature map to obtain a lane segmentation map;
[0010] The lane line segmentation map is clustered to distinguish and construct each lane line to obtain a lane line detection result of the current road surface image.
[0011] The present invention discloses a lane line detection method, which first divides the current road surface image into a first road surface image for global feature extraction and a second road surface image for local feature extraction, so as to reduce the amount of data processed by the global feature extraction through the image segmentation, thereby improving the detection efficiency. Next, by respectively extracting the global features and local features corresponding to the lane lines in different images after segmentation, the acquisition of local information and global information is taken into account, thereby improving the accuracy of lane line detection. Furthermore, in the process of lane line detection, by performing instance segmentation on the lane line feature map that integrates the lane line global features and the lane line local features, a lane line segmentation map containing only the lane lines is extracted, thereby avoiding the interference of the background image in the road surface image on the lane line detection, and improving the detection accuracy. Finally, by clustering the lane line segmentation map to distinguish and construct each lane line, the accuracy of lane line detection is improved.
[0012] As a preferred example, before segmenting the acquired current road surface image, the process includes:
[0013] Constructing an initial deep learning model and a model training data set; wherein the model training data set includes a plurality of training road images and a lane line label corresponding to each of the training road images;
[0014] Inputting the training road surface image into the initial deep learning model, so as to extract a lane line global feature map and a lane line local feature map corresponding to the training road surface image through the initial deep learning model;
[0015] The lane line global feature map and the lane line local feature map are fused by the initial deep learning model, so as to output a lane line detection result corresponding to the training road surface image according to the fused lane line feature map;
[0016] Calculating a training loss corresponding to the initial deep learning model according to the lane line label and the lane line detection result, so as to iteratively adjust a model weight of the initial deep learning model according to the training loss;
[0017] When the training loss is less than or equal to a preset loss threshold, the current model weight corresponding to the training loss is obtained to obtain a lane line detection model by combining the model weight and the initial deep learning model.
[0018] The present invention constructs an initial deep learning model and trains it so that the model learns the characteristics of the road surface image, and then performs lane line detection through the model obtained after training to ensure the accuracy and efficiency of the detection. Furthermore, during the training process, the weight of the model is continuously adjusted by calculating the difference between the output of the model and the true value, thereby improving the accuracy of the model and thus improving the accuracy of the lane line detection method.
[0019] As a preferred example, the step of segmenting the acquired current road surface image to acquire the first road surface image and the second road surface image includes:
[0020] Performing data enhancement processing on the acquired current road surface image to obtain an enhanced road surface image;
[0021] The enhanced road surface image is input into a pre-trained lane line detection model, so as to cross-segment the enhanced road surface image through the lane line detection model to obtain a first road surface image and a second road surface image.
[0022] The present invention enhances the data of the current road surface image to improve the accuracy of feature extraction of the image by the subsequent model, and then inputs the enhanced road surface image into a pre-trained lane line detection model for image segmentation to reduce the computational burden and improve the detection efficiency.
[0023] As a preferred example, the extracting the lane line global features corresponding to the first road surface image and the lane line local features corresponding to the second road surface image includes:
[0024] Performing window division on the first road surface image and the second road surface image respectively by using an image segmentation layer preset in the lane line detection model to obtain a first reduced-dimensionality road surface image corresponding to the first road surface image and a second reduced-dimensionality road surface image corresponding to the second road surface image;
[0025] Inputting the first reduced-dimensionality road surface image into a first branch preset in the lane line detection model, so as to extract the lane line global features corresponding to the first reduced-dimensionality road surface image through the first branch;
[0026] The second reduced-dimensionality road surface image is input into a second branch preset in the lane line detection model, so as to extract lane line local features corresponding to the second reduced-dimensionality road surface image through the second branch.
[0027] The present invention sets different branches in the lane line detection model for extracting different features at the same time, thereby improving the detection efficiency. At the same time, the branches are set to extract local features and global features respectively, so as to simultaneously identify the global information and local information of the lane line, thereby ensuring the accuracy of lane line detection.
[0028] As a preferred example, the extracting the lane line global features corresponding to the first dimensionality-reduced road surface image through the first branch includes:
[0029] The first branch is used to divide the first reduced-dimensional road surface image into a plurality of patches, and each of the patches is flattened and mapped into a vector of a fixed dimension in turn to obtain a spatial vector corresponding to the first reduced-dimensional road surface image;
[0030] The spatial vector is subjected to in-window feature extraction through the window multi-head self-attention mechanism and the multi-layer perceptron preset in the first branch, and information is shared through the sliding window mechanism preset in the first branch to generate the lane line global feature corresponding to the first dimensionality reduced road surface image.
[0031] In order to extract the global information in the image, the present invention first divides the image into a series of patches of fixed size and flattens them and maps them into a vector of fixed dimension to reduce the computational cost of the data. Subsequently, a window multi-head self-attention mechanism and a multi-layer perceptron are used to extract features within the window and a sliding window mechanism is used to share information across windows to establish long-range connections between features. This can enhance the feature extraction capability for longer objects in the image, thereby improving the accuracy of the extracted global features and further improving the accuracy of detection.
[0032] As a preferred example, the extracting the lane line local features corresponding to the second dimensionality-reduced road surface image through the second branch includes:
[0033] enhancing edge information of the second reduced-dimensionality road surface image by using an edge enhancement algorithm preset in the second branch to obtain an edge enhanced image;
[0034] Extracting initial local features corresponding to the edge enhanced image by using a deformable convolution preset in the second branch;
[0035] The initial local features are adjusted by a standard convolution preset in the second branch to obtain lane line local features corresponding to the second reduced-dimensionality road surface image.
[0036] The present invention enhances the edge information of local features to efficiently extract the local features of lane lines. Furthermore, deformable convolution is used to extract lane line features with complex and changeable shapes. Finally, standard convolution is used to adjust the local features to reduce the amount of calculation while maintaining the richness of the features.
[0037] As a preferred example, the fusing of the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image includes:
[0038] Merging the lane line global feature and the lane line local feature by using a merging function preset in the lane line detection model;
[0039] Performing feature fusion on the combined feature data by using a standard convolution preset in the lane line detection model to generate lane line features corresponding to the current road surface image;
[0040] The lane line feature is added to the current road surface image to obtain a lane line feature map corresponding to the current road surface image.
[0041] The present invention completes local information enhancement and global information acquisition by fusing the global features and the local features, which is beneficial to improving the accuracy of lane line detection. Furthermore, the fused features are added to the initially acquired image to achieve the effect of residual connection and improve the detection accuracy.
[0042] As a preferred example, the performing instance segmentation on the lane feature map to obtain a lane segmentation map includes:
[0043] Separating lane line pixels and background pixels from the lane line feature map through a binary segmentation network preset in the lane line detection model to obtain lane line pixels;
[0044] The lane line pixels are mapped to an embedding space through the lane line detection model to generate a lane line segmentation map.
[0045] The present invention performs pixel segmentation on the lane feature map, reduces the computational cost and avoids the influence of background pixels on lane detection, thereby improving the detection accuracy. Furthermore, by mapping lane pixels to an embedding space, pixels belonging to the same lane have similar embedding vectors, thereby helping the model to more accurately identify and separate lanes in subsequent clustering.
[0046] As a preferred example, clustering the lane line segmentation map to distinguish and construct each lane line to obtain a lane line detection result of the current road surface image includes:
[0047] Clustering each pixel in the lane line segmentation map by using a clustering algorithm preset in the lane line detection model;
[0048] According to the clustering result, each lane line in the current road surface image is constructed and outputted through the lane line detection model.
[0049] The present invention utilizes a clustering algorithm to detect lane lines, thereby improving the efficiency of lane line detection while ensuring the accuracy of lane line recognition.
[0050] On the other hand, the present invention discloses a lane line detection system, including an image segmentation module, a feature extraction module, a feature fusion module, an instance segmentation module and a lane line detection module;
[0051] The image segmentation module is used to segment the acquired current road surface image to acquire a first road surface image and a second road surface image;
[0052] The feature extraction module is used to extract the global features of the lane lines corresponding to the first road surface image and the local features of the lane lines corresponding to the second road surface image;
[0053] The feature fusion module is used to fuse the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image;
[0054] The instance segmentation module is used to perform instance segmentation on the lane feature map to obtain a lane segmentation map;
[0055] The lane line detection module is used to cluster the lane line segmentation map to distinguish and construct each lane line, so as to obtain the lane line detection result of the current road surface image.
[0056] The present invention discloses a lane line detection system, which first divides the current road surface image into a first road surface image for global feature extraction and a second road surface image for local feature extraction, so as to reduce the amount of data processed by the global feature extraction through the image segmentation, thereby improving the detection efficiency. Next, by respectively extracting the global features and local features corresponding to the lane lines in different images after segmentation, the acquisition of local information and global information is taken into account, thereby improving the accuracy of lane line detection. Furthermore, in the process of lane line detection, by performing instance segmentation on the lane line feature map that integrates the lane line global features and the lane line local features, a lane line segmentation map containing only the lane lines is extracted, thereby avoiding the interference of the background image in the road surface image on the lane line detection, and improving the detection accuracy. Finally, by clustering the lane line segmentation map to distinguish and construct each lane line, the accuracy of lane line detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 : is a flow chart of a lane line detection method disclosed in an embodiment of the present invention;
[0058] Figure 2 : is a structural schematic diagram of a lane line detection system disclosed in an embodiment of the present invention;
[0059] Figure 3 : A schematic flow chart of a lane line detection method disclosed in another embodiment of the present invention;
[0060] Figure 4 : is a structural schematic diagram of a lane line detection model disclosed in another embodiment of the present invention;
[0061] Figure 5 : A schematic diagram of the structure of a backbone network disclosed in another embodiment of the present invention;
[0062] Figure 6 : A schematic diagram of a double-branch structure disclosed in another embodiment of the present invention. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] Embodiment 1
[0065] This embodiment discloses a lane line detection method for improving the accuracy and efficiency of lane line detection. Figure 1 , mainly including steps 101 to 105, the steps are:
[0066] Step 101: segment the acquired current road surface image to acquire a first road surface image and a second road surface image.
[0067] Step 102: extracting the global features of the lane lines corresponding to the first road surface image and the local features of the lane lines corresponding to the second road surface image.
[0068] Step 103: Fusing the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image.
[0069] Step 104: Perform instance segmentation on the lane feature map to obtain a lane segmentation map.
[0070] Step 105: Clustering the lane line segmentation map to distinguish and construct each lane line, so as to obtain a lane line detection result of the current road surface image.
[0071] In some implementations of this embodiment, before executing steps 101 to 105, a lane detection model based on deep learning is first trained to execute steps 101 to 105 through the lane detection model. Specifically, the training process of the lane detection model includes steps S1 to S5;
[0072] Step S1: construct an initial deep learning model and a model training data set; wherein the model training data set includes a plurality of training road images and lane line labels corresponding to each of the training road images.
[0073] Step S2: Inputting the training road surface image into the initial deep learning model, so as to extract a lane line global feature map and a lane line local feature map corresponding to the training road surface image through the initial deep learning model.
[0074] Step S3: Fusing the lane line global feature map and the lane line local feature map through the initial deep learning model to output the lane line detection result corresponding to the training road surface image according to the fused lane line feature map.
[0075] Step S4: Calculate the training loss corresponding to the initial deep learning model according to the lane line label and the lane line detection result, so as to iteratively adjust the model weight of the initial deep learning model according to the training loss.
[0076] Step S5: When the training loss is less than or equal to a preset loss threshold, the current model weight corresponding to the training loss is obtained to obtain a lane line detection model by combining the model weight and the initial deep learning model.
[0077] In this embodiment, steps S1 to S5 construct an initial deep learning model and train it so that the model learns the features of the road image, and then performs lane line detection through the model obtained after training to ensure the accuracy and efficiency of the detection. Furthermore, during the training process, the weight of the model is continuously adjusted by calculating the difference between the output of the model and the true value, thereby improving the accuracy of the model and thus improving the accuracy of the lane line detection method.
[0078] In some implementations of this embodiment, after the lane detection model is constructed as shown in steps S1 to S5, the process of executing step 101 through the lane detection model mainly includes steps S6 and S7;
[0079] S6: Perform data enhancement processing on the acquired current road surface image to obtain an enhanced road surface image.
[0080] S7: Inputting the enhanced road surface image into a pre-trained lane line detection model to cross-segment the enhanced road surface image through the lane line detection model to obtain a first road surface image and a second road surface image.
[0081] In this embodiment, steps S6 and S7 perform data enhancement on the current road surface image to improve the accuracy of feature extraction of the image by the subsequent model, and then input the enhanced road surface image into the pre-trained lane line detection model for image segmentation to reduce the computational burden and improve the detection efficiency.
[0082] Further, after generating the first road surface image and the second road surface image, the process of extracting the lane line global features and the lane line local features in step 102 includes steps S8 to S10;
[0083] S8: Performing window division on the first road surface image and the second road surface image respectively through the image segmentation layer preset in the lane line detection model to obtain a first reduced-dimensionality road surface image corresponding to the first road surface image and a second reduced-dimensionality road surface image corresponding to the second road surface image.
[0084] S9: Inputting the first reduced-dimensionality road surface image into a first branch preset in the lane line detection model, so as to extract the lane line global features corresponding to the first reduced-dimensionality road surface image through the first branch.
[0085] S10: Inputting the second reduced-dimensionality road surface image into a second branch preset in the lane line detection model, so as to extract lane line local features corresponding to the second reduced-dimensionality road surface image through the second branch.
[0086] In this embodiment, steps S8 to S9 extract different features at the same time through different branches set in the lane line detection model, thereby improving the detection efficiency. At the same time, the branches are set to extract local features and global features respectively, so as to simultaneously identify the global information and local information of the lane line, thereby ensuring the accuracy of lane line detection.
[0087] Furthermore, in the step S9, the process of extracting the lane line global features corresponding to the first dimension-reduced road surface image through the first branch specifically includes step S91 and step S92;
[0088] Step S91: segmenting the first reduced-dimensionality road image into a plurality of patches through the first branch, and flattening each of the patches in turn and mapping them into a vector of a fixed dimension to obtain a spatial vector corresponding to the first reduced-dimensionality road image.
[0089] Step S92: Extract in-window features of the spatial vector through the window multi-head self-attention mechanism and multi-layer perceptron preset in the first branch, and share information through the sliding window mechanism preset in the first branch to generate global lane line features corresponding to the first reduced-dimensional road surface image.
[0090] In this embodiment, step S91 and step S92 are to extract global information from the image. First, the image is divided into a series of patches of fixed size and flattened and mapped into a vector of fixed dimension to reduce the computational cost of the data. Subsequently, a window multi-head self-attention mechanism and a multi-layer perceptron are used to extract features within the window and a sliding window mechanism is used to share information across windows to establish long-range connections between features. This can enhance the feature extraction capability for longer objects in the image, thereby improving the accuracy of the extracted global features and thereby improving the accuracy of detection.
[0091] The process of extracting the lane line local features corresponding to the second reduced-dimensionality road surface image through the second branch in step S10 specifically includes step S101 and step S103;
[0092] Step S101: enhancing edge information of the second reduced-dimensionality road surface image by using an edge enhancement algorithm preset in the second branch to obtain an edge enhanced image.
[0093] Step S102: extracting initial local features corresponding to the edge enhanced image through the deformable convolution preset in the second branch.
[0094] Step S103: adjusting the initial local features through the standard convolution preset in the second branch to obtain the lane line local features corresponding to the second reduced-dimensional road surface image.
[0095] In this embodiment, steps S101 to S103 enhance the edge information of local features to efficiently extract the local features of the lane lines. Furthermore, deformable convolution is used to extract lane line features with complex and changeable shapes. Finally, standard convolution is used to adjust the local features to reduce the amount of calculation while maintaining the richness of the features.
[0096] In some implementations of this embodiment, after extracting the global features and local features, the process of fusing the features in step 103 to obtain a lane feature map corresponding to the current road surface image includes steps S11 to S13;
[0097] Step S11: merging the lane line global features and the lane line local features through a merging function preset in the lane line detection model.
[0098] Step S12: performing feature fusion on the feature data obtained after merging through the standard convolution preset in the lane line detection model to generate lane line features corresponding to the current road surface image.
[0099] Step S13: Add the lane line feature to the current road surface image to obtain a lane line feature map corresponding to the current road surface image.
[0100] In this embodiment, steps S11 to S13 complete local information enhancement and global information acquisition by fusing the global features and the local features, which is beneficial to improving the accuracy of lane line detection. Furthermore, the fused features are added to the initially acquired image to achieve the effect of residual connection and improve the detection accuracy.
[0101] In some implementations of this embodiment, after obtaining the lane feature map, the process of obtaining the lane segmentation map in step 104 includes steps S14 and S15;
[0102] Step S14: Separating lane line pixels and background pixels from the lane line feature map through a binary segmentation network preset in the lane line detection model to obtain lane line pixels.
[0103] Step S15: Mapping the lane line pixels to an embedding space through the lane line detection model to generate a lane line segmentation map.
[0104] In this embodiment, step S14 and step S15 perform pixel segmentation on the lane feature map, thereby reducing the computational cost and avoiding the influence of background pixels on lane detection, thereby improving the detection accuracy. Furthermore, by mapping lane pixels to an embedding space, pixels belonging to the same lane have similar embedding vectors, thereby helping the model to more accurately identify and separate lanes in subsequent clustering.
[0105] In some implementations of this embodiment, the process of obtaining the lane line detection result of the current road surface image in step 105 includes steps S16 and S17;
[0106] Step S16: Clustering each pixel in the lane line segmentation map using a clustering algorithm preset in the lane line detection model.
[0107] Step S17: Based on the clustering result, each lane line in the current road surface image is constructed and outputted through the lane line detection model.
[0108] In this embodiment, step S16 and step S17 use a clustering algorithm to detect lane lines, thereby improving the efficiency of lane line detection while ensuring the accuracy of lane line recognition.
[0109] On the other hand, this embodiment also discloses a lane line detection system. For the specific structure of the detection system, please refer to Figure 2 , including an image segmentation module 201, a feature extraction module 202, a feature fusion module 203, an instance segmentation module 204 and a lane line detection module 205.
[0110] The image segmentation module 201 is used to segment the acquired current road surface image to acquire a first road surface image and a second road surface image.
[0111] The feature extraction module 202 is used to extract the global features of the lane lines corresponding to the first road surface image and the local features of the lane lines corresponding to the second road surface image.
[0112] The feature fusion module 203 is used to fuse the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image.
[0113] The instance segmentation module 204 is used to perform instance segmentation on the lane feature map to obtain a lane segmentation map.
[0114] The lane line detection module 205 is used to cluster the lane line segmentation map to distinguish and construct each lane line, so as to obtain the lane line detection result of the current road surface image.
[0115] The present embodiment discloses a lane line detection method and system, which first divides the current road surface image into a first road surface image for global feature extraction and a second road surface image for local feature extraction, so as to reduce the amount of data processed by the global feature extraction through the image segmentation, thereby improving the efficiency of detection. Next, by respectively extracting the global features and local features corresponding to the lane lines in different images after segmentation, the acquisition of local information and global information is taken into account, thereby improving the accuracy of lane line detection. Furthermore, in the process of lane line detection, by performing instance segmentation on the lane line feature map that integrates the lane line global features and the lane line local features, a lane line segmentation map containing only the lane lines is extracted, thereby avoiding the interference of the background image in the road surface image on the lane line detection, and improving the accuracy of detection. Finally, by clustering the lane line segmentation map to distinguish and construct each lane line, the accuracy of lane line detection is improved.
[0116] Embodiment 2
[0117] With the research on deep learning theory, applying deep learning theory to the field of vehicle driving has become an important research direction. Among them, lane line detection is a hot competitive direction in autonomous driving, and its application of deep learning theory is more extensive. However, in the existing technology, the application of deep learning theory to lane line detection is often limited to the application of a single model, and it is impossible to perform lane line detection comprehensively and multi-facetedly, resulting in the inability to take into account the requirements of real-time and accuracy in lane line detection.
[0118] To solve the above technical problems, this embodiment provides a deep learning model that can simultaneously extract global information and local information, so as to extract the global features and local features of lane lines, while taking into account the real-time requirements and accuracy requirements of lane line detection.
[0119] Specifically, in some implementations of this embodiment, the Transformer-CNN hybrid structure is used to construct the deep learning model. Preferably, the specific process of using the Transformer-CNN hybrid structure to perform lane line detection is shown in FIG. Figure 3 , mainly including steps 301 to 306, the steps are mainly:
[0120] Step 301: construct an initial deep learning model, and train the initial deep learning model using a pre-constructed training data set to generate a lane detection model.
[0121] In this embodiment, this step mainly includes: setting the training parameters of the initial deep learning model, including the learning rate, optimizer, batch size, and number of iterations. Then, data enhancement and preprocessing are performed on each image containing lane lines in the pre-constructed training data set. The training data set includes a number of road images containing lane lines and lane line labels corresponding to each of the road images; wherein the labels represent the lane line information contained in the road image.
[0122] Furthermore, the image obtained after data enhancement and preprocessing is segmented into two parts through channels, and then the two parts are respectively input into the two branches of the backbone network in the initial deep learning model for feature extraction, and finally fused to obtain fused features, which are respectively input into the segmentation prediction module and lane embedding module in the initial deep learning model. The segmentation prediction module outputs the segmentation detection result, and the lane embedding module outputs the lane instance.
[0123] The lane instances are then clustered to obtain lane line detection results. Then, the training loss of the lane line detection results and the lane line information in the corresponding lane line labels is calculated using a preset loss function. The loss function is the same as that of LaneNet. The model is then back-propagated to adjust the weights based on the training loss.
[0124] After each iteration, the model weights are saved and used for the next iteration until the network converges and the optimal model weights are obtained. The network loads the optimal model weights and generates a lane line detection model in combination with the training parameters of the initial deep learning model.
[0125] Step 302: Acquire a current road surface image and perform image segmentation on the current road surface image using the lane line detection model to generate a first road surface image and a second road surface image.
[0126] In this embodiment, this step mainly includes: collecting road images in real time and performing data enhancement processing on the road images. The data enhancement includes rotating, flipping, and randomly cropping the images. After the data enhancement processing, in order to improve the accuracy of lane line detection, the image processed by data enhancement can be preprocessed to improve the resolution of the image. Preferably, the image preprocessing is set to adjust the image resolution size after data enhancement to 576×1024.
[0127] Next, the image after the image preprocessing is input into the lane line detection model, so as to segment the image through the lane line detection model, so as to extract local features and global features respectively. Specifically, in some implementations of this embodiment, the lane line detection model is set to segment the image by cross segmentation. Preferably, the image segmentation process can refer to Figure 4 , Figure 4 A model structure diagram provided for this embodiment.
[0128] like Figure 4 As shown in FIG. 1 , the backbone network module in the lane detection model performs channel segmentation on the image as cross segmentation. The cross segmentation process is defined as in ∈R B×C×H×W , there are x i +y j =f in , where x i ∈R B×C×H×W ,(C=0,2,4…),y i ∈R B×C×H×W ,C=1,3,5…). Among them, f in is the input, f out is the output, x i ,y i are the two-part features of the segmentation, B is the batch size, C is the number of channels, H is the feature height, and W is the feature width. Through channel segmentation, the input is divided into two parts, that is, the input image is divided into the first road surface image and the second road surface image.
[0129] Step 303: extracting the lane line global features corresponding to the first road surface image and the lane line local features corresponding to the second road surface image through the lane line detection model.
[0130] In this embodiment, this step mainly includes: inputting the first road surface image into the first branch of the lane line detection model to extract the global features of the lane line corresponding to the first road surface image through the first branch, and at the same time inputting the second road surface image into the second branch of the lane line detection model to extract the local features of the lane line corresponding to the second road surface image through the second branch.
[0131] Specifically, refer to Figure 4 In some implementations of this embodiment, the first branch is set to a SwinT branch to capture global information, and the second branch is set to a CNN-based edge information enhancement branch to enhance and extract local information.
[0132] Preferably, in the process of capturing the global information and the local information through the first branch and the second branch, in order to further improve the efficiency of detection, a corresponding module can be set to reduce the dimension of the image before feature extraction to reduce the computational cost.
[0133] Specifically, in order to achieve dimensionality reduction and feature extraction of the image, in some implementations of this embodiment, a multi-feature fusion backbone network based on a Transformer-CNN hybrid structure is set in the lane line detection model, and the backbone network includes a patch embedding layer, a Swint-Conv module and a patch merging layer. The combination of the patch embedding layer and the Swint-Conv module, and the combination of the patch merging layer and the Swint-Conv module can constitute multiple stages. Preferably, in some implementations of this embodiment, refer to Figure 5 , which is a backbone network structure diagram corresponding to the lane line detection model when the lane line detection model is set to include 4 stages.
[0134] like Figure 5 As shown, in stage 1, for the input x∈R B×3×H×W , the patch embedding layer first divides it with a 4×4 window to obtain Then x ′ The following three stages are composed of a patch merging layer and several SwinT-Conv modules. We set the number of SwinT-Conv modules in the four stages to [2, 2, 4, 2] respectively. The patch merging layer will downsample the input, reducing both H and W to 1 / 2, and doubling the number of channels C. Therefore, the output of the backbone network is
[0135] Furthermore, in order to realize feature extraction corresponding to different images, a dual-branch structure based on a Transformer-CNN hybrid structure is deployed in the SwinT-Conv module, that is, a SwinT branch and a CNN-based edge information enhancement branch are deployed in the SwinT-Conv module. The specific structural composition of the dual-branch structure can be referred to Figure 6 .
[0136] Specifically, Figure 6 As shown in the figure, after the image is reduced in dimension by the patch embedding layer, the destination branch of the image input is determined according to the number of channels corresponding to the image. When the image is input to the SwinT branch, based on the fact that the SwinT branch is a backbone network based on Transformer, the SwinT branch extracts global features by introducing a sliding window mechanism, so as to learn cross-window knowledge while saving computational effort. Specifically, the SwinT branch first divides the input image into a series of Patches of fixed size and flattens them and maps them to a vector of fixed dimension, then uses window multi-head self-attention (W-MSA) and FFN to extract features within the window, and finally uses a sliding window mechanism to share cross-window information. The purpose of this branch is to establish long-range connections between features and enhance the backbone network's ability to extract features of longer objects in the image.
[0137] When the image is input to another branch, that is, to the edge information enhancement branch based on CNN, the purpose of this branch is to enhance the edge information of local features and efficiently extract the lane line features. Therefore, first of all, the edge information enhancement branch of CNN is an ESAM edge information enhancement module, which is used to enhance the edge information of the image, making the slender lane lines more obvious, which is convenient for subsequent feature extraction; then a 3×3 deformable convolution is used to capture local features. Its variability is more suitable for extracting lane line features with complex and changeable shapes, and it has stronger feature extraction capabilities than standard convolutions; finally, two 1×1 standard convolutions in series are used to adjust the depth of the feature map, while reducing the amount of calculation while maintaining the richness of the features. Through this design, the network's ability to extract lane line features locally is improved.
[0138] Step 304: Fusing the lane line global features and the lane line local features to construct a lane line feature map corresponding to the current road surface image.
[0139] In this embodiment, this step mainly includes: merging the lane line global features and the lane line local features to obtain the lane line fusion features, and adding the lane line fusion features to the current road surface image to generate a lane line feature map.
[0140] In some implementations of this example, reference Figures 4 to 5 In the lane detection model shown, after the SwinT branch and the edge information enhancement branch of the CNN are respectively calculated and output, the local features of the lane and the global features of the lane can be merged by the concatenate function preset in the SwinT-Conv module, and then the merged data is subjected to feature fusion using a 1×1 standard convolution, and finally the fused data is added to the image originally input to the module to obtain a feature map, achieving an effect similar to the residual connection in the convolutional neural network. Thus, the generated feature map completes the local information enhancement and global information acquisition, and is very suitable for extracting the features of objects such as lanes that are thin and long.
[0141] Step 305: Perform pixel segmentation and lane embedding on the lane feature map to obtain a lane instance segmentation map.
[0142] In this embodiment, this step mainly includes: separating lane line pixels and background pixels of the lane line feature map through a binary segmentation network preset in the lane line detection model to obtain lane line pixels; mapping the lane line pixels to an embedding space through the lane line detection model to generate a lane line segmentation map.
[0143] Specifically, in certain implementations of this embodiment, refer to Figures 4 to 5 The lane detection model shown in the figure separates lane line pixels from background pixels through a segmentation prediction module in the lane detection model. The segmentation prediction module is a binary segmentation network based on ENET, which is used to separate lane line pixels from background pixels and uses a standard cross entropy loss function.
[0144] Furthermore, the intrinsic relationship between lane line pixels is learned through the embedding module in the lane line detection model, and the lane line pixels are mapped to the embedding space so that pixels belonging to the same lane line in the embedding space have similar embedding vectors, thereby helping the model to more accurately identify and separate lane lines in subsequent clustering.
[0145] Step 306: Clustering the lane line instance segmentation map through the lane line detection model to construct each lane line and output the lane line detection result of the current road surface image.
[0146] In this embodiment, this step mainly includes: clustering each pixel point in the lane line segmentation map through the clustering algorithm preset in the lane line detection model; according to the clustering result, constructing and outputting each lane line in the current road surface image through the lane line detection model.
[0147] Specifically, in certain implementations of this embodiment, an improved DBSCAN clustering algorithm is applied in the lane line detection model to perform lane line analysis. The improved DBSCAN clustering algorithm takes into account the similar physical intervals between different lane lines, and thus processes each point independently to calculate whether it is in the neighborhood of a certain explicit cluster center point, and then completes the construction of the lane line based on the classification of the pixel points. The improved DBSCAN clustering algorithm does not adopt the process of calculating the neighborhood relationship between points based on recursive expansion in the standard DBSCAN algorithm.
[0148] The lane line detection method provided in this embodiment combines the global information acquisition capability of Transformer and the local feature acquisition capability of CNN to construct a new backbone network with a dual-branch structure. The backbone network uses the Transformer-based SwinT branch to extract global features and establishes global connections between features. While using the CNN-based edge information enhancement branch to enhance the edge information of the local area, the lane line information is efficiently extracted, which can effectively improve the feature extraction capability of objects such as lane lines that are thin and long, and also has good effects in complex environments (occlusion, bad weather, insufficient light, etc.). Furthermore, the channel segmentation design is used to make full use of the dual-branch structure while reducing the computational overhead. The lane line detection method provided in this embodiment has higher accuracy, speed and robustness in lane line detection tasks, and the proposed backbone network can also be used in other lane line detection algorithms.
[0149] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A lane line detection method, characterized in that: include: Segmenting the acquired current road surface image to acquire a first road surface image and a second road surface image; Extracting global features of lane lines corresponding to the first road surface image and local features of lane lines corresponding to the second road surface image; Fusion of the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image; Performing instance segmentation on the lane feature map to obtain a lane segmentation map; The lane line segmentation map is clustered to distinguish and construct each lane line to obtain a lane line detection result of the current road surface image.
2. A lane line detection method according to claim 1, characterized in that: Before segmenting the acquired current road surface image, the method includes: Constructing an initial deep learning model and a model training data set; wherein the model training data set includes a plurality of training road images and a lane line label corresponding to each of the training road images; Inputting the training road surface image into the initial deep learning model, so as to extract a lane line global feature map and a lane line local feature map corresponding to the training road surface image through the initial deep learning model; The lane line global feature map and the lane line local feature map are fused by the initial deep learning model, so as to output a lane line detection result corresponding to the training road surface image according to the fused lane line feature map; Calculating a training loss corresponding to the initial deep learning model according to the lane line label and the lane line detection result, so as to iteratively adjust a model weight of the initial deep learning model according to the training loss; When the training loss is less than or equal to a preset loss threshold, the current model weight corresponding to the training loss is obtained to obtain a lane line detection model by combining the model weight and the initial deep learning model.
3. A lane line detection method according to claim 2, characterized in that: The step of segmenting the acquired current road surface image to acquire the first road surface image and the second road surface image includes: Performing data enhancement processing on the acquired current road surface image to obtain an enhanced road surface image; The enhanced road surface image is input into a pre-trained lane line detection model, so as to cross-segment the enhanced road surface image through the lane line detection model to obtain a first road surface image and a second road surface image.
4. A lane line detection method according to claim 3, characterized in that: The extracting the lane line global features corresponding to the first road surface image and the lane line local features corresponding to the second road surface image includes: Performing window division on the first road surface image and the second road surface image respectively by using an image segmentation layer preset in the lane line detection model to obtain a first reduced-dimensionality road surface image corresponding to the first road surface image and a second reduced-dimensionality road surface image corresponding to the second road surface image; Inputting the first reduced-dimensionality road surface image into a first branch preset in the lane line detection model, so as to extract the lane line global features corresponding to the first reduced-dimensionality road surface image through the first branch; The second reduced-dimensionality road surface image is input into a second branch preset in the lane line detection model, so as to extract lane line local features corresponding to the second reduced-dimensionality road surface image through the second branch.
5. A lane line detection method according to claim 4, characterized in that: The extracting the lane line global features corresponding to the first dimension-reduced road surface image through the first branch includes: The first branch is used to divide the first reduced-dimensional road surface image into a plurality of patches, and each of the patches is flattened and mapped into a vector of a fixed dimension in turn to obtain a spatial vector corresponding to the first reduced-dimensional road surface image; The spatial vector is subjected to in-window feature extraction through the window multi-head self-attention mechanism and the multi-layer perceptron preset in the first branch, and information is shared through the sliding window mechanism preset in the first branch to generate the lane line global feature corresponding to the first dimensionality reduced road surface image.
6. A lane line detection method according to claim 4, characterized in that: The extracting the lane line local features corresponding to the second dimension-reduced road surface image through the second branch includes: enhancing edge information of the second reduced-dimensionality road surface image by using an edge enhancement algorithm preset in the second branch to obtain an edge enhanced image; Extracting initial local features corresponding to the edge enhanced image by using a deformable convolution preset in the second branch; The initial local features are adjusted by a standard convolution preset in the second branch to obtain lane line local features corresponding to the second reduced-dimensionality road surface image.
7. A lane line detection method according to claim 4, characterized in that: The fusing the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image includes: Merging the lane line global feature and the lane line local feature by using a merging function preset in the lane line detection model; Performing feature fusion on the combined feature data by using a standard convolution preset in the lane line detection model to generate lane line features corresponding to the current road surface image; The lane line feature is added to the current road surface image to obtain a lane line feature map corresponding to the current road surface image.
8. A lane line detection method according to claim 7, characterized in that: The performing instance segmentation on the lane feature map to obtain a lane segmentation map includes: Separating lane line pixels and background pixels from the lane line feature map through a binary segmentation network preset in the lane line detection model to obtain lane line pixels; The lane line pixels are mapped to an embedding space through the lane line detection model to generate a lane line segmentation map.
9. A lane line detection method according to claim 8, characterized in that: The clustering of the lane line segmentation map to distinguish and construct each lane line to obtain a lane line detection result of the current road surface image includes: Clustering each pixel in the lane line segmentation map by using a clustering algorithm preset in the lane line detection model; According to the clustering result, each lane line in the current road surface image is constructed and outputted through the lane line detection model.
10. A lane detection system, characterized in that: It includes image segmentation module, feature extraction module, feature fusion module, instance segmentation module and lane detection module; The image segmentation module is used to segment the acquired current road surface image to acquire a first road surface image and a second road surface image; The feature extraction module is used to extract the global features of the lane lines corresponding to the first road surface image and the local features of the lane lines corresponding to the second road surface image; The feature fusion module is used to fuse the lane line global features and the lane line local features to obtain a lane line feature map corresponding to the current road surface image; The instance segmentation module is used to perform instance segmentation on the lane feature map to obtain a lane segmentation map; The lane line detection module is used to cluster the lane line segmentation map to distinguish and construct each lane line, so as to obtain the lane line detection result of the current road surface image.