An end-to-end instance segmentation based lane detection method

Abstract

The application discloses an end-to-end lane detection method based on instance segmentation, which comprises the following steps: firstly, performing data enhancement processing on a lane detection dataset to obtain a training and verification dataset; then, building a lane instance segmentation network model LANet and setting a loss function; meanwhile, introducing an attention gate into the network to fuse lane features of different layers, and designing a lane self-attention module to enhance the expression of lane features; finally, training the model and testing it on the lane detection dataset to complete the lane detection task. The method can fully extract global context information and eliminate relevant lane interference by using an attention network, can cope with the challenge of complex scene interference such as serious lane occlusion and extreme light conditions, and has excellent robustness.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle autonomous driving, and in particular relates to an end-to-end lane detection method based on instance segmentation. Background Art

[0002] With the continuous development of autonomous driving technology, its corresponding environmental perception technology has also received significant attention. Lane detection based on front-of-vehicle visual sensors is one of the most important tasks in the autonomous driving field. This technology helps autonomous vehicles identify the current lane and detect lane changes, playing a vital role in Advanced Driver Assistance Systems (ADAS). However, lane detection has always been a challenging task because the slender shape of lanes and the lack of visual features such as occlusion and poor lighting conditions can hinder accurate lane detection.

[0003] Traditional lane detection algorithms obtain lane information through manual feature extraction from images. While this method offers advantages such as simplicity and fast processing speed, its reliance on manual experience in feature definition and its poor ability to cope with multiple lanes and environmental interference have hindered its development. In contrast, lane detection methods based on deep learning use models to autonomously learn lane features, offering advantages such as good stability and high detection accuracy. Segmentation-based lane detection methods, currently the mainstream and widely studied, employ a pixel-by-pixel prediction approach, extracting features from a given input image and upsampling it before outputting a prediction for each pixel. However, existing segmentation methods still perform poorly in challenging scenarios such as occlusion and poor lighting conditions. They also suffer from large parameter counts and require cumbersome post-processing procedures. Summary of the Invention

[0004] To solve the above problems, the present invention discloses an end-to-end lane detection method based on instance segmentation, which solves the problem that existing lane detection methods perform poorly when dealing with complex scenarios such as severe occlusion and extreme lighting conditions. It can achieve rapid lane detection while ensuring accuracy.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] An end-to-end lane detection method based on instance segmentation includes the following steps:

[0007] Step 1: Obtain a lane detection dataset and perform data augmentation on the road images to obtain training and validation datasets.

[0008] Step 2: Build a lane instance segmentation network model, LANet. This network model uses an encoder-decoder structure, with the encoding structure of ERFNet as the feature extraction network. The decoding network consists of a lane existence branch and a segmentation branch.

[0009] Step 3: Set the loss function to accelerate model convergence and improve the network lane feature mining capability;

[0010] Step 4: Use the training data set to train the model and obtain the best detection model after multiple iterations;

[0011] Step 5: Test on the lane detection dataset, extract lanes from the lane probability map output by the model and overlay it on the original image to visualize lane detection.

[0012] Furthermore, the data enhancement of the road images in the lane detection dataset in step 1 includes: random affine transformation (including translation, rotation and scaling) and random horizontal flipping, and the enhanced dataset image size is adjusted to 288×800, thereby obtaining the training and validation datasets.

[0013] Furthermore, the segmentation branch in step 2 will use the channel attention mechanism and the spatial attention mechanism to eliminate lane interference factors on the features output by the encoding network. The attention gate is introduced to merge the coarse information extracted from the encoder network with the relevant features in the detail information, and output the prediction result M after upsampling. pred At the same time, a lane self-attention module is added as an auxiliary part in the segmentation branch to extract and fuse the lateral and longitudinal features of the lane respectively to enhance the expression of the global contextual features of the lane.

[0014] Furthermore, the attention gate method is as follows: after the input image I is extracted by the encoder network feature, the shallower detail feature F is obtained. ab and the deeper coarse features F bb , both have the same size, and F is obtained by adding them pixel by pixel b+ After that, the activation function and Sigmod normalization are applied to F b+ After processing, the response value F is obtained bp . F bp It can be considered that it contains the semantic information of the context, which determines the importance of each pixel, thereby locating the area where attention is applied and suppressing the feature response of irrelevant areas. bp With F ab Multiply pixel by pixel to get the output result F bag , F bag It can better integrate coarse features and detailed features.

[0015] Furthermore, the auxiliary part extracts features through conv1×1 and conv3×3, and is reduced in size through two maximum pooling operations. The features F are obtained through different fully connected networks. v and F h , and expand them horizontally and vertically to obtain F ev and F eh , and finally the attention probability matrix obtained by fusing the two is M bp =γF ev +(1-γ)F eh Among them, the parameter γ is the probability weight, ranging from 0 to 1. bp More attention will be paid to the location where the lane exists, by comparing it with the segmentation output M pred Lane constraints are implemented by pixel-by-pixel multiplication.

[0016] Furthermore, the loss function in step 3 consists of three parts: segmentation loss, attention loss, and lane existence loss. seg =CE(M pred ,M true ) is used to measure the difference between the predicted value and the true label, where M pred is the segmentation output, M true is the true segmentation label; attention loss L LSA Used to constrain the attention probability output in the auxiliary part to make it pay more attention to the lane position; there is a loss L ext =BCE(F pred ,F true ) is used to further constrain the lane prediction results, F pred is the output of the lane existence branch, F true is the lane existence label. Taking α, β and λ as loss coefficients, the final total loss function can be written as: L = αL seg +βL LSA +λL ext .

[0017] Furthermore, the lane attention loss L LSA The specific expression formula is:

[0018]

[0019] in Represents the multiplication of the true label and the attention probability map, Denotes the multiplication of the predicted value and the attention probability map, n is the training batch size, η is the loss coefficient, Ω is the balance constraint factor, and κ determines the proportion of weighted lane areas and has a significant impact on model performance. The attention loss aims to constrain the difference between the two and use the true label to constrain the segmentation output results.

[0020] Furthermore, in step 4, the set loss function, loss coefficients α, β, and λ, and the learning rate of the training model are substituted into the lane instance segmentation network model, and the lane detection dataset is input into the model for training according to the batch size. After the training model converges, the detection model is obtained.

[0021] The beneficial effects of the present invention are:

[0022] The present invention proposes an end-to-end instance segmentation-based lane detection method, which fully extracts global contextual information to cope with the challenges of lanes being disturbed by complex scenes such as severe occlusion and extreme lighting conditions. The proposed method uses attention gates to effectively fuse shallow and deep features, and uses a combination of spatial and channel attention to weight lane features to remove unnecessary interference. At the same time, the designed lane self-attention module serves as an auxiliary part, which can extract global contextual information to enhance lane features. Experiments were conducted on datasets with complex scenes to evaluate the performance of the proposed method. Compared with existing lane detection and segmentation methods, the method proposed in the present invention achieved excellent performance, especially showing excellent robustness under complex lighting and occlusion conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of the method provided by the present invention;

[0024] Figure 2 It is the lane instance segmentation network model built by the present invention;

[0025] Figure 3 is a schematic diagram of the structure of the attention gate of the present invention;

[0026] Figure 4 Schematic diagram of the lane self-attention module of the present invention;

[0027] Figure 5 This is the visualization result of lane detection during the test process of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0029] like Figure 1 As shown, the present invention discloses an end-to-end lane detection method based on instance segmentation, comprising the following steps:

[0030] Step 1: Obtain a lane detection dataset and perform data augmentation on the road images. Data augmentation for the lane detection dataset includes random affine transformations (including translation, rotation, and scaling) and random horizontal flips. The augmented dataset images are resized to 288×800 to obtain the training and validation datasets.

[0031] Step 2: Build the lane instance segmentation network model LANet. The LANet network structure is as follows Figure 2 As shown in Figure 1, the network model adopts an encoder-decoder structure, using the ERFNet encoding structure as the feature extraction network. The decoding network consists of a lane presence branch and a segmentation branch. The segmentation branch uses the channel attention mechanism and the spatial attention mechanism to eliminate interference factors on the features output by the encoding network. The attention gate is introduced to merge the coarse information extracted from the encoding network with the relevant features in the detail information, and the prediction result M is output after upsampling. pred This paper uses instance segmentation for lane detection, so the network model outputs the sum of the lane category and the background. Furthermore, a lane self-attention module is added to the segmentation branch as an auxiliary component to extract and fuse the lateral and longitudinal features of the lane, enhancing the expression of lane characteristics.

[0032] The attention gate method is: input image I Figure 2 In the network model LANet shown in the figure, after the encoding network feature extraction, the shallower detail feature F is obtained. ab and the deeper coarse features F bb , both have the same size. Figure 3 As shown, F ab and F bb F is obtained by pixel-by-pixel addition b+ , for F b+ After applying the activation function and Sigmod normalization, the response value F is obtained bp . F bp It can be considered that it contains the semantic information of the context, which determines the importance of each pixel, thereby locating the area where attention is applied and suppressing the feature response of irrelevant areas. bp With F ab Multiply pixel by pixel to get the output result F bag . F bag It can better integrate coarse features and detailed features.

[0033] like Figure 4 As shown in the figure, the lane attention module in the auxiliary part extracts features through conv1×1 and conv3×3, and reduces the size through two maximum pooling operations. The features F are obtained through different fully connected networks. v and Fh , and expand in the horizontal and vertical directions to obtain F ev and F eh , which is equivalent to extracting global context features in both horizontal and vertical directions respectively. The final attention probability matrix is ​​M bp =γF ev +(1-γ)F eh Among them, the parameter γ is the probability weight, ranging from 0 to 1. bp By fusing the global feature information of the horizontal and vertical directions, the common information is retained and the erroneous features are weakened, thereby paying more attention to the location of the lanes and combining it with the segmentation output M pred Perform pixel-by-pixel multiplication to implement lane constraints.

[0034] Step 3: Set the loss function to accelerate model convergence and improve the network's lane feature mining capabilities. The loss function consists of three parts: segmentation loss, attention loss, and presence loss.

[0035] Segmentation loss L seg =CE(M pred ,M true ) is used to measure the difference between the predicted value and the true label, where M pred Represents the segmentation output, M true represents the true segmentation label.

[0036] Attention loss L LSA Used to constrain the attention probability output in the auxiliary part to make it pay more attention to the lane position characteristics, Represents the multiplication of the true label and the attention probability map, It means that the predicted value is multiplied by the attention probability map. The attention loss is intended to constrain the difference between the two. The relationship expression is:

[0037]

[0038] Among them, n is the batch size, η is the loss coefficient, Ω is the balance constraint factor, and κ determines the proportion of weighted lane area and has an important impact on the performance of the model.

[0039] There is a loss L ext =BCE(F pred ,F true ) is used to further constrain the lane prediction results, where F pred is the output of the lane existence branch, F true Is the lane presence label.

[0040] Taking α, β and λ as the loss coefficients corresponding to the above three loss functions, the final total loss function can be expressed as: L = αL seg +βLLSA +λL ext .

[0041] Step 4: Use the training dataset to train the model. After multiple iterations, a converged lane detection model is obtained. The lane detection dataset is fed into the model in batches. After the training model converges, the detection model is obtained.

[0042] Step 5: Test on the lane detection dataset, extract lanes from the lane probability map output by the model and overlay it on the original image to visualize lane detection. Figure 5 Shown is the visualization of the detection model in multiple complex scenarios of the dataset.

[0043] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end instance segmentation-based lane detection method, characterized in that: The following steps are involved: Step 1: Obtain a lane detection dataset and perform data augmentation on the road images to obtain training and validation datasets. Step 2: Build a lane instance segmentation network model, LANet. This network model uses an encoder-decoder structure, with the encoding structure of ERFNet as the feature extraction network. The decoding network consists of a lane existence branch and a segmentation branch. The segmentation branch uses the channel attention mechanism and the spatial attention mechanism to eliminate lane interference factors on the features output by the encoding network; the attention gate is introduced to merge the coarse information extracted from the encoder network with the relevant features in the detail information, and outputs the prediction result M after upsampling. pred At the same time, a lane self-attention module is added as an auxiliary part in the segmentation branch to extract and fuse the horizontal and vertical features of the lane respectively to enhance the expression of the lane's global context features; Step 3: Set the loss function to accelerate model convergence and improve the network lane feature mining capability; The loss function consists of three parts, namely segmentation loss, attention loss and lane existence loss; segmentation loss L seg =CE(M pred ,M true ) is used to measure the difference between the predicted value and the true label, where M pred is the segmentation output, M true is the true segmentation label; attention loss L LSA Used to constrain the attention probability output in the auxiliary part to make it pay more attention to the lane position; there is a loss L ext =BCE(F pred ,F true ) is used to further constrain the lane prediction results, F pred is the output of the lane existence branch, F true is the lane existence label; taking α, β and λ as loss coefficients, the final total loss function is: L = αL seg +βL LSA +λL ext ; Step 4: Use the training data set to train the model and obtain the best detection model after multiple iterations; Step 5: Test on the lane detection dataset, extract lanes from the lane probability map output by the model and overlay it on the original image to visualize lane detection.

2. The end-to-end instance segmentation-based lane detection method according to claim 1, characterized in that: The data enhancement of the road images in the lane detection dataset in step 1 includes: random affine transformation and random horizontal flipping, and the enhanced dataset image size is adjusted to 288×800, thereby obtaining the training and validation datasets.

3. The end-to-end instance segmentation-based lane detection method according to claim 1, characterized in that: The attention gate method is as follows: after the input image I is extracted by the encoder network feature, the shallower detail feature F is obtained. ab and the deeper coarse features F bb , both have the same size, and F is obtained by adding them pixel by pixel b+ After that, the activation function and Sigmod normalization are applied to F b+ After processing, the response value F is obtained bp ; F bp Contains the semantic information of the context, which determines the importance of each pixel, thereby locating the area where attention is applied and suppressing the feature response of irrelevant areas; finally, F bp With F ab Multiply pixel by pixel to get the output result F bag , F bag Can fuse coarse features and detailed features.

4. The end-to-end instance segmentation-based lane detection method according to claim 1, characterized in that: The auxiliary part extracts features through conv1×1 and conv3×3, and reduces the size through two maximum pooling operations; the features F are obtained through different fully connected networks. v and F h , and expand them horizontally and vertically to obtain F ev and F eh , and finally the attention probability matrix obtained by fusing the two is M bp =γF ev +(1-γ)F eh ; Among them, the parameter γ is the probability weight, ranging from 0 to 1, M bp More attention will be paid to the location where the lane exists, by comparing it with the segmentation output M pred Lane constraints are implemented by pixel-by-pixel multiplication.

5. The end-to-end instance segmentation-based lane detection method according to claim 1, characterized in that: Lane attention loss L in the loss function LSA The specific expression formula is: in Represents the multiplication of the true label and the attention probability map, represents the multiplication of the predicted value and the attention probability map, n is the batch size of training, η is the loss coefficient, Ω is the balance constraint factor, and κ determines the proportion of the weighted lane area, which has an important impact on the performance of the model; the attention loss aims to constrain the difference between the two and use the true label to constrain the segmentation output results.

6. The end-to-end instance segmentation-based lane detection method according to claim 1, characterized in that: In step 4, the set loss function, loss coefficients α, β, and λ, and the learning rate of the training model are substituted into the lane instance segmentation network model, and the lane detection dataset is input into the model for training according to the batch size. After the training model converges, the detection model is obtained.

Images