Road constraint based robust lane line detection and extraction method

Abstract

Aiming at the problems of wear and blockage in lane line detection, a lane line robust detection and extraction method based on road constraint is proposed. Firstly, a large number of Dongfeng park scene images are collected to construct a lane line dataset, and the road area is labeled while labeling the lane line. Secondly, the ENet-SAD network model is improved, and the lane line segmentation loss function of joint road constraint is defined to generate accurate lane line segmentation prediction results. Finally, the full connection conditional random field algorithm is introduced to post-process the output lane line probability map, correct the probability map, select N offline coordinate points on each lane line, and use the cubic spline curve fitting algorithm to obtain the fitted lane line. The experimental results show that compared with the ENet-SAD method, the method in this paper achieves good results in different scenes such as conventional, strong light, shadow, blockage, etc. At the same time, based on the loss function defined in this paper, the performance of existing lane line detection models (such as ENet-SAD, SCNN, etc.) can be further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned driving, in particular to a lane line robust detection and extraction method based on road constraints. BACKGROUND

[0002] In the implementation of automatic driving, environment perception is an important task, and the lane line detection method research occupies a major position in environment perception. In the traditional lane line detection algorithm, the feature is mainly extracted, and the feature will change with the environment, so it is easy to be affected by the environment. With the development of deep learning at home and abroad, researchers began to use deep learning technology to study lane line detection algorithm, and tried to reduce the influence of environmental factors. At the same time, because the visual sensor has low cost, and the lane line has more visual characteristics, the lane line detection research method based on computer vision gradually develops into the mainstream. At present, there are three kinds of lane line detection algorithms based on computer vision: lane detection based on image features, lane detection based on mathematical model, and emerging lane detection based on deep learning.

[0003] The lane detection method based on image features is to extract features according to the image features (color, size, etc.) of the lane line and fitting. The feature-based method first processes the image specially, and then detects the features of the lane line through artificial design algorithm. This kind of method performs well in the case of uniform illumination, unobstructed road surface, but in the case of blurred vision, occlusion and other conditions, the detection effect is poor and the generalization ability is poor.

[0004] The lane detection method based on mathematical model is to match the appropriate curve model for the lane line according to the geometric features of the lane line on the structured road, and then use the least square method, Hough transform and other methods to fit the lane line. This kind of method will appear a lot of false detection for some geometric objects, such as road edge line and column. In addition, the lane line detection method based on mathematical model is relatively complex in calculation, and the real-time performance is not high.

[0005] The emerging lane detection method based on deep learning mainly relies on the convolutional neural network with strong feature extraction ability, establishes a neural network containing multiple hidden layers, puts a large amount of data into the network to extract the essential features of the lane line, so as to improve the accuracy of the model. This kind of method will affect the detection of lane line in the case of complex road scene such as lane line wear and road congestion, resulting in poor lane line detection result. SUMMARY

[0006] In order to achieve the above purpose, the present application provides a lane line robust detection and extraction method based on road constraints, which mainly solves the problem of poor lane line detection in different scenes such as lane line wear, occlusion and road congestion.

[0007] The technical scheme adopted by the present application is a lane line robust detection and extraction method based on road constraints, comprising the following steps:

[0008] S1, construct Dongfeng park scene lane line dataset

[0009] S1.1, video acquisition is carried out through the automatic driving vehicle, the video is converted into picture dataset by using ffmpeg software, and 1 frame of video image is extracted every 10 frames as the dataset;

[0010] S1.2, label the dataset, and label the passable area while labeling the lane line.

[0011] S2, construct CULane dataset with passable area

[0012] S2.1, visualize the lane line label image, and label the passable area based on the same.

[0013] S3, propose an improved ENet-SAD model structure, improve the model detection ability, and obtain more detailed and accurate classification results

[0014] S3.1, the improved ENet-SAD model uses an encoder-decoder as the basic architecture, extracts the features of the input image, and obtains the lane line pixel probability graph from the decoder output; At the same time, a lane line branch network is connected from the encoder to obtain the lane line existence probability;

[0015] S3.2, CBAM attention mechanism is added behind each module of the encoder E2-E4, which improves the attention degree of the model to the key features in the feature map during the training process, and enhances the network recognition performance;

[0016] S3.3, a series-parallel combination strategy is adopted between the encoder E3 and E4 modules, mainly to solve the problem of gradient disappearance in deep network, to ensure the reliability of deep network.

[0017] S4, CRF post-processing is carried out on the model output probability graph, and point fitting is carried out on the optimized probability graph through cubic spline curve fitting algorithm

[0018] S5, design the loss function of the road constraint lane line detection model, and analyze the good and bad of the training result according to the size of the loss value

[0019] S5.1, the loss function of the algorithm contains three parts, which are semantic segmentation loss, existence loss and distillation loss in turn, wherein the semantic segmentation loss contains segmentation loss and joint road constraint loss.

[0020] S6, train the road constraint lane line detection model, and design the evaluation index of the road constraint lane line detection model

[0021] S6.1、In the training process, the number of iterations is set to 40k, the batch size is 12, the initial learning rate is 1e-2, the momentum and weight decay are configured to 0.9 and 2e-5 respectively, and each graphics card processes 2 images at a time;

[0022] S6.2, the average intersection over union (mIoU) is used as the evaluation index of accuracy, which is the average value of the intersection and union ratio between the predicted value and the true value, indicating that the more accurate the prediction result of the algorithm is;

[0023] S6.3, the F1 value is selected as another evaluation index of accuracy, which is a harmonic average of model accuracy and recall rate, and the value range is from 0 to 1, and the mF1 value refers to the sum of the F1 values of each class and the average result.

[0024] The beneficial effects of the present application are:

[0025] 1. Constructing Dongfeng park scene lane line dataset, and labeling the passable area on Dongfeng park scene lane line and part of CULane dataset.

[0026] 2. The lane line robust detection and extraction method based on road constraint solves the problem that the lane line detection result is not accurate in the prior art scheme under the condition of lane line wear, road congestion and other complex road scenes. In the case of ensuring high-speed identification effect, CBAM attention mechanism is added after each module of the encoder E2-E4 to improve the attention degree of key features in the feature map during the training process of the model and enhance the network identification performance. At the same time, a series-parallel combination strategy is adopted between the encoder E3 and E4 modules to solve the problem of gradient disappearance in deep network and ensure the reliability of deep network.

[0027] 3. The lane line robust detection and extraction method based on road constraint solves the problem of edge accuracy and incoherent area in the lane line prediction result of the prior art scheme. The output probability graph is post-processed by CRF, and for each lane line, uniform sampling is adopted to take points, and then based on cubic spline, curve fitting is performed to obtain more detailed and accurate classification results. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the flowchart of the present application;

[0029] Figure 2 is the overall architecture diagram of the present application;

[0030] Figure 3 is the original picture and labeled picture of Dongfeng dataset;

[0031] Figure 4 is the original picture and label visualization diagram of CULane dataset;

[0032] Figure 5 is a training and validation loss function graph;

[0033] Figure 6 is the lane line detection effect diagram of each model under the Dongfeng dataset. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0035] Please refer to Figures 1-2 , the lane line robust detection and extraction method based on road constraints, comprising the following steps:

[0036] S1, constructing a lane line dataset of Dongfeng park scene

[0037] S1.1, video acquisition by an autonomous vehicle, using ffmpeg software to convert the video into a picture dataset, taking every 10 frames to extract 1 frame of video image as the dataset;

[0038] S1.1.1, the vehicle for collecting is the autonomous driving of Dongfeng Enjoy Technology Co., Ltd., the vehicle body is equipped with 12 cameras, and the pixel of each camera is 1920x1080. In this paper, the front-end camera of the vehicle is used to collect lane line data. The road features mainly include straight road, curved road, crossroad, etc. The road scene includes conventional, shadow, strong light, and shielding, and each scene has 3 lane lines;

[0039] S1.1.2, the video accumulates for about 9 hours, and ffmpeg software is used to convert the video into a picture dataset. Since the converted video frame image has time sequence continuity, the change between video frames is small. In this paper, every 10 frames is extracted to extract 1 frame of video image as the dataset, and a total of 10741 images with clear picture quality and different scenes are constructed to construct the dataset.

[0040] S1.2, labeling the dataset, labeling the passable area while labeling the lane line, as shown in Figure 3 .

[0041] S1.2.1, first, label the lane line, and use the method of creating a polygon to label the lane line in the image from left to right, and when the lane line is worn, shielded, or still connected, label the lane line;

[0042] S1.2.2. Secondly, on the basis of the above-mentioned lane line labeling, the same method is used to label the passable area (road) in the image from left to right in turn. When labeling the passable area, it is necessary to adhere to the lane line to ensure that there is a gap between the two but the gap is small, so as to avoid excessive constraint;

[0043] S1.2.3. Finally, the labeled image is converted into a standard input data set, that is, the pixel label values of the three lane lines from left to right are set to 1, 2 and 3 respectively, the pixel value of the passable area is set to 4, and the pixel label value of the non-lane line area is set to 0, to generate a corresponding label file.

[0044] S2. Constructing a CULane data set with a passable area

[0045] S2.1. Visualizing lane line label images, and labeling passable areas on the basis thereof;

[0046] S2.1.1. Selecting 20,000 CULane data sets of different scenes, including different scenes such as normal, congestion, no lane line and shadow, and selecting an average of 5,000 images for each scene. Visualizing the lane line label images, and labeling the passable areas using software on the basis of the visualized label images, and the labeling method is the same as described above;

[0047] S2.1.2. Fusing the label images of the labeled passable areas with the lane line label images. If there is pixel point overlap during fusion, the pixel value of the lane line label image is used as the reference, as shown in Figure 4 ;

[0048] S2.1.3. Finally, the fused image is converted into a standard input data set, that is, the pixel label values of the four lane lines from left to right are set to 1, 2, 3 and 4 respectively, the pixel value of the passable area is set to 5, and the pixel label value of the non-lane line area is set to 0, to generate a corresponding label file.

[0049] S3. Proposing an improved ENet-SAD model structure to improve the model detection capability and obtain more detailed and accurate classification results

[0050] S3.1. The improved ENet-SAD model uses an encoder-decoder as the basic architecture to extract the features of the input image, and outputs a lane line pixel probability map from the decoder; at the same time, a lane line branch network is connected from the encoder to obtain a lane line existence probability.

[0051] S3.1.1、The encoder mainly includes five modules, namely the initial module and E1-E4 modules, each of which is composed of a bottleneck module. The initial module designs a pooling operation in parallel with a convolution operation with a step of 2, and combines the result feature map. The E1 module includes one down-sampling bottleneck module and four regular convolution modules; the E2 module includes one down-sampling bottleneck module and eight convolution bottleneck modules, which use different convolution strategies respectively; the E3 and E4 modules are consistent with the E2 module except that there is no down-sampling bottleneck module. Each bottleneck module has two branches, of which the main branch is used for short connection across layers, and the other branch is a regular convolution. This branch first passes through a 1x1 convolution to reduce the channel dimension of the input feature, then passes through a convolution, and then through a 1x1 convolution to restore the original dimension. There is a BN layer and a PReLU activation function between each two convolutions, and a dropout regularization is added after the last 1x1 convolution. The convolutions in different bottleneck modules in the entire network are different, and the convolutions used here include regular convolution, atrous convolution, asymmetric convolution, etc.

[0052] S3.1.2、The decoder mainly includes D1 and D2 two modules, the D1 module includes one up-sampling bottleneck module and two regular convolution bottleneck modules; the D2 module includes one up-sampling bottleneck module and one regular convolution module. The feature map is output after passing through the decoder. The probability map (w x h x c) of the original image size, where w is the image width, h is the image height, and c corresponds to the number of lane line categories;

[0053] S3.1.3、 Figure 2 The P1 is a prediction lane line branch network, which performs average pooling on the feature matrix through avgpool, and outputs a one-dimensional vector of 1*n through the full connection layer and the sigmoid activation function, which is used to predict the lane existence probability, where n corresponds to the number of detected lane line categories;

[0054] S3.1.4、The attention generator uses the feature map obtained by network forward propagation (Activation-based distillation map), which mainly converts the three-dimensional features of length, width and channel into two-dimensional features of length and width, which is represented as:

[0055]

[0056] The attention generator only participates in model training, so it will not increase the computational complexity during inference. And through the information flow between algorithm layers, the network can retain the context information of the scene in the deep layer, so as to more effectively identify the continuous target similar to the lane line with the characteristics of long and narrow.

[0057] S3.2 Add the CBAM attention mechanism after each module of encoder E2-E4 to improve the model's attention to key features in the feature map during training and enhance the network's recognition performance.

[0058] S3.3. A series-parallel combination strategy is adopted between the encoder E3 and E4 modules to mainly solve the gradient vanishing problem in deep networks and ensure the reliability of deep networks.

[0059] S4. Perform CRF post-processing on the model output probability map, and simultaneously perform point fitting on the optimized probability map using a cubic spline curve fitting algorithm.

[0060] S4.1 This paper uses a fully connected conditional random field (CRF) model to further optimize the probability map, that is, the optimization problem of the probability map is transformed into a multi-label assignment problem, assigning a unique class label x to each image pixel i. i ∈{1,2,...,k}. The category labels X={x i The result can be obtained by minimizing the following energy function:

[0061]

[0062] The unary term is defined as ψ(x) i )=-logP(x i ), probability P(x i ) is the probability vector of pixel i calculated by the feedforward DNN; the binary term is defined as:

[0063]

[0064] Where the category label x i ≠x j When, the compatibility function μ(x) i ,x j = 1; otherwise, the value is 0. The expressions within the square brackets on the right side of the above equation represent two Gaussian kernels: the first bilateral kernel is related to the position and color differences between pixels; the second Gaussian kernel depends only on the distance between pixels, where p i and p j Represents the spatial coordinates of pixel i and pixel j; I i and I j σ represents the color values ​​of pixels i and j. α σ β and σ γ is a hyperparameter. The literature utilizes a mean-field approximation algorithm to quickly solve for the aforementioned energy function.

[0065] S4.2, there will still be some missing cases after the lane line probability map is optimized by CRF, so a cubic spline curve fitting algorithm is used to take points fitting on the lane line probability map optimized by CRF. First, select N off-line coordinate points on each lane line. Select the initial point (x0, y0), where x0∈[0, 800], y0∈[0, 288]. Then take the vertical coordinate y0 of the initial point as the benchmark, search for the point with the maximum probability value along the y-axis positive direction every 10 rows, and constantly loop the above operation until y≥280 stop, and then use the cubic spline curve fitting algorithm to fit the extracted feature points.

[0066] S5, design the loss function of the lane line detection model with road constraints, analyze the pros and cons of the training results according to the size of the loss value, and the training and verification loss function graph is as shown in Figure 5

[0067] S5.1, the loss function of the algorithm contains three parts, in turn, semantic segmentation loss, existence loss and distillation loss, among which the semantic segmentation loss contains segmentation loss and joint road constraint loss.

[0068] S5.1.1, this paper adopts the combination of multiple loss functions, by setting the loss weight coefficient of each loss function, then adding the loss value, in the training process, view the change of each loss, in order to adjust the loss weight coefficient, optimize the model. The total loss function calculation formula is:

[0069]

[0070] Among them, is the segmentation loss function, is the joint road constraint lane line segmentation loss function, is the existence loss function, is the continuous hierarchical distillation loss function, β / γ is the loss weight coefficient.

[0071] S5.1.2, cross entropy loss function can measure the difference between two different probability distributions in the same random variable, which is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning. Compare the prediction result (probability distribution vector) of each pixel category with the one-hot encoding label vector. The cross entropy definition is shown in equation (5):

[0072]

[0073] Among them, m is the number of samples processed, n is the number of lane line categories, p is the true distribution, and q is the network output distribution.

[0074] ​S5.1.3, the lane line segmentation loss function combined with the road constraint is actually an intersection over union (IoU) loss function, where the IoU is the ratio of the area of the intersection of the predicted box and the true box to the area of the union. In this paper, the IoU loss function is divided into a lane line IoU loss function and a road constraint IoU loss function, and the weight coefficient of the IoU loss function is adjusted. The combined IoU loss function L join is defined as shown in equation (6):

[0075]

[0076] where s and are the coordinates of the predicted value and the true value, and liou and riou are the loss weight coefficients, and L liou and L riou are the IoU loss functions of the lane line and the road constraint, respectively. The calculation formula of the IoU loss function is:

[0077]

[0078] where N p is the number of predicted lane pixels, N g is the number of ground true lane pixels, and N o is the number of lane pixels in the overlapping area between the predicted lane area and the ground true lane area.

[0079] S5.1.4, the continuous hierarchical distillation loss function is mainly for continuous layers, and the useful context information can be extracted from the front layer to supervise the learning of the rear layer. Because the front layer attention map is used to constrain the rear layer, it plays a role in preserving detailed information in deep networks. The continuous hierarchical distillation loss calculation formula is as follows:

[0080]

[0081] where L d is an L2 loss function (mean squared error), A m and A m+1 are the outputs of the attention generator, m is the mth convolutional layer, and where C is the number of lane line categories.

[0082] S6, training the road-constrained lane line detection model, designing evaluation indicators for the road-constrained lane line detection model

[0083] S6.1、In the training process, the number of iterations is set to 40k, the batch size is 12, the initial learning rate is 1e-2, the momentum and weight decay are configured as 0.9 and 2e-5 respectively, and each graphics card processes 2 images at a time;

[0084] S6.2、The average intersection over union (mIoU) is used as an accuracy evaluation index, which is the average value of the intersection and union ratio between the predicted value and the true value, indicating that the more accurate the algorithm's prediction result is, as shown in Table 1.

[0085] S6.2.1、The intersection over union (IOU) is the ratio of the intersection and union between the predicted value and the true value of the algorithm for a certain class. The larger the IOU value, the more accurate the algorithm's prediction result is, as shown in equation (8). And MIOU is the result of summing and averaging the IOU value of each class of the algorithm, as shown in equation (9).

[0086]

[0087]

[0088] where i represents the class (i = 1,..., n); TP i represents the number of pixel points whose true class is i and whose algorithm prediction is class i; FP i represents the number of pixel points whose true class is not i and whose algorithm prediction is class i; FN i represents the number of pixel points whose true class is i and whose algorithm prediction is not class i.

[0089] S6.3、The F1 value is selected as another accuracy evaluation index, which is a harmonic mean of the model precision and recall, with a value range from 0 to 1, and mF1 value is the result of summing and averaging the F1 value of each class of the algorithm, as shown in Table 2.

[0090] S6.3.1、The F1 value is a harmonic mean of precision and recall, and the larger the F1 value range from 0 to 1, the more accurate the algorithm's prediction result is, as shown in equation (11). And MF1 is the result of summing and averaging the F1 value of each class of the algorithm, as shown in equation (12).

[0091]

[0092]

[0093] where i represents the class (i = 1,..., n); TP i represents the number of pixel points whose true class is i and whose algorithm prediction is class i; FP iTN represents the number of pixels for which the algorithm predicted a class that is not i, given that the true class is i. i FN represents the number of pixels for which the algorithm predicted a class that is i, given that the true class is not i.

[0094]

[0095] Figure 6 Lane line detection effect diagram using the model of the present application.

[0096] The above description is merely that of the preferred embodiments of the application, and is not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A road constraint-based lane line robust detection and extraction method, comprising the following steps: S1, constructing Dongfeng park scene lane line dataset; S2, constructing CULane dataset with passable area; S3, proposing an improved ENet-SAD model structure to improve the detection ability of the model and obtain more detailed and accurate classification results; Said improved ENet-SAD model structure, improving the detection ability of the model, and obtaining more detailed and accurate classification results, comprising: S3.1, the improved ENet-SAD model uses an encoder-decoder as the basic architecture to extract the features of the input image, and outputs a lane line pixel probability map from the decoder; At the same time, a lane line branch network is connected from the encoder to obtain the lane line existence probability; S3.1.1, the encoder mainly includes five modules, which are initial module and E1 to E4 modules, and each module is composed of a bottleneck module; The initial module designs a pooling operation and a convolution operation with a step of 2 in parallel, and combines the result feature map; E1 module includes 1 down-sampling bottleneck module and 4 normal convolution modules; E2 module includes 1 down-sampling bottleneck module and 8 convolution bottleneck modules, which respectively adopt different convolution strategies; E3 and E4 modules include 8 convolution bottleneck modules; Each bottleneck module has two branches, wherein the main branch is used for short connection cross layer link, and the other branch is a normal convolution, which first reduces the channel dimension of the input feature through a 1*1 convolution, then through a convolution, and then through a 1*1 convolution to restore the original dimension; There are a BN layer and a PReLU activation function between each two convolutions, and a dropout regularization is added after the last 1*1 convolution; S3.1.2, the decoder mainly includes D1 and D2 two modules, the D1 module includes 1 up-sampling bottleneck module and 2 normal convolution bottleneck modules; D2 module includes 1 up-sampling bottleneck module and 1 normal convolution module; The feature map is outputted to the probability map of the original picture size after passing through the decoder; S3.1.3, the improved ENet-SAD model also includes a lane line prediction branch network, which averages the feature matrix through avgpool, and outputs a one-dimensional vector of 1*n through the sigmoid activation function after the full connection layer, wherein n corresponds to the number of lane line categories detected; S3.1.4, the improved ENet-SAD model also includes an attention generator, which uses the feature map obtained by network forward propagation, mainly converts the three-dimensional features of length, width and channel into two-dimensional features of length and width; The attention generator only participates in model training; S3.2, CBAM attention mechanism is added after each module of the encoder E2-E4; S3.3, a series-parallel combination strategy is adopted between the encoder E3 and E4 modules; S4, the model output probability map is post-processed by CRF, and the optimized probability map is point-fitted by a cubic spline curve fitting algorithm. S5, design a loss function of the lane line detection model of the road constraint, and analyze the pros and cons of the training result according to the size of the loss value; S6, training the lane line detection model of the road constraint, and designing the evaluation index of the lane line detection model of the road constraint.

2. A method for robust lane line detection and extraction based on road constraints as claimed in claim 1 wherein, The specific implementation process of S1 is as follows: S1.1, video acquisition is performed by an automatic driving vehicle, ffmpeg software is used to convert the video into a picture data set, and every 10 frames of video images are extracted as a data set; S1.2, label the data set, and label the passable area while labeling the lane line.

3. A method for robust lane line detection and extraction based on road constraints as claimed in claim 1 wherein, The specific implementation process of S2 is as follows: S2.1, visualize the lane line label image, and label the passable area based on it.

4. The method of claim 1, wherein, The specific implementation process of S5 is as follows: S5.1, the loss function of the algorithm contains three parts, which are semantic segmentation loss, existence loss and distillation loss in turn, wherein the semantic segmentation loss contains segmentation loss and joint road constraint loss; The joint road constraint loss is an intersection over union loss function, wherein the intersection over union is the ratio of the intersection area of the predicted box and the real box to the combined area; the intersection over union loss function is divided into lane line IoU loss function and road constraint IoU loss function, and the lane line IoU loss function and the road constraint IoU loss function each have a weight coefficient.

5. A method for robust lane marking detection and extraction based on road constraints as claimed in claim 1, wherein, The specific implementation process of S6 is as follows: S6.1, in the training process, the iteration number is set to 40k, the batch size is 12, the initial learning rate is 1e-2, the momentum and weight decay are configured to 0.9 and 2e-5 respectively, and each graphics card processes 2 images at a time; S6.2, the average intersection over union (mIoU) is used as the evaluation index of the precision, which is the average value of the intersection and union ratio between the predicted value and the true value, indicating that the more accurate the algorithm prediction result is; S6.3, select F1 value as another evaluation index of precision, which is a harmonic mean of model precision and recall, and the value range is from 0 to 1, and the mF1 value refers to the result of summing and averaging the F1 value of each class of the algorithm.

Images