A lane line detection method based on UFLane-Detection

Abstract

The application discloses a lane line detection method based on UFLane-Detection, collects lane line pictures shot by a driving recorder, and obtains a training set after cleaning the pictures; a lane line detection model based on UFLane-Detection is constructed, a backbone network separates lane lines and backgrounds through downsampling, extracts lane line features, and continuously compresses image pixels; a Lane-Decoder decoder is used to restore image pixels, and meanwhile, a global channel attention module is combined to further strengthen feature information extracted by the backbone network; a Softmax classifier is used to further identify the lane lines; the training set is input into the lane line detection model for training, and finally, the model output result is rescaled back to the original size of the to-be-tested picture to obtain a final segmentation result. The application can realize identification of lane lines in a complex traffic scene, has high accuracy, good stability, strong anti-interference capability, high universality, good robustness and the like, and can be applied to a driving assistance system or an unmanned driving system.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of semantic segmentation, image processing and intelligent driving, and particularly relates to a lane line detection method based on UFLane-Detection. BACKGROUND

[0002] Semantic segmentation plays a very important role in the field of computer vision, which identifies different categories in an image through the labeling of picture pixels. With the application of intelligent systems to the field of vehicle driving, more and more vehicles are equipped with intelligent systems capable of realizing automatic driving function or auxiliary driving function. In order to realize the automatic driving or auxiliary driving function, the intelligent system on the vehicle usually needs to identify the lane line from the road image around the vehicle to determine the driving lane near the vehicle, so as to guide the driving of the vehicle.

[0003] In the existing lane line detection technology, the image is usually processed by a human-designed feature extraction method, and the lane line marking image is obtained after the edge image after operation is processed by Hough transformation, but this way can only extract features from pictures under specific conditions, and cannot effectively detect complex situations such as lane line occlusion, blur, curves and curves. At present, most of the image semantic segmentation adopts deep learning technology, but different deep learning frameworks have different focuses, and most object detection algorithms for lane line detection cannot perfectly apply to only the surface lane line, and cannot accurately identify the semantic lane line. SUMMARY

[0004] The application aims to solve the problem that the traditional lane line detection method has low accuracy in detecting lane lines under complex traffic environment, and provides a lane line detection method based on UFLane Detection, which uses a multi-feature fusion detection method to improve the lane line detection effect.

[0005] Technical scheme: The application provides a lane line detection method based on UFLane-Detection, which specifically comprises the following steps:

[0006] (1) collecting lane line pictures taken by a driving recorder to form a data set, the lane line pictures containing different categories of lane lines marked on the road;

[0007] (2) obtaining a training set after cleaning the data set;

[0008] (3) Construct a lane line detection model based on UFLane-Detection, including a backbone network, a feature flipping fusion module, a global channel attention module GCA, and a Lane-Decoder decoder; the backbone network separates the lane line and the background through downsampling, extracts the lane line features of the lane line, and continuously compresses the image pixels; the Lane-Decoder decoder is used to restore the image pixels, and the global channel attention module is combined to further strengthen the feature information extracted by the backbone network; a Softmax classifier is used to further identify the lane line;

[0009] (4) Input the training set into the lane line detection model based on UFLane-Detection for training, and then rescale the output result of the final model back to the original size of the to-be-tested picture to obtain the final segmentation result.

[0010] Further, the different types of lane lines in step (1) include lane boundary lines, guide lines, stop lines, V-shaped lines, parking lines, and zebra crossings.

[0011] Further, the step (2) is implemented as follows:

[0012] All lane line pictures in the data set are traversed, and pictures with blur, overexposure, overdarkness, and no lane line target in the picture are deleted to form a data set for training and labeling; the pictures are subjected to data enhancement processing, and are adjusted in terms of brightness, saturation, noise, contrast, cropping, and scaling to obtain more pictures.

[0013] Further, the step (3) is implemented as follows:

[0014] The pictures are input into the backbone network for lane line feature extraction. The backbone network is divided into four layers W1-W4. The feature extraction mainly uses residual structures containing 3 3x3, and the number of residual structures in each layer is 1, 3, 3, and 6, respectively.

[0015] The hierarchical feature layer Wi is sent to the feature flipping fusion module, and the feature X input by Wi is subjected to two kinds of processing. One is to flip to obtain the feature F f , and at the same time, a 1x1 convolution kernel is used to extract the original feature map to obtain the feature map F1. The feature maps after the two kinds of processing are added to obtain the feature F2. Then a 3x3 convolution is used on F2, and F1, F f are multiplied, and finally a global average pooling GAP is used to associate global features to achieve the purpose of strengthening the lane line features. The related formula is as follows:

[0016] F2(x)=concat[F1(x)+F f (x)]

[0017] F3 = Conv3[F2(x)] F1(x) F f (x)

[0018] F = GAP(F3)

[0019] The feature after feature flipping fusion is respectively sent into the Lane-Decoder decoder of each layer, the decoder includes two 3x3 convolutions and a PSCN module, and the spatial information is obtained through the PSCN module; then the long-distance dependence information is obtained by inputting the strip pooling module, and the spatial information is extracted by the PSCN module in the upward, downward, left and right four directions respectively; first, the PSCN module takes a three-dimensional tensor with a size of CxHxW as input, where C, H and W represent the number of channels, rows and columns respectively; the first slice in the segmented tensor is sent to the convolution layer of the PSCN module, where the size of the convolution kernel is Cxω, and ω is the width of the convolution kernel; then, the output of the convolution layer is added to the next layer and becomes a new slice, which is sent to the convolution layer of the next PSCN module; the above process is repeated until the last slice is updated; the upward and downward, left and right are combined in parallel; the slice formula in the PSCN is as follows:

[0020]

[0021] Specifically, it is assumed that there is a three-dimensional kernel tensor K, where the element K m,i,n represents the weight between the element in the channel m of the slice and the element in the channel i of the current slice, and the offset between the two elements is n; the element of the input three-dimensional tensor X is represented as X i,j,k , where i, j, k represent the channel, row and column indexes respectively, and X' i,j,k is the updated value, and f is the ReLU activation function.

[0022] Further, the global attention mechanism module is attached in the Lane-Decoder decoder of each layer, which is added to each 3x3 convolution of the encoder, and the weight distribution is performed on the fused feature to retain the lane feature information; the global attention module independently uses the global average pool and the global maximum pool to aggregate two different spatial context information; the pooled features are sent into the fully connected layer; finally, the element sum and the sigmoid function are used to combine the output feature vector:

[0023] M c (F) = σ(FC1(AvgPool(x)) + FC2(Max Pool(x)))

[0024] Wherein, x is an input feature, Avg Pool is an average pooling, Max Pool is a maximum pooling, FC is a full connection layer, and sigma is a sigmod activation function.

[0025] Further, the loss function of the lane line detection model based on the UFLane-Detection is:

[0026]

[0027] In the formula, W and H represent the width and height of the final output feature map of the segmentation model. represents a semantic segmentation true label; and 6 represents predicted semantic information.

[0028] Advantages: Compared with the prior art, the advantages of the present application are: the present application regards the task of detecting lane lines as a pixel-level semantic segmentation problem, divides lane lines and backgrounds through the UFLane-Detection network, then combines the characteristics of lane line continuity, thinness and symmetry, designs a lane line feature extraction module with pertinence, and makes good use of spatial information, thereby improving the robustness of lane line detection in a complex driving environment and improving the accuracy of lane line detection; then, through a global attention mechanism, high-level semantic information and low-level semantic information are fused, and the weight distribution of important information is increased, so that the lane line segmentation result in a general scene is more accurate; finally, the fitted lane line is fitted into the image through a fitting algorithm; the present application has good lane line detection effect when coping with a complex driving environment. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of the present application;

[0030] Figure 2 the present application is a lane line detection method structure diagram based on UFLane Detection;

[0031] Figure 3 is a main network structure diagram of the present application;

[0032] Figure 4 is a feature inversion fusion module structure diagram of the present application;

[0033] Figure 5 is a Lane-Decoder decoder structure diagram of the present application;

[0034] Figure 6 is a PSCN structure diagram of the present application;

[0035] Figure 7 is a GCA module structure diagram of the present application;

[0036] Figure 8is a complex traffic map;

[0037] Figure 9 is a road division line detection example of the present application;

[0038] Figure 10 is a lane line fitting schematic diagram of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings.

[0040] The present application proposes a lane line detection method based on UFLane-Detection, as shown in the following steps: Figure 1

[0041] Step 1: Collect lane line pictures taken by a driving recorder to form a data set, and the lane line pictures contain different types of lane lines marked on the road, which are mainly divided into the following 6 types: lane boundary line, guide line, stop line, V-shaped line, parking line and zebra crossing.

[0042] Step 2: Preprocess the data set; specifically, traverse all lane line pictures in the data set, delete all pictures that are blurred, overexposed, too dark, and do not contain lane line targets, form a final data set for training and labeling. Then perform data enhancement processing on the pictures, adjust them from multiple aspects such as brightness, saturation, noise, contrast, cropping, scaling, etc., to obtain more pictures and enhance the robustness of the model. The data enhancement processing is to randomly select one item from brightness, saturation, contrast, noise, cropping and scaling for a training picture, and randomly change the attribute value of the picture in this item, wherein the weight factor of brightness and saturation is randomly taken from 0.7 to 1.3, the contrast factor is randomly taken from 0.1 to 0.3, the noise is Gaussian noise, the variance factor is randomly taken from 0 to 1, and the cropping scale is 0.9.

[0043] Step 3: Construct a lane line detection model based on UFLane-Detection.

[0044] In the specific implementation, as shown in the following steps: Figure 2 ​As shown, the lane line detection model of the UFLane-Detection constructed by the embodiment of the application comprises a backbone network, a feature flipping fusion module, a global channel attention module GCA and a Lane-Decoder decoder. The backbone network is mainly a convolutional neural network for aggregating and forming image features at different image granularities, suppressing a large amount of useless background information, and collecting feature information of lane lines. The feature flipping fusion module is for the unique characteristics of lane lines, that is, continuity, elongation and symmetry. The features are vertically flipped and added to better enhance the lane line features. The Lane-Decoder decoder is to fuse features of different scales and slide through in the form of an elongated slice to further collect lane line features. The global channel attention module is attached to each layer of the encoder. In order to achieve the attention mechanism at different levels, the information of other levels can be referred to. The fused features are weighted and distributed to better retain lane feature information.

[0045] In specific implementation, as shown in Figure 3 The backbone network is divided into four layers W1-W4. Feature extraction mainly uses residual structures containing 3 3 3. The number of residual structures in each layer is 1, 3, 3 and 6 respectively. A simple residual module is used as the backbone network for feature extraction because the lane line feature information is less and the lane line target is single. The residual structure can effectively obtain the feature information and has fewer parameters, which is conducive to lightweight deployment.

[0046] In specific implementation, as shown in Figure 4 The hierarchical feature layer Wi is sent to the feature flipping fusion module. Considering that the lane line in the image often appears symmetrically in the vertical direction, the input feature X of Wi is processed in two ways. One is to flip to get the feature F f The feature map is vertically flipped. At the same time, the other is to use a 1 1 convolution kernel to extract the original feature map to get the feature map F1. Then the feature maps after the above two processing are added to get the feature F2. Then a 3 3 convolution is used on F2, and F1, F f are multiplied, and finally a global average pooling GAP is used to better associate global features and achieve the purpose of strengthening the characteristics of the lane line. The relevant formula is as follows:

[0047] F2(x)=concat[F1(x)+F f (x)]

[0048] F3=Conv3[F2(x)]·F1(x)·F f (x)

[0049] F=GAP(F3)

[0050] In specific implementation, asFigure 5 The flipped and fused features are respectively sent to the Lane-Decoder decoder of each layer, which includes two 3x3 convolutions and a PSCN module, and the spatial information is obtained through the PSCN module, as shown in Figure 6 Then, the long-distance dependency information is obtained by inputting a strip-shaped pooling module, and the spatial information is extracted by the PSCN modules in the upward, downward, leftward and rightward directions. First, the PSCN module takes a three-dimensional tensor with a size of CxHxW as input, where C, H and W represent the number of channels, rows and columns, respectively. The first slice in the segmented tensor is sent to the convolution layer of the PSCN module, where the size of the convolution kernel is Cxω, and ω is the width of the convolution kernel. Then, the output of the convolution layer is added to the next layer and becomes a new slice, which is sent to the convolution layer of the next PSCN module. The above process is repeated until the last slice is updated. In order to improve the detection speed, the upward and downward, and the leftward and rightward are combined in parallel, although the amount of calculation does not change, but the operation time is greatly reduced. The slice formula in PSCN is as follows:

[0051]

[0052] Specifically, assume that there is a three-dimensional kernel tensor K, where the element K m,i,n represents the weight between the element in the channel m of the slice and the element in the channel i of the current slice, and the offset between the two elements is n. The element of the input three-dimensional tensor X is also represented as X i,j,k , where i, j, k represent the channel, row and column indices respectively, and X' i,j,k is the updated value, and f is the ReLU activation function.

[0053] In specific implementation, as shown in Figure 7 , a global attention mechanism module GCA is attached to the Lane-Decoder decoder of each layer. In order to achieve attention mechanism at different levels and use other level information for reference, it is added to the two 3x3 convolutions of each encoder, and the information before and after the PSCN feature enhancement is weighted to better retain the lane feature information. The module is composed of maximum pooling and average pooling in parallel, and a fully connected layer is used after the pooling, then the results are added, and finally a sigmod function is used for weighting. The formula of the GCA module is as follows:

[0054] GCA(F)=σ(FC1(Avg Pool(x))+FC2(Max Pool(x)))

[0055] Where x is the input feature, AvgPool is the average pooling, MaxPool is the maximum pooling, FC is the fully connected layer, and σ is the sigmod activation function.

[0056] The total model loss is composed of lane line detection loss L lane The total model loss is composed of lane line detection loss L

[0057]

[0058] In the formula, W and H represent the width and height of the final output feature map of the segmentation model. represents the semantic segmentation true label; and 6 represents the predicted semantic information.

[0059] Step 4: input the training set into the lane line detection model based on UFLane-Detection for training, and then rescale the final model output result back to the original size of the to-be-tested picture to obtain the final segmentation result.

[0060] The UFLane-Detection network model is trained, and corresponding parameters are set, including an initial learning rate of 1e-3, a batch-size of 16, and an Epoch of 300. After the picture is trained and predicted by the model, a binary image containing lane lines is obtained from the complex traffic picture Figure 8 as shown in FIG. 6. The present application can well segment the irrelevant background (black) and accurately extract the lane lines (white). After obtaining the binary image containing the lane lines, the pixel points of the segmented lane lines are fitted into the original picture, and the fitting effect is as shown in FIG. 7. Figure 9 Figure 10

[0061] The above display and description are the basic principles and main features of the present application and the advantages of the present application. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims.​​

Claims

1. A lane line detection method based on UFLane-Detection, characterized in that, The method comprises the following steps: (1) collecting lane line pictures taken by a driving recorder to form a data set, the lane line pictures containing lane lines of different categories marked on a road; (2) obtaining a training set after cleaning the data set; (3) constructing a lane line detection model based on UFLane-Detection, including a backbone network, a feature flipping fusion module, a global channel attention module GCA and a Lane-Decoder decoder; the backbone network separates lane lines and backgrounds through downsampling, extracts lane line features and continuously compresses image pixels; the Lane-Decoder decoder restores image pixels, and the global channel attention module further strengthens the feature information extracted by the backbone network; a Softmax classifier is used to further identify lane lines; (4) inputting the training set into the lane line detection model based on UFLane-Detection for training, rescaling the output result of the final model back to the original size of the to-be-tested picture to obtain a final segmentation result; The step (3) is implemented as follows: The picture is input into a backbone network for lane line feature extraction. The backbone network is divided into four layers W1-W4. The feature extraction mainly adopts a residual structure containing 3 3, and the number of residual structures in each layer is 1, 3, 3, and 6 respectively. The layered feature layer Wi is sent to a feature flip fusion module, and two kinds of processing are performed on the feature X input by Wi, one is flip to obtain a feature F f , and the other is vertical flip. At the same time, a 1 convolution kernel is used to extract the original feature map to obtain a feature map F1, and the two kinds of processed feature maps are added to obtain a feature F2; then a 3 convolution is used on F2, and F2 is multiplied by F1 and F f , and finally global average pooling GAP is used to associate global features to achieve the purpose of strengthening the characteristics of the lane line; the related formula is as follows: The features after the feature flipping fusion are respectively sent into Lane-Decoder decoders of each layer, and the decoders include two 3 3Convolution and PSCN module, obtain spatial information through PSCN module; then input band-shaped pooling module to obtain long-distance dependence information, and the spatial information is extracted by PSCN modules in four directions of upward, downward, left and right; first, the PSCN module takes a three-dimensional tensor with a size of CxHxW as input, wherein C, H and W respectively represent the number of channels, rows and columns; the first slice in the divided tensor is sent to the convolution layer of the PSCN module, wherein the size of the convolution kernel is Cxω, and ω is the width of the convolution kernel; then, the output of the convolution layer is added to the next layer and becomes a new slice, and the new slice is sent to the convolution layer of the next PSCN module; the above process is repeated until the last slice is updated; the upward and downward and the left and right are combined in parallel; the slice formula in the PSCN is as follows: Specifically, assume a three-dimensional kernel tensor where elements represent the weights between elements in a channel of a slice and elements in a channel of the current slice, with an offset of between two elements; an element of an input three-dimensional tensor is represented as where represent the indices of the channel, row, and column, respectively, is the updated value, is a ReLU activation function; The global attention mechanism module is attached in the Lane-Decoder decoder of each layer, which is added to the convolution of each encoder 3 3, weight assignment is performed on the fused features to retain lane feature information; the global attention module independently uses global average pooling and global maximum pooling to aggregate two different spatial context information; the pooled features are sent to a fully connected layer; finally, the output feature vector is obtained by element summation and sigmoid function combination: wherein, is an input feature, is average pooling, is max pooling, is a fully connected layer, is a sigmod activation function; The loss function of the lane line detection model based on UFLane-Detection is: In the formula, and represent the width and height of the final output feature map of the segmentation model; represents the semantic segmentation true label; represents the predicted semantic information.

2. The UFLane-Detection based lane line detection method of claim 1, wherein, The different categories of lane lines in the step (1) include lane boundary lines, guide lines, stop lines, V-shaped lines, parking lines and zebra crossings.

3. The UFLane-Detection based lane line detection method of claim 1, wherein, The step (2) is implemented as follows: All lane line pictures in the data set are traversed, and pictures with blur, overexposure, overdarkness and no lane line targets in the pictures are deleted to form a data set for training and labeling; Data enhancement processing is performed on the pictures, and the pictures are adjusted from multiple aspects of brightness, saturation, noise, contrast, cropping and scaling to obtain more pictures.

Images