A multi-supervised intelligent lane semantic segmentation method integrating edge detection

Abstract

The present invention discloses a multi-supervisory lane line semantic segmentation method integrating edge detection, including: edge detection of the label truth value corresponding to the input image to obtain the lane line label edge map; the image is input into the encoding network module for feature extraction to obtain a high-resolution feature map and a high-semantic feature map; after upsampling the high-semantic feature map, a splicing operation is performed with the high-resolution feature map along the channel dimension, and the high-resolution feature map is sent to the convolution layer for dimension reduction and then upsampled twice; the high-resolution feature map is upsampled; the label truth map is used as the main supervision signal, and the label edge map is used as the auxiliary supervision signal to calculate the loss function value; the loss function value is weighted and accumulated to optimize the network structure parameters; and the lane line detection model is trained until convergence. The present invention performs semantic segmentation on the driving road condition image, and adds the traditional machine vision edge detection signal for auxiliary supervision during the network training process, thereby improving the accuracy and robustness of the detection result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a deep learning semantic segmentation model technology, and specifically to a multi-supervised intelligent lane line semantic segmentation method that integrates edge detection signals. Background Art

[0002] Lane detection is one of the research hotspots in the field of autonomous driving and is also a key technology for ADAS (Advanced Driver Assistance Systems). The camera placed on the front windshield of the vehicle is used to collect information about the surrounding environment during driving. The vision-based processing algorithm is used to extract lane information from the camera output image and use it in subsequent tasks such as track planning, lane keeping, and behavior prediction.

[0003] The traditional lane line extraction algorithm based on machine vision mainly includes two steps: manual feature extraction and curve fitting. Feature extraction uses lane line edge information, color and texture features to capture the effective pixel points of the lane line in the image, and then uses the curve model to fit the extracted lane line pixel points to obtain the lane line geometric parameter model. This traditional lane line detection method can detect lane lines in an environment with clear road signs and clear camera field of view. However, when the driving scene is affected by changes in lighting and weather, and the road features are severely obscured by vehicles, the traditional method based on machine vision is difficult to extract the effective pixel points of the lane line from the above difficulties, and the detection results cannot meet the technical indicators of autonomous driving vehicles.

[0004] In recent years, the field of computer vision has gradually emerged, providing new methods for lane line detection technology for intelligent driving cars. Lane line detection methods based on deep learning have gradually become the mainstream method in this field. Compared with traditional algorithms that rely too much on manually extracted feature information in specific environments, methods based on semantic segmentation rely on the powerful learning ability of convolutional neural networks themselves. By continuously optimizing neural network parameters, the network's perception of scenes and the network's own generalization ability are improved, which has higher robustness and applicability. At present, lane line detection methods based on semantic segmentation can realize lane line detection under the influence of occlusion and environmental changes. However, in extremely challenging driving environments such as severe occlusion and night driving, the detection results still have problems such as missed detection of lane line pixels, false detection, and unclear and discontinuous lane line boundaries, which has become a hindrance to semantic segmentation technology serving the field of autonomous driving. Summary of the invention

[0005] To solve the above problems, the present invention provides a multi-supervised intelligent lane line semantic segmentation method integrating edge detection. The present invention performs semantic segmentation on driving road condition pictures, adds traditional machine vision edge detection signals for auxiliary supervision during network training, and realizes accurate detection and regression of lane lines and their boundaries.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-supervised intelligent lane semantic segmentation method integrating edge detection includes the following steps:

[0008] Step S1: Select a road condition picture collected by the autonomous driving car and process it as an input picture;

[0009] Step S2: Perform edge detection on the label truth value corresponding to the input image to obtain a lane line label edge map;

[0010] Step S3: input the image into the encoding network module for feature extraction to obtain a high-resolution feature map and a high-semantic feature map;

[0011] Step S4: After performing an upsampling operation on the high semantic feature map, a splicing operation is performed with the high resolution feature map along the channel dimension, and the map is sent to the convolutional layer to reduce the dimension and then upsampled twice to restore the size to the original image size;

[0012] Step S5: upsample the high-resolution feature map and restore its size to the original image size;

[0013] Step S6: Use the label truth map as the main supervisory signal of the output of this path to calculate the loss function value; use the label edge map as the auxiliary supervisory signal after upsampling the high-resolution feature map to calculate the loss function value; weightedly accumulate the above loss function values, calculate the parameter gradient, and back-propagate to optimize the network structure parameters;

[0014] Step S7: Train the lane detection model until convergence.

[0015] Furthermore, the step S1 specifically includes:

[0016] The collected video is sampled in time to obtain continuous frame road surface information maps; after all images are cropped to the same size, a standardized transformation operation is performed; the standardized transformation operation calculation formula is:

[0017]

[0018] Among them, I is the value of a pixel point in a channel in the original input image, mean is the original mean of all pixels in the channel, std is the original standard deviation of all pixels in the channel, and I normalThe pixel values ​​are normalized and follow the normal distribution.

[0019] Furthermore, the step S2 specifically includes:

[0020] The Sobel edge detection operator along the vertical direction is used to perform a planar convolution operation with the label true value image to obtain the label edge image corresponding to the label. The edge information calculation formula is as follows:

[0021]

[0022] L Edge =h y *L Truth (3)

[0023] Among them, h y is the edge detection operator along the vertical direction, L Truth Represents the input label true value image, L Ed ge represents the difference in brightness of each pixel after the convolution operation, reflecting the edge information of the lane line in the label.

[0024] Furthermore, the step S3 specifically includes the following process:

[0025] (1) Input the standardized image in step S1 into the weight-sharing convolutional neural network feature extraction layer, and use convolution and pooling operations to complete the downsampling and feature extraction of the input image to obtain the original Figure 4 A low-semantic high-resolution feature map F with a size one-fifth of the original I / 4 ;

[0026] (2) F I / 4 The feature map is continuously input into the deep feature extraction layer network structure, and an abstract high-semantic feature map F with global information and downsampled to one sixteenth of the original image is gradually extracted from the high-resolution features. I / 16 .

[0027] Furthermore, the S4 specifically includes the following process:

[0028] F I / 16 The feature map is four times upsampled using the bilinear interpolation upsampling method to obtain the same I / 4 Features with the same size Along the channel dimension, the two feature maps F I / 4 and After performing concatenation and convolution operations, a secondary upsampling is performed to obtain a feature map F with the same size as the original image. I .

[0029] Furthermore, in the convolution operation, the convolution kernel size is 1*1 and the step size is 1.

[0030] Furthermore, the step S5 specifically includes the following process:

[0031] Upsample the high-resolution features obtained in step 3 to obtain a feature map of the same size as the original image. The network outputs a feature map F with the same size as the original image. I and

[0032] Furthermore, the step S6 specifically includes the following process:

[0033] A weighted multi-supervisory loss function is used to calculate the difference between the prediction and the true value. First, the true value of the label is taken as the main supervisory signal, and the cross entropy loss function is set to calculate F I With L Truth Secondly, the label edge is used as an auxiliary supervision signal, and the focus loss function is used to calculate the lane line edge regression binary classification problem. With L Edge The difference between them is as follows:

[0034]

[0035] FL(p n )=-(1-p n ) γ log(p n ) (5)

[0036] Loss=Loss CE +λ·Loss FL (6)

[0037] Among them, expression (4) calculates the cross entropy loss function value, expression (5) calculates the focus loss function value, and expression (6) balances the cross entropy loss function Loss by the weight factor λ. CE And the focal loss function Loss FL The weight ratio is used to calculate the final model loss value, which is used to update and optimize the network parameters; ic represents the predicted probability that the observed sample i belongs to category c, y ic represents the true category of sample i, N represents the total number of samples, M represents the total number of categories, and p n It indicates the probability of observing sample n being predicted as positive class, and γ represents the difficulty and easy sample adjustment parameter.

[0038] Furthermore, the upsampling operation includes restoring the size of the feature map by bilinear interpolation.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] 1. It fully integrates the advantages of traditional lane line detection methods and deep learning-based detection technology. In the model training stage, the edge detection operator is used to extract lane line boundary features and add them to the neural network as auxiliary supervision signals, guiding the neural network to refine the boundary features during the reasoning process, significantly improving the accuracy of lane line pixel point detection and the continuity of the overall lane sign detection results; 2. The present invention fully considers the different levels of feature characteristics output by the convolutional neural network encoding process, extracts global information with rich spatial features in the deep network, and extracts local information with rich geometric features in the shallow network; in the decoder stage, the features of the two are spliced ​​and fused, which slightly increases the amount of calculation while greatly improving the density of the feature map, significantly increasing the accuracy and robustness of the semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the multi-supervised intelligent lane line semantic segmentation method integrating edge detection provided by the present invention.

[0042] Figure 2 Schematic diagram of the true value of the main supervisory signal label of the present invention.

[0043] Figure 3 Schematic diagram of the auxiliary supervisory signal label edge of the present invention.

[0044] Figure 4 is the original input image verified by the example of the present invention.

[0045] Figure 5 It is a conventional semantic segmentation output image without applying the method of the present invention.

[0046] Figure 6 It is a fine semantic segmentation output image using the method of the present invention. DETAILED DESCRIPTION

[0047] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0048] The present invention discloses a multi-supervisory intelligent lane line semantic segmentation method that integrates edge detection. In the process of network parameter optimization, lane line edge signals are added for auxiliary supervision. The method proposed in the present invention mainly serves the semantic segmentation model training stage, and the effectiveness of the method of the present invention is verified by comparing experimental results in the model verification stage. The process of the method of the present invention is as follows Figure 1 As shown, the specific steps include:

[0049] Step 1: Sample the video sequence collected during the driving process of the self-driving car in the time dimension to obtain a continuous frame road surface information map. After cropping all the pictures to the same size, perform data standardization transformation operations to convert the input picture from the pixel-by-pixel value (0,255) distribution on the RGB three channels to the pixel value under each channel obeying a normal distribution with a mean of 0 and a standard deviation of 1. The data standardization calculation formula is:

[0050]

[0051] Among them, I is the value of a pixel point in a channel in the original input image, mean is the original mean of all pixels in the channel, std is the original standard deviation of all pixels in the channel, and I normal The pixel values ​​are normalized and follow the normal distribution. The deep learning annotation toolbox Labelme is used to annotate the input image and obtain the true value map of the label.

[0052] Step 2: Perform edge detection on the true value of the label corresponding to the input image to obtain the lane line label edge map. Edge detection is to extract the brightness change difference value of each pixel neighborhood. The present invention uses the Sobel edge detection operator to extract edge information in the horizontal and vertical directions. The edge detection operator along the vertical direction is used as follows Figure 2 The label true value image shown in the figure is subjected to a planar convolution operation to obtain the label edge image corresponding to the label, as shown in the figure. Figure 3 As shown, the edge information calculation formula is as follows:

[0053]

[0054] L Edge =h y *L Truth (3)

[0055] Among them, h y is the edge detection operator along the vertical direction, L Truth Represents the input label true value image, L Edge It represents the difference in brightness of each pixel after the convolution operation, reflecting the edge information of the lane line in the label.

[0056] Step 3: Input the standardized image in step 1 into the weight-sharing convolutional neural network feature extraction layer, and use convolution and pooling operations to complete the downsampling and feature extraction of the input image to obtain the original image. Figure 4 A low-semantic high-resolution feature map F with a size one-fifth of the original I / 4 .

[0057] Step 4: F I / 4The feature map is continuously input into the deep feature extraction layer network structure, and an abstract high-semantic feature map with global information is gradually extracted from the high-resolution features. At the same time, the feature map size is downsampled to one sixteenth of the original image. I / 16 At this point, the encoding module in the semantic segmentation network framework has been completed. I / 4 and F I / 16 The local feature maps with rich geometric information and the global feature maps with dense spatial context information are obtained respectively. The encoding network module includes convolutional layers, pooling layers, nonlinear activation layers and batch normalization layers, which gradually aggregate the core features of the input data.

[0058] Step 5: F I / 16 The feature map is four times upsampled using the bilinear interpolation upsampling method to obtain the same I / 4 Features with the same size Along the channel dimension, the two feature maps F I / 4 and After performing concatenation and convolution operations, a secondary upsampling is performed to obtain a feature map F with the same size as the original image. I In the convolution operation, the convolution kernel size is 1*1 and the step size is 1, which has the effect of reducing the channel dimension.

[0059] Step 6: Upsample the high-resolution features obtained in step 3 to obtain a feature map of the same size as the original image. At this point, the decoding module in the semantic segmentation framework has been completed, and the network outputs a feature map F with the same size as the original image. I and

[0060] Step 7: The loss in the convolutional neural network is used to measure the difference between the prediction result obtained by the model and the true value, which is an important indicator to measure the generalization ability of the model. The present invention uses a weighted multi-supervisory loss function to calculate the difference between the prediction and the true value. First, the true value of the label is taken as the main supervisory signal, and the cross entropy loss function is set to calculate F. I With L Truth Secondly, the label edge is used as an auxiliary supervision signal, and the focal loss function is used to calculate the lane line edge regression binary classification problem. With L Edge The difference between them is as follows:

[0061]

[0062] FL(p n )=-(1-p n ) γ log(pn ) (5)

[0063] Loss=Loss CE +λ·Loss FL (6)

[0064] Expression (4) calculates the cross entropy loss function value, expression (5) calculates the focus loss function value, and expression (6) balances the cross entropy loss function Loss by the weight factor λ CE And the focal loss function Loss FL The weight ratio of p is used to calculate the final model loss value, which is used to update and optimize the network parameters. ic represents the predicted probability that the observed sample i belongs to category c, y ic represents the true category of sample i, N represents the total number of samples, M represents the total number of categories, and p n It indicates the probability of observing sample n being predicted as positive class, and γ represents the difficulty and easy sample adjustment parameter.

[0065] Step 8: Train the lane semantic segmentation model in batches until convergence, and take the network output feature map F I Input into the Softmax activation function to obtain the final lane line semantic segmentation image.

[0066] The upsampling operation mentioned in the above steps involves restoring the feature map size using bilinear interpolation.

[0067] Example 1

[0068] The present invention provides a method for semantic segmentation of lane lines for autonomous driving vehicles, which fully integrates the edge detection feature characteristics in traditional machine vision with the powerful feature representation and learning ability advantages of convolutional neural networks, introduces the label signal after edge detection into the neural network for auxiliary supervision, and realizes more accurate detection of lane line pixels and more detailed regression of lane line boundaries. This embodiment takes the semantic segmentation process of a single image as an example, and expands the steps of the semantic segmentation method of fusion edge detection. The relevant parameter settings and steps are as follows:

[0069] During the training process of the convolutional neural network, the parameter of the focal loss function in the auxiliary supervision is set to γ ​​= 2, the weight factor λ in the overall loss function is set to 0.1, 8 images are trained in the same batch at the same time (batch size = 8), and the total batch parameter is set to 100 (epoch = 100). After the semantic segmentation model is trained to convergence, a single image ( Figure 4 ) Verify the validity and usability of the content described in this invention.

[0070] Step 1: Take one frame of the video sequence collected during vehicle driving as the original image with a size of 720*1280*3, adjust the size of the original image to 352*640*3, and use it as the neural network input image.

[0071] Step 2: Input the image obtained in step 1 into the weighted shared feature extraction network, use convolution and pooling operations to extract effective information from the input image data, and output a feature map with the original Figure 4 The specific size is 88*160*128.

[0072] Step 3: Input the feature map in step 2 into the deep feature extraction layer to further extract the core features of the input data and output a high semantic feature map with a size of one sixteenth of the original image, specifically 22*40*512.

[0073] Step 4: Perform four-fold bilinear interpolation upsampling on the output features in step 3 to obtain a feature map of size 88*160*256, and concatenate it with the high-resolution feature map output in step 2 along the channel dimension to obtain features of size 88*160*384.

[0074] Step 5: Perform convolution operation on the concatenated features obtained in step 4, and the output data size is 88*160*256.

[0075] Step 6: Perform quadruple upsampling on the feature map in step 5, input the upsampling result into the activation function layer, and obtain the final semantic segmentation output image, such as Figure 6 shown.

[0076] Depend on Figure 5 and Figure 6 It can be seen that adding an auxiliary supervision mechanism based on edge detection during network training can improve the network's ability to identify the location of lane lines, refine lane line boundary features, make detection results more continuous and detailed, and reduce the probability of missed detection and false detection of the semantic segmentation model to a certain extent. As a result, a higher lane line accuracy requirement can be achieved.

[0077] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined as here. The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned embodiments, but also include technical schemes composed of any combination of the above technical features. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present invention.

Claims

1. A multi-supervised intelligent lane semantic segmentation method integrating edge detection, characterized in that: The steps include: Step S1: Select a road condition picture collected by the autonomous driving car and process it as an input picture; Step S2: Perform edge detection on the label truth value corresponding to the input image to obtain a lane line label edge map; Step S3: input the image into the encoding network module for feature extraction to obtain a high-resolution feature map and a high-semantic feature map; Step S4: After performing an upsampling operation on the high semantic feature map, a splicing operation is performed with the high resolution feature map along the channel dimension, and the map is sent to the convolutional layer to reduce the dimension and then upsampled twice to restore the size to the original image size; Step S5: upsample the high-resolution feature map and restore its size to the original image size; Step S6: Using the label truth map as the output main supervision signal, calculate the loss function value; using the label edge map as the auxiliary supervision signal after upsampling the high-resolution feature map, calculate the loss function value; weightedly accumulate the above loss function values, calculate the parameter gradient, and back-propagate to optimize the network structure parameters; Step S7: Train the lane detection model until convergence.

2. The multi-supervised intelligent lane semantic segmentation method based on fusion edge detection according to claim 1 is characterized in that: The step S1 specifically includes: The collected video is sampled in time to obtain continuous frame road surface information maps; after all images are cropped to the same size, a standardized transformation operation is performed; the standardized transformation operation calculation formula is: Among them, I is the value of a pixel point in a channel in the original input image, mean is the original mean of all pixels in the channel, std is the original standard deviation of all pixels in the channel, and I normal The pixel values ​​are normalized and follow the normal distribution.

3. The multi-supervised intelligent lane semantic segmentation method based on fusion edge detection according to claim 1 is characterized in that: The step S2 specifically includes: The Sobel edge detection operator along the vertical direction is used to perform a planar convolution operation with the label true value image to obtain the label edge image corresponding to the label. The edge information calculation formula is as follows: L Edge h y *L Truth (3) Among them, h y is the edge detection operator along the vertical direction, L Truth Represents the input label true value image, L Edge It represents the difference in brightness of each pixel after the convolution operation, reflecting the edge information of the lane line in the label.

4. The multi-supervised intelligent lane semantic segmentation method based on fusion edge detection according to claim 1 is characterized in that: The step S3 specifically includes the following process: (1) Input the standardized image in step S1 into the weight-sharing convolutional neural network feature extraction layer, use convolution and pooling operations to complete the downsampling and feature extraction of the input image, and obtain a low-semantic high-resolution feature map F with a quarter of the size of the original image. I / 4 ; (2) F I / 4 The feature map is continuously input into the deep feature extraction layer network structure, and an abstract high-semantic feature map F with global information and downsampled to one sixteenth of the original image is gradually extracted from the high-resolution features. I / 16 .

5. The multi-supervised intelligent lane semantic segmentation method with fusion edge detection according to claim 1 is characterized in that: The S4 specifically includes the following process: F I / 16 The feature map is four times upsampled using the bilinear interpolation upsampling method to obtain the same I / 4 Features with the same size Along the channel dimension, the two feature maps F I / 4 and After performing concatenation and convolution operations, a secondary upsampling is performed to obtain a feature map F with the same size as the original image. I .

6. The multi-supervised intelligent lane semantic segmentation method based on fusion edge detection according to claim 5 is characterized in that: In the convolution operation, the convolution kernel size is 1*1 and the step size is 1.

7. The multi-supervised intelligent lane semantic segmentation method based on fusion edge detection according to claim 1 is characterized in that: The step S5 specifically includes the following process: Upsample the high-resolution features obtained in step 3 to obtain a feature map of the same size as the original image. The network outputs a feature map F with the same size as the original image. I and 8. The multi-supervised intelligent lane semantic segmentation method with fusion edge detection according to claim 1 is characterized in that: The step S6 specifically includes the following process: A weighted multi-supervisory loss function is used to calculate the difference between the prediction and the true value. First, the true value of the label is taken as the main supervisory signal, and the cross entropy loss function is set to calculate F I With L Truth Secondly, the label edge is used as an auxiliary supervision signal, and the focus loss function is used to calculate the lane line edge regression binary classification problem. With L Edge The difference between them is as follows: FL(p n )=-(1-p n ) γ log(p n ) (5) Loss=Loss CE +λ·Loss FL (6) Among them, expression (4) calculates the cross entropy loss function value, expression (5) calculates the focus loss function value, and expression (6) balances the cross entropy loss function Loss by the weight factor λ. CE And the focal loss function Loss FL The weight ratio is used to calculate the final model loss value, which is used to update and optimize the network parameters; ic represents the predicted probability that the observed sample i belongs to category c, y ic represents the true category of sample i, N represents the total number of samples, M represents the total number of categories, and p n It indicates the probability of observing sample n being predicted as positive class, and γ represents the difficulty and easy sample adjustment parameter.

9. The multi-supervised intelligent lane semantic segmentation method with fusion edge detection according to claim 1, characterized in that: The upsampling operation includes restoring the size of the feature map by a bilinear interpolation operation.

Images