A lane detection method for high-level autonomous driving

Abstract

This invention discloses a lane detection method for high-level autonomous driving. The method uses sample images containing lane lines captured by a camera to obtain ground truth data for model training. This ground truth data is then used to build and optimize an optimized convolutional neural network model. A generative adversarial network is used to perform deblurring and adaptive processing on the image to be detected, obtaining a clear image of the lane lines to be detected. The optimized convolutional neural network model then processes the clear image of the lane lines to be detected through morphological processing, connected domain analysis, and spatial clustering to obtain lane line instance segmentation results. A laser radar is used to detect the road surface to obtain lane line position information. This information is combined with the lane line instance segmentation results to obtain a complete detection result containing lane line category information and lane line position information. This method can meet the requirements of high-level autonomous driving for lane line detection accuracy, recall rate, and information integrity.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception, and more specifically to a lane line detection method, which is particularly applicable to high-level autonomous driving scenarios. Background Art

[0002] Lane detection, as one of the core functions of the perception module, is an essential component of autonomous driving systems. With the rapid development of autonomous driving technology, the application scenarios requiring lane detection and recognition are becoming increasingly widespread. Using onboard sensors to detect lane information on structured roads in real time during driving provides the underlying input for functions such as lane keeping, adaptive cruise control, lane departure warning, and path planning for advanced autonomous driving, thus ensuring critical driving safety.

[0003] Lane detection methods can be categorized into traditional image processing methods based on features or models, and deep learning-based methods. Feature-based lane detection typically utilizes underlying lane features, such as color, edges, texture, and gradient variations, to extract lane lines from the road surface that meet the required characteristics. Model-based lane detection typically constructs a geometric model based on the geometric characteristics of the lane shape. The model parameters are then calculated using methods such as the RANSAC algorithm or the least squares method, which is then used to fit the corresponding lane curve. These methods have fixed models and are relatively easy to implement, but require manual adjustment of operator parameters, such as those for grayscale transformation, Gaussian blur, edge detection, and Hough transform. Their detection results are easily affected by the road environment and are sensitive to curves, lighting, and occlusion, resulting in poor robustness and real-time performance. In recent years, with the increase in the amount of annotated data, the performance of neural network algorithms, and the computing power of GPUs, deep learning-based lane detection algorithms have gradually become mainstream. Numerous convolutional neural network models are used for lane detection, such as LaneNet, DVCNN, and VPGNet. These models primarily solve binary classification problems based on fully convolutional neural networks. This approach can be roughly divided into two main steps: first, predicting the segmentation mask of the lane markings. Then, a lane model, such as a parabola or spline curve, is fitted to the post-processed mask. This approach can effectively detect lanes, but the detection of lane information is incomplete. Detection of information such as lane category and location is still immature, and cannot meet the lane detection accuracy, recall rate, and information integrity requirements of high-level autonomous driving. Summary of the Invention

[0004] In order to avoid the shortcomings of the above-mentioned existing technologies, the present invention provides a lane line detection method for high-level autonomous driving, which can accurately detect lane line category and position information and meet the requirements of high-level autonomous driving for lane line detection accuracy, recall rate and information integrity.

[0005] The present invention adopts the following technical solutions to achieve the purpose of the invention:

[0006] The lane line detection method for high-level autonomous driving of the present invention is characterized by: using sample images containing lane lines collected by a camera to obtain true value data for model training, using the true value data to build and optimize an optimized convolutional neural network model; using a generative adversarial network to perform deblurring adaptation processing on the image to be detected to obtain a clear image of the lane line to be detected; using the optimized convolutional neural network model to process the clear image of the lane line to be detected through morphological processing, connected domain analysis and spatial clustering to obtain a lane line instance segmentation result; using a laser radar to detect the road surface to obtain lane line position information, and combining the lane line position information and the lane line instance segmentation result to obtain a complete detection result including lane line category information and lane line position information.

[0007] The lane line detection method for high-level autonomous driving of the present invention is also characterized by being performed according to the following steps:

[0008] Step 1: Define lane line categories for sample images containing lane lines captured by the camera to obtain lane line labeling specifications; use the lane line labeling specifications to label lane lines to obtain true value data for model training;

[0009] Step 2: Perform image enhancement and category ratio adaptation on the true value data to obtain enhanced true value data;

[0010] Step 3: Using the enhanced truth data, a convolutional neural network model is constructed as follows, including an inter-layer feature aggregation module and an intra-layer information loop transfer module:

[0011] Taking the enhanced ground-truth data as input, the inter-layer feature aggregation module is used to perform multi-scale feature fusion on the output feature maps of different network branches; the intra-layer information loop transfer module is used to perform information enhancement in the sparse feature space of the lane line pixels in the enhanced ground-truth data; the output dimension of the convolutional neural network model is designed based on the one-hot mechanism, a weighted comprehensive loss function including multi-classification focal loss, pixel clustering loss, and regularization term is defined, the weighted comprehensive loss function is optimized using a stochastic gradient descent algorithm, and the convolutional neural network model parameters are updated to obtain an optimized convolutional neural network model;

[0012] Step 4: Use the rain, snow, and fog sensors to determine the weather conditions. If it is rainy, snowy, or foggy, use the generative adversarial network to perform deblurring and adaptive processing on the image to be detected collected in rainy, snowy, or foggy weather to obtain a clear image of the lane line to be detected. If it is not rainy, snowy, or foggy, the image to be detected is a clear image of the lane line to be detected.

[0013] Step 5: Using the optimized convolutional neural network model to perform lane line information inference on the clear image of the lane line to be detected, obtaining a semantic segmentation result containing lane line category information, performing morphological processing, connected domain analysis, and spatial clustering on the semantic segmentation result, obtaining an instance segmentation result containing lane line category information;

[0014] Step 6: Extract echo intensity differences from the road surface information scanned by the LiDAR to obtain a grayscale image containing lane line pixels. Perform DBSCAN clustering on the grayscale image to obtain a LiDAR detection result with lane line position information.

[0015] Step 7: Perform spatiotemporal registration and pixel semantic matching fusion on the instance segmentation result and the lane line position information to obtain a complete detection result including lane line category information and position information, thereby realizing lane line detection.

[0016] The lane line detection method for high-level autonomous driving of the present invention is also characterized in that: in the step 1, the lane line colors in the lane line marking specification are divided into white and yellow, and the lane line forms are divided into solid lines, dashed lines, single lines and double lines. Then, the categories of lane lines with different feature combinations include: white single solid line, white single dashed line, white double solid line, white double dashed line, yellow single solid line, yellow single dashed line, yellow double solid line and yellow double dashed line, and the labels of lane lines of the same type are consistent.

[0017] The lane line detection method for high-level autonomous driving of the present invention is also characterized in that: in the step 2, image enhancement and category ratio adaptation of the true value data means: the original sample image of the true value data is enhanced by means of image flipping, rotation, scaling, cropping and Gaussian filtering, and the lane line categories whose number ratio in the sample image is less than 10% are expanded. After the expansion, the ratio of the number of each lane line category is 0.8-1.2, so as to achieve a uniform ratio of various lane line types and complex scenes.

[0018] The lane line detection method for high-level autonomous driving of the present invention is also characterized in that: in step 3, the inter-layer feature aggregation module has two branch structures, namely a spatial information branch and a high-level semantic branch;

[0019] The number of convolution kernel channels of the spatial information branch is 2 x , x is not less than 6, and the overall stack is 6-8 layers, which is used to extract the spatial position relationship information of lane line pixels in the clear image of the lane line to be detected, capture the lane line pixel information and obtain the feature map of the spatial information branch;

[0020] The convolution kernel channels of the high-level semantic branch are one-fourth of the channels of the convolution kernel of the spatial information branch, and the whole is stacked with 12-14 layers. This is used to extract lane line texture and category information from the clear image of the lane line to be detected, and obtain the feature map of the high-level semantic branch containing lane line texture and category information.

[0021] The inter-layer feature aggregation module aggregates the feature map containing the spatial position relationship information of the lane lines of the spatial information branch and the feature map containing the lane line texture and category information of the high-level semantic branch to obtain an aggregated feature map;

[0022] The lane line detection method for high-level autonomous driving of the present invention is also characterized in that: in step 3, the intra-layer information loop transmission module is:

[0023] The top hidden layer in the high-level semantic branch defines a slice convolution module for extracting the prior relationship between lane line rows and columns. For the three-dimensional tensor C×H×W at the top of the high-level semantic branch, the three-dimensional tensor is first decomposed into H slices along the H direction. The first H-direction slice is processed along one side of the H direction by performing a one-dimensional convolution operation on the first H-direction slice and C convolution kernel tensors of size C×u to obtain a first H-direction processed slice. The first H-direction processed slice is added to the second H-direction slice to obtain a new H-direction slice; the new H-direction slice is operated in the same manner as the first H-direction slice, and the obtained new processed slice is added to the third H-direction slice until all H slices are processed; then the same operation is performed along the other side of the H direction to obtain a first updated three-dimensional tensor of the same size of C×H×W; the first An updated three-dimensional tensor is decomposed into W slices along the W direction. The first W-directional slice is processed along one side of the W direction by performing a one-dimensional convolution operation on the first W-directional slice and C convolution kernel tensors of size C×u to obtain the first W-directional processed slice, and the first W-directional processed slice is added to the second W-directional slice to obtain a new W-directional slice; the new W-directional slice is operated on in the same way as the first W-directional slice, and the obtained new processed slice is added to the third W-directional slice until all W slices are processed; then the same operation is performed along the other side of the W direction to obtain a final updated three-dimensional tensor of the same size of C×H×W, which is used to enhance the extraction of the slender features of the lane lines; C, H and W are used to represent the number of channels, the number of rows and the number of columns of the three-dimensional tensor C×H×W in a one-to-one correspondence; u represents the width of the convolution kernel of size C×u.

[0024] The lane line detection method for high-level autonomous driving of the present invention is also characterized by:

[0025] Designing the output dimension of the convolutional neural network model according to the one-hot mechanism in step 3 and defining a weighted comprehensive loss function including a multi-classification focal loss, a pixel clustering loss, and a regularization term means: performing convolution on the aggregated feature map using a 1×1 convolution kernel, defining the number of the 1×1 convolution kernels according to the number of lane line categories, so that the number of lane line categories plus background equals the number of channels of the semantic segmentation result at the final output of the network; optimizing the weighted comprehensive loss function using the stochastic gradient descent algorithm and updating the convolutional neural network model parameters; the weighted comprehensive loss function includes a multi-classification focal loss function, a pixel clustering loss function, and a regularization term;

[0026] The multi-classification focal loss function is represented by formula (1):

[0027] FL softmax =-α c (1-p c ) γ log(p c ) (1)

[0028] In formula (1):

[0029] FL softmax is the multi-classification focal loss function value;

[0030] p c Represents the probability value of the c-th lane line output by the softmax function;

[0031] γ is the modulation factor; α c is the positive sample weight;

[0032] The pixel clustering loss function is represented by formula (2):

[0033] L cluster =L var +L dis t (2)

[0034] In formula (2):

[0035] L cluster is the pixel clustering loss function value, L var is the variance term, L dist Let be the distance term, represented by (3):

[0036]

[0037] In formula (3):

[0038] W represents the number of lanes, w represents the w-th lane, w = 1, 2, ..., W;

[0039] Nw represents the total number of pixels in the wth lane, μ w represents the average pixel value of the w-th lane;

[0040] i represents the i-th pixel, x i represents the pixel value of the i-th pixel;

[0041] w A and w B Both represent lanes, and lane w A and lane w B For different lanes;

[0042] μ wA Indicates lane w A The average pixel value, μ wB Indicates lane w B The average pixel value of

[0043] δ v and δ d All are set values;

[0044] The regularization term is represented by formula (4):

[0045]

[0046] In formula (4): L2 is the regularization term value, λ is the hyperparameter, and J is the network weight;

[0047] Then, the weighted comprehensive loss function is represented by formula (5):

[0048] Loss=ω1FL softmax +ω2L cluster +ω3L2 (5)

[0049] In formula (5): Loss is the weighted comprehensive loss function value;

[0050] ω1 is the weight of the pixel clustering loss function, ω2 is the weight of the pixel clustering loss function, and ω2 is the weight of the regularization term.

[0051] The lane line detection method for high-level autonomous driving of the present invention is also characterized by:

[0052] The morphological processing, connected domain analysis, and spatial clustering described in step 5 refer to: filling the lane line pixel holes in the instance segmentation result containing the lane line category information using an image morphological closing operation algorithm, connecting adjacent pixel regions with the same lane line pixel value in the instance segmentation result containing the lane line category information using a connected domain analysis algorithm, and further spatially clustering the feature pixels in the instance segmentation result containing the lane line category information using a spatial clustering algorithm to obtain different lane line instances.

[0053] The lane line detection method for high-level autonomous driving of the present invention is also characterized by:

[0054] The fusion of spatiotemporal registration and pixel semantic matching in step 7 refers to: first, using the time index and data interpolation method to align the camera and lidar detection timestamps, and using joint calibration and spatial conversion to spatially align the camera and lidar to obtain temporal and spatial registration data; calculating the overlap rate loss of the lane line in the instance segmentation result and the lane line in the lidar detection result, and using the overlap rate loss to establish a cost matrix; using the Hungarian algorithm to solve the cost matrix to obtain the minimum cost lane line matching pair; if the IOU of the minimum cost lane line matching pair is not less than 0.5, the matching pair is determined to be the same lane line in the instance segmentation result and the lidar detection result, and the lane line category information contained in the instance segmentation result of the same lane line is fused with the lane line position information contained in the lidar detection result to obtain complete detection information including lane line position and category;

[0055] The overlap loss is characterized by equation (6):

[0056]

[0057] In formula (6): C is the overlap loss; IOU is the overlap value;

[0058] A i Lane line A in the instance segmentation result i ;

[0059] B j Lane line B in the laser radar detection result j .

[0060] This invention implements end-to-end multi-classification detection of lane line categories and location information, meeting the requirements of high-level autonomous driving for lane line detection accuracy, recall rate, and information integrity. Compared with existing technologies, the present invention has the following advantages:

[0061] 1. The present invention provides a fusion scheme for extracting multi-scale features using a multi-branch network structure. It uses more channels and shallower network branches to retain rich spatial information and generate high-resolution feature maps. It uses fewer channels and a deeper network to quickly sample and obtain high-level semantic information. It further fuses multi-scale features to obtain a richer feature space and enhance the original feature information.

[0062] 2. This invention considers the shape continuity and spatial dependence of lane lines and establishes a parallel information transmission module within the layer, namely the slice convolution module, to supplement the lane line reasoning capability in complex scenarios and improve the accuracy and recall rate of lane line detection;

[0063] 3. The present invention embeds a defuzzification processing module, namely a generative adversarial module, at the input of the trained network to improve the robustness of lane line detection in severe weather conditions.

[0064] 4. This invention combines the Hungarian algorithm to match and fuse the same lane line from two data sources, achieving comprehensiveness and high fault tolerance for lane line detection in high-level autonomous driving.

[0065] 5. The present invention combines laser radar to obtain lane line position information and neural network to obtain lane line category information, thereby realizing comprehensive detection of lane line information including lane line category and position. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of a lane detection method based on the fusion of lidar and deep neural network in an embodiment of the present invention;

[0067] Figure 2 This is a two-branch structure diagram of a deep neural network in an embodiment of the present invention;

[0068] Figure 3 Schematic diagram of the characteristics of the slice convolution structure in an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of generating an adversarial structure in an embodiment of the present invention; DETAILED DESCRIPTION

[0070] The lane line detection method for high-level autonomous driving in this embodiment is to use the sample images containing lane lines collected by the camera to obtain the true value data for model training, and use the true value data to build and optimize the optimized convolutional neural network model; use the generative adversarial network to perform deblurring adaptation processing on the image to be detected to obtain a clear image of the lane line to be detected; use the optimized convolutional neural network model to process the clear image of the lane line to be detected through morphological processing, connected domain analysis and spatial clustering to obtain the lane line instance segmentation result; use the laser radar to detect the road surface to obtain the lane line position information, and combine the lane line position information and the lane line instance segmentation result to obtain a complete detection result including the lane line category information and the lane line position information. The complete lane line detection process is as follows: Figure 1 shown.

[0071] The lane line detection method for high-level autonomous driving in this embodiment is performed in the following steps:

[0072] Step 1: Define lane line categories for sample images containing lane lines captured by the camera and obtain lane line labeling specifications; use the lane line labeling specifications to label lane lines and obtain true value data for model training.

[0073] Step 2: Perform image enhancement and category ratio adaptation on the true value data to obtain enhanced true value data.

[0074] Step 3: Use the enhanced truth data to build a convolutional neural network model that includes an inter-layer feature aggregation module and an intra-layer information loop transfer module as follows:

[0075] Using pytorch as the deep learning framework, the batch size is set to 16, and the enhanced truth data is used as input. The inter-layer feature aggregation module is used to perform multi-scale feature fusion on the output feature maps of different network branches. The intra-layer information loop transfer module is used to enhance the information in the sparse feature space of the lane line pixels in the enhanced truth data. The output dimension of the convolutional neural network model is designed according to the one-hot mechanism, and a weighted comprehensive loss function including multi-classification focal loss, pixel clustering loss and regularization term is defined. The stochastic gradient descent algorithm is used to optimize the weighted comprehensive loss function, and the convolutional neural network model parameters are updated to obtain the optimized convolutional neural network model.

[0076] Step 4: Use the rain, snow and fog sensors to judge the weather conditions. If it is rainy, snowy or foggy, use the generative adversarial network to perform deblurring and adaptation processing on the image to be detected collected in rainy, snowy and foggy weather to obtain a clear image of the lane line to be detected; if it is not rainy, snowy or foggy, the image to be detected is a clear image of the lane line to be detected. The schematic diagram of the generative adversarial network is shown below. Figure 4 shown.

[0077] Step 5: Use the optimized convolutional neural network model to infer lane line information on the clear image of the lane line to be detected, obtain a semantic segmentation result containing lane line category information, perform morphological processing, connected domain analysis, and spatial clustering on the semantic segmentation result to obtain an instance segmentation result containing lane line category information.

[0078] Step 6: Extract the echo intensity difference of the road surface information scanned by the lidar to obtain a grayscale image containing lane line pixels. Perform DBSCAN clustering on the grayscale image to obtain the lidar detection result with lane line position information.

[0079] Step 7: Perform spatiotemporal registration and pixel semantic matching fusion on the instance segmentation results and lane line position information to obtain a complete detection result containing lane line category information and position information, thereby realizing lane line detection.

[0080] In specific implementation, the corresponding technical measures also include:

[0081] In step 1, the lane line colors in the lane line marking specification are divided into white and yellow, and the lane line forms are divided into solid lines, dashed lines, single lines, and double lines. The lane line categories with different feature combinations include: single solid white line, single dashed white line, double solid white line, double dashed white line, single solid yellow line, single dashed yellow line, double solid yellow line, and double dashed yellow line. Lane lines of the same type have the same label. The lane line marking tool uses Labelme software.

[0082] In step 2, image enhancement and category ratio adaptation of the true value data means: using image flipping, rotation, scaling, cropping and Gaussian filtering to enhance the original sample image of the true value data, and expanding the lane line categories whose number ratio in the sample image is less than 10% to achieve a uniform ratio of various lane line types and complex scenes. The original sample image enhancement increases the diversity of the training samples, and the expansion of the number of lane line categories makes the training more sufficient, thereby effectively improving the lane line detection accuracy.

[0083] In step 3, the inter-layer feature aggregation module has two branch structures, namely the spatial information branch and the high-level semantic branch. The two-branch structure of the deep neural network is as follows: Figure 2 shown.

[0084] The number of convolution kernel channels of the spatial information branch is 2 x , x is not less than 6, x can be taken as 6, 6-8 layers can be stacked, and 6 layers can be stacked, with 64 channels per layer, which is used to extract the spatial position relationship information of lane line pixels in the clear image of the lane line to be detected, capture the lane line pixel information and obtain the feature map of the spatial information branch.

[0085] The number of convolution kernel channels in the high-level semantic branch is one-fourth of that in the spatial information branch. The overall stack consists of 12-14 layers, and can be set to 13 layers. This is used to extract lane line texture and category information from the clear image of the lane line to be detected, and obtain the feature map of the high-level semantic branch containing lane line texture and category information.

[0086] The inter-layer feature aggregation module aggregates the feature map of the spatial information branch and the feature map of the high-level semantic branch containing lane line texture and category information to obtain an aggregated feature map. Aggregation is to stack the channels of the two branches, that is, to fuse the features of the two branches to promote the combination and transmission of information.

[0087] The information loop transmission module in the middle layer of step 3 is as follows: Figure 3 As shown:

[0088] A slice convolution module is defined in the top hidden layer of the high-level semantic branch to extract the prior relationship between lane line rows and columns. For the three-dimensional tensor C×H×W at the top of the high-level semantic branch, the three-dimensional tensor is first decomposed into H slices along the H direction. The first H-direction slice is processed along one side of the H direction by performing a one-dimensional convolution operation on the first H-direction slice and C convolution kernel tensors of size C×u to obtain the first H-direction processed slice. The first H-direction processed slice is added to the second H-direction slice to obtain a new H-direction slice; the new H-direction slice is operated in the same way as the first H-direction slice, and the new processed slice is added to the third H-direction slice until all H slices are processed; then the same operation is performed along the other side of the H direction to obtain The first updated three-dimensional tensor is also of size C×H×W; the first updated three-dimensional tensor is decomposed into W slices along the W direction, and the first W-direction slice is processed along one side of the W direction by performing a one-dimensional convolution operation on the first W-direction slice and C convolution kernel tensors of size C×u to obtain the first W-direction processed slice, and the first W-direction processed slice is added to the second W-direction slice to obtain a new W-direction slice; the new W-direction slice is operated in the same way as the first W-direction slice, and then the new processed slice is added to the third W-direction slice until all W slices are processed; then the same operation is performed along the other side of the W direction to obtain the final updated three-dimensional tensor of size C×H×W, so as to enhance the extraction of the slender features of the lane lines.

[0089] C, H, and W correspond one to one to represent the number of channels, rows, and columns of the three-dimensional tensor C×H×W;

[0090] Let u represent the width of the convolution kernel of size C×u, and u can be taken as 9.

[0091] In step 3, the output dimension of the convolutional neural network model is designed according to the one-hot mechanism, and the weighted comprehensive loss function including multi-classification focal loss, pixel clustering loss and regularization term is defined as:

[0092] The aggregated feature map is convolved with a 1×1 convolution kernel. The number of 1×1 convolution kernels is defined by the number of lane line categories, so that the number of lane line categories plus background equals the number of channels in the semantic segmentation result at the final output of the network. The predefined number of lane line categories is 8, so the number of channels in the semantic segmentation output is 9. The weighted comprehensive loss function is optimized using the stochastic gradient descent algorithm to update the convolutional neural network model parameters. The weighted comprehensive loss function includes a multi-classification focal loss function, a pixel clustering loss function, and a regularization term.

[0093] The multi-classification focal loss function is represented by formula (1):

[0094] FL softmax =-αc (1-p v ) γ log(p c ) (1)

[0095] In formula (1): FL softmax is the multi-classification focal loss function value;

[0096] p c Represents the probability value of the c-th lane line output by the softmax function;

[0097] γ is the modulation factor, the value is 2; α c is the positive sample weight;

[0098] The pixel clustering loss function is represented by formula (2):

[0099] L cluster =L var +L dist (2)

[0100] In formula (2):

[0101] L cluster is the pixel clustering loss function value, L var is the variance term, L dis t is the distance term, represented by (3):

[0102]

[0103] In formula (3):

[0104] W represents the number of lanes, w represents the w-th lane, w = 1, 2, ..., W;

[0105] N w represents the total number of pixels in the wth lane, μ w represents the average pixel value of the w-th lane;

[0106] i represents the i-th pixel, x i represents the pixel value of the i-th pixel;

[0107] w A and w B Both represent lanes, and lane w A and lane w B For different lanes;

[0108] μ wA Indicates lane w A The average pixel value, μ wB Indicates lane w B The average pixel value of

[0109] δv and δ d are all set values, and δ v Set to 0.5, and set δ d Set to 3;

[0110] The regularization term is represented by formula (4):

[0111]

[0112] In formula (4):

[0113] L2 is the regularization term value, λ is the hyperparameter, λ is set to 0.02, and J is the network weight;

[0114] Then, the weighted comprehensive loss function is represented by formula (5):

[0115] Loss=ω1FL softmax +ω2L cluster +ω3L2 (5)

[0116] In formula (5), Loss is the weighted comprehensive loss function value; ω1 is the weight of the pixel clustering loss function, ω2 is the weight of the pixel clustering loss function, and ω3 is the weight of the regularization term; ω1, ω2, and ω3 are all set to 1.

[0117] Morphological processing, connected domain analysis, and spatial clustering in step 5 refer to:

[0118] The image morphological closing operation algorithm is used to fill the lane line pixel holes in the instance segmentation results containing lane line category information, that is, to optimize the network output results, which is convenient for the subsequent statistics of the instance segmented lane lines; the connected domain analysis algorithm is used to connect the adjacent pixel areas with the same lane line pixel value in the instance segmentation results containing lane line category information, and the spatial clustering algorithm is further used to spatially cluster the feature pixels in the instance segmentation results containing lane line category information to obtain different lane line instances; in the specific implementation, the image morphological closing operation algorithm and the connected domain analysis algorithm are processed using the corresponding functions in the opencv library, and the spatial clustering algorithm is processed using the corresponding functions in the sklearn library.

[0119] The fusion of spatiotemporal registration and pixel semantic matching in step 7 refers to: first, using the time index and data interpolation method to align the camera and lidar detection timestamps, and using joint calibration and spatial transformation to spatially align the camera and lidar to obtain temporal and spatial registration data; calculating the overlap rate loss of the lane line in the instance segmentation result and the lane line in the lidar detection result, and using the overlap rate loss to establish a cost matrix; using the Hungarian algorithm to solve the cost matrix to obtain the minimum cost lane line matching pair; the matching pair with an overlap rate of not less than 0.5 in the minimum cost lane line matching pair is identified as the same lane line in the instance segmentation result and the lidar detection result, and the lane line category information contained in the instance segmentation result of the same lane line is fused with the lane line position information contained in the lidar detection result to obtain complete detection information including the lane line position and category.

[0120] The overlap loss is represented by equation (6):

[0121]

[0122] In formula (6): C is the overlap loss, IOU is the overlap value;

[0123] A i Representing the lane line A in the instance segmentation result i ;B j Represents the lane line B in the lidar detection results j .

[0124] The spatiotemporal registration in step 7 is the prerequisite for the coordinated work of the lidar and the camera. The key is to achieve time and space synchronization between the lidar and the camera. The specific method is as follows: the frequency of lidar data acquisition is usually 10Hz, and the frequency of camera data acquisition is usually 30Hz. In order to synchronize the lidar detection results and the camera detection results, the timestamp of each frame of lidar detection results is used as the synchronization reference time to match the camera detection results that are closest in time. Let the camera acquisition time be t, the lidar corresponding to the previous and next two frames be t1 and t2, and the lidar acquisition data corresponding to t1 and t2 are The collected data X of the millimeter-wave radar at time t is obtained by linear interpolation according to formula (5-1): t :

[0125]

[0126] In this way, the release frequency of lidar and camera information is consistent, achieving time synchronization between the two sensors.

[0127] As a whole, the multi-sensor fusion system requires a unified coordinate system to achieve data fusion, that is, the lidar and camera need to be jointly calibrated and spatially transformed. The respective coordinate systems are established according to the installation positions of the lidar and camera, and the rotation and translation matrix between the lidar and camera is obtained as follows: the automatic joint calibration method of the checkerboard calibration plate is used for calibration, that is, the coordinates of multiple sets of corresponding points of the checkerboard calibration plate are obtained in the lidar coordinate system and the camera coordinate system respectively, and the rotation and translation matrix is ​​solved using the coordinates of the corresponding points in the two coordinate systems. The lidar detection results are projected onto the camera detection results according to the rotation and translation matrix, thereby achieving time synchronization and spatial transformation between the camera and lidar.

[0128] In step 7, the Hungarian algorithm solves the cost matrix in the following steps:

[0129] Step C1: Based on the statistics of the clustering algorithm, there are n lane line clusters from the lidar and m lane line clusters from the camera. The overlap loss value is calculated for each set of data. The size of the cost matrix established using the overlap loss value is n×m.

[0130] Step C2, traverse the cost matrix;

[0131] Step C3, subtract the minimum element of each row of the cost matrix;

[0132] Step C4, subtract the minimum element in each column of the cost matrix;

[0133] Step C5: Cover all zeros in the cost matrix with the least number of lines.

[0134] Step C6: If the number of lines is equal to min(m, n), then the optimal allocation is found;

[0135] Step C7: Find the minimum cost value in the cost matrix that is not covered by the line at this time, subtract this element from the rows not covered by the line, add this element to the columns covered by the line, and jump to step C3 until the optimal matching pair is found.

[0136] The selection is made based on the cost value between the targets of the optimal matching pair. If the cost value is less than the overlap loss value, the match is considered successful and the final matching pair is output. The matched lane line information contains complete information about the lane line position and category.

[0137] This embodiment describes the entire process from data preprocessing to information fusion. The output results include information from both the lidar and the camera, specifically including lane position and category, meeting the requirements for lane information integrity for high-level autonomous driving.

Claims

1. A lane detection method for high-level autonomous driving, characterized by: Using sample images containing lane lines captured by a camera to obtain true value data for model training, and using the true value data to build and optimize an optimized convolutional neural network model; Using a generative adversarial network to perform deblurring and adaptation processing on the image to be detected, a clear image of the lane lines to be detected is obtained; using the optimized convolutional neural network model to process the clear image of the lane lines to be detected through morphological processing, connected domain analysis, and spatial clustering to obtain a lane line instance segmentation result; Lane position information is obtained by detecting the road surface using a laser radar. This lane position information is then combined with the lane instance segmentation results to obtain a complete detection result including lane category information and lane position information. The lane detection method for high-level autonomous driving is performed in the following steps: Step 1: Define lane line categories for sample images containing lane lines captured by the camera to obtain lane line labeling specifications; use the lane line labeling specifications to label lane lines to obtain true value data for model training; Step 2: Perform image enhancement and category ratio adaptation on the true value data to obtain enhanced true value data; Step 3: Using the enhanced truth data, a convolutional neural network model is constructed as follows, including an inter-layer feature aggregation module and an intra-layer information loop transfer module: Taking the enhanced true value data as input, the inter-layer feature aggregation module is used to perform multi-scale feature fusion on the output feature maps of different network branches; The intra-layer information loop transfer module is used to perform information enhancement in the sparse feature space of the lane line pixels in the enhanced true value data; the output dimension of the convolutional neural network model is designed based on the one-hot mechanism, a weighted comprehensive loss function including multi-classification focal loss, pixel clustering loss, and regularization term is defined, the weighted comprehensive loss function is optimized using a stochastic gradient descent algorithm, and the convolutional neural network model parameters are updated to obtain an optimized convolutional neural network model; Step 4: Use the rain, snow, and fog sensors to determine the weather conditions. If it is rainy, snowy, or foggy, use the generative adversarial network to perform deblurring and adaptive processing on the image to be detected collected in rainy, snowy, or foggy weather to obtain a clear image of the lane line to be detected. If it is not rainy, snowy, or foggy, the image to be detected is a clear image of the lane line to be detected. Step 5: Using the optimized convolutional neural network model to perform lane line information inference on the clear image of the lane line to be detected, obtaining a semantic segmentation result containing lane line category information, performing morphological processing, connected domain analysis, and spatial clustering on the semantic segmentation result, obtaining an instance segmentation result containing lane line category information; Step 6: Extract echo intensity differences from the road surface information scanned by the LiDAR to obtain a grayscale image containing lane line pixels. Perform DBSCAN clustering on the grayscale image to obtain a LiDAR detection result with lane line position information. Step 7: Perform spatiotemporal registration and pixel semantic matching fusion on the instance segmentation result and the lane line position information to obtain a complete detection result including lane line category information and position information, thereby realizing lane line detection.

2. The lane line detection method for high-level autonomous driving according to claim 1 is characterized in that: In step 1, the lane line colors in the lane line marking specification are divided into white and yellow, and the lane line forms are divided into solid lines, dashed lines, single lines and double lines. The lane line categories with different feature combinations include: white single solid line, white single dashed line, white double solid line, white double dashed line, yellow single solid line, yellow single dashed line, yellow double solid line and yellow double dashed line. The labels of the same type of lane lines are consistent.

3. The lane line detection method for high-level autonomous driving according to claim 1 is characterized in that: In step 2, image enhancement and category ratio adaptation of the true value data refers to: enhancing the original sample image of the true value data by means of image flipping, rotation, scaling, cropping and Gaussian filtering, and expanding the lane line categories with a number ratio of less than 10% in the sample image. After expansion, the ratio of the number of lane line categories is 0.8-1.2, achieving a uniform ratio of various lane line types and complex scenes.

4. The lane detection method for high-level autonomous driving according to claim 1, wherein: exist In step 3, the inter-layer feature aggregation module has two branch structures, namely a spatial information branch and a high-level semantic branch; The number of convolution kernel channels of the spatial information branch is 2 x , x is not less than 6, and the overall stack is 6-8 layers, which is used to extract the spatial position relationship information of lane line pixels in the clear image of the lane line to be detected, capture the lane line pixel information and obtain the feature map of the spatial information branch; The convolution kernel channels of the high-level semantic branch are one-fourth of the channels of the convolution kernel of the spatial information branch, and the whole is stacked with 12-14 layers. This is used to extract lane line texture and category information from the clear image of the lane line to be detected, and obtain the feature map of the high-level semantic branch containing lane line texture and category information. The inter-layer feature aggregation module aggregates the feature map of the spatial information branch containing the spatial position relationship information of the lane lines and the feature map of the high-level semantic branch containing the lane line texture and category information to obtain an aggregated feature map.

5. The lane detection method for high-level autonomous driving according to claim 4, wherein: In step 3, the intra-layer information loop transmission module is as follows: a slice convolution module is defined in the top hidden layer of the high-level semantic branch to extract the prior relationship between lane line rows and columns. For the three-dimensional tensor C×H×W at the top of the high-level semantic branch, the three-dimensional tensor is first decomposed into H slices along the H direction, and the first H-direction slice is processed along one side of the H direction by performing a one-dimensional convolution operation on the first H-direction slice and C convolution kernel tensors of size C×u to obtain the first H-direction processed slice, and the first H-direction processed slice is added to the second H-direction slice to obtain a new H-direction slice; the new H-direction slice is operated in the same manner as the first H-direction slice, and the obtained new processed slice is added to the third H-direction slice until all H slices are processed; then the H-direction slice is processed. The same operation is performed on the other side to obtain a first updated three-dimensional tensor of the same size of C×H×W; the first updated three-dimensional tensor is decomposed into W slices along the W direction, and the first W slice is processed on one side along the W direction by performing a one-dimensional convolution operation on the first W slice and C convolution kernel tensors of size C×u to obtain a first W-direction processed slice, and the first W-direction processed slice is added to the second W-direction slice to obtain a new W-direction slice; the new W-direction slice is operated on in the same manner as the first W-direction slice, and then the obtained new processed slice is added to the third W-direction slice until all W slices are processed; then the same operation is performed on the other side along the W direction to obtain a final updated three-dimensional tensor of the same size of C×H×W, which is used to enhance the extraction of the slender features of the lane lines; C, H, and W correspond one to one to represent the number of channels, rows, and columns of the three-dimensional tensor C×H×W; Let u represent the width of the convolution kernel of size C×u.

6. The lane detection method for high-level autonomous driving according to claim 4, wherein: Designing the output dimension of the convolutional neural network model according to the one-hot mechanism in step 3 and defining a weighted comprehensive loss function including a multi-classification focal loss, a pixel clustering loss, and a regularization term means: performing convolution on the aggregated feature map using a 1×1 convolution kernel, defining the number of the 1×1 convolution kernels according to the number of lane line categories, so that the number of lane line categories plus background equals the number of channels of the semantic segmentation result at the final output of the network; optimizing the weighted comprehensive loss function using the stochastic gradient descent algorithm and updating the convolutional neural network model parameters; the weighted comprehensive loss function includes a multi-classification focal loss function, a pixel clustering loss function, and a regularization term; The multi-classification focal loss function is represented by formula (1): FL softmax =-a c (1-p c ) γ log(p c ) (1) In formula (1): FL softmax is the multi-classification focal loss function value; p c Represents the probability value of the c-th lane line output by the softmax function; γ is the modulation factor; α c is the positive sample weight; The pixel clustering loss function is represented by formula (2): L cluster =L var +L dist (2) In formula (2): L cluster is the pixel clustering loss function value, L var is the variance term, L dist Let be the distance term, represented by (3): In formula (3): W represents the number of lanes, w represents the w-th lane, w = 1, 2, ..., W; N w represents the total number of pixels in the wth lane, μ w represents the average pixel value of the w-th lane; i represents the i-th pixel, x i represents the pixel value of the i-th pixel; w A and w B Both represent lanes, and lane w A and lane w B For different lanes; Indicates lane w A The average pixel value, Indicates lane w B The average pixel value of δ v and δ d All are set values; The regularization term is represented by formula (4): In formula (4): L2 is the regularization term value, λ is the hyperparameter, and J is the network weight; Then, the weighted comprehensive loss function is represented by formula (5): Loss=ω1FL softmax +ω2L cluster +ω3L2 (5) In formula (5): Loss is the weighted comprehensive loss function value; ω1 is the weight of the pixel clustering loss function, ω2 is the weight of the pixel clustering loss function, and ω3 is the weight of the regularization term.

7. The lane detection method for high-level autonomous driving according to claim 1, wherein: The morphological processing, connected domain analysis, and spatial clustering described in step 5 refer to: filling the lane line pixel holes in the instance segmentation result containing the lane line category information using an image morphological closing operation algorithm, connecting adjacent pixel regions with the same lane line pixel value in the instance segmentation result containing the lane line category information using a connected domain analysis algorithm, and further spatially clustering the feature pixels in the instance segmentation result containing the lane line category information using a spatial clustering algorithm to obtain different lane line instances.

8. The lane detection method for high-level autonomous driving according to claim 1, wherein: The fusion of spatiotemporal registration and pixel semantic matching in step 7 refers to: first, using the time index and data interpolation method to align the camera and lidar detection timestamps, and using joint calibration and spatial conversion to spatially align the camera and lidar to obtain temporal and spatial registration data; calculating the overlap rate loss of the lane line in the instance segmentation result and the lane line in the lidar detection result, and using the overlap rate loss to establish a cost matrix; using the Hungarian algorithm to solve the cost matrix to obtain the minimum cost lane line matching pair; if the IOU of the minimum cost lane line matching pair is not less than 0.5, the matching pair is determined to be the same lane line in the instance segmentation result and the lidar detection result, and the lane line category information contained in the instance segmentation result of the same lane line is fused with the lane line position information contained in the lidar detection result to obtain complete detection information including lane line position and category; The overlap loss is characterized by equation (6): In formula (6): C is the overlap loss; IOU is the overlap value; A i Lane line A in the instance segmentation result i ; B j Lane line B in the laser radar detection result j .

Images