Lane Line Detection and Fitting Method Based on Deep Learning and Vehicle

Abstract

The present invention discloses a lane line detection and fitting method, system, vehicle and storage medium based on deep learning, and also completes the geometric transformation and polynomial fitting parts in the lane line information acquisition task through a lightweight deep learning network, realizing end-to-end lane line information acquisition. During the process of realizing fitting, the consideration of image features is added, and the stability of information acquisition is improved through the training of a large amount of data. The present invention effectively reduces the influence of factors such as terrain changes and vehicle body postures on the final output result.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and particularly relates to a lane line detection and fitting method and vehicle based on deep learning. Background Technique

[0002] In recent years, the development and application of autonomous driving-related technologies have become increasingly mature. Among them, the detection and information extraction of lane lines are one of the key issues in the field of autonomous driving environment perception. Its effect will directly affect the stability and accuracy of subsequent autonomous driving path planning and vehicle control.

[0003] The existing detection method is to perform recognition and detection by extracting image features from the video collected by the vehicle-mounted front camera. According to different principles, it is divided into traditional image detection methods and deep learning-based methods. Given that the traditional method has weak adaptability to complex environmental factors such as light and occlusion, the actual implementation effect is often poor. Therefore, at present, the lane line detection method based on deep learning has become the main research and application direction.

[0004] In the prior art, a deep learning model is used to complete the task of obtaining the lane line coordinates in the image space, and then the image is transformed into the top-down physical space through camera calibration and geometric transformation, and then the lane line parameter information is obtained by fitting with a polynomial (usually a third-order polynomial) and provided to the subsequent planning and control module. However, during the actual operation of the vehicle, the unevenness and slope of the road surface and the body jitter will all affect the accuracy and stability of the geometric transformation, and thus affect the fitting result of the lane line. For example, a lane line detection algorithm based on an instance segmentation neural network framework disclosed in patent document CN109740465A proposes a method of dynamically outputting a bird's-eye projection transformation matrix by using a neural network, which can optimize the above problems to a certain extent. However, the main task of the neural network is still to obtain the lane line, and the fitting process still uses the traditional mathematical fitting method, and its adaptability to complex scenarios still needs to be strengthened.

[0005] Therefore, it is necessary to develop a new lane line detection and fitting method and vehicle based on deep learning. Summary of the Invention

[0006] The purpose of the present invention is to provide a lane line detection and fitting method and vehicle based on deep learning, which can improve the stability of the attribute parameters of the finally output lane line.

[0007] In the first aspect, a lane line detection and fitting method based on deep learning according to the present invention includes the following steps:

[0008] Step 1: Collect lane line image data and label the lane line position coordinates;

[0009] Step 2: Use the calibration parameters of the camera to perform perspective transformation on the collected images to obtain the lane line coordinates from a top-down perspective. Then, fit the lane lines to obtain the parameters of the third-order polynomial of the lane lines, and use these parameters as the annotation data of the images to form the required dataset;

[0010] Step 3: Preprocess the collected image data;

[0011] Step 6: Construct a backbone convolutional neural network model, and use the image data processed in Step 3 as the input of the backbone convolutional neural network model. The backbone convolutional neural network model extracts the image feature information;

[0012] Step 5: Construct a lane line position prediction model, and use the image feature information output in Step 4 as the input of this lane line position prediction model. The lane line position prediction model outputs the lane line position information;

[0013] Step 6: Construct a lane line fitting parameter prediction model, and use the image feature information in Step 4 and the lane line position information obtained in Step 5 as the input of the lane line fitting parameter prediction model together. The lane line fitting parameter prediction model outputs the predicted lane line fitting result;

[0014] Step 7: Train the constructed backbone convolutional neural network model, lane line position prediction model, and lane line fitting parameter prediction model to make the error reach the expected value, and obtain the trained lane line fitting parameter prediction model;

[0015] Step 8: Use the lane line fitting parameter prediction model obtained in Step 7. After processing the collected image data through Step 3, input it into the trained lane line fitting parameter prediction model to obtain the prediction result of the lane line fitting parameters.

[0016] Optionally, in Step 1, repeat image collection multiple times in various road surface terrains, and enrich the sample size of the data by flipping and translating the data.

[0017] Optionally, Step 2 is specifically as follows:

[0018] Through camera calibration, obtain the perspective transformation matrix of the collected images:

[0019]

[0020] where: a, b, c, d, e, f are transformation parameters. Given a pixel point p = [x, y, 1]T with coordinates (x, y) in the original image, its projected coordinates are p' = H * p = [x', y', 1]T, where T represents the transpose of the matrix;

[0021] Then, the points on the projected lane line are fitted by the least squares method to obtain the parameters in the third-order polynomial x’ = f(y’) = A0 + A1*y’ + A2*y’2 + A3*y’3 corresponding to the lane line, that is, [A0, A1, A2, A3]. These parameters will be used as the true value data for subsequent model training.

[0022] Optionally, in step 3, it specifically includes: cropping the region of interest, scaling the image size, and normalizing the data.

[0023] Optionally, in step 5, the cross-entropy between the predicted lane line position information and the lane line position information marked in step 1 is used as the first loss function to obtain the error between the predicted lane line position information and the true lane line position information.

[0024] Optionally, in step 6, assuming that n lane lines are predicted, the lane line fitting parameter prediction model will predict n sets of fitting parameters A0′ n , A1′ n , A2′ n , A3′ n , combined with the true value data A0 of the lane line fitting parameters obtained in step 2 n , A1 n , A2 n , A3 n , set the second loss function Loss as:

[0025]

[0026] Use the second loss function to obtain the error between the predicted value of the lane line fitting parameter and the true value of the lane line fitting parameter.

[0027] Optionally, in step 7, using the image data obtained in step 1 and the true value data of the fitting parameters obtained in step 2, the backbone convolutional neural network model, the lane line position prediction model, and the lane line fitting parameter prediction model are trained and optimized by the backpropagation algorithm until the errors obtained by the first loss function and the second loss function reach the corresponding expected values, and a trained lane line fitting parameter prediction model is obtained.

[0028] In a second aspect, a lane line detection and fitting system based on deep learning according to the present invention includes a memory and a controller, and a computer-readable program is stored therein. When the computer-readable program is called by the controller, it can execute the steps of the lane line detection and fitting method based on deep learning according to the present invention.

[0029] In a third aspect, a vehicle according to the present invention uses the lane line detection and fitting system based on deep learning according to the present invention.

[0030] In a fourth aspect, a storage medium according to the present invention stores a computer-readable program which, when called, can execute the steps of the lane line detection and fitting method based on deep learning according to the present invention.

[0031] The present invention has the following advantages:

[0032] (1) On the basis of obtaining the lane line coordinates by using a deep learning model, a lightweight fitting model is designed to directly output the fitting result of the lane line, that is, the polynomial coefficients; by introducing image features and combining them with the lane line coordinate information, the stability of the finally output attribute parameters of the lane line is improved.

[0033] (2) In the present invention, the geometric transformation and polynomial fitting parts in the lane line information acquisition task are also completed by a lightweight deep learning network, realizing end-to-end lane line information acquisition; the consideration of image features is added in the process of realizing fitting, and the stability of information acquisition is improved through the training of a large amount of data, effectively reducing the influence of factors such as terrain changes and vehicle body postures on the finally output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart in this embodiment;

[0035] Figure 2 It is a schematic diagram of this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The present invention will be described in detail below with reference to the accompanying drawings.

[0037] As Figure 1 and Figure 2 shown, in this embodiment, a lane line detection and fitting method based on deep learning includes the following steps:

[0038] Step 1: Collect lane line image data and label the lane line position coordinates;

[0039] Step 2: Use the calibration parameters of the camera to perform perspective transformation on the collected image to obtain the lane line coordinates from a top-down perspective, and then fit the lane line to obtain the parameters of the third-order polynomial of the lane line, and use this parameter as the annotation data of the image to form the required data set;

[0040] Step 3: Preprocess the collected image data;

[0041] Step 4: Construct a backbone convolutional neural network model, use the image data processed in Step 3 as the input of the backbone convolutional neural network model, and the backbone convolutional neural network model extracts the image feature information;

[0042] Step 5: Construct a lane line position prediction model, using the image feature information output in Step 4 as the input of this lane line position prediction model, and the lane line position prediction model outputs lane line position information;

[0043] Step 6: Construct a lane line fitting parameter prediction model, using the image feature information in Step 4 and the lane line position information obtained in Step 5 as the input of the lane line fitting parameter prediction model together, and the lane line fitting parameter prediction model outputs the predicted lane line fitting result;

[0044] Step 7: Train the constructed backbone convolutional neural network model, lane line position prediction model and lane line fitting parameter prediction model to make the error reach the expected value, and obtain the trained lane line fitting parameter prediction model;

[0045] Step 8: Using the lane line fitting parameter prediction model obtained in Step 7, after processing the collected image data through Step 3, input it into the trained lane line fitting parameter prediction model, and then obtain the prediction result of the lane line fitting parameter.

[0046] In this method, the geometric transformation and polynomial fitting parts in the lane line information acquisition task are also completed by a lightweight deep learning network, realizing end-to-end lane line information acquisition. The consideration of image features is added during the process of fitting, and the stability of information acquisition is improved through the training of a large amount of data. The present invention effectively reduces the influence of factors such as terrain changes and vehicle body posture on the final output result.

[0047] In this embodiment, an in-vehicle front-view camera is used to collect vehicle driving images with lane lines and mark the lane line position coordinates on the images. During this process, image collection is repeatedly performed in various road terrains, and data augmentation means such as flipping and translation are used to enrich the data sample size.

[0048] In this embodiment, the specific content of Step 2 is as follows:

[0049] Calibrate the camera. Based on the obtained calibration parameters, project the lane line coordinates obtained in Step 1 from the two-dimensional image space to the bird's-eye physical space in the top view through perspective transformation. The specific method is as follows:

[0050] Through camera calibration, the perspective transformation matrix of the collected original image can be obtained:

[0051]

[0052] Where: a, b, c, d, e, f are transformation parameters. Given a pixel point p = [x, y, 1]T with coordinates (x, y) in the original image, its projected coordinates are p' = H * p = [x', y', 1]T, where T represents the transpose of the matrix;

[0053] Then, the least squares method is used to fit the points on the projected lane line, and the parameters in the third-order polynomial x' = f(y') = A0 + A1 * y' + A2 * y'^2 + A3 * y'^3 corresponding to the lane line are obtained, that is, [A0, A1, A2, A3]. These parameters will be used as the true value data for subsequent model training.

[0054] In this embodiment, the image data collected in step 1 is preprocessed as follows: the region of interest (ROI) is cropped, the image size is scaled, and then normalization processing is performed to make the image data distributed in the interval [-1, 1] to adapt to the input of the subsequent deep learning model.

[0055] In this embodiment, in step 4, a backbone convolutional neural network model based on a convolutional neural network is constructed for image feature extraction. The MobileNet model is used as the backbone convolutional neural network model in this invention. This backbone convolutional neural network model uses depthwise separable convolution as the basic unit. Compared with ordinary convolutional operations, under the premise of equivalent effects, the computational amount and the number of model parameters are greatly reduced, making it more suitable for scenarios such as vehicles that are sensitive to computing resources. This backbone convolutional neural network model takes the preprocessed image data in step 3 as the input and outputs an image feature map.

[0056] In this embodiment, in step 5, a lane line position prediction model based on a fully connected layer is constructed. The image feature information output in step 4 is used as the input of this lane line position prediction model, and the lane line position prediction model outputs lane line position information.

[0057] The cross entropy between the predicted lane line position information and the lane line position information marked in step 1 is used as the first loss function to obtain the error between the predicted lane line position information and the true lane line position information.

[0058] In this embodiment, in step 6, the image features extracted by the backbone convolutional neural network model in step 4 are fused with the lane line information predicted by the lane line position prediction model in step 5. For example, the image feature data and the lane line prediction data can be concatenated into a group of multi-dimensional vectors. A lane line fitting parameter prediction model based on a fully connected layer is constructed, and the fused data is used as the input to output the predicted value of the lane line fitting parameter.

[0059] Assume that n lane lines are predicted, then the lane line fitting parameter prediction model will predict n groups of fitting parameters A0' n , A1' n , A2' n , A3' n , combined with the true value data A0 of the lane line fitting parameter obtained in step 2 n, A1 n , A2 n , A3 n , set the second loss function Loss as follows:

[0060]

[0061] Use the second loss function to obtain the error between the predicted value of the lane line fitting parameter and the true value of the lane line fitting parameter.

[0062] In this embodiment, in step 7, using the image data obtained in step 1, that is, the true value of the lane line, and the true value data of the fitting parameter obtained in step 2, train and optimize the backbone convolutional neural network model, the lane line position prediction model, and the lane line fitting parameter prediction model through the backpropagation algorithm until the errors obtained by the first loss function and the second loss function reach the corresponding expected values, and obtain the trained lane line fitting parameter prediction model.

[0063] In this embodiment, a lane line detection and fitting system based on deep learning includes a memory and a controller, and a computer-readable program is stored therein. When the computer-readable program is called by the controller, it can execute the steps of the lane line detection and fitting method based on deep learning as described in this embodiment.

[0064] In this embodiment, a vehicle adopts the lane line detection and fitting system based on deep learning as described in this embodiment.

[0065] In this embodiment, a storage medium stores a computer-readable program. When the computer-readable program is called, it can execute the steps of the lane line detection and fitting method based on deep learning as described in this embodiment.

[0066] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A lane line detection and fitting method based on deep learning, characterized in that: The following steps are involved: Step 1: Collect lane line image data and mark the lane line position coordinates; Step 2: Use the camera calibration parameters to perform perspective transformation on the captured image to obtain the lane line coordinates from a bird's-eye view. Then, fit the lane lines to obtain the parameters of a third-order polynomial for the lane lines. This parameter is used as the image annotation data to form the required data set. Step 3: Preprocess the collected image data, including cropping the region of interest, image resizing, and data normalization; Step 4: Construct a backbone convolutional neural network model, use the image data processed in step 3 as the input of the backbone convolutional neural network model, and extract image feature information through the backbone convolutional neural network model; Step 5: Construct a lane position prediction model, using the image feature information output in step 4 as input to the lane position prediction model, which outputs lane position information. Step 6: Construct a lane fitting parameter prediction model. The image feature information in step 4 and the lane position information obtained in step 5 are used as inputs of the lane fitting parameter prediction model. The lane fitting parameter prediction model outputs a predicted lane fitting result. Step 7: Train the constructed backbone convolutional neural network model, lane position prediction model, and lane fitting parameter prediction model to achieve the expected error, thereby obtaining a trained lane fitting parameter prediction model. Step 8: Using the lane line fitting parameter prediction model obtained in step 7, the collected image data is processed in step 3 and input into the trained lane line fitting parameter prediction model to obtain the prediction result of the lane line fitting parameters.

2. The lane line detection and fitting method based on deep learning according to claim 1, characterized in that: In step 1, image acquisition is repeated multiple times in various road terrains, and the sample size of the data is enriched by flipping and translating the data.

3. The lane line detection and fitting method based on deep learning according to claim 1 or 2, characterized in that: The step 2 is specifically as follows: Through camera calibration, the perspective transformation matrix of the captured image is obtained: Where a, b, c, d, e, and f are transformation parameters. Given a pixel point p = [x, y, 1]T with coordinates (x, y) in the original image, its projected coordinates are p' = H*p = [x', y', 1]T, where T represents the transpose of the matrix. The points on the projected lane line are then fitted using the least squares method to obtain the parameters of the third-order polynomial x'=f(y')=A0+A1*y'+A2*y'2+A3*y'3 corresponding to the lane line, that is, [A0, A1, A2, A3]. These parameters will be used as the true value data for subsequent model training.

4. The lane line detection and fitting method based on deep learning according to claim 3, characterized in that: The step 3 specifically includes: cropping the region of interest, scaling the image size, and normalizing the data.

5. The lane line detection and fitting method based on deep learning according to claim 4, characterized in that: In step 5, the cross entropy between the predicted lane line position information and the lane line position information marked in step 1 is used as the first loss function to obtain the error between the predicted lane line position information and the actual lane line position information.

6. The lane line detection and fitting method based on deep learning according to claim 5, characterized in that: In step 6, Assuming that n lane lines are predicted, the lane line fitting parameter prediction model will predict n sets of fitting parameters A0′ n , A1′ n , A2′ n , A3′ n , combined with the lane line fitting parameter true value data A0 obtained in step 2 n , A1 n , A2 n , A3 n , set the second loss function Loss to: The second loss function is used to obtain the error between the predicted value of the lane line fitting parameter and the true value of the lane line fitting parameter.

7. The lane line detection and fitting method based on deep learning according to claim 6, characterized in that: In step 7, the image data obtained in step 1 and the true value data of the fitting parameters obtained in step 2 are used to train and optimize the backbone convolutional neural network model, the lane line position prediction model and the lane line fitting parameter prediction model through a back propagation algorithm until the errors obtained by the first loss function and the second loss function reach the corresponding expected values, thereby obtaining a trained lane line fitting parameter prediction model.

8. A lane detection and fitting system based on deep learning, characterized by: It includes a memory and a controller, in which a computer-readable program is stored. When the computer-readable program is called by the controller, it can execute the steps of the lane line detection and fitting method based on deep learning as described in any one of claims 1 to 7.

9. A vehicle, characterized in that: Adopt the lane line detection and fitting system based on deep learning as described in claim 8.

10. A storage medium, characterized in that: A computer-readable program is stored therein, and when the computer-readable program is called, it can execute the steps of the lane line detection and fitting method based on deep learning as described in any one of claims 1 to 7.

Images