3D lane line detection method and device

Abstract

The invention discloses a 3D lane line detection method and device, and the method comprises the steps: obtaining the image data of the environment in front of a vehicle in real time; inputting the acquired image data of the environment in front of the vehicle into an open source 3D lane line detection model to obtain an initial prediction lane line; performing offset mapping on the initial prediction lane line through a correction module to obtain a corrected lane line; wherein the correction module is formed by carrying out iterative regression training through a preset loss function and a pre-marked real value of the self-mining lane line. On one hand, a large amount of open source data is effectively utilized, and the training difficulty of real scene data is reduced; and on the other hand, training is carried out by using a small amount of self-collected marking data, so that the problem of lane line deviation caused by the adoption of an open source 3D lane line detection model is avoided, a high-precision actual scene 3D lane line detection model is obtained, and the 3D detection precision of lane lines in a real scene is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a 3D lane line detection method and device. Background Art

[0002] Lane detection plays an important role in autonomous driving systems. In order to ensure the safety of the car during driving, the autonomous driving system needs to keep the car moving along the lane lines on the road, which requires accurate perception of the lane lines.

[0003] Lane line perception is an important part of achieving L4 level autonomous driving technology. Traditional lane line perception methods include edge detection plus Hough transform, color threshold, perspective transform and other methods. Most of these methods require manual parameter adjustment, which is labor-intensive and ineffective, especially in some complex road scenes, and are almost unusable. In recent years, thanks to the improvement of computer computing power, the capabilities of deep learning technology have become stronger and stronger, and there are precedents for the successful application of deep learning technology in various fields. In the field of lane line perception, spatial convolutional neural network (SCNN), Lane-Net network, H-Net network, etc. all use semantic segmentation to detect lane lines and have achieved good results.

[0004] In order to solve the problem of inaccurate lane layout estimation in autonomous driving scenarios (uphill / downhill, bumpy), 3D lane line detection came into being, such as 3D-LaneNet network, Gen-LaneNet network and Persformer network, all of which use anchor regression based on the bird's-eye view BEV perspective to complete 3D lane line detection. The difficulty in the 3D lane line detection task is the acquisition and labeling of lane line 3D data. When there is less self-collected data and labeled data, it is difficult to obtain an accurate lane line perception model. Even if a large amount of open source labeled data is available, the scenarios of open source labeled data are quite different from those used in actual projects, and most of the open source data are foreign street scenes, and the camera models, parameters, and layouts are also very different. The open source data model obtained based on the open source data set is difficult to have high accuracy in real scenes, and usually has a large position offset. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a 3D lane line detection method and device to improve the 3D lane line detection accuracy and reduce the difficulty of model training.

[0006] In order to solve the above technical problems, the present invention provides a 3D lane line detection method, comprising:

[0007] Acquire image data of the environment in front of the vehicle in real time;

[0008] Input the acquired image data of the environment in front of the vehicle into the open source 3D lane line detection model to obtain the initial predicted lane line;

[0009] The initial predicted lane line is offset mapped through a correction module to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

[0010] Preferably, the initial predicted lane line is represented in the form of an M×3N matrix, wherein M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line.

[0011] Preferably, the correction module is a linear mapping module, which is represented in the form of a 3N×3N matrix, where N is the number of lane line 3D points corresponding to each anchor line.

[0012] Preferably, the initial predicted lane line is offset mapped by a correction module to obtain a corrected lane line in the following manner:

[0013] Assume that the initial predicted lane line is Pred and the linear mapping module is M 3N×3N , the corrected lane line is Corr M×3N ,but:

[0014] Corr M×3N =Pred×M 3N×3N .

[0015] Preferably, the correction module performs iterative regression training using a preset loss function and pre-labeled true values ​​of self-collected lane lines, specifically including:

[0016] Using the pre-loss function, the differences between the 3D coordinates of the corrected lane line output by the correction module and the true value of the pre-marked self-collected lane line are calculated respectively, and then the differences are summed to obtain the total loss value, and the total loss value is minimized.

[0017] The present invention also provides a 3D lane line detection device, comprising:

[0018] The vehicle-mounted perception module is used to obtain image data of the environment in front of the vehicle in real time;

[0019] An initial detection module, used to input the image data of the vehicle's front environment acquired by the vehicle-mounted perception module into an open source 3D lane line detection model to obtain an initial predicted lane line;

[0020] A correction module is used to offset and map the initial predicted lane line to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

[0021] Preferably, the initial predicted lane line is represented in the form of an M×3N matrix, wherein M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line.

[0022] Preferably, the correction module is a linear mapping module, which is represented in the form of a 3N×3N matrix, where N is the number of lane line 3D points corresponding to each anchor line.

[0023] Preferably, the initial predicted lane line is offset mapped by a correction module to obtain a corrected lane line in the following manner:

[0024] Assume that the initial predicted lane line is Pred and the linear mapping module is M 3N×3N , the corrected lane line is Corr M×3N ,but:

[0025] Corr M×3N =Pred×M 3N×3N .

[0026] Preferably, the correction module performs iterative regression training using a preset loss function and pre-labeled true values ​​of self-collected lane lines, specifically in the following manner:

[0027] Using the pre-loss function, the differences between the 3D coordinates of the corrected lane line output by the correction module and the true value of the pre-marked self-collected lane line are calculated respectively, and then the differences are summed to obtain the total loss value, and the total loss value is minimized.

[0028] The implementation of the present invention has the following beneficial effects: on the one hand, the present invention effectively utilizes a large amount of open source data, reducing the difficulty of training real scene data; on the other hand, it uses a small amount of self-collected labeled data for training, avoiding the lane line offset problem caused by the use of an open source 3D lane line detection model, thereby obtaining a high-precision actual scene 3D lane line detection model, and improving the 3D detection accuracy of lane lines in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flowchart of a 3D lane line detection method according to Embodiment 1 of the present invention.

[0031] Figure 2 It is a schematic diagram of the principle of a 3D lane line detection method according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0032] The following descriptions of the embodiments refer to the accompanying drawings to illustrate specific embodiments in which the present invention may be implemented.

[0033] Please refer to Figure 1 As shown, the first embodiment of the present invention provides a 3D lane line detection method, including:

[0034] Acquire image data of the environment in front of the vehicle in real time;

[0035] Input the acquired image data of the environment in front of the vehicle into the open source 3D lane line detection model to obtain the initial predicted lane line;

[0036] The initial predicted lane line is offset mapped through a correction module to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

[0037] From the above, it can be seen that, on the one hand, the present invention effectively utilizes a large amount of open source data to reduce the difficulty of training real scene data; on the other hand, it uses a small amount of self-collected labeled data for training to avoid the lane line offset problem caused by the use of an open source 3D lane line detection model, thereby obtaining a high-precision actual scene 3D lane line detection model, which improves the 3D detection accuracy of lane lines in real scenes.

[0038] Specifically, please combine Figure 2 As shown, in this embodiment, the image data is mainly obtained by using a vehicle-mounted sensor such as a vehicle-mounted camera. The image data of the environment in front of the vehicle is usually used for training and prediction of the anchor-based 3D lane line detection model. The image data is usually collected by a vehicle-mounted camera, which provides information such as the position and shape of the lane line on the image.

[0039] In this embodiment, the open source 3D lane detection model is an anchor-based 3D lane detection model. This model usually regards the lane line as a line segment composed of multiple anchor lines and multiple feature points, so it is also called "anchor-based". During the training process, the model needs to learn how to identify the anchor line from the input data, and then predict the position of the lane line based on the anchor line. Specifically, as an example, this model first uses a clustering algorithm to identify the anchor line in the image data and pair it with the corresponding image feature points. The number of anchor lines is usually fixed, while the number and position of feature points vary according to different image inputs. Then, these anchor lines and feature points are input into the neural network, and feature extraction and classification are performed using convolutional layers and fully connected layers. Finally, the model outputs the predicted result of the lane line position at each anchor line. In the testing and prediction stage, the model will predict the lane line position by inputting sensor perception data. The prediction can usually be expressed as a 3D coordinate or a 2D pixel position, depending on the type of sensor and data format used.

[0040] For the embodiment of the present invention, the image data of the environment in front of the vehicle and the internal and external parameters of the corresponding on-board camera are input into the open source 3D lane line detection model to obtain the output result of the open source 3D lane line detection model, that is, the initial predicted lane line Pred; it can be understood that the initial predicted lane line Pred is represented in the form of an M×3N matrix, where M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line. Usually, the number of 3D points corresponding to each anchor line is fixed. The M and N parameters can also be set in combination with the actual road scene and the current data situation. As an example, M is generally 16 and N is generally 80.

[0041] One of the improvements made by the embodiment of the present invention compared to the prior art is that a correction module is provided to correct the lane line deviation caused by the use of an open source 3D lane line detection model. The specific form of the correction module can be selected according to the actual complexity of the self-collected lane line true value GT (Ground Truth), such as selecting a linear mapping module, or selecting CNN (implemented by a fully connected layer); in the embodiment of the present invention, through testing and observing the data deviation law, it is found that the usual lane line deviation mapping relationship is linear, so a linear mapping module (Mapping function) is selected as the correction module, which is expressed in the form of a 3N×3N matrix, where N, as mentioned above, is the number of lane line 3D points corresponding to each anchor line.

[0042] After the initial predicted lane line Pred is linearly mapped by the linear mapping module, the correction lane line (Correction line) is obtained, which is also expressed in the form of an M×3N matrix, which has the same matrix structure as the initial predicted lane line Pred: Let the correction lane line Correctionline be Corr M×3N , the linear mapping module is M 3N×3N ,but:

[0043] Corr M×3N =Pred×M 3N×3N .

[0044] It should be noted that the correction module is iteratively regressed through the preset loss function L1-norm and the pre-labeled true value GT of the self-collected lane line (obtained by collecting image data and 3D point cloud data from the collection vehicle and manually labeling).

[0045] The L1-norm loss function is also called the Least Abosulote Error, which is used to measure the difference between the predicted value of the open source 3D lane detection model and the true value of the self-collected data. The formula of the L1-norm loss function is as follows:

[0046]

[0047] It can be understood that during the training process of the correction module, the image data of the vehicle's front environment and the corresponding internal and external parameters of the on-board camera are also first input into the open source 3D lane line detection model to obtain the initial predicted lane line Pred; the initial predicted lane line Pred is then offset mapped through the correction module to obtain the correction lane line Correctionline; assuming that the 3D coordinate values ​​of the correction lane line Correctionline output by the correction module are x, y, and z respectively, and the 3D coordinate values ​​of the pre-annotated self-collected lane line true value GT are x, y, and z respectively. gt ,y gt 、z gt , calculate the losses of x, y, and z respectively through the L1-norm loss function, and add them up to get the total loss value D L1 , the formula is as follows:

[0048]

[0049] As mentioned above, M is the number of anchor lines, and N is the number of lane line 3D points corresponding to each anchor line. The input during reasoning only requires the image data collected by the camera in real time and the internal and external parameters of the corresponding camera, and the corresponding 3D lane line can be output without inputting 3D point cloud data. This patent is a pure visual lane line detection task. During training, the input sources are image data, camera internal and external parameters, and the GT used for supervision is the 3D coordinates corresponding to the lane line in the image, that is, the 3D coordinates x, y, and z corresponding to each point in the lane line.

[0050] In this way, after each batch of training data is sent to the open source 3D lane detection model, the predicted value is output through forward propagation, and then the loss function calculates the difference between the predicted value and the true value in the three-dimensional coordinates, that is, the loss value, and then sums up to get the total loss value, and minimizes the total loss value as much as possible. After that, the model updates each parameter through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model is closer to the true value, thereby achieving the purpose of learning.

[0051] After the correction module is trained, its parameters are fixed. In the actual lane line detection process, it is only necessary to offset and map the initial predicted lane line Pred through the correction module, that is, to correct the initial predicted lane line Pred output by the open source 3D lane line detection model, and then a high-precision correction lane line Correctionline can be obtained. During implementation, the correction module of the embodiment of the present invention can be added after the existing open source data 3D lane line detection model.

[0052] Corresponding to the 3D lane line detection method described in the first embodiment of the present invention, the embodiment of the present invention provides a 3D lane line detection device, including:

[0053] The vehicle-mounted perception module is used to obtain image data of the environment in front of the vehicle in real time;

[0054] An initial detection module, used to input the image data of the vehicle's front environment acquired by the vehicle-mounted perception module into an open source 3D lane line detection model to obtain an initial predicted lane line;

[0055] A correction module is used to offset and map the initial predicted lane line to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

[0056] Preferably, the initial predicted lane line is represented in the form of an M×3N matrix, wherein M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line.

[0057] Preferably, the correction module is a linear mapping module, which is represented in the form of a 3N×3N matrix, where N is the number of lane line 3D points corresponding to each anchor line.

[0058] Preferably, the initial predicted lane line is offset mapped by a correction module to obtain a corrected lane line in the following manner:

[0059] Assume that the initial predicted lane line is Pred and the linear mapping module is M 3N×3N , the corrected lane line is Corr M×3N ,but:

[0060] Corr M×3N =Pred×M 3N×3N .

[0061] Preferably, the correction module performs iterative regression training using a preset loss function and pre-labeled true values ​​of self-collected lane lines, specifically in the following manner:

[0062] Using the pre-loss function, the differences between the 3D coordinates of the corrected lane line output by the correction module and the true value of the pre-marked self-collected lane line are calculated respectively, and then the differences are summed to obtain the total loss value, and the total loss value is minimized.

[0063] For the working principle and process of this embodiment, please refer to the description of the first embodiment of the present invention, which will not be repeated here.

[0064] From the above description, it can be seen that compared with the prior art, the beneficial effects of the present invention are: on the one hand, the present invention effectively utilizes a large amount of open source data to reduce the difficulty of training real scene data; on the other hand, it uses a small amount of self-collected labeled data for training to avoid the lane line offset problem caused by the use of an open source 3D lane line detection model, thereby obtaining a high-precision actual scene 3D lane line detection model, which improves the 3D detection accuracy of lane lines in real scenes.

[0065] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A 3D lane line detection method, characterized in that: include: Acquire image data of the environment in front of the vehicle in real time; Input the acquired image data of the environment in front of the vehicle into the open source 3D lane line detection model to obtain the initial predicted lane line; The initial predicted lane line is offset mapped through a correction module to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

2. The 3D lane line detection method according to claim 1, characterized in that: The initial predicted lane line is represented in the form of an M×3N matrix, where M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line.

3. The 3D lane line detection method according to claim 2, characterized in that: The correction module is a linear mapping module, which is represented in the form of a 3N×3N matrix, where N is the number of lane line 3D points corresponding to each anchor line.

4. The 3D lane line detection method according to claim 3, characterized in that: The initial predicted lane line is offset mapped by the correction module to obtain the corrected lane line in the following manner: Assume that the initial predicted lane line is Pred and the linear mapping module is M 3N×3N , the corrected lane line is Corr M×3N ,but: Corr M×3N =Pred×M 3N×3N 。 5. The 3D lane line detection method according to claim 1, characterized in that: The correction module performs iterative regression training using a preset loss function and the true value of the pre-labeled self-collected lane line, specifically including: Using the pre-loss function, the differences between the 3D coordinates of the corrected lane line output by the correction module and the true value of the pre-marked self-collected lane line are calculated respectively, and then the differences are summed to obtain the total loss value, and the total loss value is minimized.

6. A 3D lane line detection device, characterized in that: include: The vehicle-mounted perception module is used to obtain image data of the environment in front of the vehicle in real time; An initial detection module, used to input the image data of the vehicle's front environment acquired by the vehicle-mounted perception module into an open source 3D lane line detection model to obtain an initial predicted lane line; A correction module is used to offset and map the initial predicted lane line to obtain a corrected lane line; wherein the correction module is formed by iterative regression training using a preset loss function and pre-labeled true values ​​of the self-collected lane lines.

7. The 3D lane line detection device according to claim 6, characterized in that: The initial predicted lane line is represented in the form of an M×3N matrix, where M is the number of anchor lines identified by the open source 3D lane line detection model, and N is the number of lane line 3D points corresponding to each anchor line.

8. The 3D lane line detection device according to claim 7, characterized in that: The correction module is a linear mapping module, which is represented in the form of a 3N×3N matrix, where N is the number of lane line 3D points corresponding to each anchor line.

9. The 3D lane line detection device according to claim 8, characterized in that: The initial predicted lane line is offset mapped by the correction module to obtain the corrected lane line in the following manner: Assume that the initial predicted lane line is Pred and the linear mapping module is M 3N×3N , the corrected lane line is Corr M×3N ,but: Corr M×3N =Pred×M 3N×3N 。 10. The 3D lane line detection device according to claim 6, characterized in that: The correction module performs iterative regression training using a preset loss function and the true value of the pre-labeled self-collected lane line. The specific method is as follows: Using the pre-loss function, the differences between the 3D coordinates of the corrected lane line output by the correction module and the true value of the pre-marked self-collected lane line are calculated respectively, and then the differences are summed to obtain the total loss value, and the total loss value is minimized.

Images