Lane line 3D coordinate generation method and device

Abstract

The invention provides a lane line 3D coordinate generation method and device. The method comprises the steps of obtaining a to-be-recognized lane line 2D image and a lane line prediction coordinate obtained through prediction based on the to-be-recognized lane line 2D image; performing vanishing point value reasoning on the 2D image of the lane line to be recognized according to a pre-trained reasoning model to obtain vanishing point values of different positions of a road surface under an image coordinate system; establishing a road slope model according to the vanishing point value; and calculating 3D coordinates of the lane line points according to the lane line prediction coordinates and the road slope model. Therefore, the method and the device can automatically generate the 3D lane line coordinates, do not need manual processing, and save a large amount of time and labor cost.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a method and device for generating 3D coordinates of lane lines. Background Art

[0002] At present, in the field of autonomous driving and advanced driver assistance systems, it is crucial to accurately and efficiently generate 3D lane line information. Existing 3D lane line generation methods mainly rely on the spatiotemporal synchronization of multi-frame point cloud images and camera images, jointly mapping the spatiotemporal synchronized multi-frame point cloud images with image information to generate an RGB driving scene point cloud map, and then use the offline lane line large model to pre-mark the lane lines in the RGB driving scene point cloud map, and combine manual adjustments and updates to the lane line pre-marking. However, in practice, it is found that the existing methods require manual participation in specific data screening and labeling work, which requires a large amount of manpower and time. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for generating lane line 3D coordinates, which can automatically generate 3D lane line coordinates without the need for manual processing, thus saving a lot of time and labor costs.

[0004] The first aspect of the present application provides a method for generating 3D coordinates of a lane line, comprising:

[0005] Acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0006] Performing vanishing point value inference on the 2D image of the lane line to be identified according to the pre-trained inference model to obtain vanishing point values ​​at different positions of the road surface in the image coordinate system;

[0007] Establishing a road surface slope model according to the vanishing point value;

[0008] The 3D coordinates of the lane line point are calculated according to the lane line predicted coordinates and the road surface slope model.

[0009] In the above implementation process, the method can first obtain the 2D image of the lane line to be identified and the predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified; then, according to the pre-trained inference model, the vanishing point value inference is performed on the 2D image of the lane line to be identified to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system; then, the road surface slope model is established according to the vanishing point value; finally, according to the predicted coordinates of the lane line and the road surface slope model, the 3D coordinates of the lane line point are calculated. Based on this, the method can automatically generate 3D lane line coordinates without manual processing, saving a lot of time and labor costs.

[0010] Furthermore, the acquiring of the 2D image of the lane line to be identified and the predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified includes:

[0011] Acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified;

[0012] The 2D lane position is predicted based on the lane line annotation data to obtain the lane line prediction coordinates.

[0013] Furthermore, the vanishing point value inference is performed on the 2D image of the lane line to be identified according to the pre-trained inference model to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system, including:

[0014] Segmenting and extracting the effective road surface portion in the 2D image of the lane line to be identified to obtain an effective road surface image to be identified;

[0015] Acquire laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified;

[0016] Convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates;

[0017] Get a pre-trained inference model;

[0018] The vanishing point value is inferred based on the point cloud coordinates of the effective road surface position and the inference model to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system.

[0019] Furthermore, the vanishing point value reasoning is performed according to the effective road surface position point cloud coordinates and the reasoning model to obtain the vanishing point values ​​of different road surface positions in the image coordinate system, including:

[0020] Calculate the normal vector corresponding to each point cloud coordinate of the valid road position using the K-nearest neighbor method;

[0021] Setting a fixed target rectangular area in the 2D image of the lane line to be identified;

[0022] Dividing the image in the target rectangular area into a plurality of small rectangular areas;

[0023] Determine the nearest point cloud coordinates of the center of each small rectangular area in the image coordinate system according to the small rectangular area and the effective road surface position point cloud coordinates;

[0024] Obtaining the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each of the valid road position point cloud coordinates;

[0025] The vanishing point value is inferred based on the normal vector corresponding to the coordinates of the nearest point cloud in the center and the inference model to obtain the vanishing point value corresponding to each of the small rectangular area positions in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular area positions is the vanishing point value at different positions of the road surface in the image coordinate system.

[0026] Furthermore, establishing a road surface slope model according to the vanishing point value includes:

[0027] Determine the lower edge position and the upper edge position of each small rectangular area;

[0028] Calculate the first position coordinates of the vehicle coordinates corresponding to the lower edge position of each area;

[0029] Converting the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates;

[0030] According to the vanishing point value and the second position coordinates corresponding to each of the small rectangular area positions, sequentially calculating the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the areas;

[0031] A road surface slope model corresponding to each of the small rectangular areas is constructed according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the areas.

[0032] Further, the calculating the 3D coordinates of the lane line point according to the lane line predicted coordinates and the road surface slope model includes:

[0033] Acquire the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate;

[0034] Determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate;

[0035] Calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0036] The position coordinates of each lane line position point in the camera coordinate system are converted to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0037] A second aspect of the present application provides a lane line 3D coordinate generation device, the lane line 3D coordinate generation device comprising:

[0038] An acquisition unit, used to acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0039] An inference unit, configured to perform vanishing point value inference on the 2D image of the lane line to be identified according to a pre-trained inference model, and obtain vanishing point values ​​at different positions of the road surface in the image coordinate system;

[0040] A model building unit, used to build a road surface slope model according to the vanishing point value;

[0041] A calculation unit is used to calculate the 3D coordinates of the lane line point according to the lane line predicted coordinates and the road surface slope model.

[0042] Furthermore, the acquisition unit includes:

[0043] A first acquisition subunit is used to acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified;

[0044] The prediction subunit is used to perform 2D lane position prediction based on the lane line annotation data to obtain lane line prediction coordinates.

[0045] Furthermore, the reasoning unit comprises:

[0046] An extraction subunit is used to segment and extract the effective road surface part in the 2D image of the lane line to be identified, so as to obtain an effective road surface image to be identified;

[0047] A second acquisition subunit is used to acquire laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified;

[0048] A first conversion subunit, used to convert the laser point cloud data from a vehicle coordinate system to an image coordinate system to obtain point cloud coordinates of a valid road surface position;

[0049] A third acquisition subunit is used to acquire a pre-trained inference model;

[0050] The reasoning subunit is used to perform vanishing point value reasoning according to the effective road surface position point cloud coordinates and the reasoning model to obtain the vanishing point values ​​at different positions of the road surface in the image coordinate system.

[0051] Furthermore, the reasoning subunit includes:

[0052] A calculation module, used for calculating the normal vector corresponding to each of the valid road surface position point cloud coordinates by using a K-nearest method;

[0053] A setting module, used for setting a fixed target rectangular area in the 2D image of the lane line to be identified;

[0054] A division module, used for dividing the image in the target rectangular area into a plurality of small rectangular areas;

[0055] A determination module, used to determine the nearest point cloud coordinates of the center corresponding to each of the small rectangular areas in the image coordinate system according to the small rectangular areas and the point cloud coordinates of the valid road position;

[0056] An acquisition module, used for acquiring the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each of the valid road position point cloud coordinates;

[0057] An inference module is used to infer the vanishing point value based on the normal vector corresponding to the coordinates of the nearest point cloud in the center and the inference model, so as to obtain the vanishing point value corresponding to each of the small rectangular area positions in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular area positions is the vanishing point value at different positions of the road surface in the image coordinate system.

[0058] Furthermore, the model building unit includes:

[0059] A first determining subunit is used to determine the lower edge position and the upper edge position of each small rectangular area;

[0060] A first calculation subunit, used for calculating a first position coordinate under the vehicle coordinate corresponding to the lower edge position of each area;

[0061] A second conversion subunit, used for converting the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates;

[0062] A second calculation subunit is used to calculate, in sequence, the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the small rectangular areas according to the vanishing point value and the second position coordinates corresponding to the position of each of the small rectangular areas;

[0063] The construction subunit is used to construct a road surface slope model corresponding to each of the small rectangular areas according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the areas.

[0064] Furthermore, the calculation unit comprises:

[0065] A fourth acquisition subunit, configured to acquire a lane line y coordinate of each lane line position point in a 2D image coordinate system according to the lane line predicted coordinate;

[0066] A second determining subunit is used to determine a target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate;

[0067] A third calculation subunit is used to calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0068] The third conversion subunit is used to convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0069] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method described in any one of the first aspects of the present application.

[0070] A fourth aspect of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the lane line 3D coordinate generation method described in any one of the first aspect of the present application is executed.

[0071] The beneficial effects of the present application are as follows: the method and device can obtain the lane line position in the image coordinate system and the vanishing point value at different positions of the road surface through model reasoning, and combine the two reasoning results to obtain the 3D position of the lane line, thereby realizing the fully automatic generation of the 3D coordinates of the lane line, avoiding manual processing, and saving a lot of time and labor costs. At the same time, the method and device can also improve the accuracy of the confirmation of the Z direction value in the traditional method of directly projecting the 2D points of the lane line back to 3D through the application of the slope model, thereby improving the generation accuracy of the 3D coordinates of the lane line. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of a process for generating 3D coordinates of lane lines provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of a flow chart of another method for generating 3D coordinates of lane lines provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of the structure of a lane line 3D coordinate generating device provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of the structure of another lane line 3D coordinate generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0078] At present, the existing 3D lane line generation methods can be divided into the following two types, and both methods have their own problems, as follows:

[0079] (1) Obtain point cloud data in three-dimensional space; convert the point cloud data into image data, where the image data is pixel values ​​on a two-dimensional plane; perform lane line detection on the image data according to the algorithm model, and the result is a 2D detection result obtained by image-based 2D lane line detection; perform coordinate conversion on the detection result to obtain a 3D lane line based on the point cloud.

[0080] It can be seen that this method only predicts the lane line results in the 2D image. When the 2D lane line is converted back to 3D through coordinate conversion, the Z-direction height in the vehicle coordinate system is calculated as 0 because the road height information cannot be known, and the obtained 3D value is inaccurate. This is exactly the technical problem that this application can solve.

[0081] (2) Perform spatiotemporal synchronization processing on the collected multi-frame point cloud images and camera images; jointly construct a map by combining the spatiotemporal synchronized multi-frame point cloud images with image information to generate an RGB driving scene point cloud map, i.e., a three-dimensional reconstruction of the driving scene; use the offline lane line large model to pre-mark the lane lines in the RGB driving scene point cloud map, and adjust and update the lane line pre-marking manually.

[0082] It can be seen that this method is only used as a pre-labeling, and requires manual participation in specific data screening and labeling work, which requires a large amount of manpower and time. This is also the technical problem that this application can solve.

[0083] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0084] Example 1

[0085] Please see Figure 1 , Figure 1 A schematic diagram of a process flow of a lane line 3D coordinate generation method provided in this embodiment. The lane line 3D coordinate generation method includes:

[0086] S101, obtaining a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified.

[0087] S102: Perform vanishing point value inference on the 2D image of the lane line to be identified according to the pre-trained inference model to obtain vanishing point values ​​at different positions of the road surface in the image coordinate system.

[0088] In this embodiment, the method can generate true values ​​of vanishing points at different positions of the image.

[0089] S103: Establish a road surface slope model according to the vanishing point value.

[0090] S104: Calculate the 3D coordinates of the lane line points according to the predicted lane line coordinates and the road surface slope model.

[0091] By implementing this embodiment, the method can predict the 2D position of the lane line and the road surface slope based on the 2D image, and convert the 2D lane line into a road surface slope model to automatically generate relatively accurate 3D lane lines.

[0092] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0093] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0094] It can be seen that the lane line 3D coordinate generation method described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, thus saving a lot of time and labor costs.

[0095] Example 2

[0096] Please see Figure 2 , Figure 2 A schematic diagram of a process flow of a lane line 3D coordinate generation method provided in this embodiment. The lane line 3D coordinate generation method includes:

[0097] S201, obtaining a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified.

[0098] S202: Predict the 2D lane position based on the lane line marking data to obtain the lane line prediction coordinates.

[0099] In this embodiment, the method can predict the x and y coordinates of the lane position in the 2D image through the lane line annotation data on the existing 2D image.

[0100] S203 , segment and extract the effective road surface part in the 2D image of the lane line to be identified, and obtain an effective road surface image to be identified.

[0101] In this embodiment, the method can segment and extract the effective road surface part in the 2D image.

[0102] S204: Obtain laser point cloud data based on the vehicle coordinate system in the valid road surface image to be identified.

[0103] S205 , converting the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates.

[0104] In this embodiment, the method can perform coordinate transformation on the laser point cloud data, transform the point cloud data from the vehicle coordinate system to the image coordinate system, and only retain the point cloud results of the valid road surface position.

[0105] S206: Obtain a pre-trained inference model.

[0106] S207, using the K-nearest neighbor method to calculate the normal vector corresponding to the point cloud coordinates of each valid road surface position.

[0107] In this embodiment, the method may use the k-nearest neighbor method to calculate the normal vector corresponding to each retained point cloud.

[0108] S208: Setting a fixed target rectangular area in the 2D image of the lane line to be identified.

[0109] S209: Divide the image in the target rectangular area into a plurality of small rectangular areas.

[0110] In this embodiment, the method can fix a rectangular area on the image (the area generally includes the road surface), and divide the area into a number of small rectangular areas N1, N2, ... N from near to far. k .

[0111] S210 , determining the nearest point cloud coordinates of the center corresponding to each small rectangular area in the image coordinate system according to the small rectangular area and the effective road surface position point cloud coordinates.

[0112] In this embodiment, the method can select the point cloud closest to the center point of each small area (in the image coordinate system).

[0113] S211. Obtain the normal vector corresponding to the point cloud coordinates closest to the center according to the normal vector corresponding to the point cloud coordinates of each valid road surface position.

[0114] S212, performing vanishing point value inference according to the normal vector corresponding to the coordinates of the nearest point cloud at the center and the inference model, to obtain the vanishing point value corresponding to the position of each small rectangular area in the image coordinate system.

[0115] In this embodiment, the method can calculate the vanishing point of the position according to the normal vector of the point cloud at the nearest position, and obtain K vanishing point values ​​of different small rectangular areas in the image coordinate system.

[0116] In this embodiment, the vanishing point value corresponding to the position of each small rectangular area is the vanishing point value of different positions of the road surface in the image coordinate system.

[0117] In this embodiment, the method can send the vanishing point values ​​at different road surface positions in the image coordinate system into the network for training to obtain a model, so that the inference result of the model is the vanishing point values ​​at different road surface positions in the image coordinate system.

[0118] As an optional implementation, the method may also use the lane line fitting straight line in the 2D image coordinate system to find the intersection point, and determine the intersection point as the vanishing point for use.

[0119] S213: Determine the lower edge position and the upper edge position of each small rectangular area.

[0120] S214, calculating the first position coordinates under the vehicle coordinates corresponding to the lower edge position of each area.

[0121] S215 . Convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates.

[0122] In this embodiment, the method can calculate the vehicle coordinate system position corresponding to the lower edge position of the small rectangular area N1 at the bottom of the image. Assuming that the Z-direction height of this position is 0, that is, assuming that the lower edge of the rectangular frame is the current road position, the vehicle coordinate system of this point is converted to the camera coordinate system, which is recorded as P 1-bottom .

[0123] S216 , according to the vanishing point value and the second position coordinate corresponding to the position of each small rectangular area, sequentially calculate the third position coordinate and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0124] In this embodiment, the method can use the vanishing point value to calculate the slope angle between the position and the next image position (ie, the upper edge of the current small rectangular area).

[0125] S217: Construct a road surface slope model corresponding to each small rectangular area according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0126] In this embodiment, the method can be performed by slope angle and P 1-bottom , calculate the camera coordinate position P of the upper edge of the current small rectangular area 1-top .

[0127] In this embodiment, the upper edge P of the current N1 small rectangular area1-top As the lower edge P of the next small rectangular area N2 2-bottom , repeating the above steps, the upper edge camera coordinate system position can be gradually calculated through the lower edge camera coordinate system of each small rectangular area, that is, the camera coordinate system position of the edge of each area is calculated from near to far, and this position includes the road slope, so as to establish the road slope model.

[0128] S218. Obtain the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate.

[0129] S219, determining the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate.

[0130] S220, calculating the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area.

[0131] S221. Convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0132] In this embodiment, the method can determine which small rectangular area a certain lane line point corresponds to based on the y coordinate of the lane line in the 2D image coordinate system, and then calculate the camera coordinate system position of the lane line point through the area model, and convert the camera coordinate system position to the vehicle coordinate system, which is the 3D coordinate of the lane line point, thereby obtaining the 3D coordinates of all lane line points.

[0133] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0134] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0135] It can be seen that the lane line 3D coordinate generation method described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, thus saving a lot of time and labor costs.

[0136] Example 3

[0137] Please see Figure 3 , Figure 3 Schematic diagram of the structure of a lane line 3D coordinate generating device provided in this embodiment. Figure 3 As shown, the lane line 3D coordinate generating device includes:

[0138] An acquisition unit 310 is used to acquire a 2D image of a lane line to be identified and a predicted coordinate of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0139] An inference unit 320 is used to perform vanishing point value inference on the 2D image of the lane line to be identified according to a pre-trained inference model to obtain vanishing point values ​​at different positions of the road surface in the image coordinate system;

[0140] A model building unit 330, used to build a road surface slope model according to the vanishing point value;

[0141] The calculation unit 340 is used to calculate the 3D coordinates of the lane line point according to the lane line prediction coordinates and the road surface slope model.

[0142] In this embodiment, the explanation of the lane line 3D coordinate generating device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0143] It can be seen that the lane line 3D coordinate generation device described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, thus saving a lot of time and labor costs.

[0144] Example 4

[0145] Please see Figure 4 , Figure 4 Schematic diagram of the structure of a lane line 3D coordinate generating device provided in this embodiment. Figure 4 As shown, the lane line 3D coordinate generating device includes:

[0146] An acquisition unit 310 is used to acquire a 2D image of a lane line to be identified and a predicted coordinate of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0147] An inference unit 320 is used to perform vanishing point value inference on the 2D image of the lane line to be identified according to a pre-trained inference model to obtain vanishing point values ​​at different positions of the road surface in the image coordinate system;

[0148] A model building unit 330, used to build a road surface slope model according to the vanishing point value;

[0149] The calculation unit 340 is used to calculate the 3D coordinates of the lane line point according to the lane line prediction coordinates and the road surface slope model.

[0150] As an optional implementation, the acquiring unit 310 includes:

[0151] The first acquisition subunit 311 is used to acquire the 2D image of the lane line to be identified and the lane line annotation data of the 2D image of the lane line to be identified;

[0152] The prediction subunit 312 is used to perform 2D lane position prediction based on the lane line annotation data to obtain the lane line prediction coordinates.

[0153] As an optional implementation, the reasoning unit 320 includes:

[0154] An extraction subunit 321 is used to segment and extract the effective road surface part in the 2D image of the lane line to be identified, so as to obtain an effective road surface image to be identified;

[0155] The second acquisition subunit 322 is used to acquire laser point cloud data based on the vehicle coordinate system in the valid road surface image to be identified;

[0156] The first conversion subunit 323 is used to convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates;

[0157] A third acquisition subunit 324 is used to acquire a pre-trained inference model;

[0158] The reasoning subunit 325 is used to perform vanishing point value reasoning based on the point cloud coordinates of the valid road surface position and the reasoning model to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system.

[0159] As an optional implementation, the reasoning subunit 325 includes:

[0160] A calculation module, used to calculate the normal vector corresponding to the point cloud coordinates of each valid road position using a K-nearest method;

[0161] A setting module, used for setting a fixed target rectangular area in the 2D image of the lane line to be identified;

[0162] A division module, used for dividing the image in the target rectangular area into a plurality of small rectangular areas;

[0163] A determination module, used to determine the nearest point cloud coordinates of the center corresponding to each small rectangular area in the image coordinate system according to the point cloud coordinates of the small rectangular area and the effective road position;

[0164] An acquisition module, used for acquiring the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to the point cloud coordinates of each valid road position;

[0165] The inference module is used to infer the vanishing point value based on the normal vector corresponding to the coordinates of the nearest point cloud in the center and the inference model, so as to obtain the vanishing point value corresponding to the position of each small rectangular area in the image coordinate system; wherein the vanishing point value corresponding to the position of each small rectangular area is the vanishing point value at different positions of the road surface in the image coordinate system.

[0166] As an optional implementation, the model building unit 330 includes:

[0167] A first determining subunit 331 is used to determine the lower edge position and the upper edge position of each small rectangular area;

[0168] The first calculation subunit 332 is used to calculate the first position coordinates of the vehicle coordinates corresponding to the lower edge position of each area;

[0169] A second conversion subunit 333 is used to convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates;

[0170] The second calculation subunit 334 is used to calculate the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area in sequence according to the vanishing point value and the second position coordinates corresponding to the position of each small rectangular area;

[0171] The construction subunit 335 is used to construct a road surface slope model corresponding to each small rectangular area according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0172] As an optional implementation, the calculation unit 340 includes:

[0173] The fourth acquisition subunit 341 is used to acquire the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line prediction coordinate;

[0174] The second determining subunit 342 is used to determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate;

[0175] The third calculation subunit 343 is used to calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0176] The third conversion subunit 344 is used to convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0177] In this embodiment, the explanation of the lane line 3D coordinate generating device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0178] It can be seen that the lane line 3D coordinate generation device described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, thus saving a lot of time and labor costs.

[0179] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method in embodiment 1 or embodiment 2 of the present application.

[0180] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the lane line 3D coordinate generation method in embodiment 1 or embodiment 2 of the present application is executed.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0183] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0185] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0186] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for generating 3D coordinates of lane lines, characterized in that: include: Acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified; Performing vanishing point value inference on the 2D image of the lane line to be identified according to the pre-trained inference model to obtain vanishing point values ​​at different positions of the road surface in the image coordinate system; Establishing a road surface slope model according to the vanishing point value; The 3D coordinates of the lane line point are calculated according to the lane line predicted coordinates and the road surface slope model.

2. The method for generating lane line 3D coordinates according to claim 1, characterized in that: The acquiring of a 2D image of a lane line to be identified and a predicted coordinate of a lane line obtained by prediction based on the 2D image of the lane line to be identified includes: Acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified; The 2D lane position is predicted based on the lane line annotation data to obtain the lane line prediction coordinates.

3. The method for generating lane line 3D coordinates according to claim 1, characterized in that: The vanishing point value inference is performed on the 2D image of the lane line to be identified according to the pre-trained inference model to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system, including: Segmenting and extracting the effective road surface portion in the 2D image of the lane line to be identified to obtain an effective road surface image to be identified; Acquire laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified; Convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates; Get a pre-trained inference model; The vanishing point value is inferred based on the point cloud coordinates of the effective road surface position and the inference model to obtain the vanishing point values ​​of different positions of the road surface in the image coordinate system.

4. The method for generating lane line 3D coordinates according to claim 3, characterized in that: The performing vanishing point value reasoning according to the effective road surface position point cloud coordinates and the reasoning model to obtain the vanishing point values ​​at different positions of the road surface in the image coordinate system includes: Calculate the normal vector corresponding to each point cloud coordinate of the valid road position using the K-nearest neighbor method; Setting a fixed target rectangular area in the 2D image of the lane line to be identified; Dividing the image in the target rectangular area into a plurality of small rectangular areas; Determine the nearest point cloud coordinates of the center of each small rectangular area in the image coordinate system according to the small rectangular area and the effective road surface position point cloud coordinates; Obtaining the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each of the valid road position point cloud coordinates; The vanishing point value is inferred based on the normal vector corresponding to the coordinates of the nearest point cloud in the center and the inference model to obtain the vanishing point value corresponding to each of the small rectangular area positions in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular area positions is the vanishing point value at different positions of the road surface in the image coordinate system.

5. The method for generating lane line 3D coordinates according to claim 4, characterized in that: The step of establishing a road surface slope model according to the vanishing point value comprises: Determine the lower edge position and the upper edge position of each small rectangular area; Calculate the first position coordinates of the vehicle coordinates corresponding to the lower edge position of each area; Converting the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates; According to the vanishing point value and the second position coordinates corresponding to each of the small rectangular area positions, sequentially calculating the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the areas; A road surface slope model corresponding to each of the small rectangular areas is constructed according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each of the areas.

6. The method for generating lane line 3D coordinates according to claim 4, characterized in that: The calculating the 3D coordinates of the lane line point according to the lane line prediction coordinates and the road surface slope model includes: Acquire the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate; Determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate; Calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area; The position coordinates of each lane line position point in the camera coordinate system are converted to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

7. A lane line 3D coordinate generating device, characterized in that: The lane line 3D coordinate generating device comprises: An acquisition unit, used to acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified; An inference unit, configured to perform vanishing point value inference on the 2D image of the lane line to be identified according to a pre-trained inference model, and obtain vanishing point values ​​at different positions of the road surface in the image coordinate system; A model building unit, used to build a road surface slope model according to the vanishing point value; A calculation unit is used to calculate the 3D coordinates of the lane line point according to the lane line predicted coordinates and the road surface slope model.

8. The lane line 3D coordinate generating device according to claim 7, characterized in that: The acquisition unit comprises: A first acquisition subunit is used to acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified; The prediction subunit is used to perform 2D lane position prediction based on the lane line annotation data to obtain lane line prediction coordinates.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method according to any one of claims 1 to 6.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method for generating lane line 3D coordinates according to any one of claims 1 to 6 is executed.

Images