A method and device for generating 3D coordinates of lane lines

Abstract

This application provides a method and device for generating 3D lane line coordinates. The method comprises: obtaining a 2D image of a lane line to be identified and predicting lane line coordinates based on the 2D image; performing vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different road surface locations within the image coordinate system; establishing a road surface slope model based on the vanishing point values; and calculating the 3D coordinates of lane line points based on the predicted lane line coordinates and the road surface slope model. This method and device can automatically generate 3D lane line coordinates without requiring manual processing, saving significant time and labor costs.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a method and device for generating 3D coordinates of lane lines. Background Art

[0002] Currently, in the field of autonomous driving and advanced driver assistance systems, accurate and efficient generation of 3D lane information is crucial. Existing 3D lane generation methods primarily rely on the spatiotemporal synchronization of multi-frame point cloud images and camera images. These synchronized multi-frame point cloud images are combined with image information to generate an RGB point cloud map of the driving scene. Lane lines within the RGB point cloud map are then pre-labeled using a large offline lane model, and manually adjusted and updated. However, in practice, existing methods require manual data screening and labeling, resulting in significant labor and time costs. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for generating 3D lane line coordinates, which can automatically generate 3D lane line coordinates without the need for manual processing, saving a lot of time and labor costs.

[0004] In a first aspect, the present application provides a method for generating 3D coordinates of lane lines, comprising:

[0005] Acquire a 2D image of a lane line to be identified and predicted lane line coordinates obtained by prediction based on the 2D image of the lane line to be identified;

[0006] Performing vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different positions on the road surface in the image coordinate system;

[0007] Establishing a road surface slope model according to the vanishing point value;

[0008] The 3D coordinates of the lane line points are calculated based on the lane line predicted coordinates and the road surface slope model.

[0009] In this implementation, the method first obtains a 2D image of the lane line to be identified and the predicted lane line coordinates based on the 2D image. Then, using a pre-trained inference model, it infers vanishing point values ​​on the 2D image to determine the vanishing point values ​​at different road surface locations within the image coordinate system. A road surface slope model is then established based on the vanishing point values. Finally, the 3D coordinates of the lane line points are calculated based on the predicted lane line coordinates and the road surface slope model. This method automatically generates 3D lane line coordinates without manual processing, saving significant time and labor costs.

[0010] Furthermore, the acquiring of a 2D image of a lane line to be identified and the predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified includes:

[0011] Obtaining a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified;

[0012] 2D lane position prediction is performed based on the lane line annotation data to obtain lane line prediction coordinates.

[0013] Furthermore, the vanishing point value inference is performed on the 2D image of the lane line to be identified based on the pre-trained inference model to obtain the vanishing point values ​​at different positions of the road surface in the image coordinate system, including:

[0014] Segmenting and extracting the effective road surface portion in the 2D image of the lane line to be identified to obtain an effective road surface image to be identified;

[0015] Acquiring laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified;

[0016] Converting the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain effective road surface position point cloud coordinates;

[0017] Get a pre-trained inference model;

[0018] Vanishing point value inference is performed based on the valid road surface position point cloud coordinates and the inference model to obtain vanishing point values ​​at different road surface positions in the image coordinate system.

[0019] Furthermore, the vanishing point value reasoning is performed based on the valid road surface position point cloud coordinates and the reasoning model to obtain the vanishing point values ​​at different road surface positions in the image coordinate system, including:

[0020] Calculate the normal vector corresponding to each valid road surface position point cloud coordinate using the K-nearest nearest neighbor method;

[0021] Setting a fixed target rectangular area in the 2D image of the lane line to be identified;

[0022] Dividing the image in the target rectangular area into a plurality of small rectangular areas;

[0023] Determine the nearest point cloud coordinates of the center of each small rectangular area in the image coordinate system according to the small rectangular area and the effective road surface position point cloud coordinates;

[0024] Obtaining the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each valid road surface position point cloud coordinate;

[0025] The vanishing point value is inferred based on the normal vector corresponding to the coordinates of the nearest point cloud of the center and the inference model to obtain the vanishing point value corresponding to each position of the small rectangular area in the image coordinate system; wherein the vanishing point value corresponding to each position of the small rectangular area is the vanishing point value of different positions of the road surface in the image coordinate system.

[0026] Furthermore, establishing a road surface slope model according to the vanishing point value includes:

[0027] Determine the lower edge position and the upper edge position of each small rectangular area;

[0028] Calculating the first position coordinates of the vehicle corresponding to the lower edge position of each area;

[0029] Converting the first position coordinate from the vehicle coordinate system to the camera coordinate system to obtain a second position coordinate;

[0030] Calculating, in sequence, the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each small rectangular area according to the vanishing point value and the second position coordinates corresponding to each small rectangular area position;

[0031] A road surface slope model corresponding to each of the small rectangular areas is constructed based on the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0032] Furthermore, the calculating of the 3D coordinates of the lane line points according to the lane line predicted coordinates and the road surface slope model includes:

[0033] Obtaining the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate;

[0034] Determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate;

[0035] Calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0036] The position coordinates of each lane line position point in the camera coordinate system are converted to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0037] A second aspect of the present application provides a lane line 3D coordinate generation device, the lane line 3D coordinate generation device comprising:

[0038] An acquisition unit, configured to acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0039] An inference unit, configured to perform vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model, and obtain vanishing point values ​​at different positions on the road surface in the image coordinate system;

[0040] A model building unit, configured to build a road surface slope model according to the vanishing point value;

[0041] A calculation unit is used to calculate the 3D coordinates of the lane line point based on the lane line predicted coordinates and the road surface slope model.

[0042] Furthermore, the acquisition unit includes:

[0043] A first acquisition subunit is configured to acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified;

[0044] The prediction subunit is used to predict the 2D lane position based on the lane line annotation data to obtain the lane line prediction coordinates.

[0045] Furthermore, the inference unit includes:

[0046] An extraction subunit, configured to segment and extract a valid road surface portion in the 2D image of the lane line to be identified, to obtain a valid road surface image to be identified;

[0047] A second acquisition subunit is used to acquire laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified;

[0048] A first conversion subunit is used to convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates;

[0049] A third acquisition subunit is used to acquire a pre-trained inference model;

[0050] The inference subunit is used to perform vanishing point value inference based on the valid road surface position point cloud coordinates and the inference model to obtain the vanishing point values ​​at different road surface positions in the image coordinate system.

[0051] Furthermore, the reasoning subunit includes:

[0052] A calculation module, configured to calculate a normal vector corresponding to each of the valid road surface position point cloud coordinates using a K-nearest nearest neighbor method;

[0053] A setting module, configured to set a fixed target rectangular area in the 2D image of the lane line to be identified;

[0054] A division module, used for dividing the image in the target rectangular area into multiple small rectangular areas;

[0055] A determination module, configured to determine the nearest point cloud coordinates of the center corresponding to each of the small rectangular areas in the image coordinate system based on the small rectangular areas and the point cloud coordinates of the valid road surface position;

[0056] An acquisition module, configured to acquire the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each valid road surface position point cloud coordinate;

[0057] An inference module is configured to infer the vanishing point value based on the normal vector corresponding to the coordinates of the nearest point cloud of the center and the inference model, to obtain the vanishing point value corresponding to each of the small rectangular area positions in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular area positions is the vanishing point value at different positions of the road surface in the image coordinate system.

[0058] Furthermore, the model building unit includes:

[0059] A first determining subunit is configured to determine a lower edge position and an upper edge position of each of the small rectangular areas;

[0060] A first calculation subunit is used to calculate the first position coordinates of the vehicle corresponding to the lower edge position of each area;

[0061] a second conversion subunit, configured to convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates;

[0062] a second calculating subunit, configured to sequentially calculate, based on the vanishing point value and the second position coordinates corresponding to each of the small rectangular area positions, the third position coordinates in the camera coordinate system and the slope angle corresponding to the upper edge position of each of the areas;

[0063] The construction subunit is used to construct a road surface slope model corresponding to each of the small rectangular areas according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0064] Furthermore, the calculation unit includes:

[0065] A fourth acquisition subunit is configured to acquire a lane line y coordinate of each lane line position point in a 2D image coordinate system according to the lane line predicted coordinate;

[0066] A second determining subunit is configured to determine a target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y-coordinate;

[0067] A third calculation subunit is used to calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0068] The third conversion subunit is used to convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0069] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method described in any one of the first aspects of the present application.

[0070] In a fourth aspect, the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the method for generating lane line 3D coordinates described in any one of the first aspects of the present application is executed.

[0071] The beneficial effects of this application are as follows: the method and device can use model reasoning to determine the lane line position in the image coordinate system and the vanishing point values ​​at different road surface locations. These two reasoning results are then combined to determine the lane line's 3D position, thereby achieving fully automated generation of the lane line's 3D coordinates, eliminating manual processing and saving significant time and labor costs. Furthermore, by applying a slope model, the method and device can improve the accuracy of Z-direction value confirmation compared to the traditional method of directly projecting 2D lane line points back into 3D, thereby increasing the accuracy of 3D lane line coordinate generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0073] Figure 1 A schematic diagram of a process for generating 3D coordinates of lane lines provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of a process for generating 3D lane coordinates according to another embodiment of the present invention;

[0075] Figure 3 A schematic diagram of the structure of a lane line 3D coordinate generation device provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of the structure of another lane line 3D coordinate generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0078] Currently, existing 3D lane line generation methods can be divided into the following two types, and both methods have their own problems, as follows:

[0079] (1) Obtain point cloud data in three-dimensional space; convert the point cloud data into image data, which is the pixel value on a two-dimensional plane; perform lane line detection on the image data according to the algorithm model, and the result is a 2D detection result obtained by image-based 2D lane line detection; perform coordinate conversion on the detection result to obtain a 3D lane line based on the point cloud.

[0080] As can be seen, this method only predicts lane lines in 2D images. When the 2D lane lines are converted back to 3D through coordinate conversion, the Z-direction height in the vehicle coordinate system is calculated as 0 because the road surface height information is not known, resulting in inaccurate 3D values. This is precisely the technical problem that this application can solve.

[0081] (2) Perform spatiotemporal synchronization processing on the collected multi-frame point cloud images and camera images; jointly construct a map of the spatiotemporal synchronized multi-frame point cloud image with the image information to generate an RGB driving scene point cloud map, that is, a three-dimensional reconstruction of the driving scene; use the offline lane line large model to pre-mark the lane lines in the RGB driving scene point cloud map, and combine manual adjustment and update of the lane line pre-marking.

[0082] It can be seen that this method is only used for pre-labeling and requires manual participation in the specific data screening and labeling work, which requires a large amount of manpower and time. This is also the technical problem that this application can solve.

[0083] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0084] Example 1

[0085] Please see Figure 1 , Figure 1 The following is a flow chart of a method for generating 3D coordinates of lane lines provided in this embodiment. The method for generating 3D coordinates of lane lines includes:

[0086] S101: Acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified.

[0087] S102 : Perform vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different positions on the road surface in the image coordinate system.

[0088] In this embodiment, the method can generate true values ​​of vanishing points at different positions of the image.

[0089] S103: Establish a road surface slope model according to the vanishing point value.

[0090] S104: Calculate the 3D coordinates of the lane line points based on the predicted lane line coordinates and the road surface slope model.

[0091] By implementing this embodiment, the method can predict the 2D position of the lane line and the road surface slope based on the 2D image, and convert the 2D lane line into a road surface slope model to automatically generate relatively accurate 3D lane lines.

[0092] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and this is not limited in this embodiment.

[0093] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0094] It can be seen that the lane line 3D coordinate generation method described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, saving a lot of time and labor costs.

[0095] Example 2

[0096] Please see Figure 2 , Figure 2 The following is a flow chart of a method for generating 3D coordinates of lane lines provided in this embodiment. The method for generating 3D coordinates of lane lines includes:

[0097] S201: Obtain a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified.

[0098] S202: Perform 2D lane position prediction based on the lane line marking data to obtain lane line prediction coordinates.

[0099] In this embodiment, the method can predict the x and y coordinates of the lane position in the 2D image through the lane line annotation data on the existing 2D image.

[0100] S203 : Segment and extract the effective road surface portion in the 2D image of the lane line to be identified to obtain an effective road surface image to be identified.

[0101] In this embodiment, the method can segment and extract the effective road surface portion in the 2D image.

[0102] S204: Obtain laser point cloud data based on the vehicle coordinate system in the valid road surface image to be identified.

[0103] S205 , converting the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates.

[0104] In this embodiment, the method can perform coordinate conversion on the laser point cloud data, converting the point cloud data from the vehicle coordinate system to the image coordinate system, and only retaining the point cloud results of the valid road surface position.

[0105] S206: Obtain a pre-trained inference model.

[0106] S207: Calculate the normal vector corresponding to each valid road surface position point cloud coordinate using the K-nearest nearest neighbor method.

[0107] In this embodiment, the method may use the k-nearest neighbor method to calculate the normal vector corresponding to each retained point cloud.

[0108] S208: Setting a fixed target rectangular area in the 2D image of the lane line to be identified.

[0109] S209: Divide the image in the target rectangular area into multiple small rectangular areas.

[0110] In this embodiment, the method can fix a rectangular area on the image (the area generally includes the road surface), and divide the area into several small rectangular areas N1, N2...N from near to far. k .

[0111] S210 , determining the nearest point cloud coordinates of the center corresponding to each small rectangular area in the image coordinate system according to the point cloud coordinates of the small rectangular area and the valid road surface position.

[0112] In this embodiment, the method can select the point cloud closest to the center point of each small area (in the image coordinate system).

[0113] S211. Obtain the normal vector corresponding to the nearest point cloud coordinates at the center according to the normal vector corresponding to the point cloud coordinates of each valid road surface position.

[0114] S212 , performing vanishing point value inference based on the normal vector corresponding to the nearest point cloud coordinates and the inference model, to obtain the vanishing point value corresponding to each small rectangular area position in the image coordinate system.

[0115] In this embodiment, the method can calculate the vanishing point of the position based on the normal vector of the point cloud at the nearest position, and obtain K vanishing point values ​​of different small rectangular areas in the image coordinate system.

[0116] In this embodiment, the vanishing point value corresponding to the position of each small rectangular area is the vanishing point value at different positions of the road surface in the image coordinate system.

[0117] In this embodiment, the method can send the vanishing point values ​​at different road surface positions in the image coordinate system into the network for training to obtain a model, so that the inference result of the model is the vanishing point values ​​at different road surface positions in the image coordinate system.

[0118] As an optional implementation, the method may also use the lane line fitting line in the 2D image coordinate system to find the intersection point, and determine the intersection point as the vanishing point for use.

[0119] S213: Determine the lower edge position and the upper edge position of each small rectangular area.

[0120] S214: Calculate the first position coordinates of the vehicle corresponding to the lower edge position of each area.

[0121] S215 : Convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates.

[0122] In this embodiment, the method can calculate the vehicle coordinate system position corresponding to the lower edge position of the small rectangular area N1 at the bottom of the image. Assuming that the Z-direction height of this position is 0, that is, assuming that the lower edge of the rectangular frame is the current road position, the vehicle coordinate system of this point is converted to the camera coordinate system, which is recorded as P 1-bottom .

[0123] S216 , sequentially calculating the third position coordinates and slope angle in the camera coordinate system corresponding to the upper edge position of each small rectangular area according to the vanishing point value and the second position coordinates corresponding to the position of each small rectangular area.

[0124] In this embodiment, the method may use the vanishing point value to calculate the slope angle between the position and the next image position (ie, the upper edge of the current small rectangular area).

[0125] S217: Construct a road slope model corresponding to each small rectangular area according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0126] In this embodiment, the method can be performed by slope angle and P 1-bottom , calculate the camera coordinate position P of the upper edge of the current small rectangular area 1-top .

[0127] In this embodiment, the upper edge P of the current N1 small rectangular area1-top As the lower edge P of the next small rectangular area N2 2-bottom By repeating the above steps, the upper edge camera coordinate system position can be gradually calculated through the lower edge camera coordinate system of each small rectangular area, that is, the camera coordinate system position of the edge of each area is calculated from near to far. This position includes the road slope, so as to establish the road slope model.

[0128] S218. Obtain the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate.

[0129] S219 , determining the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate.

[0130] S220 , calculating the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area.

[0131] S221. Convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0132] In this embodiment, the method can determine which small rectangular area a lane line point corresponds to based on the y-coordinate of the lane line in the 2D image coordinate system, and then calculate the camera coordinate system position of the lane line point through the area model. The camera coordinate system position is converted to the vehicle coordinate system, which is the 3D coordinate of the lane line point, thereby obtaining the 3D coordinates of all lane line points.

[0133] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and this is not limited in this embodiment.

[0134] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0135] It can be seen that the lane line 3D coordinate generation method described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, saving a lot of time and labor costs.

[0136] Example 3

[0137] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a lane line 3D coordinate generation device provided in this embodiment. Figure 3 As shown, the lane line 3D coordinate generating device includes:

[0138] An acquisition unit 310 is configured to acquire a 2D image of a lane line to be identified and predicted lane line coordinates obtained by prediction based on the 2D image of the lane line to be identified;

[0139] An inference unit 320 is configured to perform vanishing point inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different locations on the road surface in the image coordinate system;

[0140] A model building unit 330 is used to build a road slope model according to the vanishing point value;

[0141] The calculation unit 340 is used to calculate the 3D coordinates of the lane line points based on the lane line predicted coordinates and the road surface slope model.

[0142] In this embodiment, the explanation of the lane line 3D coordinate generating device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0143] It can be seen that the lane line 3D coordinate generation device described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, saving a lot of time and labor costs.

[0144] Example 4

[0145] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a lane line 3D coordinate generation device provided in this embodiment. Figure 4 As shown, the lane line 3D coordinate generating device includes:

[0146] An acquisition unit 310 is configured to acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified;

[0147] An inference unit 320 is configured to perform vanishing point inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different locations on the road surface in the image coordinate system;

[0148] A model building unit 330 is used to build a road slope model according to the vanishing point value;

[0149] The calculation unit 340 is used to calculate the 3D coordinates of the lane line points based on the lane line predicted coordinates and the road surface slope model.

[0150] As an optional implementation, the acquiring unit 310 includes:

[0151] The first acquisition subunit 311 is used to acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified;

[0152] The prediction subunit 312 is used to perform 2D lane position prediction based on the lane line annotation data to obtain the lane line prediction coordinates.

[0153] As an optional implementation, the reasoning unit 320 includes:

[0154] The extraction subunit 321 is used to segment and extract the effective road surface portion in the 2D image of the lane line to be identified, so as to obtain the effective road surface image to be identified;

[0155] The second acquisition subunit 322 is used to acquire laser point cloud data based on the vehicle coordinate system in the valid road surface image to be identified;

[0156] The first conversion subunit 323 is used to convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates;

[0157] A third acquisition subunit 324 is used to acquire a pre-trained inference model;

[0158] The inference subunit 325 is used to infer the vanishing point value based on the valid road surface position point cloud coordinates and the inference model to obtain the vanishing point values ​​at different road surface positions in the image coordinate system.

[0159] As an optional implementation, the reasoning subunit 325 includes:

[0160] A calculation module is used to calculate the normal vector corresponding to the point cloud coordinates of each valid road position using the K-nearest method;

[0161] A setting module, used to set a fixed target rectangular area in the 2D image of the lane line to be identified;

[0162] A division module, used for dividing the image in the target rectangular area into multiple small rectangular areas;

[0163] A determination module is used to determine the nearest point cloud coordinates of the center corresponding to each small rectangular area in the image coordinate system based on the small rectangular area and the effective road surface position point cloud coordinates;

[0164] An acquisition module is used to obtain the normal vector corresponding to the center nearest point cloud coordinates based on the normal vector corresponding to each valid road surface position point cloud coordinates;

[0165] The inference module is used to infer the vanishing point value based on the normal vector corresponding to the coordinates of the nearest point cloud in the center and the inference model, and obtain the vanishing point value corresponding to the position of each small rectangular area in the image coordinate system. The vanishing point value corresponding to the position of each small rectangular area is the vanishing point value at different positions on the road surface in the image coordinate system.

[0166] As an optional implementation, the model building unit 330 includes:

[0167] The first determining subunit 331 is configured to determine the lower edge position and the upper edge position of each small rectangular area;

[0168] The first calculation subunit 332 is used to calculate the first position coordinates of the vehicle corresponding to the lower edge position of each area;

[0169] A second conversion subunit 333 is used to convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates;

[0170] The second calculation subunit 334 is used to calculate the third position coordinates and slope angle in the camera coordinate system corresponding to the upper edge position of each small rectangular area according to the vanishing point value and the second position coordinates corresponding to the position of each small rectangular area;

[0171] The construction subunit 335 is used to construct a road surface slope model corresponding to each small rectangular area according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area.

[0172] As an optional implementation, the calculation unit 340 includes:

[0173] The fourth acquisition subunit 341 is configured to acquire the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate;

[0174] The second determining subunit 342 is configured to determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified based on the lane line y-coordinate;

[0175] The third calculation subunit 343 is used to calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area;

[0176] The third conversion subunit 344 is used to convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

[0177] In this embodiment, the explanation of the lane line 3D coordinate generating device can refer to the description in Example 1 or Example 2, and will not be further elaborated in this embodiment.

[0178] It can be seen that the lane line 3D coordinate generation device described in this embodiment can automatically generate 3D lane line coordinates without the need for manual processing, saving a lot of time and labor costs.

[0179] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method in embodiment 1 or embodiment 2 of the present application.

[0180] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the lane line 3D coordinate generation method in embodiment 1 or embodiment 2 of the present application is executed.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0182] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0183] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0185] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0186] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for generating 3D coordinates of lane lines, characterized in that: include: Acquire a 2D image of a lane line to be identified and predicted lane line coordinates obtained by prediction based on the 2D image of the lane line to be identified; Performing vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model to obtain vanishing point values ​​at different positions on the road surface in the image coordinate system; Establishing a road surface slope model according to the vanishing point value; Calculating the 3D coordinates of the lane line points based on the lane line predicted coordinates and the road surface slope model; The method of performing vanishing point value inference on the 2D image of the lane line to be identified based on the pre-trained inference model to obtain vanishing point values ​​at different positions on the road surface in the image coordinate system includes: Segmenting and extracting the effective road surface portion in the 2D image of the lane line to be identified to obtain an effective road surface image to be identified; Acquiring laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified; Converting the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain effective road surface position point cloud coordinates; Get a pre-trained inference model; Performing vanishing point value inference based on the valid road surface position point cloud coordinates and the inference model to obtain vanishing point values ​​at different road surface positions in the image coordinate system; The step of performing vanishing point value reasoning based on the valid road surface position point cloud coordinates and the inference model to obtain vanishing point values ​​at different road surface positions in the image coordinate system includes: Calculate the normal vector corresponding to each valid road surface position point cloud coordinate using the K-nearest nearest neighbor method; Setting a fixed target rectangular area in the 2D image of the lane line to be identified; Dividing the image in the target rectangular area into a plurality of small rectangular areas; Determine the nearest point cloud coordinates of the center of each small rectangular area in the image coordinate system according to the small rectangular area and the effective road surface position point cloud coordinates; Obtaining the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each valid road surface position point cloud coordinate; Performing vanishing point value inference based on the normal vector corresponding to the coordinates of the nearest point cloud at the center and the inference model to obtain a vanishing point value corresponding to each of the small rectangular areas in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular areas is the vanishing point value at different positions of the road surface in the image coordinate system; The step of establishing a road surface slope model according to the vanishing point value includes: Determine the lower edge position and the upper edge position of each small rectangular area; Calculating the first position coordinates of the vehicle corresponding to the lower edge position of each area; Converting the first position coordinate from the vehicle coordinate system to the camera coordinate system to obtain a second position coordinate; Calculating, in sequence, the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each small rectangular area according to the vanishing point value and the second position coordinates corresponding to each small rectangular area position; Constructing a road slope model corresponding to each of the small rectangular areas according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area; Calculating the 3D coordinates of lane line points based on the predicted lane line coordinates and the road surface slope model includes: Obtaining the lane line y coordinate of each lane line position point in the 2D image coordinate system according to the lane line predicted coordinate; Determine the target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y coordinate; Calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area; The position coordinates of each lane line position point in the camera coordinate system are converted to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

2. The method for generating lane line 3D coordinates according to claim 1, characterized in that: The acquiring of a 2D image of a lane line to be identified and a predicted lane line coordinate obtained by prediction based on the 2D image of the lane line to be identified includes: Obtaining a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified; 2D lane position prediction is performed based on the lane line annotation data to obtain lane line prediction coordinates.

3. A lane line 3D coordinate generation device, characterized in that: The lane line 3D coordinate generating device includes: An acquisition unit, configured to acquire a 2D image of a lane line to be identified and predicted coordinates of the lane line obtained by prediction based on the 2D image of the lane line to be identified; An inference unit, configured to perform vanishing point value inference on the 2D image of the lane line to be identified based on a pre-trained inference model, and obtain vanishing point values ​​at different positions on the road surface in the image coordinate system; A model building unit, configured to build a road surface slope model according to the vanishing point value; a calculation unit, configured to calculate the 3D coordinates of a lane line point based on the lane line predicted coordinates and the road surface slope model; Wherein, the inference unit includes: An extraction subunit, configured to segment and extract a valid road surface portion in the 2D image of the lane line to be identified, to obtain a valid road surface image to be identified; A second acquisition subunit is used to acquire laser point cloud data based on a vehicle coordinate system in the valid road surface image to be identified; A first conversion subunit is used to convert the laser point cloud data from the vehicle coordinate system to the image coordinate system to obtain the effective road surface position point cloud coordinates; A third acquisition subunit is used to acquire a pre-trained inference model; an inference subunit, configured to perform vanishing point value inference based on the valid road surface position point cloud coordinates and the inference model, to obtain vanishing point values ​​at different road surface positions in the image coordinate system; Wherein, the reasoning subunit includes: A calculation module, configured to calculate a normal vector corresponding to each of the valid road surface position point cloud coordinates using a K-nearest nearest neighbor method; A setting module, configured to set a fixed target rectangular area in the 2D image of the lane line to be identified; A division module, used for dividing the image in the target rectangular area into multiple small rectangular areas; A determination module, configured to determine the nearest point cloud coordinates of the center corresponding to each of the small rectangular areas in the image coordinate system based on the small rectangular areas and the point cloud coordinates of the valid road surface position; An acquisition module, configured to acquire the normal vector corresponding to the center nearest point cloud coordinates according to the normal vector corresponding to each valid road surface position point cloud coordinate; an inference module, configured to infer the vanishing point value based on the normal vector corresponding to the coordinates of the nearest point cloud and the inference model, to obtain the vanishing point value corresponding to each of the small rectangular areas in the image coordinate system; wherein the vanishing point value corresponding to each of the small rectangular areas is the vanishing point value at different positions on the road surface in the image coordinate system; Wherein, the model building unit includes: A first determining subunit is configured to determine a lower edge position and an upper edge position of each of the small rectangular areas; A first calculation subunit is used to calculate the first position coordinates of the vehicle corresponding to the lower edge position of each area; a second conversion subunit, configured to convert the first position coordinates from the vehicle coordinate system to the camera coordinate system to obtain second position coordinates; a second calculating subunit, configured to sequentially calculate, based on the vanishing point value and the second position coordinates corresponding to each of the small rectangular area positions, the third position coordinates in the camera coordinate system and the slope angle corresponding to the upper edge position of each of the areas; A construction subunit, configured to construct a road slope model corresponding to each of the small rectangular areas according to the third position coordinates and the slope angle in the camera coordinate system corresponding to the upper edge position of each area; Wherein, the calculation unit includes: A fourth acquisition subunit is configured to acquire a lane line y coordinate of each lane line position point in a 2D image coordinate system according to the lane line predicted coordinate; A second determining subunit is configured to determine a target small rectangular area where each lane line position point is located in the 2D image of the lane line to be identified according to the lane line y-coordinate; A third calculation subunit is used to calculate the position coordinates of each lane line position point in the camera coordinate system according to the lane line y coordinate and the road slope model corresponding to the target small rectangular area; The third conversion subunit is used to convert the position coordinates of each lane line position point in the camera coordinate system to the vehicle coordinate system to obtain the lane line 3D coordinates of each lane line position point.

4. The lane line 3D coordinate generating device according to claim 3, characterized in that: The acquisition unit includes: A first acquisition subunit is configured to acquire a 2D image of a lane line to be identified and lane line annotation data of the 2D image of the lane line to be identified; The prediction subunit is used to predict the 2D lane position based on the lane line annotation data to obtain the lane line prediction coordinates.

5. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lane line 3D coordinate generation method according to any one of claims 1 to 2.

6. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method for generating lane line 3D coordinates according to any one of claims 1 to 2 is executed.

Images