Three-dimensional lane line labeling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a three-dimensional lane line labeling method and device, electronic equipment and storage medium. The three-dimensional lane line labeling method comprises: acquiring first laser point cloud data and image data collected synchronously, wherein the first laser point cloud data and the image data both comprise information of lane lines; labeling lane line images in the image data to obtain a two-dimensional lane line semantic segmentation labeling result; projecting the first laser point cloud data to the image data, and obtaining point cloud attribute information according to the projection result and the two-dimensional lane line semantic segmentation labeling result; adding the point cloud attribute information to the first laser point cloud data to obtain second laser point cloud data; and obtaining a laser point cloud labeling result of labeled three-dimensional lane lines according to the second laser point cloud data. The present scheme can realize automatic labeling of three-dimensional lane lines and save costs.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a three-dimensional lane line annotation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Lane labeling refers to the process of assigning lane line labels to lane line points in a laser point cloud and assigning road surface or background labels to other points such as the road surface. Lane labeling plays a very important role in lane detection.

[0003] For lane marking, image data is typically acquired using cameras, and laser point cloud data is acquired using LiDAR. Furthermore, semantic segmentation algorithms are generally used to process the point cloud data. In current technology, taking the semantic segmentation and annotation of a single laser point cloud containing approximately 200,000 points as an example, an experienced annotator spends 3-6 hours on annotation, with verification and quality assurance taking approximately 0.5-1 hour. The cost of annotating a single laser point cloud file reaches tens to hundreds of yuan. For training multimodal deep perception learning models in intelligent assisted driving, hundreds of thousands or more of high-quality annotated laser point cloud data are generally required; for complex models, the data requirements are even greater.

[0004] It is evident that manual labeling would incur significant costs in terms of manpower, finances, and time. Summary of the Invention

[0005] In view of this, embodiments of this application provide a three-dimensional lane line annotation method, apparatus, electronic device and storage medium, which can realize automated annotation of three-dimensional lane lines and save costs.

[0006] According to a first aspect of the embodiments of this application, a three-dimensional lane line annotation method is provided, comprising: acquiring synchronously acquired first laser point cloud data and image data, wherein both the first laser point cloud data and the image data include lane line information; annotating the lane line image in the image data to obtain a two-dimensional lane line semantic segmentation annotation result; projecting the first laser point cloud data onto the image data, and obtaining point cloud attribute information based on the projection result and the two-dimensional lane line semantic segmentation annotation result, wherein the point cloud attribute information includes color information and label information corresponding to the projection points of at least some laser points in the first laser point cloud data in the image data, and the label information is used to indicate the entity type of the corresponding projection point; adding the point cloud attribute information to the first laser point cloud data to obtain second laser point cloud data; and obtaining a laser point cloud annotation result with three-dimensional lane lines annotated based on the second laser point cloud data.

[0007] In one possible implementation, obtaining the laser point cloud annotation result for marking the three-dimensional lane lines based on the second laser point cloud data includes: fusing the color information and label information corresponding to different image data in the second laser point cloud data to obtain a first laser point cloud annotation result, wherein the different image data are acquired by different cameras; stitching the first laser point cloud annotation result with historical laser point cloud annotation results to obtain a second laser point cloud annotation result, wherein the number of laser points included in the second laser point cloud annotation result is greater than the number of laser points included in the first laser point cloud annotation result; and eliminating the conflict of label information between different laser points in the second laser point cloud annotation result to obtain the laser point cloud annotation result for marking the three-dimensional lane lines.

[0008] In one possible implementation, fusing the color information and label information corresponding to different image data in the second laser point cloud data to obtain the first laser point cloud annotation result includes:

[0009] For each laser point in the second laser point cloud data, perform the following:

[0010] The average color value of the laser point is calculated based on the multiple color information corresponding to the laser point included in the second laser point cloud data.

[0011] Based on the average color value of the laser point and the multiple color information corresponding to the laser point in the second laser point cloud data, calculate the color deviation of the multiple color information corresponding to the laser point respectively.

[0012] Based on the multiple color deviations corresponding to the laser point, the first label of the laser point is calculated using the following formula:

[0013]

[0014] The first label used to characterize the j-th laser point in the second laser point cloud data, (Δr) ij , Δg ij Δb ij This is used to characterize the color deviation of the j-th laser point in the second laser point cloud data corresponding to the projection point in the i-th image data. The label information used to characterize the projection point of the j-th laser point in the second laser point cloud data in the i-th image data, where N is the number of image data, j is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N;

[0015] If the label information corresponding to the laser point in the second laser point cloud data is the same as the first label, then the first label of the laser point is determined as the fused label of the laser point.

[0016] If at least one interference label information corresponding to the laser point in the second laser point cloud data is different from the first label, then the interference label information and the corresponding color information are deleted from the second laser point cloud data, and the first label of the laser point is recalculated based on the remaining color information and label information corresponding to the laser point in the second laser point cloud data, until all the label information corresponding to the laser point in the second laser point cloud data is the same as the first label.

[0017] The average color value of each laser point and the fused label are added to the second laser point cloud data to obtain the first laser point cloud annotation result.

[0018] In one possible implementation, the step of concatenating the first laser point cloud annotation result with the historical laser point cloud annotation result to obtain the second laser point cloud annotation result includes: mapping the laser points in the first laser point cloud data to the world coordinate system based on the vehicle's position and attitude matrix when the first laser point cloud data was collected; mapping the laser points in the historical laser point cloud data to the world coordinate system based on the vehicle's position and attitude matrix when the historical laser point cloud data was collected; and concatenating the first laser point cloud annotation result, the historical laser point cloud annotation result corresponding to the historical laser point cloud data, the coordinate values ​​of the laser points in the first laser point cloud data in the world coordinate system, and the coordinate values ​​of the laser points in the historical laser point cloud data in the world coordinate system to obtain the second laser point cloud annotation result.

[0019] In one possible implementation, eliminating conflicts in the label information between different laser points in the second laser point cloud annotation result to obtain a laser point cloud annotation result with the three-dimensional lane lines annotated includes:

[0020] For any laser point in the second laser point cloud annotation result, determine the set of neighboring points corresponding to the laser point. The set of neighboring points includes at least one laser point whose Euclidean distance from the laser point is less than a distance threshold.

[0021] If the fused label of a laser point is different from the fused label of at least one laser point in the set of neighboring points corresponding to that laser point, then the second label is calculated by the following formula, and the fused labels of each laser point in the set of neighboring points corresponding to that laser point are updated to the second label. The set of neighboring points corresponding to each laser point is then redefined until the fused label of any laser point is the same as the fused label of each laser point in the set of neighboring points corresponding to that laser point, and the laser point cloud annotation result is obtained.

[0022]

[0023] The second label is used to characterize the j-th laser point in the second laser point cloud annotation result, and K is used to characterize the number of laser points in the neighbor set corresponding to the j-th laser point. The fused label used to characterize the k-th laser point within the set of neighboring points corresponding to the j-th laser point (x k y k , z k The coordinates of the l-th laser point within the set of neighboring points corresponding to the k-th laser point are used to characterize the three-dimensional coordinates of the l-th laser point.

[0024] In one possible implementation, the method further includes: acquiring laser point cloud data collected by lidar; sequentially performing null value removal, infinite value removal, and statistical outlier removal on the laser point cloud data to obtain the first laser point cloud data.

[0025] According to a second aspect of the embodiments of this application, a lane line marking device is provided, comprising: an acquisition module, configured to acquire synchronously acquired first laser point cloud data and image data, wherein both the first laser point cloud data and the image data include lane line information; an annotation module, configured to annotate the lane line image in the image data to obtain a two-dimensional lane line semantic segmentation annotation result; a projection module, configured to project the first laser point cloud data onto the image data, and obtain point cloud attribute information based on the projection result and the two-dimensional lane line semantic segmentation annotation result, wherein the point cloud attribute information includes color information and label information corresponding to the projection points of at least some laser points in the first laser point cloud data in the image data, and the label information is used to indicate the entity type of the corresponding projection point; an addition module, configured to add the point cloud attribute information to the first laser point cloud data to obtain second laser point cloud data; and an acquisition module, configured to obtain a laser point cloud annotation result with three-dimensional lane lines marked based on the second laser point cloud data.

[0026] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.

[0027] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0028] According to a fifth aspect of the present application, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, cause at least one processor to perform the method as described in the first aspect.

[0029] Based on the 3D lane line annotation method provided by the above scheme, the first laser point cloud data and image data are collected simultaneously. The 3D first laser point cloud data is projected onto the 2D image data. Since the 2D image data, after annotation, clearly defines the lane line position and assigns label and color information to the lane line, the projection of the 3D first laser point cloud data onto the 2D image data assigns point cloud attribute information to the laser points in the first laser point cloud data. Adding this point cloud attribute information to the first laser point cloud data yields the second laser point cloud data. Therefore, the laser point cloud annotation result for the 3D lane line can be obtained based on the second laser point cloud data. Thus, by assigning point cloud attribute information to the 3D first laser point cloud data, the second laser point cloud data is obtained, and the laser point cloud annotation result for the 3D lane line is obtained based on the second laser point cloud data. The entire process reduces manual annotation and verification steps, resulting in high annotation efficiency and reduced costs. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 A flowchart illustrating a three-dimensional lane line annotation method provided in one embodiment of this application;

[0032] Figure 2 A flowchart illustrating a three-dimensional lane line annotation method provided in yet another embodiment of this application;

[0033] Figure 3 A flowchart of the three-dimensional lane line annotation method for each laser point in the second laser point cloud data provided in this application;

[0034] Figure 4 A flowchart of the three-dimensional lane line annotation method for obtaining the second laser point cloud annotation result provided in this application;

[0035] Figure 5 A flowchart of the three-dimensional lane line annotation method provided in this application for obtaining laser point cloud annotation results with three-dimensional lane lines annotated;

[0036] Figure 6 A flowchart illustrating a three-dimensional lane line annotation method provided in another embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a three-dimensional lane marking device provided in one embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0039] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0042] 3D lane line annotation method

[0043] Example 1

[0044] Reference Figure 1 The diagram illustrates a flowchart of a three-dimensional lane marking method according to Embodiment 1 of the present invention. Figure 1 As shown, the three-dimensional lane line annotation method includes steps 101 to 105:

[0045] Step 101: Acquire the first laser point cloud data and image data acquired synchronously.

[0046] Both the first laser point cloud data and the image data include lane line information.

[0047] By configuring a first device on the vehicle to acquire laser point cloud data, first laser point cloud data, including lane line information, is obtained. Specifically, the first device for acquiring the laser point cloud data can be a lidar system.

[0048] By configuring a second device on the vehicle to acquire image data, image data including lane line information can be obtained. Specifically, the second device can be a surround-view camera; for example, by arranging six cameras on the vehicle—front-view, rear-view, left-front-view, left-rear-view, right-front-view, and right-rear-view—360-degree surround-view image acquisition can be achieved, thereby acquiring image data from various angles.

[0049] By employing hardware synchronization, the first and second devices are synchronized, and images and laser point cloud data are acquired synchronously at a certain acquisition frequency, thereby achieving synchronous acquisition of images and laser point cloud data.

[0050] Step 102: Label the lane line image in the image data to obtain the two-dimensional lane line semantic segmentation labeling results.

[0051] The acquired two-dimensional image data includes lane line information as well as non-lane information, such as buildings, road surfaces (other than lane lines), and plants. By annotating the lane line images in the image data, the two-dimensional lane line semantic segmentation annotation results are obtained.

[0052] Image data can be semantically segmented using traditional methods or deep learning methods to obtain two-dimensional lane line semantic segmentation table annotation results. There are many traditional semantic segmentation methods, mainly including Normalized-cut, Structured Random Forest, and SVM, etc.

[0053] In this embodiment, lane line images are labeled in image data using deep learning methods. Specifically, lane lines are labeled in image data by training a two-dimensional lane line labeling model. First, (1) a public lane line dataset or a lane line simulation dataset is obtained, and then a two-dimensional lane line first labeling model is trained based on the public lane line dataset or the lane line simulation dataset. Optionally, a public lane line dataset or a lane line simulation dataset can be obtained through the CULane dataset, Tusimple dataset, or CurveLanes dataset. (2) The image data is labeled using the two-dimensional lane line first labeling model. From the perspective of ensuring labeling quality, manual verification and correction can be used to obtain labeled image data; however, manual verification and correction can also be avoided. (3) The two-dimensional lane line first labeling model is trained using the labeled image data to obtain a two-dimensional lane line labeling model, thereby labeling more image data.

[0054] Two-dimensional lane line labeling models can use semantic segmentation-based algorithms such as LaneNet and SCNN, or anchor-based methods such as LaneATT and CLRNet. Semantic segmentation-based algorithms output whether each pixel is a lane line, and this type of output can efficiently and conveniently count the number of pixels containing lane lines in the image data. Anchor-based methods output discrete sampling points of lane lines, and the boundaries of lane lines can be obtained through lateral expansion, thus obtaining labels similar to semantic segmentation. Lateral expansion refers to obtaining the rectangular boundary (envelope) of the lane line by extending the detected lane line centerline to a certain width.

[0055] The semantic segmentation and annotation results of two-dimensional lane lines are statistically analyzed to calculate the richness of lane line information. If the image data corresponding to the semantic segmentation and annotation results of two-dimensional lane lines does not contain lane line information, the image data will not participate in the next step, thereby reducing the calculation time and improving the annotation efficiency.

[0056] Step 103: Project the first laser point cloud data onto the image data, and obtain the point cloud attribute information based on the projection result and the two-dimensional lane line semantic segmentation annotation result.

[0057] The point cloud attribute information includes color information and label information corresponding to the projection points of at least some laser points in the first laser point cloud data in the image data. The label information is used to indicate the entity type of the corresponding projection point.

[0058] As mentioned above, all image data contains lane line information. Since the first laser point cloud data is three-dimensional, while the image data is two-dimensional, the three-dimensional first laser point cloud data is projected onto the two-dimensional image data. Based on the projection result and the two-dimensional lane line semantic segmentation and annotation result, the point cloud attribute information is obtained.

[0059] In a feasible example, for instance, the image data contains lane lines, trees, and roads. Based on the semantic segmentation and labeling results of the two-dimensional lane lines in the image data, the lane lines are red, the trees are green, and the roads are white. The first laser point cloud data is projected onto the image data, and the point cloud data labeled as lane lines is red, the point cloud data labeled as trees is green, and the point cloud data labeled as roads is white. That is, the point cloud attribute information includes the color information and label information corresponding to the projection points of at least some laser points in the first laser point cloud data onto the image data.

[0060] Optionally, the first laser point cloud data can be projected into the image data according to the following formula (1).

[0061]

[0062] [uij v ij 1] T This represents the coordinates of the projection point of the j-th laser point in the i-th image data; [x j y j z j 1] T This represents the coordinates of the j-th laser point in the first laser point cloud, where the subscript j indicates the j-th point in a point cloud file. For example, with a 64-line or 128-line radar, the number of points in a single point cloud file is typically between 150,000 and 300,000; K i Y represents the intrinsic parameter matrix of the i-th second device; i This represents the transformation matrix from the laser radar to the i-th second device.

[0063] Step 104: Add the point cloud attribute information to the first laser point cloud data to obtain the second laser point cloud data.

[0064] In one embodiment, when the point cloud attribute information includes color information, label information, and reflection intensity, the point cloud attribute information is added to the first laser point cloud data to obtain the coordinates of the second laser point cloud data. The second laser point cloud data can be expressed as the following formula (2):

[0065]

[0066] [x j y j z j 1] T f represents the coordinates of the j-th laser point in the first laser point cloud data. j The value represents the reflection intensity of the j-th laser point in the point cloud attribute information, [r ij g ij b ij ] T This represents the color information corresponding to the projection point of the j-th laser point in the i-th image data. This represents the label information corresponding to the projection point of the j-th laser point in the i-th image data.

[0067] Step 105: Based on the second laser point cloud data, obtain the laser point cloud annotation results of the three-dimensional lane lines.

[0068] Based on the second laser point cloud data, the final laser point cloud annotation results of the three-dimensional lane lines with no label conflicts are obtained.

[0069] In this embodiment, firstly, synchronously acquired 3D first laser point cloud data and 2D image data are obtained. Secondly, lane line images are annotated on the 2D image data to obtain 2D lane line semantic segmentation annotation results. Then, the 3D first laser point cloud data is projected onto the image data, and point cloud attribute information is obtained based on the projection result and the 2D lane line semantic segmentation annotation results. Finally, the obtained point cloud attribute information is added to the 3D first laser point cloud data to obtain second laser point cloud data. Based on the second laser point cloud data, the laser point cloud annotation result with 3D lane lines is obtained. This solution realizes the transformation from 3D (first laser point cloud data) to 2D (projecting the first laser point cloud data onto image data) and then to 3D (obtaining the laser point cloud annotation result with 3D lane lines), ultimately obtaining the laser point cloud annotation result with 3D lane lines. This reduces the steps of manual annotation and verification, improves annotation efficiency, and reduces costs.

[0070] Example 2

[0071] Reference Figure 2 The diagram shows a flowchart of another three-dimensional lane line marking method according to Embodiment 2 of the present invention.

[0072] In this embodiment of the application, the three-dimensional lane line marking method includes the aforementioned steps 101 to 105, wherein step 105, based on the second laser point cloud data, obtains the laser point cloud marking result of the three-dimensional lane line, which can be achieved through sub-steps 1051 to 1053.

[0073] Sub-step 1051: Fuse the color information and label information of the different image data in the second laser point cloud data to obtain the first laser point cloud annotation result.

[0074] Different image data were captured by different cameras.

[0075] For points in the second laser point cloud data within the same frame, assume they correspond to two different image datasets acquired by different cameras, such as a surround-view camera on a vehicle, obtained from a front-view camera and a left-front side-view camera, respectively. Because 2D lane line detection may contain errors (such as projection errors), the same point might have different color and label information in different image datasets. However, in 3D point clouds, a point can only have one type of label and one type of color information. Therefore, conflict resolution is necessary. That is, the color and label information corresponding to the different image datasets in the second laser point cloud data are fused to obtain the first laser point cloud annotation result.

[0076] The first laser point cloud annotation result is represented by the following formula (3):

[0077]

[0078] Among them, PC 1-j This represents the annotation result of the j-th laser point cloud in the first laser point cloud annotation result. f represents the tag information of the j-th laser point. j Represents the reflection intensity of the j-th laser point, (r j g j b j ) represents the color information of the j-th laser point, (x j y j , z j ) represents the coordinates of the j-th laser point.

[0079] In one feasible approach, for each laser point in the second laser point cloud data in sub-step 1051, steps 201 to 203 are performed, referring to... Figure 3 A flowchart of a three-dimensional lane line annotation method for each laser point in a second laser point cloud data according to the present invention is shown.

[0080] Step 201: Calculate the average color value of the laser point based on the multiple color information corresponding to the laser point included in the second laser point cloud data;

[0081] Since the second laser point cloud data corresponds to multiple different image data, and different image data display different color information due to projection errors, i.e., different RGB values, the average color value of the laser point is calculated based on the multiple color information corresponding to the laser point included in the second laser point cloud data. This is calculated using the following formula (4):

[0082]

[0083] Among them, (r ij g ij b ij (r) represents the color information corresponding to the projection point of the j-th laser point in the i-th image data. j g j b j ) represents the average color value of the j-th laser point in the second laser point cloud data, and N represents the number of image data.

[0084] Step 202: Calculate the color deviation of the multiple color information corresponding to the laser point based on the average color value of the laser point and the multiple color information corresponding to the laser point in the second laser point cloud data.

[0085] Compared to the average color of the laser point, multiple color information corresponding to the laser point in the second laser point cloud data all have color deviations. These deviations are calculated using the following formula (5):

[0086]

[0087] Based on the multiple color deviations corresponding to the laser point, the first label of the laser point is calculated using the following formula (6):

[0088]

[0089] This represents the first label of the j-th laser point in the second laser point cloud data, (Δr) ij , Δg ij Δb ij The ) represents the color deviation of the j-th laser point in the second laser point cloud data corresponding to the projection point in the i-th image data. This represents the label information corresponding to the projection point of the j-th laser point in the i-th image data in the second laser point cloud data. N is the number of image data, j is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N.

[0090] If the label information corresponding to the laser point in the second laser point cloud data is the same as the first label, then the first label of the laser point is determined as the fused label of the laser point.

[0091] If at least one interfering label information corresponding to the laser point in the second laser point cloud data is different from the first label, the interfering label information and the corresponding color information are deleted from the second laser point cloud data. Based on the remaining color information and label information corresponding to the laser point in the second laser point cloud data, the first label of the laser point is recalculated until all the label information corresponding to the laser point in the second laser point cloud data is the same as the first label.

[0092] Step 203: Add the color mean of each laser point and the fused label to the second laser point cloud data to obtain the first laser point cloud annotation result.

[0093] The calculated color mean and the fused label are added to the second laser point cloud data, realizing the migration from two-dimensional label to three-dimensional label. The fused first laser point cloud labeling results, including lane lines, are more accurate, improving the labeling effect.

[0094] Sub-step 1052: Combine the first laser point cloud annotation result with the historical laser point cloud annotation result to obtain the second laser point cloud annotation result.

[0095] The laser point cloud data collected at different times is different. For example, the laser point cloud annotation result collected at the first time has 250,000 laser points, of which 3,000 laser points are labeled as lane lines; the laser point cloud annotation result collected at the second time has 250,000 laser points, of which 2,500 laser points are labeled as lane lines. Lane line laser point clouds marked with the same label in the historical laser point cloud annotation results from these two time points need to be stitched together to obtain a denser laser point cloud annotation result. That is, the first laser point cloud annotation result is stitched together with the historical laser point cloud annotation results to obtain the second laser point cloud annotation result.

[0096] Specifically, point cloud stitching can be achieved by adding the number of points in the point cloud and adding the dimensions of the point cloud.

[0097] In one feasible approach, sub-step 1052 involves concatenating the first laser point cloud annotation result with the historical laser point cloud annotation result to obtain the second laser point cloud annotation result. This can be achieved through steps 301 to 303, as described below. Figure 4 A flowchart of a three-dimensional lane line annotation method for obtaining a second laser point cloud annotation result according to the present invention is shown.

[0098] Step 301: Based on the vehicle's position and attitude matrix when collecting the first laser point cloud data, map the laser points in the first laser point cloud data to the world coordinate system.

[0099] Step 302: Based on the vehicle's position and attitude matrix when collecting historical laser point cloud data, map the laser point cloud in the historical laser point cloud data to the world coordinate system.

[0100] The laser points in the first laser point cloud data are mapped to the world coordinate system using formula (7). middle.

[0101] The laser points in the historical laser point cloud data are mapped to the world coordinate system using formula (7). middle.

[0102]

[0103] Among them, P t This represents the position and attitude matrix of the vehicle at time t, with a dimension of 4*4.

[0104] The laser point [x] collected at any time j y j z j 1] T Switch to a unified world coordinate system The laser points collected at any given time include the first laser point cloud data and historical laser point cloud data.

[0105] Step 303: Combine the first laser point cloud annotation result, the historical laser point cloud annotation result corresponding to the historical laser point cloud data, the coordinate values ​​of the laser points in the world coordinate system in the first laser point cloud data, and the coordinate values ​​of the laser points in the world coordinate system in the historical laser point cloud data to obtain the second laser point cloud annotation result.

[0106] Any point in the laser point cloud of the second annotation result can be represented in the following form:

[0107]

[0108] PC2 represents the laser point cloud of the second annotation result.

[0109] By stitching and fusing multiple frames of laser point cloud data, dozens or even hundreds of point cloud frames can be stitched together to obtain a second laser point cloud annotation result containing a large amount of data, thus improving annotation efficiency.

[0110] Sub-step 1053: Eliminate the conflict of label information between different laser points in the second laser point cloud annotation result to obtain the laser point cloud annotation result with the three-dimensional lane line annotated.

[0111] The label information between different laser points in the second laser point cloud annotation result may conflict. Since lane lines have a certain length and width, it is assumed that the labels of two laser points that are very close to each other in the second laser point cloud annotation result including lane lines should be consistent. However, in reality, they are inconsistent; that is, one laser point is labeled as lane line, and the other laser point is labeled as tree. It is necessary to eliminate the conflict of label information between these two laser points in the second laser point cloud annotation result in order to obtain the laser point cloud annotation result that annotates the 3D lane lines.

[0112] In one feasible approach, sub-step 1053 eliminates conflicts in label information between different laser points in the second laser point cloud annotation result, obtaining the laser point cloud annotation result with the 3D lane lines annotated, including steps 401 to 402, referring to... Figure 5 The flowchart illustrates a three-dimensional lane line annotation method according to the present invention for obtaining laser point cloud annotation results of three-dimensional lane lines.

[0113] Step 401: For any laser point in the second laser point cloud annotation result, determine the set of neighboring points corresponding to the laser point. The set of neighboring points includes at least one laser point whose Euclidean distance from the laser point is less than a distance threshold.

[0114] Since there may be more than one laser point corresponding to a single label, it is necessary to determine the set of neighboring points corresponding to that laser point.

[0115] Assuming a distance threshold of 0.5 meters, for any laser point in the second laser point cloud annotation result, search for the set of neighboring points corresponding to the laser point according to the Euclidean distance. Assume that the set of neighboring points has K neighboring points (including any laser point in the second laser point cloud annotation result).

[0116] Step 402: If the fused label of a laser point is different from the fused label of at least one laser point in the set of neighboring points corresponding to the laser point, then calculate the second label using the following formula (8), update the fused labels of each laser point in the set of neighboring points corresponding to the laser point to the second label, and redetermine the set of neighboring points corresponding to each laser point until the fused label of any laser point is the same as the fused label of each laser point in the set of neighboring points corresponding to the laser point, and obtain the laser point cloud labeling result.

[0117]

[0118] This represents the second label corresponding to the j-th laser point in the second laser point cloud annotation result, and K represents the number of laser points in the neighbor set corresponding to the j-th laser point. This represents the fused label of the k-th laser point within the set of neighboring points corresponding to the j-th laser point, (x k y k , z k ) represents the three-dimensional coordinates of the k-th laser point within the set of neighboring points corresponding to the j-th laser point.

[0119] The fused label of any laser point is determined to be the same as the fused labels of all laser points in the set of neighboring points corresponding to that laser point, thus ensuring that the labels of adjacent laser points in the second laser point cloud annotation result are consistent, thereby improving annotation efficiency.

[0120] In this embodiment, a first laser point cloud annotation result is obtained by fusing color and label information from different image data into the second laser point cloud data. Then, the first laser point cloud annotation result is stitched together with historical laser point cloud annotation results to obtain a second laser point cloud annotation result. Finally, conflicts in label information between different laser points in the second laser point cloud annotation result are eliminated to obtain a laser point cloud annotation result with labeled 3D lane lines. Through these three steps, a laser point cloud annotation result with no label information conflicts is finally obtained, thereby reducing manual annotation steps, improving annotation efficiency, and reducing annotation costs.

[0121] Example 3

[0122] Reference Figure 6 The diagram shows a flowchart of another three-dimensional lane line marking method according to Embodiment 2 of the present invention.

[0123] In this embodiment of the application, the three-dimensional lane line annotation method includes the aforementioned steps 101 to 105, and the three-dimensional lane line annotation method further includes steps 501 to 502:

[0124] Step 501: Acquire the laser point cloud data collected by the lidar.

[0125] Step 502: Perform null value removal, infinite value removal, and statistical outlier removal on the laser point cloud data in sequence to obtain the first laser point cloud data.

[0126] The first laser point cloud data is obtained by preprocessing the laser point cloud data acquired by the lidar. Specifically, the preprocessing includes sequentially removing null values, removing infinite values, and removing statistical outliers from the laser point cloud data. Statistical outlier removal involves selecting several laser points around a central laser point in the laser point cloud data, calculating the distances of these laser points to the central laser point, and deleting any laser point whose deviation from the average exceeds a certain multiple of the variance.

[0127] Optionally, laser points that are far from and / or high above a central laser point in the laser point cloud data can be removed or pre-labeled as non-interest points: for example, laser points that are more than 50 meters away from the central laser point and / or more than 5 meters high can be removed. These points may appear in other frames or are not important, and can be directly labeled as non-interest points.

[0128] In this embodiment, laser point cloud data collected by lidar is acquired. First laser point cloud data is obtained by sequentially removing null values, infinite values, and statistical outliers from the laser point cloud data. Preprocessing the acquired laser point cloud data removes invalid laser points, thereby improving annotation efficiency.

[0129] 3D lane marking device

[0130] like Figure 7 As shown, Figure 7 This is a schematic diagram of the three-dimensional lane marking device provided in the embodiment of the application. The three-dimensional lane marking device includes:

[0131] The acquisition module 601 is used to acquire synchronously acquired first laser point cloud data and image data, both of which include lane line information.

[0132] The annotation module 602 is used to annotate lane line images in image data to obtain two-dimensional lane line semantic segmentation annotation results.

[0133] The projection module 603 is used to project the first laser point cloud data onto the image data, and obtain point cloud attribute information based on the projection result and the two-dimensional lane line semantic segmentation and annotation result. The point cloud attribute information includes color information and label information corresponding to the projection points of at least some laser points in the first laser point cloud data in the image data. The label information is used to indicate the entity type of the corresponding projection point.

[0134] Module 604 is used to add point cloud attribute information to the first laser point cloud data to obtain the second laser point cloud data.

[0135] The acquisition module 605 is used to obtain the laser point cloud annotation results of the three-dimensional lane lines based on the second laser point cloud data.

[0136] The 3D lane marking device also includes the following modules:

[0137] Data acquisition module: used to acquire first laser point cloud data and image data.

[0138] The data storage module is used to store the first laser point cloud data, image data, two-dimensional lane line semantic segmentation annotation results, point cloud attribute information, second laser point cloud data, and laser point cloud annotation results of the three-dimensional lane lines.

[0139] The multi-view display and interaction module is used to read and display the laser point cloud annotation results of the three-dimensional lane lines generated by the acquisition module 605.

[0140] In this embodiment, module 601 acquires synchronously collected first laser point cloud data and image data; annotation module 602 annotates lane line images in the image data to obtain two-dimensional lane line semantic segmentation annotation results; projection module 603 projects the three-dimensional first laser point cloud data onto the two-dimensional image data to obtain projection results. Since the obtained two-dimensional image data, after annotation, clearly defines the position of the lane lines and assigns label and color information to them, after the three-dimensional first laser point cloud data is projected onto the two-dimensional image data, module 604 assigns point cloud attribute information to the laser points in the first laser point cloud data to obtain second laser point cloud data; and acquisition module 605 obtains the laser point cloud annotation results of the three-dimensional lane lines based on the second laser point cloud data. Therefore, by assigning point cloud attribute information to the three-dimensional first laser point cloud data, the second laser point cloud data is obtained, and the laser point cloud annotation results of the three-dimensional lane lines are obtained based on the second laser point cloud data. The entire process reduces the steps of manual annotation and verification, resulting in high annotation efficiency and reduced costs.

[0141] It should be noted that the information interaction and execution process between the modules of the above-mentioned lane line marking device are based on the same concept as the three-dimensional lane line marking method embodiment. For details, please refer to the description in the three-dimensional lane line marking method embodiment, and will not be repeated here.

[0142] electronic devices

[0143] Reference Figure 8 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0144] like Figure 8 As shown, the electronic device may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0145] in:

[0146] The processor 802, communication interface 804, and memory 806 communicate with each other through the communication bus 808.

[0147] Communication interface 804 is used to communicate with other electronic devices or servers.

[0148] The processor 802 is used to execute program 810, which can specifically execute the relevant steps in the above-described three-dimensional lane line marking method embodiment.

[0149] Specifically, program 810 may include program code that includes computer operation instructions.

[0150] The processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0151] Memory 806 is used to store program 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0152] Specifically, program 810 can be used to cause processor 802 to execute the three-dimensional lane line annotation method in any of the foregoing embodiments.

[0153] The specific implementation of each step in program 810 can be found in the corresponding steps and units described in any of the aforementioned lane marking device embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0154] In this embodiment, firstly, synchronously acquired 3D first laser point cloud data and 2D image data are obtained. Secondly, lane line images are annotated on the 2D image data to obtain 2D lane line semantic segmentation annotation results. Then, the 3D first laser point cloud data is projected onto the image data, and point cloud attribute information is obtained based on the projection result and the 2D lane line semantic segmentation annotation results. Finally, the obtained point cloud attribute information is added to the 3D first laser point cloud data to obtain second laser point cloud data. Based on the second laser point cloud data, the laser point cloud annotation result with 3D lane lines is obtained. This solution realizes the transformation from 3D (first laser point cloud data) to 2D (projecting the first laser point cloud data onto image data) and then to 3D (obtaining the laser point cloud annotation result with 3D lane lines), ultimately obtaining the laser point cloud annotation result with 3D lane lines. This reduces the steps of manual annotation and verification, improves annotation efficiency, and reduces costs.

[0155] Computer storage media

[0156] This application also provides a computer-readable storage medium storing instructions for causing a machine to execute the three-dimensional lane line annotation method as described herein. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0157] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of this application.

[0158] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0159] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0160] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0161] Computer program products

[0162] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0163] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0164] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the three-dimensional lane marking method described herein. Furthermore, when a general-purpose computer accesses the code used to implement the three-dimensional lane marking method shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the checksum generation method shown herein.

[0165] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0166] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A three-dimensional lane marking method, comprising: obtaining first laser point cloud data and image data synchronously collected, the first laser point cloud data and the image data both comprising information of lane lines; marking lane line images in the image data to obtain a two-dimensional lane line semantic segmentation annotation result; projecting the first laser point cloud data to the image data and obtaining point cloud attribute information according to the projection result and the two-dimensional lane line semantic segmentation annotation result, the point cloud attribute information comprising color information and label information corresponding to at least part of laser points in the first laser point cloud data, the label information being used to indicate an entity type of a corresponding projection point; adding the point cloud attribute information to the first laser point cloud data to obtain second laser point cloud data; fusing the color information and the label information corresponding to different image data in the second laser point cloud data to obtain a first laser point cloud annotation result, the different image data being collected by different cameras; splicing the first laser point cloud annotation result with historical laser point cloud annotation results to obtain a second laser point cloud annotation result, the second laser point cloud annotation result comprising a larger number of laser points than the first laser point cloud annotation result; eliminating conflicts of the label information between different laser points in the second laser point cloud annotation result to obtain a laser point cloud annotation result in which three-dimensional lane lines are marked.

2. The method of claim 1, wherein, The fusing the color information and the label information corresponding to different image data in the second laser point cloud data to obtain a first laser point cloud annotation result comprises: for each laser point in the second laser point cloud data, performing: calculating a color mean value of the laser point according to a plurality of color information corresponding to the laser point included in the second laser point cloud data; calculating a color deviation of the plurality of color information corresponding to the laser point respectively according to the color mean value of the laser point and the plurality of color information corresponding to the laser point in the second laser point cloud data; calculating a first label of the laser point according to the plurality of color deviations corresponding to the laser point through the following formula: , a first label for representing the jth laser point in the second laser point cloud data, , , a color deviation corresponding to the jth laser point in the second laser point cloud data projected in the ith image data, the label information corresponding to the jth laser point in the second laser point cloud data projected in the ith image data, N is the number of the image data, j is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to N. if the label information corresponding to the laser point in the second laser point cloud data are all the same as the first label, determining the first label of the laser point as a fused label of the laser point; if there is at least one interference label information corresponding to the laser point in the second laser point cloud data different from the first label, deleting the interference label information and corresponding color information from the second laser point cloud data, and recalculating the first label of the laser point based on the remaining color information and label information corresponding to the laser point in the second laser point cloud data until the label information corresponding to the laser point in the second laser point cloud data are all the same as the first label; adding the color mean value and the fused label of each laser point to the second laser point cloud data to obtain the first laser point cloud annotation result.

3. The method of claim 1, wherein, The first laser point cloud annotation result is spliced with historical laser point cloud annotation results to obtain a second laser point cloud annotation result, including: According to the position and posture matrix of the vehicle when the first laser point cloud data is collected, the laser points in the first laser point cloud data are mapped to a world coordinate system; According to the position and posture matrix of the vehicle when the historical laser point cloud data is collected, the laser points in the historical laser point cloud data are mapped to the world coordinate system; The first laser point cloud annotation result, the historical laser point cloud data corresponding historical laser point cloud annotation result, the coordinate values of the laser points in the first laser point cloud data in the world coordinate system, and the coordinate values of the laser points in the historical laser point cloud data in the world coordinate system are spliced to obtain the second laser point cloud annotation result.

4. The method of claim 1, wherein, The conflicts of the label information between different laser points in the second laser point cloud annotation result are eliminated to obtain a laser point cloud annotation result in which three-dimensional lane lines are labeled, including: For any laser point in the second laser point cloud annotation result, a set of adjacent points corresponding to the laser point is determined, and the set of adjacent points includes at least one laser point with a Euclidean distance less than a distance threshold from the laser point; If the fused label of a laser point is different from the fused label of at least one laser point in the set of adjacent points corresponding to the laser point, a second label is calculated by the following formula, the fused label of each laser point in the set of adjacent points corresponding to the laser point is updated to the second label, and the set of adjacent points corresponding to each laser point is re-determined until the fused label of any laser point is the same as the fused label of each laser point in the set of adjacent points corresponding to the laser point, and the laser point cloud annotation result is obtained; , Used to characterize the second laser point cloud annotation result. The second label corresponding to each laser dot Used to characterize the The number of laser points within the set of neighboring points corresponding to each laser point Used to characterize the Within the set of neighboring points corresponding to the i-th laser point The fused label of the laser points, ( ) is used to characterize the first Within the set of neighboring points corresponding to the i-th laser point The three-dimensional coordinates of the laser point.

5. The method of any one of claims 1-4, further comprising: acquiring laser point cloud data collected by a laser radar; performing null value elimination, infinite value elimination, and statistical outlier elimination on the laser point cloud data in sequence to obtain the first laser point cloud data.

6. A three-dimensional lane line labeling device, comprising: an acquisition module configured to acquire first laser point cloud data and image data collected synchronously, the first laser point cloud data and the image data both comprising information of lane lines; a labeling module configured to label lane line images in the image data to obtain a two-dimensional lane line semantic segmentation annotation result; a projection module configured to project the first laser point cloud data to the image data, and to obtain point cloud attribute information according to a projection result and the two-dimensional lane line semantic segmentation annotation result, the point cloud attribute information comprising color information and label information of at least part of the projection points of the laser points in the first laser point cloud data in the image data, the label information being used to indicate the entity type of the corresponding projection points; an adding module configured to add the point cloud attribute information to the first laser point cloud data to obtain second laser point cloud data. The acquisition module is configured to fuse the color information and the label information corresponding to different image data in the second laser point cloud data, to obtain a first laser point cloud annotation result, to splice the first laser point cloud annotation result with a historical laser point cloud annotation result, to obtain a second laser point cloud annotation result, to eliminate the conflicts of the label information between different laser points in the second laser point cloud annotation result, and to obtain a laser point cloud annotation result in which three-dimensional lane lines are labeled. The different image data is collected by different cameras. The number of laser points included in the second laser point cloud annotation result is greater than the number of laser points included in the first laser point cloud annotation result. 7.An electronic device comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of any one of claims 1-5. 8.A computer storage medium having stored thereon a computer program, the program being executed by a processor to implement the method of any one of claims 1-5. 9.A computer program product tangibly stored on a computer readable medium and comprising computer executable instructions that, when executed, cause at least one processor to perform the method of any one of claims 1-5.

Images