A three-dimensional lane line labeling method and device based on scene reconstruction
By fusing laser point cloud and 2D image data through scene reconstruction methods, semantic segmentation, clustering, and collinear fitting are performed, solving the sparsity problem of 3D lane line annotation and achieving efficient and accurate lane line annotation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Z-ONE TECH CO LTD
- Filing Date
- 2023-07-11
- Publication Date
- 2026-07-03
AI Technical Summary
In existing technologies, 3D lane line annotation is difficult. The sparsity and irregular distribution of laser point clouds make annotation difficult. The amount of lane line point cloud data is scarce, resulting in inaccurate annotation results, low efficiency, and high cost.
A scene-based 3D lane line annotation method is adopted. By acquiring laser point cloud data and 2D image data, semantic segmentation, clustering and collinear fitting are performed to achieve automatic lane line annotation.
It improves the density and accuracy of lane line point cloud annotation, realizes time-series consistent four-dimensional scene reconstruction, and improves annotation efficiency and accuracy.
Smart Images
Figure CN116895060B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of point cloud data processing technology, and in particular to a method and apparatus for three-dimensional lane line annotation based on scene reconstruction. Background Technology
[0002] Lane detection is receiving increasing attention and importance in the perception tasks of intelligent assisted driving. In order to improve the safety redundancy of the vehicle and due to the restrictions of relevant regulations, intelligent assisted driving tends to adopt a "heavy perception, light map" approach, that is, by improving the perception ability of intelligent driving vehicles to perceive environmental information such as roads, vehicles, and pedestrians, the application of high-precision maps in assisted driving is avoided or reduced.
[0003] Currently, the primary method for labeling lane lines in 3D and BEV (Bird's Eye View) maps is to use laser point clouds collected by LiDAR. The industry-standard approach involves using data collection vehicles to gather multimodal data, including laser point clouds, surround-view images, GPS, IMU, and various transformation matrices, to perform semi-automatic lane line labeling.
[0004] The difficulties in labeling lane lines using laser point clouds are mainly due to two factors. First, the sparseness and irregular distribution of laser point clouds, along with the lack of color and texture information, make labeling difficult, and single-frame labeling cannot meet application requirements. Second, lane line labeling itself is a very challenging task, as lane lines are generally long and thin, and the number of laser points scanned on each lane line is very small, resulting in a small amount of lane line point cloud data. Summary of the Invention
[0005] To address the problems of inaccurate 3D lane line annotation results, low annotation efficiency, high annotation cost, and small number of lane line point clouds, this invention provides a 3D lane line annotation method and apparatus based on scene reconstruction.
[0006] In a first aspect, the present invention provides a three-dimensional lane line annotation method based on scene reconstruction, which adopts the following technical solution:
[0007] A 3D lane line annotation method based on scene reconstruction includes:
[0008] Acquire first laser point cloud data containing several data sequences and corresponding two-dimensional image data acquired by the camera;
[0009] Two-dimensional lane line semantic segmentation labels are obtained based on the two-dimensional image data;
[0010] The first laser point cloud data is projected onto the two-dimensional image data to obtain the first semantic lane line point cloud;
[0011] The first semantic lane line point cloud of the continuous frames is transformed from the vehicle coordinate system to the world coordinate system and clustered to obtain the first instance lane line point cloud.
[0012] The second instance of lane line point cloud is obtained by collinear fitting the first instance of lane line point cloud.
[0013] The second instance lane line point cloud is projected onto the two-dimensional image data to obtain a third instance lane line point cloud with a consistent data sequence.
[0014] Furthermore, in the above-mentioned method for 3D lane line annotation based on scene reconstruction, the step of obtaining first laser point cloud data containing several data sequences and its corresponding 2D image data includes:
[0015] Collect road data and establish a self-collected dataset containing several data sequences;
[0016] The first laser point cloud data is obtained by preprocessing the original laser point cloud in the self-acquired dataset.
[0017] Furthermore, in the above-mentioned method for three-dimensional lane line annotation based on scene reconstruction, the step of projecting the first laser point cloud data onto the two-dimensional image data to obtain the first semantic lane line point cloud includes:
[0018] The first laser point cloud data is projected from the lidar coordinate system to the camera coordinate system, so that the first laser point cloud data obtains lane line semantic segmentation labels. The set of laser point clouds with the lane line semantic segmentation labels is the first semantic lane line point cloud.
[0019] Furthermore, in the aforementioned 3D lane line annotation method based on scene reconstruction, the step of transforming the first semantic lane line point cloud of consecutive frames from the vehicle coordinate system to the world coordinate system and performing clustering processing to obtain the first instance lane line point cloud includes:
[0020] Projecting the first semantic lane line point cloud onto the world coordinate system yields the second semantic lane line point cloud;
[0021] Cluster the second semantic lane line point cloud and divide it into several closely spaced subsets to obtain the first instance lane line point cloud.
[0022] Furthermore, in the above-mentioned 3D lane line annotation method based on scene reconstruction, the step of performing collinear fitting on the first instance lane line point cloud to obtain the second instance lane line point cloud includes:
[0023] Curve fitting is performed on each lane line within the lane line point cloud of the first instance.
[0024] The fitted lane lines are sorted by length and number of laser points.
[0025] Determine whether short-line instances and long-line instances are collinear one by one. If they are collinear, merge them and update their fitting results.
[0026] The process continues until no further merging is possible to obtain a second instance of lane line point cloud. A unique identifier is assigned to each lane line point cloud within the second instance of lane line point cloud.
[0027] Secondly, the present invention also provides a three-dimensional lane line marking device based on scene reconstruction, which adopts the following technical solution:
[0028] A 3D lane marking device based on scene reconstruction includes:
[0029] The data acquisition and acquisition module acquires first laser point cloud data containing several data sequences and corresponding two-dimensional image data acquired by the camera;
[0030] Data storage module, used to store the first laser point cloud data and the two-dimensional image data;
[0031] The lane line segmentation model training and annotation module is used to train a two-dimensional lane line semantic segmentation model and use the segmentation model to annotate lane lines in two-dimensional image data to obtain lane line semantic segmentation labels.
[0032] The laser point cloud preprocessing module is used to preprocess the acquired first laser point cloud data;
[0033] The lane line point cloud semantic acquisition module is used to project the first laser point cloud data onto the coordinate system where the two-dimensional image data is located, thereby obtaining the first semantic lane line point cloud;
[0034] The lane line instance construction module is used to process the first semantic lane line point cloud to obtain the second instance lane line point cloud;
[0035] The laser point cloud reprojection module is used to project the second instance lane line point cloud onto the two-dimensional image data to obtain the third instance lane line point cloud.
[0036] Furthermore, in the aforementioned three-dimensional lane line annotation device based on scene reconstruction, the lane line instance construction module includes a laser point cloud four-dimensional lane line reconstruction module, a laser point cloud clustering and fitting module, and a laser point cloud collinear merging module.
[0037] The laser point cloud four-dimensional lane line reconstruction module is used to transform the first semantic lane line point cloud to the world coordinate system to obtain the second semantic lane line point cloud.
[0038] The laser point cloud clustering and fitting module is used to cluster the second semantic lane line point cloud to obtain the first instance lane line point cloud.
[0039] The laser point cloud collinearity merging module is used to determine the collinearity of lane lines in the first instance lane line point cloud, and to perform collinearity fitting on the collinear lane lines to generate the second instance lane line point cloud.
[0040] Furthermore, the aforementioned 3D lane marking device based on scene reconstruction also includes:
[0041] The four-dimensional interactive display and correction module is used to display the annotation results for annotation personnel to check and confirm.
[0042] Thirdly, the present invention also provides a readable storage medium, which adopts the following technical solution:
[0043] A computer-readable storage medium storing computer instructions that, when executed by a processor, implement a three-dimensional lane line annotation method based on scene reconstruction as described in any one of the first aspects above.
[0044] In summary, the present invention has at least one of the following beneficial technical effects:
[0045] 1. The laser point cloud-based 3D lane line annotation method described in this invention achieves automatic 3D lane line annotation by fusing laser point cloud and 2D image data. This method has advantages such as dense lane line point cloud annotation results, high annotation accuracy and efficiency, and strong consistency of sequence annotation results.
[0046] 2. During the annotation process, the annotated lane lines not only include laser points at three-dimensional spatial locations, but also all laser point clouds at different times. Therefore, the annotation effect achieves four-dimensional scene reconstruction with time series. The lane lines annotated based on the four-dimensional reconstructed scene are annotated on point clouds of multiple consecutive frames, greatly increasing the amount of point cloud data for each lane line instance. Attached Figure Description
[0047] Figure 1 This is a flowchart of an embodiment of a three-dimensional lane line annotation method based on scene reconstruction according to the present invention.
[0048] Figure 2 This is a flowchart of an embodiment of step S1 of a three-dimensional lane line annotation method based on scene reconstruction according to the present invention.
[0049] Figure 3 This is a flowchart of an embodiment of step S2 of a three-dimensional lane line annotation method based on scene reconstruction according to the present invention.
[0050] Figure 4 This is a flowchart of an embodiment of step S4 of a three-dimensional lane line annotation method based on scene reconstruction according to the present invention.
[0051] Figure 5 This is a flowchart of an embodiment of step S5 of a three-dimensional lane line annotation method based on scene reconstruction according to the present invention.
[0052] Figure 6 This is a schematic diagram of an embodiment of a three-dimensional lane line marking device based on scene reconstruction according to the present invention.
[0053] Figure labeling: 101, Data acquisition and processing module; 102, Data storage module; 103, Lane segmentation model training and annotation module; 104, Laser point cloud preprocessing module; 105, Lane point cloud semantic acquisition module; 106, Laser point cloud four-dimensional lane line reconstruction module; 107, Laser point cloud clustering and fitting module; 108, Laser point cloud collinearity merging module; 109, Laser point cloud reprojection module; 110, Four-dimensional interactive display and correction module. Detailed Implementation
[0054] To more clearly illustrate the objectives, technical solutions, and advantages of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] The method steps described in this embodiment of the invention can be executed in the order described in the specific implementation, or the execution order of each step can be adjusted according to actual needs, provided that the technical problem can be solved. These are not listed one by one here.
[0056] The main reasons for the difficulty in labeling lane lines using laser point clouds are twofold: first, the sparseness and irregular distribution of laser point clouds, along with the lack of color and texture information, make labeling difficult, and single-frame labeling cannot meet application requirements; second, lane line labeling itself is a highly challenging task, as lane lines are generally slender structures, and the number of laser points scanned on each lane line is very small.
[0057] To address the problems of difficulty in ensuring annotation quality, low annotation efficiency, and high annotation cost in current laser point cloud 3D lane line annotation, this invention proposes a 3D lane line annotation method and device based on scene reconstruction, utilizing related technologies such as scene reconstruction, semantic segmentation, and data processing.
[0058] The following is in conjunction with the appendix Figure 1-6The present invention will be described in further detail below.
[0059] This invention discloses a 3D lane line annotation method based on scene reconstruction, referring to... Figure 1 The above includes:
[0060] S1, acquire the first laser point cloud data containing several data sequences and the corresponding two-dimensional image data acquired by the camera;
[0061] S2, Obtain two-dimensional lane line semantic segmentation labels based on the two-dimensional image data;
[0062] S3, project the first laser point cloud data onto the two-dimensional image data to obtain the first semantic lane line point cloud;
[0063] S4, transform the first semantic lane line point cloud of the continuous frames from the vehicle coordinate system to the world coordinate system, and perform clustering to obtain the first instance lane line point cloud;
[0064] S5, perform collinear fitting on the first instance lane line point cloud to obtain the second instance lane line point cloud;
[0065] S6, Project the second instance lane line point cloud onto the two-dimensional image data to obtain the third instance lane line point cloud.
[0066] During vehicle operation, firstly, sensors such as LiDAR and cameras acquire a data sequence containing first laser point cloud data and 2D image data, where each frame contains a laser point cloud and a corresponding 2D image. Next, a deep learning model is used to segment the 2D image to obtain lane line semantic segmentation labels. Then, the 2D laser point cloud is projected onto the corresponding 2D image plane, and a first semantic lane line point cloud is obtained based on the lane line semantic segmentation labels. Next, the first semantic lane line point cloud undergoes 3D projection and clustering processing to obtain an instantiated representation of the lane lines, i.e., the first instance lane line point cloud. Then, collinear fitting is performed on the lane line instances in the first instance lane line point cloud to obtain a second instance lane line point cloud. Finally, the second instance lane line point cloud is projected onto each 2D image to obtain a third instance lane line point cloud consistent throughout the sequence.
[0067] The laser point cloud-based 3D lane line annotation method described in this invention achieves automatic 3D lane line annotation by fusing laser point cloud and 2D image data. After automatic annotation, the initially annotated lane lines are further collinearly fitted to obtain continuous and consistent second and third instance lane line point clouds. This method has advantages such as dense lane line point cloud annotation results, high annotation accuracy and efficiency, and strong consistency of sequential annotation results, and can be widely applied in fields such as autonomous driving and intelligent transportation.
[0068] Furthermore, as one embodiment of the present invention, refer to Figure 2 Step S1, acquire first laser point cloud data containing several data sequences and corresponding two-dimensional image data acquired by the camera, including:
[0069] S11, Collect road data and establish a self-collected dataset containing several data sequences;
[0070] S12, preprocess the original laser point cloud in the self-acquired dataset to obtain the first laser point cloud data.
[0071] Furthermore, in step S11, the road data includes at least: laser point cloud data, surround-view camera image data at the corresponding time, a transformation matrix from the vehicle coordinate system to the lidar coordinate system, a transformation matrix between the lidar coordinate system and the camera coordinate system, an intrinsic parameter matrix of the surround-view camera, and a pose matrix of the number of vehicles collected at each acquisition time, etc. The above road data is collected to form the self-collected dataset.
[0072] Optionally, in order to improve annotation efficiency and ensure annotation effect, data segmentation can be performed according to a certain time length (e.g., 30 seconds), that is, the self-collected dataset can be divided into several self-collected data sequences; before each data collection, the pose, external parameter matrix, etc. can be calibrated and adjusted, and the trigger signals of LiDAR and camera acquisition can be synchronized by software.
[0073] Furthermore, in step S12, the preprocessing method includes motion compensation and data removal. The motion compensation preprocessing refers to the fact that the lidar uses a scanning method for distance measurement, with a scanning frequency typically of 10Hz. Since the vehicle collecting the data travels at a certain speed, the acquisition time for each azimuth angle of a single frame of point cloud varies, leading to distortion in the point cloud. Compensation is needed to reduce its impact on scene accuracy. The data removal preprocessing involves sequentially performing operations such as null value removal, infinite value removal, and statistical outlier removal on the lidar point cloud to reduce data noise and improve data quality.
[0074] Furthermore, as one embodiment of the present invention, refer to Figure 3 Step S2, obtaining two-dimensional lane line semantic segmentation labels based on the two-dimensional image data, including:
[0075] S21, Determine the two-dimensional lane line segmentation model;
[0076] S22, Train the two-dimensional lane line segmentation model based on the lane line self-collected dataset and / or public dataset;
[0077] S23, use the trained lane line segmentation model to perform semantic segmentation on the two-dimensional image data to obtain the lane line semantic segmentation label.
[0078] Furthermore, as one embodiment of the present invention, the two-dimensional lane line segmentation model can use semantic segmentation models such as BiSeNet, InternImage, and HRNet.
[0079] Furthermore, as one embodiment of the present invention, the two-dimensional lane line segmentation model can also use instance segmentation models such as LaneNet, OneFormer, and Panoptic-DeepLab.
[0080] It should be noted that the segmentation algorithm based on the semantic segmentation model outputs the pixels of all lane lines in the image, but cannot distinguish which pixels belong to the same lane line. The segmentation algorithm based on the instance segmentation model outputs lane line instances and their corresponding pixels, which can obtain the number of lane lines in each image and the pixel position of each lane line. However, the instances here only apply to a single frame image, and it is still impossible to accurately determine which lane lines in consecutive frames belong to the same instance.
[0081] Furthermore, as one embodiment of the present invention, step S3, projecting the first laser point cloud data onto the two-dimensional image data in two dimensions to obtain the first semantic lane line point cloud, includes:
[0082] The first laser point cloud data is projected from the lidar coordinate system to the camera coordinate system, so that the first laser point cloud data obtains lane line semantic segmentation labels. The set of laser point clouds with the lane line semantic segmentation labels is the first semantic lane line point cloud.
[0083] Optionally, the first semantic lane line point cloud can be represented as k represents the total number of frames of the lidar point cloud in the sequence. N i Represents the total number of lane line semantic points in the i-th frame, [x j y j z j ] T Indicates the laser point.
[0084] Furthermore, as one embodiment of the present invention, refer to Figure 4 Step S4 involves transforming the first semantic lane line point cloud of consecutive frames from the vehicle coordinate system to the world coordinate system and performing clustering processing to obtain the first instance lane line point cloud, including:
[0085] S41, Project the first semantic lane line point cloud into the world coordinate system in three dimensions to obtain the second semantic lane line point cloud;
[0086] S42, cluster the second semantic lane line point cloud, divide the second semantic lane line point cloud into several closely spaced subsets, and obtain the first instance lane line point cloud.
[0087] Furthermore, step S41 specifically involves transforming the first semantic lane line point cloud to the world coordinate system based on the transformation matrix from the lidar coordinate system to the vehicle coordinate system and the vehicle pose, denoted as the second semantic lane line point cloud. Here, the second semantic lane line point cloud is the four-dimensional reconstructed scene containing lane lines.
[0088] Optionally, the second semantic lane line point cloud can be specifically represented as N seq S represents the total number of laser points representing all lane line semantics in the entire self-collected data sequence. laneseg-world For the entire sequence The aggregation not only includes laser points in three-dimensional space, but also a mixture of laser point clouds at different times, thus forming a four-dimensional reconstructed scene. The lane lines labeled based on the four-dimensional reconstructed scene are labeled on a continuous series of point clouds with consistent sequence.
[0089] Optionally, the clustering method in step S42 can be processed using algorithms such as Gaussian mixture model clustering, density-based spatial clustering, and K-means clustering; after completing the above processing, S can be... laneseg-world The system is divided into several pseudo lane line instances, denoted as the first instance lane line point cloud. The term "pseudo lane line instance" refers to the fact that clustering is a very coarse processing method that only performs a rough division. If the two-dimensional lane line segmentation model used when performing semantic segmentation on the two-dimensional image data is an instance segmentation model, then step S42 can be omitted. After transforming the first semantic lane line point cloud to the world coordinate system, the resulting second semantic lane line point cloud is the first instance lane line point cloud.
[0090] Optionally, the first instance lane line point cloud can be represented as N laneinstance1 This represents the number of pseudo-instances after clustering, and the set of laser point clouds of lane line pseudo-instances. It contains several laser points.
[0091] Furthermore, as one embodiment of the present invention, refer to Figure 5 Step S5, performing collinear fitting on the first instance lane line point cloud to obtain the second instance lane line point cloud, including:
[0092] S51, Perform curve fitting on each lane line in the first instance lane line point cloud;
[0093] S52, sort the fitted lane lines by length and number of laser points;
[0094] S53, determine whether short line instances and long line instances are collinear one by one. If they are collinear, merge and fit them. Continue until merging is no longer possible to obtain the second instance lane line point cloud.
[0095] Optionally, the curve fitting in step S51 can be performed using methods such as polynomial fitting, spline curve fitting, or Bézier curve fitting.
[0096] Optionally, the overall ranking in step S52 can be measured by the Euclidean distance between the two endpoints in the world coordinate system. Furthermore, to address scenarios where lane lines are curved, the number of point clouds in each lane line instance can be incorporated for weighted overall ranking.
[0097] Optionally, collinearity in step S53 refers to two curves being relatively close. Specifically, the mean Euclidean distance from all laser points in the shorter line to the fitted curve of the longer line can be used as a metric. Due to interference from factors such as the curve itself, the threshold can be set below a certain range, meaning that the two lane line instances are considered collinear.
[0098] Optionally, the second instance lane line point cloud S laneinstance2 The representation of the first instance lane line point cloud S laneinstance1 The format is similar, the difference lies in the number of instances N. laneinstance2 Fewer, but more accurate lane marking examples. Generally, taking a 30-second self-collected data sequence as an example, its real-world acquisition distance reaches approximately 500 meters, N laneinstance2 The quantity is generally between 10 and 50, N laneinstance1 The number depends on the clustering method and is generally between 200 and 3000.
[0099] Furthermore, as a specific embodiment of the present invention, step S5, performing collinear fitting on the first instance lane line point cloud to obtain the second instance lane line point cloud, further includes:
[0100] S54, assign a unique identifier to each lane line point cloud in the second instance lane line point cloud.
[0101] Specifically, for the second instance, the lane line point cloud S laneinstance2 Each N in laneinstance2 Assign a unique identifier to the entire self-collected data sequence to ensure that each N... laneinstance2 Each lane line has a unique identifier, which can be a number or a string. An auto-incrementing number, starting from 1, can be used to assign an incrementing integer as the identifier for each lane line.
[0102] Optionally, the results of the second instance lane line point cloud can be displayed interactively after visualization, making it easier for annotators to confirm and correct the second instance lane line point cloud.
[0103] Further, step S6 specifically involves: projecting the lane line point cloud onto each camera image according to the relationship between the pose matrix, the transformation matrix of the LiDAR to the acquisition vehicle, and the camera intrinsic and extrinsic parameter matrices, to obtain a third instance lane line point cloud that is consistent throughout the entire sequence. The consistency of the (acquired data) sequence refers to the fact that for any image acquired by the surround-view cameras at different times, as long as it belongs to the same lane line in the physical world, its unique identifier is the same in the image annotation results. Specifically, any instance is contained within the N of the second instance lane line point cloud. laneinstance2 In each instance, the sequence here is consistent, which can guarantee the consistency of instances of the panoramic cameras.
[0104] Furthermore, the annotation results of the first instance lane line point cloud and / or the second instance lane line point cloud and / or the third instance lane line point cloud can be saved in Json, xml, txt and other file formats for easy reading and use later.
[0105] Based on the above-mentioned three-dimensional lane line annotation method based on scene reconstruction, this invention also discloses a three-dimensional lane line annotation device based on scene reconstruction.
[0106] Reference Figure 6 A 3D lane line annotation device based on scene reconstruction includes a data acquisition and acquisition module 101, a data storage module 102, a lane line segmentation model training and annotation module 103, a laser point cloud preprocessing module 104, a lane line point cloud semantic acquisition module 105, a lane line instance construction module, and a laser point cloud reprojection module 109.
[0107] The data acquisition and acquisition module 101 is used to collect radar data, image data, and other vehicle body data for calculation. Specifically, this module includes vehicle-side and cloud-side components. The data acquisition vehicle is generally equipped with LiDAR, surround-view cameras, GPS / IMU, etc. The vehicle-side component of the data acquisition module can collect information from all sensors; the cloud-side component of the data acquisition module receives information stored in the vehicle-side component and can also download publicly available lane line datasets from public websites; all of the above data is stored in the data storage module 102.
[0108] The data storage module 102 is used to store various data during the lane line annotation process. Specifically, this module is responsible for storing all data in the laser point cloud 3D lane line annotation device based on scene reconstruction, including but not limited to the collected laser point cloud, surround view image, publicly available lane line dataset, deep pre-trained model, deep model network model, intermediate processing results of laser point cloud and image, annotation result data, as well as intermediate data and log data during software operation.
[0109] The lane line segmentation model training and annotation module 103 is used to train a two-dimensional lane line semantic segmentation model and use the segmentation model to annotate lane lines in two-dimensional image data to obtain lane line semantic segmentation labels. Specifically, this module reads publicly available data from the data storage module 102 to complete the training of the lane line semantic segmentation model, and then uses the trained lane line semantic segmentation model to automatically annotate the surround-view images in the self-collected dataset. The lane line segmentation labels are stored in the data storage module 102.
[0110] The laser point cloud preprocessing module 104 is used to preprocess the acquired first laser point cloud data. Specifically, this module reads the raw laser point cloud data from the data storage module 102 and performs preprocessing operations such as motion compensation and outlier removal on the lidar point cloud data.
[0111] The lane line point cloud semantic acquisition module 105 is used to project the first laser point cloud data onto the coordinate system where the two-dimensional image data is located, thereby obtaining the first semantic lane line point cloud. Specifically, this module reads the first laser point cloud data and related transformation matrices from the data storage module 102, completes steps such as lane line segmentation label projection, obtains the first semantic lane line point cloud, and stores it in the data storage module 102.
[0112] The lane line instance construction module is used to process the first semantic lane line point cloud to obtain the first instance lane line point cloud. Specifically, the lane line instance construction module includes a laser point cloud four-dimensional lane line reconstruction module 106, a laser point cloud clustering and fitting module 107, and a laser point cloud collinearity merging module 108.
[0113] The laser point cloud four-dimensional lane line reconstruction module 106 is used to transform the first semantic lane line point cloud to the world coordinate system to obtain the second semantic lane line point cloud. Specifically, the module reads the first semantic lane line point cloud and related transformation matrix of each sequence from the data storage module 102 in sequence, completes the world coordinate system transformation and four-dimensional lane line point cloud reconstruction steps, and stores the second semantic lane line point cloud into the data storage module 102.
[0114] The laser point cloud clustering and fitting module 107 is used to cluster the second semantic lane line point cloud to obtain a first instance lane line point cloud. Specifically, this module reads the second semantic lane line point cloud from the data storage module 102, performs point cloud clustering and fitting sorting, and stores the sorted first instance lane line point cloud into the data storage module 102.
[0115] The laser point cloud collinearity merging module 108 is used to determine the collinearity of lane lines in the first instance lane line point cloud, and merge and fit the collinear lane lines to generate a second instance lane line point cloud. Specifically, this module reads the first instance lane line point cloud from the data storage module 102, determines and merges the collinearity of the lane line instance laser point clouds in the first instance lane line point cloud, and stores the second instance lane line point cloud in the data storage module 102.
[0116] The laser point cloud reprojection module 109 is used to project the second instance lane line point cloud onto the two-dimensional image data to obtain the third instance lane line point cloud. Specifically, this module reads the second instance lane line point cloud from the data storage module 102, reprojects the second instance lane line point cloud onto the two-dimensional image data to obtain the third instance lane line point cloud, and stores the third instance lane line point cloud into the data storage module 102.
[0117] Furthermore, the laser point cloud-based 3D lane marking device based on scene reconstruction also includes:
[0118] The four-dimensional interactive display and correction module 110 is used to display the annotation results for annotation personnel to check and confirm. Specifically, this module reads the second instance lane line point cloud and the third instance lane line point cloud from the data storage module 102 and displays the annotation results on the computer front-end device. Annotators can check and confirm the annotation results through the front-end interactive software.
[0119] This invention also discloses a computer-readable storage medium.
[0120] A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a scene reconstruction-based 3D lane line annotation method as described in any of the above embodiments. The computer-readable storage medium may include any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. The computer program includes computer program code. The computer program code may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable storage medium may include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.
[0121] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A 3D lane line annotation method based on scene reconstruction, characterized in that, include: Acquire a first lidar point cloud containing several data sequences and its corresponding two-dimensional image data acquired by the camera; Two-dimensional lane line semantic segmentation labels are obtained based on the two-dimensional image data; The first lidar point cloud is projected onto the two-dimensional image data to obtain the first semantic lane line point cloud; The first semantic lane line point cloud of consecutive frames is transformed from the vehicle coordinate system to the world coordinate system to obtain the second semantic lane line point cloud of the four-dimensional reconstructed scene containing lane lines. The second semantic lane line point cloud is then clustered to obtain the first instance lane line point cloud. Curve fitting is performed on each lane line in the first instance lane line point cloud; the fitted lane lines are sorted by length and number of laser points; it is determined one by one whether short line instances and long line instances are collinear. If they are collinear, they are merged and fitted until they cannot be merged to obtain the second instance lane line point cloud. The second instance lane line point cloud is projected onto the two-dimensional image data to obtain a third instance lane line point cloud with a consistent data sequence.
2. The three-dimensional lane line annotation method based on scene reconstruction according to claim 1, characterized in that, The acquisition of a first lidar point cloud containing several data sequences and its corresponding two-dimensional image data acquired by the camera includes: Collect road data and establish a self-collected dataset containing several data sequences; The first lidar point cloud is obtained by preprocessing the original laser point cloud in the self-acquired dataset.
3. The three-dimensional lane line annotation method based on scene reconstruction according to claim 1, characterized in that, The step of projecting the first lidar point cloud onto the two-dimensional image data to obtain the first semantic lane line point cloud includes: The first lidar point cloud is projected from the lidar coordinate system to the camera coordinate system, so that the first lidar point cloud obtains lane line semantic segmentation labels. The set of lidar point clouds with the lane line semantic segmentation labels is the first semantic lane line point cloud.
4. The three-dimensional lane line annotation method based on scene reconstruction according to claim 1, characterized in that, The process of transforming the first semantic lane line point cloud of consecutive frames from the vehicle coordinate system to the world coordinate system and performing clustering to obtain the first instance lane line point cloud includes: Projecting the first semantic lane line point cloud onto the world coordinate system yields the second semantic lane line point cloud; Clustering is performed on the second semantic lane line point cloud to divide the second semantic lane lines into several closely spaced subsets, thus obtaining the first instance lane line point cloud.
5. The three-dimensional lane line annotation method based on scene reconstruction according to claim 1, characterized in that, The step of performing collinear fitting on the first instance lane line point cloud to obtain the second instance lane line point cloud includes: A unique identifier is assigned to each lane line point cloud within the second instance lane line point cloud.
6. A three-dimensional lane marking device based on scene reconstruction, characterized in that, include: The data acquisition and processing module is used to collect radar data and image data. The data storage module is used to store various data during the lane marking process; The lane line segmentation model training and annotation module is used to train a two-dimensional lane line semantic segmentation model and use the segmentation model to annotate lane lines in two-dimensional image data to obtain lane line semantic segmentation labels. The laser point cloud preprocessing module is used to preprocess the first laser radar point cloud collected. The lane line point cloud semantic acquisition module is used to project the first lidar point cloud onto the coordinate system where the two-dimensional image data is located, thereby obtaining the first semantic lane line point cloud. The lane line instance construction module is used to process the first semantic lane line point cloud to obtain the second instance lane line point cloud; A laser point cloud reprojection module is used to project the second instance lane line point cloud onto the two-dimensional image data to obtain a third instance lane line point cloud; The lane line instance construction module includes a laser point cloud four-dimensional lane line reconstruction module, a laser point cloud clustering and fitting module, and a laser point cloud collinear merging module. The laser point cloud four-dimensional lane line reconstruction module is used to transform the first semantic lane line point cloud to the world coordinate system to obtain the second semantic lane line point cloud of the four-dimensional reconstruction scene containing lane lines. The laser point cloud clustering and fitting module is used to cluster the second semantic lane line point cloud to obtain the first instance lane line point cloud. The laser point cloud collinear merging module is used to perform curve fitting on each lane line in the first instance lane line point cloud; sort the fitted lane lines according to their length and the number of laser points; and determine whether short line instances and long line instances are collinear one by one. If they are collinear, they are merged and fitted until they can no longer be merged into the second instance lane line point cloud.
7. A three-dimensional lane marking device based on scene reconstruction according to claim 6, characterized in that, The device further includes: The four-dimensional interactive display and correction module is used to display the annotation results for annotation personnel to check and confirm.
8. A readable storage medium, characterized in that, The readable storage medium stores computer instructions, which, when executed by a processor, implement a three-dimensional lane line annotation method based on scene reconstruction as described in any one of claims 1-5.