A road scene automatic labeling method

Abstract

The application discloses a kind of road scene automatic labeling method, it is related to the field of automatic driving, comprising: obtaining multimodal sensor data, time synchronization camera image and laser radar point cloud;The multimodal sensor data is preprocessed;The camera image and laser radar point cloud after pre-processing are input into pre-labeling model, and pre-labeling result containing target category, three-dimensional space information and uncertainty score is generated;According to the uncertainty score, the pre-labeling result is sorted to obtain the final labeling result.High-efficiency multi-sensor fusion is realized, and more rich labeling of semantic and geometric information is generated;Through uncertainty quantification, the labeling efficiency is greatly improved, and the labeling quality is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an automated road scene annotation method. Background Technology

[0002] The perception module of an autonomous driving system relies heavily on a large amount of precisely labeled data for model training. These labels not only need to identify various elements on the road (vehicles, pedestrians, traffic signs, etc.), but also need to provide their precise geometric location, shape, and even semantic information.

[0003] Traditional annotation methods include manual annotation and automated / semi-automated methods based on traditional computer vision. Manual annotation of a single frame of high-quality point cloud data or image segmentation label can take tens of minutes, while an autonomous driving system may require hundreds of millions or even billions of frames for training. This severely slows down the entire industry's R&D progress, resulting in extremely low efficiency. Automated / semi-automated methods based on traditional computer vision are highly sensitive to changes in lighting, weather conditions (rain, snow, fog), and occlusion, requiring frequent adjustments to algorithm parameters. They struggle to handle complex shapes, cannot provide reliable semantic information (such as vehicle type and pedestrian pose), exhibit poor robustness, and have limited accuracy. Traditional annotation methods have become a major bottleneck in technological development. Summary of the Invention

[0004] In view of the above, the present invention aims to provide an automated road scene annotation method to solve the aforementioned technical problems.

[0005] The technical solution adopted in this invention is as follows:

[0006] This invention provides an automated road scene annotation method, including:

[0007] Acquire multimodal sensor data, including time-synchronized camera images and LiDAR point clouds;

[0008] The multimodal sensor data is preprocessed;

[0009] The preprocessed camera images and LiDAR point clouds are input into the pre-labeled model to generate pre-labeled results containing target category, 3D spatial information and uncertainty score;

[0010] The pre-labeling results are sorted according to the uncertainty scores to obtain the final labeling results.

[0011] Optionally, the multimodal sensor data is preprocessed, including:

[0012] After removing isolated noise points from the lidar point cloud, the lidar point cloud is divided into ground points and non-ground points;

[0013] Assign color information to each point to generate a colored point cloud.

[0014] Optionally, the preprocessed camera images and LiDAR point clouds are input into the pre-labeled model to generate pre-labeled results containing target category, 3D spatial information, and uncertainty score, including:

[0015] Extracting 2D image features from multiple camera images;

[0016] Extracting 3D voxel point cloud features from lidar point clouds;

[0017] The 2D image features are mapped to the bird's-eye view coordinate system, and the 2D image features are stitched together with the 3D voxel point cloud features to obtain a fused feature map.

[0018] The pre-labeled model predicts the target category, 3D spatial information, and uncertainty score based on the fused feature map.

[0019] Optionally, the pre-labeled model is a deep fusion network based on a bird's-eye view, including:

[0020] The image feature extraction branch is used to extract multi-scale 2D image features from multiple camera images;

[0021] The point cloud feature extraction branch is used to extract 3D voxel features from LiDAR point clouds;

[0022] The viewpoint transformation module is used to map 2D image features to the bird's-eye view coordinate system;

[0023] The feature fusion module is used to stitch together 2D image features with 3D voxel point cloud features;

[0024] The detection head module is used to predict the target category, 3D spatial information, and uncertainty score.

[0025] Optionally, the pre-labeling results are sorted according to the uncertainty scores to obtain the final labeling results, including:

[0026] Set a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0027] Pre-labeled results with uncertainty scores greater than the first threshold are determined as the final labeling results;

[0028] Pre-labeled results with uncertainty scores less than the second threshold are sent to the manual verification platform;

[0029] Pre-labeled results with uncertainty scores between the first and second thresholds are retained for further processing.

[0030] Optional, automated road scene annotation methods also include:

[0031] The labeled data corrected by the manual verification platform is used to build an incremental learning database.

[0032] The pre-labeled model is periodically adjusted and its parameters are updated using the incremental learning database.

[0033] The above-described solution of the present invention has at least the following beneficial effects:

[0034] The above-described solution of the present invention acquires multimodal sensor data, including time-synchronized camera images and LiDAR point clouds; preprocesses the multimodal sensor data; inputs the preprocessed camera images and LiDAR point clouds into a pre-annotation model to generate pre-annotation results containing target category, 3D spatial information, and uncertainty score; and sorts the pre-annotation results according to the uncertainty score to obtain the final annotation result. This achieves efficient multi-sensor fusion, generating annotations with richer semantic and geometric information; and through uncertainty quantification, it greatly improves annotation efficiency while ensuring annotation quality. Attached Figure Description

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0036] Figure 1 A flowchart of an automated road scene annotation method provided in an embodiment of the present invention. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0038] This invention proposes an embodiment of an automated road scene annotation method, specifically, as follows: Figure 1 As shown, it includes:

[0039] Step 11: Acquire multimodal sensor data, including time-synchronized camera images and LiDAR point clouds;

[0040] Step 12: Preprocess the multimodal sensor data;

[0041] Step 13: Input the preprocessed camera image and LiDAR point cloud into the pre-labeled model to generate a pre-labeled result containing target category, 3D spatial information and uncertainty score;

[0042] Step 14: Sort the pre-labeling results according to the uncertainty scores to obtain the final labeling results.

[0043] In this embodiment, a self-iterative and iteratively optimized automated annotation pipeline is formed by constructing a "multimodal fusion annotation model" and a "human verification closed-loop feedback system." This method does not pursue "fully automated" operation that requires no human intervention, but rather aims to minimize human intervention while ensuring extremely high annotation quality, and to allow the results of human verification to feed back into the model, making it increasingly intelligent.

[0044] In this embodiment, camera images and LiDAR point clouds are acquired through vehicle-mounted cameras and LiDAR, and time synchronization processing is performed on the camera images and LiDAR point clouds to ensure that each frame of image and its corresponding point cloud frame are acquired at the same moment or within a very short time interval.

[0045] Next, the acquired camera images and LiDAR point clouds are preprocessed. The original point cloud is preprocessed to remove isolated noise points, and the ground is initially segmented according to distance and density, dividing the point cloud into ground points and non-ground points. Using the pre-calibrated camera-LiDAR extrinsic parameter matrix, the 3D point cloud is projected onto the 2D image plane, and each 3D point is assigned corresponding RGB color information to generate a color point cloud.

[0046] Furthermore, the pre-processed camera images and LiDAR point clouds are input into the pre-annotation model, which generates 2D / 3D annotations based on a deep fusion network of bird's-eye view (BEV) images.

[0047] First, feature extraction is performed:

[0048] Using CNNs, such as ResNet-50, multi-scale 2D image features are extracted from multiple camera images.

[0049] Use point cloud neural networks, such as VoxelNet or PointPillars, to voxelize 3D point clouds and extract 3D voxel features.

[0050] The 2D features extracted from the image branches are uplifted and "pasted" onto a raster in the BEV coordinate system generated from the point cloud, based on camera intrinsic and extrinsic parameters. This process effectively aligns 2D semantic information with 3D geometric space.

[0051] Within the BEV space, feature maps from images and feature maps from point clouds are "channel stitched together" to achieve feature fusion.

[0052] Next, the fused BEV feature map is fed into a detection head, such as CenterNet or BEVFormer, to predict the category of each target, such as car, pedestrian, or cyclist; 3D information, including the centerline point, length, width, height, and orientation angle; 2D bounding boxes are obtained by backprojecting the 3D boxes back into the image, thus obtaining 2D information; and uncertainty scores are obtained through multiple inferences using prediction variance or Dropout. The uncertainty scores are used to quantify the model's confidence in each pre-labeled result.

[0053] Optionally, in the steps of the above embodiments, the pre-labeling model includes an image feature extraction branch for extracting multi-scale 2D image features from multiple camera images; a point cloud feature extraction branch for extracting 3D voxel features from LiDAR point clouds; a viewpoint conversion module for mapping 2D image features to a bird's-eye view coordinate system; a feature fusion module for stitching 2D image features with 3D voxel point cloud features; and a detection head module for predicting target category, three-dimensional spatial information, and uncertainty score.

[0054] Furthermore, based on the uncertainty scores output from the above steps, all pre-labeled results are sorted.

[0055] First, a low uncertainty threshold is set, for example, an uncertainty score > 0.95; and a high uncertainty threshold is set, for example, an uncertainty score < 0.7. When the uncertainty score is greater than the low uncertainty threshold, it is automatically accepted as the final annotation without manual intervention. This typically covers most simple and clear scenarios (such as vehicles with no obstructions in front). When the uncertainty score is less than the high uncertainty threshold, such as severely occluded objects, strangely shaped vehicles, or objects corresponding to sparse point clouds in the distance, the system automatically pushes them to a manual verification platform for priority verification and correction.

[0056] All high-quality labeled data (including raw sensor data and final labeled files) that have been manually verified or automatically accepted are stored in an "incremental learning database". Every so often, for example, every 1,000 new labeled scenes are accumulated, the system will use the new "incremental learning database" to fine-tune the pre-labeled model.

[0057] To overcome catastrophic forgetting, the training data will be mixed with some of the original base data and a large amount of newly labeled target domain data.

[0058] The automated road scene annotation method in this embodiment continuously absorbs target domain data through incremental learning, enabling the model to adapt to new environments, sensors, and regional characteristics. The BEV fusion paradigm combines the semantic advantages of images with the geometric advantages of point clouds at the optimal perspective (bird's-eye view), directly outputting accurate 3D annotations. This avoids fuzzy reasoning from 2D to 3D, achieving efficient multi-sensor fusion and generating annotations with richer semantic and geometric information. By quantifying uncertainty, valuable human effort is precisely invested in the most valuable 10%-20% of difficult cases, solving more than 80% of simple annotations and greatly improving annotation efficiency. The manual verification process ensures that the quality of the final output annotations is equal to or even higher than that of purely manual annotations, guaranteeing annotation quality.

[0059] It should be noted that this device is the same as the method described above. All implementations in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0060] An embodiment of the present invention also provides a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described in the above embodiments. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0061] In this embodiment of the invention, a computer-readable storage medium is also provided, storing instructions that, when executed on a computer, cause the computer to perform the method described in the above embodiments. All implementations of the methods described in the above embodiments are applicable to this embodiment and can achieve the same technical effect.

[0062] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0063] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0064] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0065] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0066] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0067] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0068] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0069] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0070] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automated annotation of road scenes, characterized in that, include: Acquire multimodal sensor data, including time-synchronized camera images and LiDAR point clouds; The multimodal sensor data is preprocessed; The preprocessed camera images and LiDAR point clouds are input into the pre-labeled model to generate pre-labeled results containing target category, 3D spatial information and uncertainty score; The pre-labeling results are sorted according to the uncertainty scores to obtain the final labeling results.

2. The automated road scene annotation method according to claim 1, characterized in that, Preprocessing the multimodal sensor data includes: After removing isolated noise points from the lidar point cloud, the lidar point cloud is divided into ground points and non-ground points; Assign color information to each point to generate a colored point cloud.

3. The automated road scene annotation method according to claim 1, characterized in that, The preprocessed camera images and LiDAR point clouds are input into the pre-labeled model to generate pre-labeled results containing target category, 3D spatial information, and uncertainty score, including: Extracting 2D image features from multiple camera images; Extracting 3D voxel point cloud features from lidar point clouds; The 2D image features are mapped to the bird's-eye view coordinate system, and the 2D image features are stitched together with the 3D voxel point cloud features to obtain a fused feature map. The pre-labeled model predicts the target category, 3D spatial information, and uncertainty score based on the fused feature map.

4. The automated road scene annotation method according to claim 3, characterized in that, The pre-labeled model is a deep fusion network based on bird's-eye view, including: The image feature extraction branch is used to extract multi-scale 2D image features from multiple camera images; The point cloud feature extraction branch is used to extract 3D voxel features from LiDAR point clouds; The viewpoint transformation module is used to map 2D image features to the bird's-eye view coordinate system; The feature fusion module is used to stitch together 2D image features with 3D voxel point cloud features; The detection head module is used to predict the target category, 3D spatial information, and uncertainty score.

5. The automated road scene annotation method according to claim 1, characterized in that, The pre-labeling results are sorted according to the uncertainty scores to obtain the final labeling results, including: Set a first threshold and a second threshold, wherein the first threshold is greater than the second threshold; Pre-labeled results with uncertainty scores greater than the first threshold are determined as the final labeling results; Pre-labeled results with uncertainty scores less than the second threshold are sent to the manual verification platform; Pre-labeled results with uncertainty scores between the first and second thresholds are retained for further processing.

6. The automated road scene annotation method according to claim 1, characterized in that, Also includes: The labeled data corrected by the manual verification platform is used to build an incremental learning database. The pre-labeled model is periodically adjusted and its parameters are updated using the incremental learning database.

Images