Intelligent labeling method and system for 3D point cloud data

Abstract

The invention discloses an intelligent labeling method and system for 3D point cloud data, relates to the technical field of 3D point cloud intelligent labeling, and aims to solve the technical problems that a current traditional manual labeling method is low in efficiency, high in cost and difficult to meet the requirement for large-scale data processing. Performing multi-layer filtering processing on the original 3D point cloud data of the multi-source sensor, including voxel gridding to reduce data volume and statistical filtering to remove noise points; a point cloud down-sampling system is applied, and key features are reserved while data sparsity is kept; performing super-resolution reconstruction, color correction and image enhancement processing on the camera image data to improve the image quality; a timestamp synchronization algorithm is introduced to ensure the time consistency of the point cloud data and the image data; s2, high-precision data alignment is carried out; and external calibration parameters and sensor internal parameters are utilized. The method has the advantage of realizing automatic labeling of the target object.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D point cloud intelligent annotation, and more specifically, to an intelligent annotation method and system for 3D point cloud data. Background Art

[0002] 3D point cloud annotation refers to the annotation of point cloud data collected from three-dimensional objects, and further used for the training and application of computer vision algorithms. Point cloud is a "cloud" composed of "points". It is a collection of a large number of points collected by sensors such as lidar, and is used to represent the surface of objects in three-dimensional space. These points contain three-dimensional coordinates X, Y, Z, and possible other information such as color, classification value, intensity value, time, etc. Compared with 2D image data, 3D point cloud data provides rich geometry, shape and scale information, and is not easily affected by changes in light intensity and occlusion by other objects. It can restore the surrounding environment of the machine more comprehensively and specifically. 3D point cloud annotation provides basic data for computer vision algorithms and is an important support for fields such as autonomous driving, robot navigation, and three-dimensional modeling.

[0003] In applications such as autonomous driving, robot navigation and 3D modeling, 3D point cloud data annotation is a crucial link. At present, in the three-dimensional space collected by lidar, the judgment and annotation of targets are usually performed manually. The traditional manual annotation method is not only inefficient, but also costly, and it is difficult to meet the needs of large-scale data processing. The traditional annotation method is to manually select the target using a 3D box. However, since the shape and size of the target are usually irregular, the traditional 3D box can only select the approximate range of the target and cannot fit the outline of the target. Training the intelligent driving system under this annotation method is likely to cause errors in the system's judgment of the passability of the target, thereby affecting the normal operation of the vehicle. In view of this, we propose an intelligent annotation method and system for 3D point cloud data. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent annotation method and system for 3D point cloud data, so as to solve the technical problem that the current traditional manual annotation method is not only inefficient but also costly and difficult to meet the needs of large-scale data processing.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for intelligently labeling 3D point cloud data, comprising the following steps:

[0006] S1, refined data preprocessing;

[0007] Perform multi-layer filtering on the raw 3D point cloud data from multiple source sensors, including voxel gridding to reduce the amount of data and statistical filtering to remove noise points;

[0008] Apply a point cloud downsampling system to preserve key features while maintaining data sparsity;

[0009] Perform super-resolution reconstruction, color correction and image enhancement on camera image data to improve image quality;

[0010] Introducing a timestamp synchronization algorithm to ensure the time consistency between point cloud data and image data;

[0011] S2, high-precision data alignment;

[0012] Using external calibration parameters and sensor internal parameters, the transformation matrix between sensors is optimized through the least squares method to achieve accurate spatial alignment between point cloud and image;

[0013] The alignment error minimization formula is as follows:

[0014]

[0015] Among them, R is the rotation matrix, t is the translation vector, pi and qi are the coordinates of the corresponding points in the point cloud and image respectively, and N is the number of corresponding points;

[0016] S3, deep learning target detection;

[0017] Deploy a 3D object detection network that combines VoxelNet with a ResNet feature extractor to process point cloud data and output the object’s 3D bounding box, category label, and confidence level.

[0018] At the same time, the image data is processed using EfficientDet’s 2D object detection network to output the object’s 2D bounding box, category label, and confidence level.

[0019] S4, complex multi-source data fusion;

[0020] The data fusion method based on the probabilistic graph model of conditional random field (CRF) is used to combine the 3D and 2D detection results to perform association and consistency verification between targets.

[0021] During the fusion process, a weighted fusion algorithm is used. The weight calculation takes into account confidence, target size, and shape similarity. The fusion weight calculation formula is as follows:

[0022]

[0023] Where M is the total number of detected targets;

[0024] S5, refined 3D frame optimization annotation;

[0025] Based on the fused target position and category information, a 3D box is preliminarily marked on the 3D point cloud;

[0026] The shape-aware ICP iterative optimization algorithm based on shape is introduced, and the local geometric characteristics such as surface normal and curvature of point cloud data are combined to fine-tune the 3D frame. The optimization function is as follows:

[0027]

[0028] Where b is the 3D box parameter to be optimized, T(b) is the transformation matrix, pk is the point in the point cloud, ck(s) is the fitting center based on the shape parameter s, and b init is the initial 3D box parameter, λ1 and λ2 are regularization coefficients, ShapeConstraint(b, s) is the shape constraint term;

[0029] S6, labeling result output and post-processing;

[0030] Output optimized 3D box annotation results, including detailed 3D properties of center point, size, orientation, and category.

[0031] An intelligent annotation system for 3D point cloud data, including a data acquisition system;

[0032] The data acquisition system is connected to a preprocessing system for preprocessing data, the preprocessing system is connected to a high-precision spatial system, the high-precision spatial alignment system is connected to a deep learning target detection system, the deep learning target detection system outputs target data and is connected to a multi-source data fusion and association system, the multi-source data fusion and association system is connected to a 3D annotation and edge optimization system, the 3D annotation and edge optimization system is connected to a user interaction and feedback loop system, the user interaction and feedback loop system is connected to a result output and post-processing system, and the result output and post-processing system are connected to a smart learning and self-optimization system.

[0033] Preferably, the data acquisition system is used to acquire raw data from multi-source sensors including a laser radar, a camera, and an IMU inertial measurement unit.

[0034] Preferably, the preprocessing system performs multi-layer filtering, downsampling, super-resolution reconstruction, color correction, image enhancement, and timestamp synchronization on the data collected from the data acquisition system to ensure the accuracy and consistency of the data and output high-quality 3D point cloud and image data.

[0035] Preferably, the high-precision spatial alignment system utilizes external calibration parameters and sensor internal parameters to achieve high-precision spatial alignment of 3D point cloud and image data through an optimization algorithm, and the high-precision spatial alignment system supports real-time calibration and dynamic adjustment to adapt to environmental changes or changes in sensor position.

[0036] Preferably, the deep learning target detection system includes deploying advanced 3D and 2D deep learning target detection networks to process point cloud and image data respectively, and complete the output of the target's bounding box, category label and confidence, providing key information for multi-source data fusion.

[0037] Preferably, the multi-source data fusion and association system adopts a probabilistic graphical model or a weighted fusion algorithm, combines 3D and 2D detection results to perform target association and consistency verification, considers the temporal continuity of the target, performs cross-frame tracking and matching, and improves the stability of annotation.

[0038] Preferably, the 3D annotation and edge optimization system further includes a dynamic adjustment module, which can adjust the parameters of the optimization algorithm in real time according to the motion state of the target object or environmental changes, thereby ensuring the real-time and accuracy of the annotation results.

[0039] Preferably, the user interaction and feedback loop system provides an intuitive user interface, allowing the user to view, edit and verify the annotation results, receive user feedback, and automatically adjust the annotation parameters, optimize the algorithm or re-execute part of the processing flow accordingly, supports real-time preview and gradual correction functions, improves the convenience and efficiency of user operations, and the user interaction and feedback loop system also includes an annotation history record module, which can record the user's annotation operation history and support fast retrieval and reuse of historical annotations.

[0040] Preferably, the intelligent learning and self-optimization system includes a knowledge base construction module, which can collect and organize historical annotation data, user feedback and system optimization strategies, and construct a knowledge base to support the system's intelligent learning and decision-making process.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention pre-processes the 3D point cloud data acquired by multi-source sensors by designing refined data pre-processing steps, and optimizes the transformation matrix of the sensor device by the least squares method to achieve accurate spatial alignment between the cloud and the image, and deploys a 3D target detection network and a 2D target detection network to process the point cloud data, output the 3D and 2D bounding boxes, category labels and confidence levels of the target, introduce the target data into the data fusion method of the probabilistic graph model, combine the 3D and 2D detection results, perform association and consistency verification between targets, generate target position and category information through a weighted fusion algorithm, and preliminarily mark the 3D box on the 3D point cloud, thereby achieving automatic labeling of the target, solving the problem that the traditional manual labeling method is not only inefficient but also costly and difficult to meet the needs of large-scale data processing.

[0043] 2. The present invention also realizes fine adjustment of the 3D frame by designing a refined 3D frame optimization annotation processing step, combining local geometric characteristics such as surface normals and curvature of point cloud data, thereby realizing fine annotation of the contour shape of the target object, solving the problem that the traditional 3D frame can only select the approximate range of the target object but cannot select the contour of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the system framework of the present invention;

[0045] Figure 2 It is a schematic diagram of the framework of the data acquisition system of the present invention;

[0046] Figure 3 It is a schematic diagram of the framework of the pretreatment system of the present invention;

[0047] Figure 4 It is a schematic diagram of the framework of the high-precision spatial alignment system of the present invention;

[0048] Figure 5 Schematic diagram of the framework of the deep learning target detection system of the present invention;

[0049] Figure 6 Schematic diagram of the framework output of the deep learning target detection system of the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings of the specification.

[0051] An intelligent annotation method for 3D point cloud data comprises the following steps:

[0052] S1, refined data preprocessing;

[0053] Perform multi-layer filtering on the raw 3D point cloud data from multiple source sensors, including voxel gridding to reduce the amount of data and statistical filtering to remove noise points;

[0054] Apply a point cloud downsampling system to preserve key features while maintaining data sparsity;

[0055] Perform super-resolution reconstruction, color correction and image enhancement on camera image data to improve image quality;

[0056] Introducing a timestamp synchronization algorithm to ensure the time consistency between point cloud data and image data;

[0057] S2, high-precision data alignment;

[0058] Using external calibration parameters and sensor internal parameters, the transformation matrix between sensors is optimized through the least squares method to achieve accurate spatial alignment between point cloud and image;

[0059] The alignment error minimization formula is as follows:

[0060]

[0061] Among them, R is the rotation matrix, t is the translation vector, pi and qi are the coordinates of the corresponding points in the point cloud and image respectively, and N is the number of corresponding points;

[0062] S3, deep learning target detection;

[0063] Deploy a 3D object detection network that combines VoxelNet with a ResNet feature extractor to process point cloud data and output the object’s 3D bounding box, category label, and confidence level.

[0064] At the same time, the image data is processed using EfficientDet’s 2D object detection network to output the object’s 2D bounding box, category label, and confidence level.

[0065] S4, complex multi-source data fusion;

[0066] The data fusion method based on the probabilistic graph model of conditional random field (CRF) is used to combine the 3D and 2D detection results to perform association and consistency verification between targets.

[0067] During the fusion process, a weighted fusion algorithm is used. The weight calculation takes into account confidence, target size, and shape similarity. The fusion weight calculation formula is as follows:

[0068]

[0069] Where M is the total number of detected targets;

[0070] S5, refined 3D frame optimization annotation;

[0071] Based on the fused target position and category information, a 3D box is preliminarily marked on the 3D point cloud;

[0072] The shape-aware ICP iterative optimization algorithm based on shape is introduced, and the local geometric characteristics such as surface normal and curvature of point cloud data are combined to fine-tune the 3D frame. The optimization function is as follows:

[0073]

[0074] Where b is the 3D box parameter to be optimized, T(b) is the transformation matrix, pk is the point in the point cloud, ck(s) is the fitting center based on the shape parameter s, and b initis the initial 3D box parameter, λ1 and λ2 are regularization coefficients, ShapeConstraint(b, s) is the shape constraint term;

[0075] S6, labeling result output and post-processing;

[0076] Output optimized 3D box annotation results, including detailed 3D properties of center point, size, orientation, and category.

[0077] An intelligent annotation system for 3D point cloud data, including a data acquisition system;

[0078] The data acquisition system is connected to a preprocessing system for preprocessing data, the preprocessing system is connected to a high-precision spatial system, the high-precision spatial alignment system is connected to a deep learning target detection system, the deep learning target detection system outputs target data and is connected to a multi-source data fusion and association system, the multi-source data fusion and association system is connected to a 3D annotation and edge optimization system, the 3D annotation and edge optimization system is connected to a user interaction and feedback loop system, the user interaction and feedback loop system is connected to a result output and post-processing system, and the result output and post-processing system are connected to an intelligent learning and self-optimization system.

[0079] like Figure 1-Figure 6 As shown in the figure, the data acquisition system is used to obtain raw data from multi-source sensors such as lidar, camera and IMU inertial measurement unit. The preprocessing system performs multi-layer filtering, downsampling, super-resolution reconstruction, color correction, image enhancement and timestamp synchronization on the data collected from the data acquisition system to ensure the accuracy and consistency of the data and output high-quality 3D point cloud and image data. The high-precision spatial alignment system uses external calibration parameters and sensor internal parameters to achieve high-precision spatial alignment of 3D point cloud and image data through optimization algorithms. The high-precision spatial alignment system supports real-time calibration and dynamic adjustment to adapt to environmental changes or changes in sensor position. Deep learning The target detection system includes the deployment of advanced 3D and 2D deep learning target detection networks, which process point cloud and image data respectively, and complete the output of the target's bounding box, category label and confidence, providing key information for multi-source data fusion. The multi-source data fusion and association system uses a probabilistic graph model or a weighted fusion algorithm, combines 3D and 2D detection results to perform target association and consistency verification, considers the temporal continuity of the target, performs cross-frame tracking and matching, and improves the stability of annotation. The 3D annotation and edge optimization system also includes a dynamic adjustment module, which can adjust the parameters of the optimization algorithm in real time according to the motion state of the target or environmental changes, to ensure the real-time and accuracy of the annotation results.

[0080] like Figure 1-Figure 6As shown, the user interaction and feedback loop system provides an intuitive user interface, allowing users to view, edit and verify annotation results, receive user feedback, and automatically adjust annotation parameters, optimize algorithms or re-execute part of the processing flow accordingly, supporting real-time preview and gradual correction functions to improve the convenience and efficiency of user operations. The user interaction and feedback loop system also includes an annotation history module, which can record the user's annotation operation history and support fast retrieval and reuse of historical annotations. The intelligent learning and self-optimization system includes a knowledge base construction module, which can collect and organize historical annotation data, user feedback and system optimization strategies, and build a knowledge base to support the system's intelligent learning and decision-making process.

[0081] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. An intelligent annotation method for 3D point cloud data, characterized in that: The following steps are involved: S1, refined data preprocessing; Perform multi-layer filtering on the raw 3D point cloud data from multiple source sensors, including voxel gridding to reduce the amount of data and statistical filtering to remove noise points; Apply a point cloud downsampling system to preserve key features while maintaining data sparsity; Perform super-resolution reconstruction, color correction and image enhancement on camera image data to improve image quality; Introducing a timestamp synchronization algorithm to ensure the time consistency between point cloud data and image data; S2, high-precision data alignment; Using external calibration parameters and sensor internal parameters, the transformation matrix between sensors is optimized through the least squares method to achieve accurate spatial alignment between point cloud and image; The alignment error minimization formula is as follows: Among them, R is the rotation matrix, t is the translation vector, pi and qi are the coordinates of the corresponding points in the point cloud and image respectively, and N is the number of corresponding points; S3, deep learning target detection; Deploy a 3D object detection network that combines VoxelNet with a ResNet feature extractor to process point cloud data and output the object’s 3D bounding box, category label, and confidence level. At the same time, the image data is processed using EfficientDet’s 2D object detection network to output the object’s 2D bounding box, category label, and confidence level. S4, complex multi-source data fusion; The data fusion method based on the probabilistic graph model of conditional random field (CRF) is used to combine the 3D and 2D detection results to perform association and consistency verification between targets. During the fusion process, a weighted fusion algorithm is used. The weight calculation takes into account confidence, target size, and shape similarity. The fusion weight calculation formula is as follows: Where M is the total number of detected targets; S5, refined 3D frame optimization annotation; Based on the fused target position and category information, a 3D box is preliminarily marked on the 3D point cloud; The shape-aware ICP iterative optimization algorithm based on shape is introduced, and the local geometric characteristics such as surface normal and curvature of point cloud data are combined to fine-tune the 3D frame. The optimization function is as follows: Where b is the 3D box parameter to be optimized, T(b) is the transformation matrix, pk is the point in the point cloud, ck(s) is the fitting center based on the shape parameter s, and b init is the initial 3D box parameter, λ1 and λ2 are regularization coefficients, ShapeConstraint(b, s) is the shape constraint term; S6, labeling result output and post-processing; Output optimized 3D box annotation results, including detailed 3D properties of center point, size, orientation, and category.

2. An intelligent annotation system for 3D point cloud data, characterized in that: Including data acquisition system; The data acquisition system is connected to a preprocessing system for preprocessing data, the preprocessing system is connected to a high-precision spatial system, the high-precision spatial alignment system is connected to a deep learning target detection system, the deep learning target detection system outputs target data and is connected to a multi-source data fusion and association system, the multi-source data fusion and association system is connected to a 3D annotation and edge optimization system, the 3D annotation and edge optimization system is connected to a user interaction and feedback loop system, the user interaction and feedback loop system is connected to a result output and post-processing system, and the result output and post-processing system are connected to a smart learning and self-optimization system.

3. The intelligent annotation system for 3D point cloud data according to claim 2, characterized in that: The data acquisition system is used to obtain raw data from multiple source sensors including laser radar, camera and IMU inertial measurement unit.

4. The intelligent annotation system for 3D point cloud data according to claim 3, characterized in that: The preprocessing system performs multi-layer filtering, downsampling, super-resolution reconstruction, color correction, image enhancement, and timestamp synchronization on the data collected from the data acquisition system to ensure the accuracy and consistency of the data and output high-quality 3D point cloud and image data.

5. The intelligent annotation system for 3D point cloud data according to claim 4, characterized in that: The high-precision spatial alignment system uses external calibration parameters and sensor internal parameters to achieve high-precision spatial alignment of 3D point cloud and image data through an optimization algorithm, and the high-precision spatial alignment system supports real-time calibration and dynamic adjustment to adapt to environmental changes or changes in sensor position.

6. The intelligent annotation system for 3D point cloud data according to claim 5, characterized in that: The deep learning target detection system includes deploying advanced 3D and 2D deep learning target detection networks to process point cloud and image data respectively, and complete the output of the target's bounding box, category label and confidence, providing key information for multi-source data fusion.

7. The intelligent annotation system for 3D point cloud data according to claim 6, characterized in that: The multi-source data fusion and association system adopts a probabilistic graph model or a weighted fusion algorithm, combines 3D and 2D detection results to perform target association and consistency verification, considers the temporal continuity of the target, performs cross-frame tracking and matching, and improves the stability of annotation.

8. The intelligent annotation system for 3D point cloud data according to claim 7, characterized in that: The 3D annotation and edge optimization system also includes a dynamic adjustment module, which can adjust the parameters of the optimization algorithm in real time according to the motion state of the target object or environmental changes, thereby ensuring the real-time and accuracy of the annotation results.

9. The intelligent annotation system for 3D point cloud data according to claim 8, characterized in that: The user interaction and feedback loop system provides an intuitive user interface, allowing users to view, edit and verify annotation results, receive user feedback, and automatically adjust annotation parameters, optimize algorithms or re-execute part of the processing flow accordingly, supporting real-time preview and gradual correction functions to improve the convenience and efficiency of user operations. The user interaction and feedback loop system also includes an annotation history record module, which can record the user's annotation operation history and support rapid retrieval and reuse of historical annotations.

10. The intelligent annotation system for 3D point cloud data according to claim 2, characterized in that: The intelligent learning and self-optimization system includes a knowledge base construction module, which can collect and organize historical annotation data, user feedback and system optimization strategies, and build a knowledge base to support the system's intelligent learning and decision-making process.

Images