A 3D Data Annotation Method for Embossed Intelligent Robots

Abstract

This invention relates to a 3D data annotation method for embodied intelligent robots, which addresses the problems of tedious, time-consuming, and inefficient manual annotation processes. This method combines image segmentation and point cloud clustering to enable robots to accurately and efficiently complete annotations on various objects in different application scenarios. Furthermore, it enables robots to track the annotated objects without repetitive annotation, significantly reducing annotation time and improving annotation efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a 3D data annotation method for embodied intelligent robots. Background Technology

[0002] In recent years, significant progress has been made in the field of 3D perception, greatly enhancing robots' ability to understand and process 3D space. However, high-quality 3D perception models often require large and diverse datasets for training. Currently, most 3D perception datasets are concentrated in the field of autonomous driving, and the labeled objects are mostly vehicles and pedestrians—objects commonly found on roads or in daily life. However, for embodied intelligent robots, due to the complex and diverse application scenarios, the targets to be perceived vary greatly from scenario to scenario. Therefore, 3D perception datasets for corresponding scenarios are currently extremely scarce and difficult to obtain for embodied intelligent robots.

[0003] Currently, manual annotation can ensure the accuracy of annotation results and can be flexibly adjusted according to specific task requirements, adapting to diverse annotation requirements and standards. However, manual dataset annotation requires a large amount of manpower, and the annotation process is often tedious, time-consuming, and inefficient. Summary of the Invention

[0004] The purpose of this invention is to address the problems of cumbersome, time-consuming, and inefficient manual annotation processes in the past, and to propose a 3D data annotation method for embodied intelligent robots.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A 3D data annotation method for embodied intelligent robots includes the following steps:

[0007] Step 1: Acquire image data and point cloud data of the robot's working environment, respectively;

[0008] Step 2: Obtain the position of the target object in the image data;

[0009] Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud;

[0010] Step 4: Cluster the non-ground point clouds and divide them into clusters;

[0011] Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection. Then, calculate the intersection-union ratio (IoU) of the target object position obtained in Step 2 and the minimum bounding box.

[0012] Step 6: Select the smallest bounding box with the largest intersection-union ratio as the 3D bounding box and label the 3D bounding box.

[0013] Furthermore, the specific steps of step three are as follows:

[0014] Step 31: Divide the point cloud data into multiple polar coordinate subgrids, each containing N... r,m ×N θ,m N grid cells, where N r,m N represents the number of loops in the polar coordinate subgrid. θ,m Indicates the number of sectors in the polar coordinate subgrid;

[0015] Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point to obtain the ground point set. Represented as:

[0016]

[0017] Among them, z(p k ) represents the height threshold of the return point, p k S represents a point in a grid cell. n This represents the set of points within n grid cells of a polar coordinate subgrid. z represents the average height of the seed point. seed This indicates the height threshold for selecting seed points;

[0018] Step 33: Obtain the ground point set Mean of the midpoint cloud and centralized matrix;

[0019] Steps 3 and 4: Obtain the covariance matrix of the centered matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value. Then, use the eigenvector corresponding to the minimum singular value as the normal vector of the plane. Finally, the mean and the normal vector of the plane Perform a dot product to obtain the projected value. Projection value Represented as:

[0020]

[0021] Step 35: The normal vector of the plane and p k Perform a dot product to obtain the projected value. Projection value Represented as:

[0022]

[0023] Step 36: Utilizing projection values and projection value Obtain the ground point set Ground point set Represented as:

[0024]

[0025] Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set.

[0026] Step 38: Based on ground point set The ground point set is obtained by utilizing verticality, height, and flatness. Ground point set Represented as:

[0027]

[0028] Wherein, φ(v 3,n () indicates verticality. Indicates altitude, N represents flatness. C N represents the number of grid cells contained in all polar coordinate subgrids. Z Indicates the number of polar coordinate subgrids. Indicates an intermediate variable.

[0029] Furthermore, the verticality is expressed as:

[0030]

[0031] Where z = [0,0,1] T θ τ Represents the angle threshold, v 3,n express The unit normal vector.

[0032] Furthermore, the height is represented as:

[0033]

[0034] in, r represents the average height. n γ(r) represents the distance between the centroid of Sn and the origin, where Sn represents the set of points in n grid cells within the polar coordinate subgrid. n ) indicates that r n An exponentially growing adaptive midpoint function, L τ This indicates a parameter with a constant range.

[0035] Furthermore, the flatness is expressed as:

[0036]

[0037] in, and σ τ,m These represent the gain magnitude and the set smoothness threshold, respectively, σ n λ represents an intermediate variable. 1,n , λ 2,n , λ 3,n PCA features of grid cells.

[0038] Furthermore, the specific steps of step four are as follows:

[0039] Step 41: Set the thresholds MinPts for the neighborhood radius ∈ and the number of samples;

[0040] Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. Points whose Euclidean distance is not greater than the neighborhood radius are considered to be in the point cloud of the candidate point's neighborhood. If the number of points in the neighborhood is greater than or equal to MinPts, then the candidate point is the core point; otherwise, reselect the candidate point.

[0041] Step 43: Add other points within the neighborhood of the candidate point to the cluster corresponding to the core point;

[0042] Step 44: Select all points in the cluster except the core point as candidate points, and repeat steps 42 and 43 until the cluster cannot be expanded further, obtaining the final cluster. Remove all points in this cluster from the non-terrestrial point cloud and proceed to step 45:

[0043] Steps 4 and 5: Based on the non-ground point cloud after removing points, repeat steps 42 to 44 until no core points can be obtained, then end the process and obtain several clusters.

[0044] Furthermore, the specific steps of step five are as follows:

[0045] The point clouds in each cluster after clustering are projected onto the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel of the point cloud projection, and the intersection-union ratio (IoU) is calculated with the position of the target object.

[0046] The projection matrix P is obtained through the following steps:

[0047] The extrinsic parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated based on the extrinsic parameters to obtain the projection matrix P.

[0048] Furthermore, the image data in step one is obtained through an RGB camera.

[0049] Furthermore, the point cloud data in step one is obtained using the LeiShen C16-151B lidar.

[0050] Furthermore, the position of the target object in step two is obtained through a 2D target detection model or a segmentation model.

[0051] The beneficial effects of this invention are:

[0052] This application combines multiple methods such as image segmentation and point cloud clustering, enabling robots to accurately and efficiently complete annotation when facing various labeled objects in different application scenarios. It can also track labeled objects without repeating annotations, greatly reducing annotation time and improving annotation efficiency. Attached Figure Description

[0053] Figure 1 This is the overall flowchart of this application;

[0054] Figure 2 This is a schematic diagram of the point cloud clustering process. Detailed Implementation

[0055] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.

[0056] Specific implementation method one: as follows Figure 1 The 3D data annotation method for embodied intelligent robots described in this embodiment includes the following steps:

[0057] Step 1: Acquire image data and point cloud data of the robot's working environment, respectively;

[0058] Step 2: Obtain the position of the target object in the image data;

[0059] Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud;

[0060] Step 4: Cluster the non-ground point clouds and divide them into clusters;

[0061] Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection. Then, calculate the intersection-union ratio (IoU) of the target object position obtained in Step 2 and the minimum bounding box.

[0062] Step 6: Select the smallest bounding box with the largest intersection-union ratio as the 3D bounding box and label the 3D bounding box.

[0063] First, we used an RGB camera and a Leishen C16-151B LiDAR to collect image data and point cloud data respectively, and recorded the external parameters of the two sensors.

[0064] 2D Annotated Object Segmentation

[0065] For the acquired image data, we input the labels to be annotated, such as pedestrians, cars, etc., into a 2D object detection model, such as YOLOv8, to generate 2D bounding boxes for the corresponding labeled objects in the image. Alternatively, we can annotate key points in the image, and the model draws bounding boxes based on these key points to select the objects to be annotated (the bounding boxes do not need to perfectly match the object edges). Then, we convert the annotated 2D bounding boxes into the model's input format and pass them into the image segmentation model. The model will segment the annotated objects and output a mask for the target objects. We then extract the pixel information from the mask and generate 2D bounding boxes based on the edge pixels of the segmentation results. This annotates the position and contour of the target objects in the image.

[0066] Specific Implementation Method Two: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the specific steps of step three are as follows:

[0067] Step 31: Divide the point cloud data into multiple polar coordinate subgrids, each containing N... r,n ×N θ,m N grid cells, where N r,m N represents the number of loops in the polar coordinate subgrid. θ,m Indicates the number of sectors in the polar coordinate subgrid;

[0068] Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point to obtain the ground point set. Represented as:

[0069]

[0070] Among them, z(p k ) represents the height threshold of the return point, p k S represents a point in a grid cell. n This represents the set of points within n grid cells of a polar coordinate subgrid. z represents the average height of the seed point. seed This indicates the height threshold for selecting seed points;

[0071] Step 33: Obtain the ground point set Mean of the midpoint cloud and centralized matrix;

[0072] Steps 3 and 4: Obtain the covariance matrix of the centered matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value. Then, use the eigenvector corresponding to the minimum singular value as the normal vector of the plane. Finally, the mean and the normal vector of the plane Perform a dot product to obtain the projected value. Projection value Represented as:

[0073]

[0074] Step 35: The normal vector of the plane and p k Perform a dot product to obtain the projected value. Projection value Represented as:

[0075]

[0076] Step 36: Utilizing projection values and projection value Obtain the ground point set Ground point set Represented as:

[0077]

[0078] Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set.

[0079] Step 38: Based on ground point set The ground point set is obtained by utilizing verticality, height, and flatness. Ground point set Represented as:

[0080]

[0081] Wherein, φ(v 3,n ) indicates verticality (Uprightness). Indicates elevation. N represents flatness. C N represents the number of grid cells contained in all polar coordinate subgrids. Z Indicates the number of polar coordinate subgrids. Indicates an intermediate variable.

[0082] The input point cloud P is divided into multiple sub-meshes according to a predefined mesh size, so as to process different regions and save computation. The sub-mesh is denoted by Z. m express.

[0083] Specific Implementation Method Three: This implementation method is a further explanation of Specific Implementation Method Two. The difference between this implementation method and Specific Implementation Method Two is that the verticality is expressed as:

[0084]

[0085] Where z = [0,0,1] T θ τ Represents the angle threshold, v 3,n express The unit normal vector (finding method and step three) The method is the same.

[0086] Specific Implementation Method Four: This implementation method is a further explanation of Specific Implementation Method Three. The difference between this implementation method and Specific Implementation Method Three is that the height is represented as follows:

[0087]

[0088] in, r represents the average height. n γ(r) represents the distance between the centroid of Sn and the origin, where Sn represents the set of points in n grid cells within the polar coordinate subgrid. n ) indicates that r n An exponentially growing adaptive midpoint function, L τ This indicates a parameter with a constant range.

[0089] Specific Implementation Method Five: This implementation method is a further explanation of Specific Implementation Method Four. The difference between this implementation method and Specific Implementation Method Four is that the flatness is expressed as:

[0090]

[0091] in, and σ τ,m These represent the gain magnitude and the set smoothness threshold, respectively, σ n λ represents an intermediate variable. 1,n , λ 2,n , λ 3,n PCA features of grid cells.

[0092] Specific Implementation Method Six: This implementation method is a further explanation of Specific Implementation Method Five. The difference between this implementation method and Specific Implementation Method Five is that the specific steps of step four are as follows:

[0093] Step 41: Set the thresholds MinPts for the neighborhood radius ∈ and the number of samples;

[0094] Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. Points whose Euclidean distance is not greater than the neighborhood radius are considered to be in the point cloud of the candidate point's neighborhood. If the number of points in the neighborhood is greater than or equal to MinPts, then the candidate point is the core point; otherwise, reselect the candidate point.

[0095] Step 43: Add other points within the neighborhood of the candidate point to the cluster corresponding to the core point;

[0096] Step 44: Select all points in the cluster except the core point as candidate points, and repeat steps 42 and 43 until the cluster cannot be expanded further, obtaining the final cluster. Remove all points in this cluster from the non-terrestrial point cloud and proceed to step 45:

[0097] Steps 4 and 5: Based on the non-ground point cloud after removing points, repeat steps 42 to 44 until no core points can be obtained, then end the process and obtain several clusters.

[0098] Point cloud clustering process as follows Figure 2 As shown.

[0099] Specific Implementation Method Seven: This implementation method is a further explanation of Specific Implementation Method Six. The difference between this implementation method and Specific Implementation Method Six is ​​that the specific steps of step five are as follows:

[0100] The point clouds in each cluster after clustering are projected onto the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel of the point cloud projection, and the intersection-union ratio (IoU) is calculated with the position of the target object.

[0101] The projection matrix P is obtained through the following steps:

[0102] The extrinsic parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated based on the extrinsic parameters to obtain the projection matrix P.

[0103] Finally, the LiDAR data and camera data are fused using the following formula to obtain the final annotation result. Based on the extrinsic parameters recorded during data acquisition, we perform spatial alignment calibration between the LiDAR and camera to obtain the projection matrix P. For point clouds with 3D coordinates [x...],... i ,y i ,z i ] T Each point q i It can be projected onto the corresponding pixel of the image [u i ,v i ,1] T .

[0104] The point clouds in each cluster after clustering are projected onto the corresponding image through the projection matrix P. The minimum bounding box is taken according to the position of the edge pixels of the point cloud projection. The intersection-union ratio (IoU) is calculated with the 2D bounding box of the labeled object. Finally, the cluster with the largest IoU value is selected. The 3D bounding box is generated using the point cloud in the cluster and labeled.

[0105] s[u i ,v i ,1] T =P[x i ,y i ,z i ] T

[0106] In the above formula, P is the projection matrix and s is the scaling factor.

[0107] Specific Implementation Method Eight: This implementation method is a further explanation of Specific Implementation Method Seven. The difference between this implementation method and Specific Implementation Method Seven is that the image data in step one is obtained through an RGB camera.

[0108] Specific Implementation Method Nine: This implementation method is a further explanation of Specific Implementation Method Eight. The difference between this implementation method and Specific Implementation Method Eight is that the point cloud data in step one is obtained through the Leishen LiDAR C16-151B.

[0109] Specific Implementation Method Ten: This implementation method is a further explanation of Specific Implementation Method Nine. The difference between this implementation method and Specific Implementation Method Nine is that the position of the target object in step two is obtained through a 2D target detection model or a segmentation model.

[0110] This application mainly includes 2D labeled object segmentation, point cloud clustering segmentation, and image point cloud fusion. 2D labeled object segmentation mainly involves using 2D object detection algorithms such as YOLOv8 to identify common objects in daily life and using image segmentation algorithms such as Sam2 to label uncommon objects. Point cloud clustering segmentation mainly involves removing ground points and clustering non-ground points, then fusing multimodal data, and finally determining the position and pose of the target object and completing the labeling. The overall technical roadmap of this invention is shown below. Figure 1 As shown.

[0111] This application aims to address the challenges faced by embodied intelligent robots, such as the scarcity and difficulty in obtaining 3D perception datasets for their complex and diverse application scenarios, and the high cost and difficulty of labeling objects in three-dimensional space. This application utilizes image segmentation or 2D object detection methods to annotate the position and contour of the target object in the image by inputting the desired label or roughly selecting the target object. Then, based on the fitted surface and statistical features, it identifies which points belong to the ground and removes them from the point cloud to prevent interference with subsequent annotation. Subsequently, non-ground points are clustered, and the results are projected onto the corresponding image to obtain their minimum bounding boxes. The intersection-over-union (IoU) ratio of these bounding boxes is calculated with the 2D bounding boxes of the labeled objects. Finally, the cluster with the highest IoU value is selected, and its point cloud is labeled with a 3D bounding box. This method simplifies the 3D data annotation process, reduces annotation costs, and enables the annotation of objects in various scenarios. This application utilizes an image segmentation model to achieve accurate and efficient annotation of any object, even objects that have never been seen before, thereby enabling embodied intelligent robots to complete the annotation of 3D perception data in different scenarios.

[0112] By employing a multimodal fusion method, which combines the rich visual features provided by images with the spatial structural information contained in point clouds, the data annotation process is not only faster, but also reduces the errors caused by a single modality.

[0113] This application achieves IoU values ​​exceeding 50% across various object categories. For example, from a bird's-eye view, our method achieves IoUs of 83.28% and 84.47% for automobiles and trucks, respectively. BEV .

[0114] In three-dimensional space metrics, such as in the automotive category, this application achieved an IoU of 73.35%. 3D This value means that it can capture the outline and position of an object in three-dimensional space very well.

[0115] Compared with manual annotation results, the method in this application performs remarkably well, even surpassing, the accuracy of manual annotation in multiple categories. This is particularly evident in the IoU (Intersection over Union) in three-dimensional space. 3D Under the specified criteria, the method described in this application surpasses the level of manual annotation in the categories of automobiles, trucks, and pedestrians. From a bird's-eye view perspective, the method surpasses the level of manual annotation in the categories of trucks, cyclists, and pedestrians. For categories lagging behind manual annotation, the difference is only a few percentage points. For example, in the automobile category, the IoU of manual annotation is significantly lower. BEVThe accuracy rate of manual annotation was 86.33%, while that of the method in this application was 83.28%, a difference of only 3.05%. In the truck category, it even reached 84.47%, exceeding the 82.97% of manual annotation. These results demonstrate that the method in this application can approach the high accuracy of manual annotation, and even surpass it in some cases.

[0116] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.

Claims

1. A 3D data annotation method for embodied intelligent robots, characterized in that... Includes the following steps: Step 1: Acquire image data and point cloud data of the robot's working environment, respectively; Step 2: Obtain the position of the target object in the image data; Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud; Step 4: Cluster the non-ground point clouds and divide them into clusters; Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection. Then, calculate the intersection-union ratio (IoU) of the target object position obtained in Step 2 and the minimum bounding box. Step 6: Select the smallest bounding box with the largest intersection-union ratio as the 3D bounding box and label the 3D bounding box; The specific steps of step three are as follows: Step 31: Divide the point cloud data into multiple polar coordinate subgrids, each containing N... r,m ×N θ,m N grid cells, where N r,m N represents the number of loops in the polar coordinate subgrid. θ,m Indicates the number of sectors in the polar coordinate subgrid; Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point to obtain the ground point set. Represented as: Among them, z(p k ) represents the height threshold of the return point, p k S represents a point in a grid cell. n This represents the set of points within n grid cells of a polar coordinate subgrid. z represents the average height of the seed point. seed This indicates the height threshold for selecting seed points; Step 33: Obtain the ground point set Mean of the midpoint cloud and centralized matrix; Steps 3 and 4: Obtain the covariance matrix of the centered matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value. Then, use the eigenvector corresponding to the minimum singular value as the normal vector of the plane. Finally, the mean and the normal vector of the plane Perform a dot product to obtain the projected value. Projection value Represented as: Step 35: The normal vector of the plane and p k Perform a dot product to obtain the projected value. Projection value Represented as: Step 36: Utilizing projection values and projection value Obtain the ground point set Ground point set Represented as: Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set. Step 38: Based on ground point set The ground point set is obtained by utilizing verticality, height, and flatness. Ground point set Represented as: Wherein, φ(v 3,n () indicates verticality. Indicates altitude, N represents flatness. C N represents the number of grid cells contained in all polar coordinate subgrids. Z Indicates the number of polar coordinate subgrids. Indicates intermediate variables; The verticality is expressed as: Where z = [0,0,1] T θ τ Represents the angle threshold, v 3,n express The unit normal vector; The height is represented as: in, r represents the average height. n γ(r) represents the distance between the centroid of Sn and the origin, where Sn represents the set of points in n grid cells within the polar coordinate subgrid. n ) indicates that r n An exponentially growing adaptive midpoint function, L τ Indicates a parameter with a constant value range; The flatness is expressed as: in, and σ τ,m These represent the gain magnitude and the set smoothness threshold, respectively, σ n λ represents an intermediate variable. 1,n , λ 2,n , λ 3,n PCA features of grid cells.

2. The 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that... The specific steps of step four are as follows: Step 41: Set the thresholds MinPts for the neighborhood radius ∈ and the number of samples; Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. Points whose Euclidean distance is not greater than the neighborhood radius are considered to be in the point cloud of the candidate point's neighborhood. If the number of points in the neighborhood is greater than or equal to MinPts, then the candidate point is the core point; otherwise, reselect the candidate point. Step 43: Add other points within the neighborhood of the candidate point to the cluster corresponding to the core point; Step 44: Select all points in the cluster except the core point as candidate points, and repeat steps 42 and 43 until the cluster cannot be expanded further, obtaining the final cluster. Remove all points in this cluster from the non-terrestrial point cloud and proceed to step 45: Steps 4 and 5: Based on the non-ground point cloud after removing points, repeat steps 42 to 44 until no core points can be obtained, then end the process and obtain several clusters.

3. The 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that... The specific steps of step five are as follows: The point clouds in each cluster after clustering are projected onto the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel of the point cloud projection, and the intersection-union ratio (IoU) is calculated with the position of the target object. The projection matrix P is obtained through the following steps: The extrinsic parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated based on the extrinsic parameters to obtain the projection matrix P.

4. The 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that... In step one, the image data is obtained through an RGB camera.

5. A 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that... In step one, the point cloud data is obtained using the LeiShen C16-151B lidar.

6. A 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that... In step two, the position of the target object is obtained through a 2D target detection model or a segmentation model.

Images