3D data labeling method for intelligent robot with body

Abstract

The invention discloses a 3D data labeling method for an intelligent robot with a body, relates to the technical field of image processing, and aims to solve the problems that an existing manual labeling process is often tedious, time-consuming and low in labeling efficiency. According to the technical scheme, the robot can accurately and efficiently complete labeling when facing various labeled objects in different application scenes, the labeled objects can be tracked, repeated labeling is not needed, the labeling time is greatly shortened, and the labeling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a 3D data labeling method for an embodied intelligent robot. Background Art

[0002] In recent years, significant progress has been made in the field of 3D perception tasks, greatly enhancing robots' ability to understand and process 3D space. However, high-quality 3D perception models often require large and diverse datasets for training. Currently, most 3D perception datasets are concentrated in the field of autonomous driving, and the labeled objects are mostly common objects on the road or in life, such as vehicles and pedestrians. However, for embodied intelligent robots, due to the complexity and diversity of robot application scenarios, the targets that need to be perceived in different scenarios are different and have great differences. Therefore, for embodied intelligent robots, the corresponding 3D perception datasets are extremely scarce and difficult to obtain.

[0003] At present, manual labeling can ensure the accuracy of the labeling results, and can be flexibly adjusted according to specific task requirements to adapt to diverse labeling requirements and standards. However, manual data set labeling requires a lot of manpower input, and the labeling process is often cumbersome and time-consuming, with low labeling efficiency. Summary of the invention

[0004] The purpose of the present invention is to propose a 3D data labeling method for embodied intelligent robots in order to address the problems that the existing manual labeling process is often cumbersome and time-consuming, and the labeling efficiency is low.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A 3D data annotation method for an embodied intelligent robot comprises the following steps:

[0007] Step 1: Obtain image data and point cloud data of the robot's working environment respectively;

[0008] Step 2: Obtain the position of the target object in the image data;

[0009] Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud;

[0010] Step 4: Cluster the non-ground point cloud and divide it into clusters;

[0011] Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection, and calculate the intersection and union ratio between the position of the target object obtained in step 2 and the minimum bounding box;

[0012] Step 6: Take the smallest bounding box with the largest intersection-over-union ratio as the 3D bounding box and label the 3D bounding box.

[0013] Furthermore, the specific steps of step three are:

[0014] Step 31: Divide the point cloud data into multiple polar coordinate sub-grids, each polar coordinate sub-grid contains N r,m ×N θ,m grid cells, where N r,m Represents the number of rings in the polar coordinate subgrid, N θ,m Indicates the number of sectors in the polar coordinate subgrid;

[0015] Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point, and then obtain the ground point set It is expressed as:

[0016]

[0017] Among them, z(p k ) represents the height threshold of the returned point, p k represents the point in the grid cell, S n represents the total set of points in n grid cells within the polar coordinate subgrid, Indicates the average height of the seed points, z seed Indicates the height threshold for selecting seed points;

[0018] Step 33: Get the ground point set Mean of the point cloud and the centralization matrix;

[0019] Step 3 and 4: Get the covariance matrix of the centralized matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value, and then use the eigenvector corresponding to the minimum singular value as the normal vector of the plane Finally, the mean and the plane normal vector Perform dot multiplication to get the projection value Projection value It is expressed as:

[0020]

[0021] Step 35: Change the plane's normal vector and p k Perform dot multiplication to get the projection value Projection value It is expressed as:

[0022]

[0023] Step 36: Using projection values and projection value Get ground point set Ground point set It is expressed as:

[0024]

[0025] Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set.

[0026] Step 38: Based on ground point set And use verticality, height and flatness to get the ground point set Ground point set It is expressed as:

[0027]

[0028] Among them, φ(v 3,n ) represents verticality, Indicates height, Indicates flatness, N C Indicates the number of grid cells contained in all polar coordinate subgrids, N Z represents the number of polar coordinate subgrids, Represents an intermediate variable.

[0029] Further, the verticality is expressed as:

[0030]

[0031] Where z = [0, 0, 1] T ,θ τ Indicates the angle threshold, v 3,n express The unit normal vector of .

[0032] Further, the height is expressed as:

[0033]

[0034] in, represents the average height, r n represents the distance between the centroid and the origin of Sn, Sn represents the total set of points in n grid cells in the polar coordinate subgrid, γ(r n ) means to make r n Exponentially growing adaptive midpoint function, L τ Represents a constant range parameter.

[0035] Further, the flatness is expressed as:

[0036]

[0037] in, and σ τ,m Respectively represent the size of the gain and the set flatness threshold, σ n represents the intermediate variable, λ 1,n , 2,n , 3,n is the PCA feature of the grid unit.

[0038] Furthermore, the specific steps of step 4 are:

[0039] Step 41: Set the neighborhood radius ∈ and the threshold MinPts of the number of samples;

[0040] Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. The point whose Euclidean distance is not greater than the neighborhood radius ∈ is a point cloud in the neighborhood of the candidate point. If the number of points in the neighborhood ≥ MinPts, the candidate point is a core point, otherwise, reselect the candidate point;

[0041] Step 43: Add other points in the neighborhood of the candidate point to the cluster corresponding to the core point;

[0042] Step 44: Take the points in the cluster other than the core point as candidate points in turn, repeat steps 42 and 43 until the cluster cannot be expanded, get the final cluster, remove all points in the cluster from the non-ground point cloud, and execute step 45:

[0043] Step 45: Based on the non-ground point cloud after removing the points, repeat steps 42 to 44 until no core points can be obtained, then end and obtain several clusters.

[0044] Furthermore, the specific steps of step five are:

[0045] The point cloud in each cluster after point cloud clustering is projected to the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel points of the point cloud projection, and the intersection over union (IoU) is calculated with the position of the target object.

[0046] The projection matrix P is obtained by the following steps:

[0047] The external parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated according to the external parameters of the LiDAR and RGB camera to obtain the projection matrix P.

[0048] Furthermore, the image data in step 1 is obtained through an RGB camera.

[0049] Furthermore, the point cloud data in step 1 is obtained by the Leishen laser radar C16-151B.

[0050] Furthermore, the position of the target object in step 2 is obtained through a 2D target detection model or a segmentation model.

[0051] The beneficial effects of the present invention are:

[0052] This application combines multiple methods such as image segmentation and point cloud clustering, which enables the robot to accurately and efficiently complete labeling when facing various labeled objects in different application scenarios, and can track the labeled objects without repeated labeling, greatly reducing the labeling time and improving labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall flow chart of this application;

[0054] Figure 2 Schematic diagram of the point cloud clustering process. DETAILED DESCRIPTION

[0055] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0056] Specific implementation method 1: Figure 1 The 3D data labeling method for an embodied intelligent robot described in this embodiment includes the following steps:

[0057] Step 1: Obtain image data and point cloud data of the robot's working environment respectively;

[0058] Step 2: Obtain the position of the target object in the image data;

[0059] Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud;

[0060] Step 4: Cluster the non-ground point cloud and divide it into clusters;

[0061] Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection, and calculate the intersection and union ratio between the position of the target object obtained in step 2 and the minimum bounding box;

[0062] Step 6: Take the smallest bounding box with the largest intersection-over-union ratio as the 3D bounding box and label the 3D bounding box.

[0063] First, we use the RGB camera and the LeiShen LiDAR C16-151B to collect image data and point cloud data respectively, and record the external parameters of the two sensors.

[0064] 2D Annotated Object Segmentation

[0065] For the collected image data, we input the required labels, such as pedestrians, cars, etc., into the 2D target detection model, such as YOLOv8, to generate a 2D bounding box for the objects with the corresponding labels in the image. We can also mark key points in the image, and the model will draw a bounding box based on the key points to select the objects to be marked (it does not need to be completely aligned with the edge of the object). Then we convert the marked 2D bounding box into the input format of the model and pass it into the image segmentation model. The model will segment the marked object and output the mask of the target object. We then extract the pixel information of the mask and generate a 2D bounding box based on the edge pixels of the segmentation result. Mark the position and outline of the target object in the image.

[0066] Specific implementation method 2: This implementation method is a further description of the specific implementation method 1. The difference between this implementation method and the specific implementation method 1 is that the specific steps of step 3 are:

[0067] Step 31: Divide the point cloud data into multiple polar coordinate sub-grids, each polar coordinate sub-grid contains N r,n ×N θ,m grid cells, where N r,m Represents the number of rings in the polar coordinate subgrid, N θ,m Indicates the number of sectors in the polar coordinate subgrid;

[0068] Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point, and then obtain the ground point set It is expressed as:

[0069]

[0070] Among them, z(p k ) represents the height threshold of the returned point, p k represents the point in the grid cell, S n represents the total set of points in n grid cells within the polar coordinate subgrid, Indicates the average height of the seed points, z seed Indicates the height threshold for selecting seed points;

[0071] Step 33: Get the ground point set Mean of the point cloud and the centralization matrix;

[0072] Step 3 and 4: Get the covariance matrix of the centralized matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value, and then use the eigenvector corresponding to the minimum singular value as the normal vector of the plane Finally, the mean and the plane normal vector Perform dot multiplication to get the projection value Projection value It is expressed as:

[0073]

[0074] Step 35: Change the plane's normal vector and p k Perform dot multiplication to get the projection value Projection value It is expressed as:

[0075]

[0076] Step 36: Using projection values and projection value Get ground point set Ground point set It is expressed as:

[0077]

[0078] Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set.

[0079] Step 38: Based on ground point set And use verticality, height and flatness to get the ground point set Ground point set It is expressed as:

[0080]

[0081] Among them, φ(v 3,n ) indicates verticality (Uprightness), Indicates the height (Elevation), Indicates flatness, N C Indicates the number of grid cells contained in all polar coordinate subgrids, N Z represents the number of polar coordinate subgrids, Represents an intermediate variable.

[0082] The input point cloud P is divided into multiple sub-grids according to the predefined grid size, so as to process different areas and save calculation. m express.

[0083] Specific implementation method three: This implementation method is a further description of specific implementation method two. The difference between this implementation method and specific implementation method two is that the verticality is expressed as:

[0084]

[0085] Where z = [0, 0, 1] T ,θ τ Indicates the angle threshold, v 3,n express The unit normal vector of The method is the same).

[0086] Specific implementation method 4: This implementation method is a further description of specific implementation method 3. The difference between this implementation method and specific implementation method 3 is that the height is expressed as:

[0087]

[0088] in, represents the average height, r n represents the distance between the centroid and the origin of Sn, Sn represents the total set of points in n grid cells in the polar coordinate subgrid, γ(r n ) means to make r n Exponentially growing adaptive midpoint function, L τ Represents a constant range parameter.

[0089] Specific implementation mode 5: This implementation mode is a further description of specific implementation mode 4. The difference between this implementation mode and specific implementation mode 4 is that the flatness is expressed as:

[0090]

[0091] in, and σ τ,m Respectively represent the size of the gain and the set flatness threshold, σ n represents the intermediate variable, λ 1,n , 2,n , 3,n is the PCA feature of the grid unit.

[0092] Specific implementation method 6: This implementation method is a further description of the specific implementation method 5. The difference between this implementation method and the specific implementation method 5 is that the specific steps of step 4 are:

[0093] Step 41: Set the neighborhood radius ∈ and the threshold MinPts of the number of samples;

[0094] Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. The point whose Euclidean distance is not greater than the neighborhood radius ∈ is a point cloud in the neighborhood of the candidate point. If the number of points in the neighborhood ≥ MinPts, the candidate point is a core point, otherwise, reselect the candidate point;

[0095] Step 43: Add other points in the neighborhood of the candidate point to the cluster corresponding to the core point;

[0096] Step 44: Take the points in the cluster other than the core point as candidate points in turn, repeat steps 42 and 43 until the cluster cannot be expanded, get the final cluster, remove all points in the cluster from the non-ground point cloud, and execute step 45:

[0097] Step 45: Based on the non-ground point cloud after removing the points, repeat steps 42 to 44 until no core points can be obtained, then end and obtain several clusters.

[0098] The point cloud clustering process is as follows Figure 2 shown.

[0099] Specific implementation method 7: This implementation method is a further description of specific implementation method 6. The difference between this implementation method and specific implementation method 6 is that the specific steps of step 5 are:

[0100] The point cloud in each cluster after point cloud clustering is projected to the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel points of the point cloud projection, and the intersection over union (IoU) is calculated with the position of the target object.

[0101] The projection matrix P is obtained by the following steps:

[0102] The external parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated according to the external parameters of the LiDAR and RGB camera to obtain the projection matrix P.

[0103] Finally, the following formula is used to fuse the lidar data information and the camera data information to obtain the final annotation result. We align and calibrate the lidar and the camera in space based on the external parameter data recorded when collecting data to obtain the projection matrix P. For the point cloud with 3D coordinates [x i ,y i ,z i ] T Each point q i , which can be projected to the pixel point [u i ,v i ,1] T .

[0104] The point cloud in each cluster after point cloud clustering is projected into the corresponding image through the projection matrix P. The minimum bounding box is taken according to the position of the edge pixel points of the point cloud projection, and the intersection over union (IoU) is calculated with the 2D bounding box of the labeled object. Finally, the cluster with the largest IoU value is taken, and the point cloud in the cluster is used to generate a 3D bounding box, and the 3D bounding box is labeled.

[0105] s[u i ,v i ,1] T =P[x i ,y i ,z i ] T

[0106] In the above formula, P is the projection matrix and s is the scale factor.

[0107] Specific implementation eight: This implementation is a further description of specific implementation seven. The difference between this implementation and specific implementation seven is that the image data in step one is obtained by an RGB camera.

[0108] Specific implementation method nine: This implementation method is a further explanation of specific implementation method eight. The difference between this implementation method and specific implementation method eight is that the point cloud data in step one is obtained by the Leishen laser radar C16-151B.

[0109] Specific implementation method ten: This implementation method is a further explanation of specific implementation method nine. The difference between this implementation method and specific implementation method nine is that the position of the target object in step two is obtained through a 2D target detection model or a segmentation model.

[0110] This application mainly includes 2D annotated object segmentation, point cloud clustering segmentation and image point cloud fusion. 2D annotated object segmentation mainly includes using 2D target detection algorithms such as YOLOv8 to identify common objects in life and using image segmentation algorithms such as Sam2 to annotate uncommon objects. Point cloud clustering segmentation mainly includes ground point removal and non-ground point clustering, and then fusing multimodal data, and finally determining the position and posture of the target object and completing the annotation. The overall technical roadmap of the patent of this invention is as follows Figure 1 shown.

[0111] This application intends to solve the problems that for embodied intelligent robots, due to the complexity and diversity of their application scenarios, the 3D perception data sets for the corresponding scenes are extremely scarce and difficult to obtain, and the 3D data annotation objects are located in three-dimensional space, which makes annotation difficult and costly. This application uses image segmentation or 2D target detection methods to mark the position and outline of the target object in the image by inputting the required annotation labels or roughly selecting the target object. Then, based on the fitted surface and statistical features, it is confirmed which points belong to the ground and removed from the point cloud to prevent them from interfering with subsequent annotations. Subsequently, the non-ground points are clustered, and the results are projected into the corresponding image to take the minimum bounding box, and the intersection over union (IoU) is calculated with the 2D bounding box of the annotated object. Finally, we take the cluster with the largest IoU value and annotate its point cloud with a 3D bounding box. Through the above method, the 3D data annotation process is simplified, the annotation cost is reduced, and the annotation of objects in a variety of scenarios is achieved. This application uses an image segmentation model to achieve accurate and efficient labeling of any object, even unseen objects, thereby enabling embodied intelligent robots to complete the labeling of 3D perception data in different scenarios.

[0112] The multimodal fusion method combines the rich visual features provided by the image with the spatial structure information contained in the point cloud, which not only makes the data annotation process faster, but also reduces the errors caused by a single modality.

[0113] This application can achieve an IoU value of more than 50% on objects of different categories. For example, from a bird's eye view, our method achieves an IoU of 83.28% and 84.47% on cars and trucks, respectively. BEV .

[0114] In the three-dimensional space, for example, in the car category, this application achieved an IoU of 73.35%. 3D values, which means it is able to capture the outline and position of objects in 3D space very well.

[0115] Compared with the results of manual annotation, the performance of the proposed method in multiple categories is very close to or even exceeds the accuracy of manual annotation. In particular, the IoU in three-dimensional space 3D In the 100% and 100% scalability index, the proposed method surpasses the manual annotation level in the car, truck and pedestrian categories. In the bird's eye view, the proposed method surpasses the manual annotation level in the truck, cyclist and pedestrian categories. For the categories that lag behind the manual annotation level, the gap of the proposed method is only a few percentage points. For example, in the car category, the manually annotated IoU BEVThe accuracy of the proposed method is 86.33%, while that of the proposed method is 83.28%, with a difference of only 3.05%. In the truck category, it even reaches 84.47%, exceeding the 82.97% of manual annotation. These results show that the proposed method can approach the high accuracy of manual annotation, and even surpass manual annotation in some cases.

[0116] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. A 3D data annotation method for embodied intelligent robots, characterized in that The following steps are involved: Step 1: Obtain image data and point cloud data of the robot's working environment respectively; Step 2: Obtain the position of the target object in the image data; Step 3: Remove the ground point cloud from the point cloud data to obtain the non-ground point cloud; Step 4: Cluster the non-ground point cloud and divide it into clusters; Step 5: Project the point cloud in each cluster into the image data, and obtain the minimum bounding box based on the edge pixels of the point cloud projection, and calculate the intersection and union ratio between the position of the target object obtained in step 2 and the minimum bounding box; Step 6: Take the smallest bounding box with the largest intersection-over-union ratio as the 3D bounding box and label the 3D bounding box.

2. A 3D data annotation method for an embodied intelligent robot according to claim 1, characterized in that The specific steps of step three are: Step 31: Divide the point cloud data into multiple polar coordinate sub-grids, each polar coordinate sub-grid contains N r,m ×N θ,m grid cells, where N r,m Represents the number of rings in the polar coordinate subgrid, N θ,m Indicates the number of sectors in the polar coordinate subgrid; Step 32: Select the point cloud with the lowest height in each polar coordinate subgrid as the initial seed point, and then obtain the ground point set It is expressed as: Among them, z(p k ) represents the height threshold of the returned point, p k represents the point in the grid cell, S n represents the total set of points in n grid cells within the polar coordinate subgrid, Indicates the average height of the seed points, z seed Indicates the height threshold for selecting seed points; Step 33: Get the ground point set Mean of the point cloud and the centralization matrix; Step 3 and 4: Get the covariance matrix of the centralized matrix, and perform singular value decomposition on the covariance matrix to obtain the minimum singular value, and then use the eigenvector corresponding to the minimum singular value as the normal vector of the plane Finally, the mean and the plane normal vector Perform dot multiplication to get the projection value Projection value It is expressed as: Step 35: Change the plane's normal vector and p k Perform dot multiplication to get the projection value Projection value It is expressed as: Step 36: Using projection values and projection value Get ground point set Ground point set It is expressed as: Step 37: Order Repeat steps 33 to 36 until the set number of iterations is reached to obtain the ground point set. Step 38: Based on ground point set And use verticality, height and flatness to get the ground point set Ground point set It is expressed as: Among them, φ(v 3,n ) represents verticality, Indicates height, Indicates flatness, N C Indicates the number of grid cells contained in all polar coordinate subgrids, N Z represents the number of polar coordinate subgrids, Represents an intermediate variable.

3. A 3D data annotation method for an embodied intelligent robot according to claim 2, characterized in that The verticality is expressed as: Where z = [0, 0, 1] T ,θ τ Indicates the angle threshold, v 3,n express The unit normal vector of .

4. A 3D data annotation method for an embodied intelligent robot according to claim 3, characterized in that The height is expressed as: in, represents the average height, r n represents the distance between the centroid and the origin of Sn, Sn represents the total set of points in n grid cells in the polar coordinate subgrid, γ(r n ) means to make r n Exponentially growing adaptive midpoint function, L τ Represents a constant range parameter.

5. A 3D data annotation method for an embodied intelligent robot according to claim 4, characterized in that The flatness is expressed as: in, and σ τ,m Respectively represent the size of the gain and the set flatness threshold, σ n represents the intermediate variable, λ 1,n , 2,n , 3,n is the PCA feature of the grid unit.

6. A 3D data annotation method for an embodied intelligent robot according to claim 5, characterized in that The specific steps of step 4 are: Step 41: Set the neighborhood radius ∈ and the threshold MinPts of the number of samples; Step 42: Randomly select a point in the non-ground point cloud as a candidate point, and calculate the Euclidean distance between the candidate point and other points. The point whose Euclidean distance is not greater than the neighborhood radius ∈ is a point cloud in the neighborhood of the candidate point. If the number of points in the neighborhood ≥ MinPts, the candidate point is a core point, otherwise, reselect the candidate point; Step 43: Add other points in the neighborhood of the candidate point to the cluster corresponding to the core point; Step 44: Take the points in the cluster other than the core point as candidate points in turn, repeat steps 42 and 43 until the cluster cannot be expanded, get the final cluster, remove all points in the cluster from the non-ground point cloud, and execute step 45: Step 45: Based on the non-ground point cloud after removing the points, repeat steps 42 to 44 until no core points can be obtained, then end and obtain several clusters.

7. The 3D data annotation method for an embodied intelligent robot according to claim 1, characterized in that The specific steps of step five are: The point cloud in each cluster after point cloud clustering is projected to the corresponding image data through the projection matrix P. Then, the minimum bounding box is taken according to the position of the edge pixel points of the point cloud projection, and the intersection over union (IoU) is calculated with the position of the target object. The projection matrix P is obtained by the following steps: The external parameters of the LiDAR and RGB camera are obtained respectively, and the LiDAR and RGB camera are spatially aligned and calibrated according to the external parameters of the LiDAR and RGB camera to obtain the projection matrix P.

8. The 3D data annotation method for an embodied intelligent robot according to claim 1, characterized in that The image data in step 1 is obtained through an RGB camera.

9. The 3D data annotation method for an embodied intelligent robot according to claim 1, characterized in that The point cloud data in step 1 is obtained by the Leishen laser radar C16-151B.

10. The 3D data annotation method for embodied intelligent robots according to claim 1, characterized in that The position of the target object in step 2 is obtained through a 2D target detection model or a segmentation model.

Images