A synthetic point cloud automatic labeling method and system based on a mechanical CAD model

Abstract

The application belongs to the technical field of point cloud labeling, and discloses a synthetic point cloud automatic labeling method and system based on a mechanical CAD model. A three-dimensional point cloud graph of a workpiece to be processed under different perspectives is obtained to form a real domain; in a virtual scene, a point cloud graph of the workpiece to be processed under the same perspective as the real domain is obtained to form a simulation domain; coordinate information of the workpiece to be processed in each point cloud graph in the simulation domain is obtained; a generative adversarial network is trained using all the point cloud graphs in the real domain and the simulation domain; a point cloud graph A in the simulation domain is input into the trained generative adversarial network to obtain a synthetic point cloud graph A*, and the coordinate information of the workpiece to be processed in the point cloud graph A in the simulation domain is assigned to the synthetic point cloud graph A*, so that automatic labeling of the synthetic point cloud is realized. Through the application, the problems of time-consuming and laborious manual labeling, low efficiency and easy errors are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud labeling, and more particularly to a synthetic point cloud automatic labeling method and system based on a mechanical CAD model. BACKGROUND

[0002] With the development of computer vision technology, three-dimensional target detection technology gradually emerges, and various neural networks for point cloud semantic segmentation and target detection are constantly emerging. However, the point cloud data used to train these neural networks needs to be labeled, and the semantic segmentation labeling of point cloud data is time-consuming and laborious, with particularly low labeling efficiency.

[0003] In the mechanical field, before processing a part, a CAD three-dimensional model of the part is usually first established, which contains all three-dimensional information of the part such as size and shape, and then the part is processed on a numerical control machine according to the three-dimensional CAD model. Since these three-dimensional CAD models naturally have three-dimensional information of the part, it is inconvenient to disassemble some already assembled components to collect data such as taking pictures and scanning for each single component. Moreover, when manually labeling point cloud data, due to the unorderedness, sparseness and uneven distribution of point cloud data, it is inevitable to have missing labels and wrong labels during manual labeling, which will bring great errors and low accuracy to the learning of neural networks. Therefore, there is an urgent need for an automatic labeling method of point cloud based on CAD three-dimensional model to collect point cloud information of parts in the model domain. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the application provides a synthetic point cloud automatic labeling method and system based on a mechanical CAD model, which solves the problems of time-consuming and laborious manual labeling, low efficiency and easy errors.

[0005] To achieve the above-mentioned purpose, according to one aspect of the application, a synthetic point cloud automatic labeling method based on a mechanical CAD model is provided, which comprises the following steps:

[0006] acquiring three-dimensional point cloud images of a workpiece to be processed at different angles to form a real domain; in a virtual scene, acquiring point cloud images of the workpiece to be processed at the same angle as the real domain to form a simulation domain;

[0007] acquiring coordinate information of the workpiece to be processed in each point cloud image in the simulation domain;

[0008] training a generative adversarial network using all point cloud images in the real domain and the simulation domain, the input of the generative adversarial network being the point cloud images in the real domain and the simulation domain, and the output being a synthetic point cloud image, and through training, the output synthetic point cloud image gradually approaches the point cloud image in the real domain;

[0009] The point cloud graph A in the simulation domain is input into the trained generative adversarial network to obtain a synthetic point cloud graph A*, and the coordinate information of the workpiece to be processed in the point cloud graph A in the simulation domain is assigned to the synthetic point cloud graph A*, so as to realize automatic labeling of the synthetic point cloud.

[0010] Further preferably, the step of acquiring the point cloud graph of the workpiece to be processed in the virtual scene with the same perspective as the real domain is as follows:

[0011] A depth map of the workpiece to be processed in the virtual scene is photographed and acquired with the same perspective as the real domain.

[0012] The depth map of each perspective is converted into a point cloud graph.

[0013] Further preferably, the conversion of the depth map into the point cloud graph is performed according to the following steps:

[0014] The coordinates of each point in the point cloud graph in the camera coordinate system are calculated according to a preset formula using the depth value of each pixel point in the depth map.

[0015] The coordinates of each point in the point cloud graph in the camera coordinate system are converted into coordinates in the world coordinate system using the conversion relationship between the world coordinate system in the virtual scene and the camera coordinate system of the virtual camera, so as to obtain the coordinates of each point in the point cloud graph in the world coordinate system, i.e., to obtain the point cloud graph.

[0016] Further preferably, the preset formula is as follows:

[0017] x c =(u-u0)·z c / f

[0018] y c =(v-v0)·z c / f

[0019] z c =d / s

[0020] wherein x c , y c and z c are the coordinates of the pixel point in the X, Y and Z directions of the camera coordinate system, u and v are the horizontal and vertical coordinate values of the pixel point, u0 and v0 are the horizontal and vertical differences in pixel number between the center point pixel coordinate of the image and the origin pixel coordinate of the image coordinate system, d is the depth value of the pixel point, s is the scaling factor of the depth map, and f is the focal length of the virtual camera.

[0021] Further preferably, the acquisition of the coordinate information of the workpiece to be processed in each point cloud graph in the simulation domain is performed according to the following steps:

[0022] acquiring coordinate information of the workpiece to be processed in the virtual scene;

[0023] assigning the coordinate information of the workpiece to be processed acquired in the virtual scene to each point cloud image in the simulation domain, thereby obtaining the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain.

[0024] Further preferably, the conversion relationship between the world coordinate system in the virtual scene and the coordinate system in the point cloud image in the simulation image is obtained according to the following steps:

[0025] acquiring an extrinsic parameter matrix of a virtual camera for shooting a depth map of the workpiece to be processed in the virtual scene;

[0026] obtaining the conversion relationship between the point cloud image in the world coordinate system and the camera coordinate system according to the following formula:

[0027]

[0028] wherein Me is the extrinsic parameter matrix of the virtual camera, x w , y w and z w are respectively the coordinates of a point in the world coordinate system along the x, y and z directions.

[0029] Further preferably, the coordinate information of the workpiece to be processed includes the center coordinates of each component contained in the workpiece to be processed, the size of each component and the posture of each component.

[0030] Further preferably, the system comprises a data acquisition module, a training module and an automatic labeling module, wherein:

[0031] the data acquisition module is configured to acquire the point cloud images in the real domain and the simulation domain and acquire the coordinate information of the workpiece to be processed in the point cloud images in the simulation domain;

[0032] the training module is configured to train the generative adversarial network by using the point cloud images in the real domain and the simulation domain;

[0033] the automatic labeling module is configured to input the point cloud image A in the simulation domain into the trained generative adversarial network to obtain a synthetic point cloud image A*, and then assign the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain to the synthetic point cloud image A*, so as to realize labeling of the synthetic point cloud image.

[0034] According to another aspect of the present application, there is provided a synthetic point cloud automatic labeling system based on a mechanical CAD model, comprising a processor configured to execute a synthetic point cloud automatic labeling method based on a mechanical CAD model as described above.

[0035] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned automatic labeling method of synthetic point cloud based on mechanical CAD model.

[0036] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0037] 1. The present application proposes a method for automatically generating point cloud labeling data based on a three-dimensional CAD model. In a virtual scene, a virtual camera is used to collect point cloud images, and point cloud data synthesized by a three-dimensional CAD model placed in the scene is automatically labeled. Traditionally, the labeling of point cloud data needs to be done manually, which consumes time and human resources. Through the present application, the labeling process of point cloud data can be automated, improving efficiency and accuracy to simplify the labeling process of point cloud data and improve the efficiency of training neural networks for three-dimensional target detection.

[0038] 2. In the present application, when training the generative adversarial network, both point cloud images in the simulation domain and point cloud images in the real domain are used. The generative adversarial network is trained by using two kinds of point cloud images, so that the network structure after training can convert the simulation domain point cloud image into a synthetic point cloud image that is closer to the real distribution.

[0039] 3. In the present application, a virtual camera is used to collect the depth map of the model, eliminating the need for traditional on-site collection of point cloud data. This allows for rapid acquisition of a large amount of point cloud data in a virtual environment without the need for on-site actual operation, greatly speeding up data acquisition and providing more available data for subsequent research, development and analysis.

[0040] 4. Since the labeling of point cloud data is automated, human errors in the manual labeling process are reduced. Traditional manual labeling processes are easily affected by human subjective factors, which may result in inconsistent or incorrect labeling results. The present application provides more consistent and reliable labeling results through the automatic labeling process.

[0041] 5. The present application can be applied to various parts and models. Whether it is a small part or a large assembly, the present application can process and automatically label point cloud data. Through automatic labeling of point cloud data, the present application provides more possibilities for various applications. For example, automatic labeling of point cloud data can be used in target detection, object recognition, three-dimensional reconstruction and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of an automatic labeling method of synthetic point cloud based on mechanical CAD model according to the preferred embodiment of the present application;

[0043] Figure 2 This is a flowchart of automatically labeling workpieces in a virtual scene according to a preferred embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0045] like Figure 1 As shown, an automatic annotation method for synthetic point clouds based on mechanical CAD models is proposed, which includes the following steps:

[0046] S1 acquires 3D point cloud images of the workpiece to be processed from different perspectives to form the real domain;

[0047] The 3D CAD assembly model of the workpiece to be processed is placed in a virtual scene, and depth maps are captured from different angles and positions using a virtual camera. Based on the pre-placed position of the 3D CAD model, the coordinate information of the workpiece to be processed is obtained. This coordinate information includes the 3D coordinates, dimensions, and orientation information of the centers of each component in the workpiece. 3D annotation information, such as annotation frames, is then generated using this coordinate information.

[0048] The depth map is combined with camera intrinsic and extrinsic parameters to convert it into a point cloud map of a 3D CAD assembly model. Each depth map is converted into a point cloud map, and since the point cloud map is acquired from the depth image, the occluded parts of the model do not have point clouds.

[0049] The specific steps for converting a depth map into a point cloud map are as follows:

[0050] S11 obtains the intrinsic parameter matrix M of the virtual camera. i As shown in formula (1), f is the focal length of the virtual camera, dx is the width of one pixel in the x-direction in millimeters, and 1 / dx represents the number of pixels within 1 millimeter in the x-direction. f / dx represents the pixel length of the focal length in the x-axis direction, and f / dy represents the pixel length of the focal length in the y-axis direction. u0 and v0 represent the horizontal and vertical differences between the pixel coordinates of the image center point and the pixel coordinates of the image origin, respectively, which are 1 / 2 of the camera resolution.

[0051]

[0052] S12 Obtain the virtual camera's extrinsic parameter matrix M eAs shown in equation (2), where R is a 3x3 rotation matrix, and T is a 3x1 translation vector. If the world coordinates of the camera are [x0, y0, z0] T , the yaw angle in the pose information of the camera is θ, the pitch angle is , and the roll angle is ω.

[0053]

[0054] Rotating θ around the Z axis, the rotation matrix is shown in equation (3).

[0055]

[0056] Rotating around the Y axis, the rotation matrix is shown in equation (4).

[0057]

[0058] Rotating ω around the X axis, the rotation matrix is shown in equation (5).

[0059]

[0060] The rotation matrix R is obtained by sequentially multiplying Rz, Ry, and Rx, as shown in equation (6).

[0061] R = R z · R y · R x (6)

[0062] The translation vector T is related to the world coordinates of the camera, as shown in equation (7).

[0063]

[0064] S13 combines the depth value d of each pixel point with pixel coordinates (u, v) in the depth image, and the intrinsic matrix M i and the extrinsic matrix M e of the virtual camera to complete the conversion from the depth image to the point cloud image. Specifically, the conversion from the pixel coordinate system to the camera coordinate system can be achieved by equations (8), (9), and (10).

[0065] z c = d / s (8)

[0066] x c = (u-u0)·z c / f (9)

[0067] y c = (v-v0)·z c / f (10)

[0068] In the formula, u, v represent the horizontal and vertical coordinate values of the pixel point, u0 and v0 represent the horizontal and vertical difference in pixel number between the center pixel coordinate of the image and the origin pixel coordinate of the image, respectively. d is the depth value of the pixel point, and s is the scaling factor of the depth map. The unit of d is usually mm, and s is usually 1000. c c c x, y, and z are the x, y, and z coordinates of the pixel point mapped into the camera coordinate system.

[0069] Then the conversion from the camera coordinate system to the world coordinate system can be completed by formula (11). In the formula, x w w w x, y, and z represent the x, y, and z coordinates of the point mapped into the world coordinate system. At this point, the conversion from the depth map to the point cloud map is completed.

[0070]

[0071] In an embodiment of the present application, the virtual scene can be any software that renders and processes three-dimensional models, such as Blender, Unity, Unreal Engine, etc. The virtual camera can be a perspective camera or a panoramic camera built into the software. The shooting angle can be overhead or overhead, etc., and the position of the camera can be arbitrarily placed. It should be noted that the shooting angle and position of the depth camera should be as close as possible to the actual detection of mechanical parts, so that the size and integrity of the generated point cloud are as consistent as possible with the point cloud collected in reality. The focal length and sensor size of the virtual camera can be configured independently, and appropriate parameters can be selected for shooting and sampling.

[0072] In an embodiment of the present application, the virtual camera collects the same view angle in the simulation domain as the view angle in the reality domain.

[0073] For each point cloud generated from the depth map, the three-dimensional coordinates, size, and attitude information of each part in the camera coordinate system are calculated according to the part serial number that can be observed in the point cloud.

[0074] In an embodiment of the present application, each part has a unique serial number i, and the three-dimensional coordinates of the center point of the part with serial number i are denoted as: x icenter icenter icenter The length, width, and height of the part with serial number i are denoted as: l i i i The rotation angle of the part with serial number i is denoted as θ i .

[0075] ​​​​​​​​An array of observable serial numbers of parts is [i1, i2, …] (1≤i1, i2..≤N, the total number of parts is N), and the formed labeling box parameters are {[x i1center , y i1center , z i1center , l i1 , w i1 , h i1 , θ i1 ],[x i2center , y i2center , z i2center , l i2 , w i2 , h i2 , θ i2 ]…}。

[0076] S2 introduces a module using a generative adversarial network (GAN) to generate synthetic point cloud data from the model domain point cloud data, so as to output a synthetic data set closer to the real domain.

[0077] The generative adversarial network is trained by using all the point cloud graphs in the real domain and the simulation domain, the input of the generative adversarial network is the point cloud graph in the real domain and the simulation domain, and the output is a synthetic point cloud graph;

[0078] The generative adversarial network includes three parts: a generator G, a discriminator F and a feature encoder E / decoder D. The generator takes the encoded features of the point cloud and the randomly distributed Gaussian encoded features as inputs, and can generate multiple possible outputs to improve the quality of the generated object. The method uses an adversarial learning method to convert the model domain data into simulated real domain data to obtain labeled simulated real domain data.

[0079] Specifically, first, the encoder E is used to encode the input model domain point cloud X s or the real domain point cloud X t , to obtain the feature representation z s of the model domain point cloud and the feature representation z t of the real domain point cloud, respectively. A random vector z is sampled from a standard normal distribution N=(0, I) as a random Gaussian noise as a conditional input to control the diversity of the generated point cloud.

[0080] The model domain point cloud feature z s and the Gaussian noise z are input into the generator G to generate a new synthetic feature representation z' = G(z s , z), and the goal of the generator is to make the synthetic feature z' close to the real domain point cloud feature z t , so as to realize the migration of the distribution.

[0081] The synthetic feature z' is input into the decoder D to reconstruct the synthetic point cloud X' s= D(z'), the synthetic point cloud keeps the approximate geometry and labels of the model domain point cloud, while the distribution is closer to the real domain point cloud.

[0082] The adversarial training adjusts the distribution, the discriminator F takes the input feature, and tries to distinguish whether it is a point cloud feature z from the real domain t or a synthetic feature z'. The discriminator is forced to generate features that are closer to the real domain distribution by the adversarial loss L GAN The generator G aims to deceive the discriminator so that it cannot distinguish whether the generated feature z' is synthetic or from the real domain. Through adversarial training, the generator constantly adjusts the generation strategy, so that the distribution of the synthetic point cloud gradually approaches the real domain point cloud.

[0083] S3 randomly selects a point cloud graph A in the simulation domain and inputs it into the trained generative adversarial network to obtain a synthetic point cloud graph A*. The coordinate information of the workpiece to be processed in the point cloud graph A in the simulation domain is assigned to the synthetic point cloud graph A*, so as to realize automatic labeling of the synthetic point cloud.

[0084] As shown in Figure 2 , the CAD model is placed in a virtual scene, a depth map under a certain view angle is collected using a virtual camera, and the depth map is converted into a point cloud graph in the world coordinate system in combination with the intrinsic and extrinsic parameters of the camera. A three-dimensional labeling box is automatically generated in combination with the position, posture and other information of each part in the CAD model.

[0085] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for automatic annotation of synthetic point clouds based on mechanical CAD models, characterized in that, The method includes the following steps: A real-world domain is formed by acquiring 3D point cloud images of the workpiece to be processed from different perspectives; in a virtual scene, a point cloud image of the workpiece to be processed from the same perspective as the real-world domain is acquired to form a simulation domain. Obtain the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain; The coordinate information of the workpiece to be processed in each point cloud image in the simulation domain is obtained according to the following steps: Obtain the coordinate information of the workpiece to be processed in a virtual scene; The coordinate information of the workpiece to be processed obtained in the virtual scene is assigned to each point cloud map in the simulation domain, thereby obtaining the coordinate information of the workpiece to be processed in each point cloud map in the simulation domain. A generative adversarial network is trained using all point cloud maps in the real domain and the simulated domain. The input of the generative adversarial network is the point cloud maps in the real domain and the simulated domain, and the output is a synthetic point cloud map. Through training, the output synthetic point cloud map gradually approximates the point cloud map in the real domain. The point cloud map A in the simulated domain is input into the trained generative adversarial network to obtain the synthetic point cloud map A*. The coordinate information of the workpiece to be processed in the point cloud map A in the simulated domain is assigned to the synthetic point cloud map A* to achieve automatic annotation of the synthetic point cloud.

2. The automatic annotation method for synthetic point clouds based on mechanical CAD models as described in claim 1, characterized in that, The steps for obtaining a point cloud image of the workpiece to be processed in a virtual scene, with the same viewpoint as the real-world domain, are as follows: In a virtual scene, a depth map of the workpiece to be processed is captured and acquired from the same perspective as the real-world domain. The depth maps from each viewpoint are converted into point cloud maps.

3. The automatic annotation method for synthetic point clouds based on mechanical CAD models as described in claim 2, characterized in that, To convert a depth map into a point cloud map, follow these steps: Using the depth values ​​of each pixel in the depth map, calculate the coordinates of each point in the point cloud map in the camera coordinate system according to a preset formula; By utilizing the transformation relationship between the world coordinate system in the virtual scene and the camera coordinate system of the virtual camera, the coordinates of each point in the point cloud map in the camera coordinate system are converted into coordinates in the world coordinate system, thereby obtaining the coordinates of each point in the point cloud map in the world coordinate system, i.e., obtaining the point cloud map.

4. The automatic annotation method for synthetic point clouds based on mechanical CAD models as described in claim 3, characterized in that, The preset formula is performed as follows: in, , and These are the X, Y, and Z coordinates of the pixel in the camera coordinate system, respectively, while u and v are the horizontal and vertical coordinates of the pixel, respectively. and These are the horizontal and vertical pixel differences between the center pixel coordinates of the image and the origin pixel coordinates of the image coordinate system, respectively. d is the depth value of the pixel, s is the scaling factor of the depth map, and f is the focal length of the virtual camera.

5. The automatic annotation method for synthetic point clouds based on mechanical CAD models as described in claim 4, characterized in that, The transformation relationship between the world coordinate system in the virtual scene and the camera coordinate system of the virtual camera is obtained according to the following steps: The extrinsic parameter matrix of a virtual camera is used to capture a depth map of the workpiece to be processed in a virtual scene. The transformation relationship between the point cloud image and the camera coordinate system can be obtained using the following formula: Where Me is the extrinsic parameter matrix of the virtual camera. , and These are the coordinates of the point in the world coordinate system along the x, y, and z directions, respectively.

6. The automatic annotation method for synthetic point clouds based on mechanical CAD models as described in claim 1, characterized in that, The coordinate information of the workpiece to be processed includes: the center coordinates of each component contained in the workpiece, the size of each component, and the orientation of each component.

7. A system for automatic annotation using the synthetic point cloud annotation method based on mechanical CAD models as described in any one of claims 1-6, characterized in that, The system includes a data acquisition module, a training module, and an automatic annotation module, among which: The data acquisition module is used to acquire point cloud maps in the real domain and the simulated domain, and to acquire the coordinate information of the workpiece to be processed in the point cloud map in the simulated domain; The training module is used to train and generate adversarial networks using point cloud maps in the real and simulated domains. The automatic annotation module is used to input the point cloud map A in the simulated domain into the trained generative adversarial network to obtain a synthetic point cloud map A*, and then assign the coordinate information of the workpiece to be processed in the point cloud map A in the simulated domain to the synthetic point cloud map A*, thereby realizing the annotation of the synthetic point cloud map.

8. An automatic point cloud annotation system based on mechanical CAD models, characterized in that, Includes a processor, the processor being configured to execute a method for automatically annotating synthetic point clouds based on a mechanical CAD model as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automatic annotation method for synthetic point clouds based on mechanical CAD models as described in any one of claims 1-6.

Images