Synthetic point cloud automatic labeling method and system based on mechanical CAD model

Abstract

The invention belongs to the related technical field of point cloud labeling, and discloses a mechanical CAD model-based synthetic point cloud automatic labeling method and system. Obtaining three-dimensional point cloud pictures of the to-be-processed workpiece under different visual angles to form a real domain; in a virtual scene, obtaining a point cloud picture of a to-be-processed workpiece with the same visual angle as the real domain so as to form a simulation domain; acquiring coordinate information of a to-be-processed workpiece in each point cloud picture in the simulation domain; training a generative adversarial network by using all the point cloud pictures in the real domain and the simulation domain; and inputting the point cloud image A in the simulation domain into the trained generative adversarial network to obtain a synthetic point cloud image A *, and endowing the synthetic point cloud image A * with coordinate information of a to-be-processed workpiece in the point cloud image A in the simulation domain to realize automatic labeling of the synthetic point cloud. According to the invention, the problems of time and labor waste, low efficiency and error proneness of manual labeling are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to point cloud annotation, and more specifically, relates to a method and system for automatic annotation of synthetic point clouds based on mechanical CAD models. Background Art

[0002] With the development of computer vision technology, 3D object detection technology has gradually emerged, and various neural networks for semantic segmentation and object detection of point clouds have emerged. However, the point cloud data for training these neural networks needs to be labeled, and the labeling of semantic segmentation of point cloud data is time-consuming and labor-intensive, and the labeling efficiency is particularly low.

[0003] In the field of machinery, before processing parts, CAD three-dimensional modeling is often performed on the parts first. The three-dimensional CAD model contains all the three-dimensional information of the parts, such as the size and shape, and then the parts are processed on the CNC machine tool according to the three-dimensional CAD model. Since these three-dimensional CAD models naturally have the three-dimensional information of the parts, it is inconvenient to disassemble some assembled components and take pictures, scan and other data collection operations for each single component. In addition, when manually annotating point cloud data, due to the disorder, sparsity and uneven distribution of the point cloud data itself, it is inevitable that missing labels and wrong labels will occur during manual annotation, which will bring large errors and low precision to the learning of neural networks. Therefore, there is an urgent need for a method to collect point cloud information of parts in the model domain based on the CAD three-dimensional model to realize automatic annotation of point clouds. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method and system for automatic annotation of synthetic point clouds based on mechanical CAD models, which solves the problems that manual annotation is time-consuming, labor-intensive, inefficient and error-prone.

[0005] To achieve the above object, according to one aspect of the present invention, a method for automatic annotation of synthetic point clouds based on a mechanical CAD model is provided, the method comprising the following steps:

[0006] Acquire a three-dimensional point cloud image of the workpiece to be processed at different viewing angles to form a real domain; in a virtual scene, acquire a point cloud image of the workpiece to be processed at the same viewing angle as the real domain to form a simulation domain;

[0007] Acquire coordinate information of a workpiece to be processed in each point cloud image in the simulation domain;

[0008] Using all the point cloud images in the real domain and the simulated domain to train a generative adversarial network, the input of the generative adversarial network is the point cloud images in the real domain and the simulated domain, and the output is a synthetic point cloud image, and through training, the output synthetic point cloud image gradually approaches the point cloud image in the real domain;

[0009] The point cloud image A in the simulation domain is input into the trained generative adversarial network to obtain a synthetic point cloud image A*, and the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain is assigned to the synthetic point cloud image A* to achieve automatic labeling of the synthetic point cloud.

[0010] Further preferably, the step of obtaining a point cloud image of the workpiece to be processed in the virtual scene at the same viewing angle as the real domain is as follows:

[0011] Shooting and acquiring a depth map of the workpiece to be processed at the same viewing angle as that of the real domain in the virtual scene;

[0012] The depth map of each viewing angle is converted into a point cloud map.

[0013] Further preferably, the converting of the depth map into a point cloud map is performed according to the following steps:

[0014] The depth value of each pixel in the depth map is used to calculate the coordinates of each point in the point cloud map in the camera coordinate system according to a preset formula;

[0015] By utilizing the conversion relationship between the world coordinate system in the virtual scene and the camera coordinate system of the virtual camera, the coordinates of each point in the point cloud image in the camera coordinate system are converted into coordinates in the world coordinate system, thereby obtaining the coordinates of each point in the point cloud image in the world coordinate system, that is, obtaining a point cloud image.

[0016] Further preferably, the preset formula is performed as follows:

[0017] x c =(u-u0)·z c / f

[0018] y c =(v-v0)·z c / f

[0019] z c =d / s

[0020] Among them, x c ,y c and z c are the X, Y and Z coordinates of the pixel in the camera coordinate system, u and v are the horizontal and vertical coordinate values ​​of the pixel, u0 and v0 are the horizontal and vertical difference in pixels between the pixel coordinates of the center point of the image and the pixel coordinates of the origin of the image coordinate system, d is the depth value of the pixel, s is the scaling factor of the depth map, and f is the focal length of the virtual camera.

[0021] Further preferably, the obtaining of the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain is performed according to the following steps:

[0022] Obtaining coordinate information of the workpiece to be processed in the virtual scene;

[0023] The coordinate information of the workpiece to be processed obtained in the virtual scene is assigned to each point cloud image in the simulation domain, thereby obtaining the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain.

[0024] Further preferably, the conversion relationship between the world coordinate system in the virtual scene and the coordinate system in the point cloud image in the simulation image is obtained according to the following steps:

[0025] Obtaining an external parameter matrix of a virtual camera for photographing a depth map of a workpiece to be processed in a virtual scene;

[0026] The conversion relationship between the point cloud image in the world coordinate system and the camera coordinate system is obtained according to the following formula:

[0027]

[0028] Among them, Me is the external parameter matrix of the virtual camera, x w ,y w and z w are the coordinates of the point in the world coordinate system along the x, y, and z directions respectively.

[0029] Further preferably, the coordinate information of the workpiece to be processed includes: the center coordinates of each component contained in the workpiece to be processed, the size of each component and the posture of each component.

[0030] Further preferably, the system includes a data acquisition module, a training module, and an automatic labeling module, wherein:

[0031] The data acquisition module is used to acquire point cloud images in the real domain and the simulation domain, and to acquire coordinate information of the workpiece to be processed in the point cloud image in the simulation domain;

[0032] The training module is used to train a generative adversarial network using point cloud images in the real domain and the simulated domain;

[0033] The automatic labeling module is used to input the point cloud image A in the simulation domain into the trained generative adversarial network to obtain a synthetic point cloud image A*, and then assign the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain to the synthetic point cloud image A*, so as to realize the labeling of the synthetic point cloud image.

[0034] According to another aspect of the present invention, a synthetic point cloud automatic annotation system based on a mechanical CAD model is provided, comprising a processor, wherein the processor is used to execute the synthetic point cloud automatic annotation method based on a mechanical CAD model as described above.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for automatically labeling a synthetic point cloud based on a mechanical CAD model described above is implemented.

[0036] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0037] 1. The present invention proposes a method for automatically generating point cloud annotation data based on a 3D CAD model. A virtual camera is used to collect a point cloud image in a virtual scene, and the point cloud data synthesized by the 3D CAD model placed in the scene is automatically annotated. Traditionally, the annotation of point cloud data needs to be done manually, which consumes time and human resources. Through the present invention, the annotation process of point cloud data can be automated, which improves efficiency and accuracy, thereby simplifying the annotation process of point cloud data and improving the efficiency of training neural networks for 3D target detection.

[0038] 2. When training the generative adversarial network, the present invention utilizes both the point cloud map in the simulation domain and the point cloud map in the real domain. The generative adversarial network is trained by the two point cloud maps, so that the trained network structure converts the point cloud map in the simulation domain into a synthetic point cloud map that is closer to the real distribution.

[0039] 3. The present invention adopts a virtual camera to collect the depth map of the model, eliminating the need for traditional on-site point cloud data collection, so that a large amount of point cloud data can be quickly acquired in a virtual environment without the need for actual on-site operations; this greatly speeds up the data collection process and provides more available data for subsequent research, development and analysis.

[0040] 4. Since the annotation of point cloud data is automated, human errors in the manual annotation process are reduced. The traditional manual annotation process is easily affected by human subjective factors, which may lead to inconsistent or erroneous annotation results. The present invention provides more consistent and reliable annotation results through an automated annotation process.

[0041] 5. The present invention can be applied to various parts and models. Whether it is a small part or a large assembly, the present invention can process and automatically annotate point cloud data. By automatically annotating point cloud data, the present invention provides more possibilities for various applications. For example, the automatic annotation of point cloud data can be used in target detection, object recognition, 3D reconstruction and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a method for automatic annotation of synthetic point clouds based on mechanical CAD models constructed according to a preferred embodiment of the present invention;

[0043] Figure 2 It is a flow chart for automatically marking a workpiece to be processed in a virtual scene constructed according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] like Figure 1 As shown, a method for automatic annotation of synthetic point clouds based on a mechanical CAD model comprises the following steps:

[0046] S1 obtains the three-dimensional point cloud images of the workpiece to be processed at different viewing angles to form a real domain;

[0047] The 3D CAD assembly model of the workpiece to be processed is placed in the virtual scene, and the depth map taken at different angles and positions is collected by the virtual camera. According to the pre-placed position of the 3D CAD model, the coordinate information of the workpiece to be processed is obtained, which includes: the 3D coordinates, size and posture information of the center of each component in the workpiece to be processed, and the 3D annotation box and other annotation information are generated with the coordinate information.

[0048] The depth map is combined with the camera's internal and external parameters to convert into a point cloud of the 3D CAD assembly model. Each depth map is converted into a point cloud. Since the point cloud is collected from the depth image, there is no point cloud in the occluded part of the model.

[0049] The specific steps for converting a depth map into a point cloud map are as follows:

[0050] S11 obtains the intrinsic parameter matrix M of the virtual camera i As shown in formula (1), f is the focal length of the virtual camera, dx is the width of a pixel in the x direction, in millimeters, and 1 / dx represents the number of pixels within 1 millimeter in the x direction. f / dx represents the pixel length of the focal length in the x-axis direction, and f / dy represents the pixel length of the focal length in the y-axis direction. u0 and v0 represent the number of pixels that differ in the horizontal and vertical directions between the pixel coordinates of the center point of the image and the pixel coordinates of the origin of the image, respectively, which are 1 / 2 of the camera resolution.

[0051]

[0052] S12 obtains the external parameter matrix M of the virtual camera eAs shown in formula (2), R is a 3×3 rotation matrix and T is a 3×1 translation vector. If the world coordinates of the camera are [x0, y0, z0] T , in the camera’s attitude information, the yaw angle is θ and the pitch angle is , the roll angle is ω.

[0053]

[0054] Rotate θ around the Z axis, the rotation matrix is ​​shown in formula (3).

[0055]

[0056] Rotate around the Y axis The rotation matrix is ​​shown in formula (4).

[0057]

[0058] Rotate ω around the X-axis, the rotation matrix is ​​shown in formula (5).

[0059]

[0060] The rotation matrix R is obtained by multiplying Rz, Ry, and Rx in sequence, as shown in formula (6).

[0061] R=R z ·R y ·R x (6)

[0062] The translation vector T is related to the world coordinates of the camera as shown in formula (7).

[0063]

[0064] S13 combines the depth value d of each pixel with the pixel coordinates (u, v) in the depth image and the intrinsic parameter matrix M of the virtual camera i , the external parameter matrix M e The transformation from the depth map to the point cloud map is completed. Specifically, the transformation from the pixel coordinate system to the camera coordinate system can be realized by formulas (8), (9), and (10).

[0065] z c =d / s (8)

[0066] x c =(u-u0)·z c / f (9)

[0067] y c =(v-v0)·z c / f (10)

[0068] Where u and v represent the horizontal and vertical coordinates of the pixel point, u0 and v0 represent the horizontal and vertical difference in pixels between the pixel coordinates of the center point of the image and the pixel coordinates of the origin of the image, respectively. d is the depth value of the pixel point, s is the scaling factor of the depth map, the unit of d is usually mm, and s is usually 1000. c ,y c , z c Map the pixel to the x, y, and z coordinates in the camera coordinate system.

[0069] Then, the conversion from the camera coordinate system to the world coordinate system can be completed by formula (11). w ,y w , z w Represents the x, y, and z coordinates of the pixel mapped to the point in the world coordinate system. At this point, the conversion from the depth map to the point cloud map is completed.

[0070]

[0071] In one embodiment of the present invention, the virtual scene can be any software capable of rendering and processing three-dimensional models, such as Blender, Unity, Unreal Engine, etc. The virtual camera can be a perspective camera or a panoramic camera built into the software. The shooting angle can be looking up or looking down, etc., and the position of the camera can be placed arbitrarily. It should be noted that the placement angle and position of the depth camera during the actual inspection of mechanical parts should be as close as possible, so that the size and integrity of the generated point cloud are as consistent as possible with the point cloud collected in reality. The focal length and sensor size of the virtual camera can be configured independently, and appropriate parameters can be selected for shooting and sampling.

[0072] In one embodiment of the present invention, the perspective of the virtual camera capturing the simulation domain is the same as the perspective of the virtual camera constructing the real domain.

[0073] For each point cloud image generated by the depth map, the three-dimensional coordinates, size, and posture information of each component in the camera coordinate system are calculated based on the component serial numbers that can be observed in the point cloud image.

[0074] In one embodiment of the present invention, each part has a unique serial number i, and the three-dimensional coordinates of the center point of the part with serial number i are: x icenter ,y icenter , z icenter The length, width and height of the part with serial number i are respectively: l i , w i ,h i The rotation angle of the part numbered i is denoted by θ i .

[0075] The array of the serial numbers of the observed parts is [i1, i2, ...] (1≤i1, i2..≤N, the total number of parts is N), then the parameters of the annotation box formed are {[x i1center ,y i1center , z i1center , l i1 , w i1 ,h i1 ,θ i1 ],[x i2center ,y i2center , z i2center , l i2 , w i2 ,h i2 ,θ i2 ]…}.

[0076] S2 introduces a module that uses a generative adversarial network (GAN) to generate synthetic point cloud data from model domain point cloud data, thereby outputting a synthetic dataset that is closer to the real domain.

[0077] Use all the point cloud images in the real domain and the simulated domain to train the Generative Adversarial Network. The input of the Generative Adversarial Network is the point cloud images in the real domain and the simulated domain, and the output is the synthetic point cloud image.

[0078] The generative adversarial network consists of three parts: generator G, discriminator F and feature encoder E / decoder D. The generator takes the encoded features of the point cloud and the randomly distributed Gaussian encoded features as input, and can generate multiple possible outputs to improve the quality of the generator output object. This method uses the adversarial learning method to convert the model domain data into simulated real domain data to obtain labeled simulated real domain data.

[0079] Specifically, we first use the encoder E to transform the input model domain point cloud X s Or real-world point cloud X t Encode and obtain the feature representation z of the model domain point cloud respectively. s and the feature representation of the real domain point cloud z t , a random vector z is sampled from the standard normal distribution N = (0, I) as random Gaussian noise as a conditional input to control the diversity of the generated point cloud.

[0080] The model domain point cloud feature z s and Gaussian noise z are input into the generator G to generate a new synthetic feature representation z'=G(z s , z), the goal of the generator is to make the synthesized feature z' close to the real domain point cloud feature z t , thereby achieving distributed migration.

[0081] Input the synthetic feature z' into the decoder D to reconstruct the synthetic point cloud X' s= D(z'), the synthesized point cloud maintains the approximate geometric shape and labels of the model domain point cloud, while the distribution is closer to the real domain point cloud.

[0082] Introduce adversarial training to adjust the distribution. The discriminator F accepts the input features and tries to distinguish them from the point cloud features z from the real domain. t Still from the synthetic feature z', the discriminator uses the adversarial loss L GAN Force the generator G to generate features that are more in line with the real domain distribution. The goal of the generator is to deceive the discriminator so that it cannot distinguish whether the generated feature z' is synthetic or from the real domain. Through adversarial training, the generator continuously adjusts the generation strategy so that the distribution of the synthetic point cloud gradually approaches the real domain point cloud.

[0083] S3 arbitrarily selects a point cloud image A in the simulation domain and inputs it into the trained generative adversarial network to obtain a synthetic point cloud image A*, and assigns the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain to the synthetic point cloud image A*, thereby realizing automatic labeling of the synthetic point cloud.

[0084] like Figure 2 As shown in the figure, the CAD model is placed in a virtual scene, a virtual camera is used to collect a depth map at a certain perspective, and the depth map is converted into a point cloud map in the world coordinate system by combining the camera's internal and external parameters. The 3D annotation box is automatically generated by combining the position, posture and other information of each component in the CAD model.

[0085] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic annotation of synthetic point clouds based on mechanical CAD models, characterized in that: The method comprises the following steps: Acquire a three-dimensional point cloud image of the workpiece to be processed at different viewing angles to form a real domain; in a virtual scene, acquire a point cloud image of the workpiece to be processed at the same viewing angle as the real domain to form a simulation domain; Acquire coordinate information of a workpiece to be processed in each point cloud image in the simulation domain; Using all the point cloud images in the real domain and the simulated domain to train a generative adversarial network, the input of the generative adversarial network is the point cloud images in the real domain and the simulated domain, and the output is a synthetic point cloud image, and through training, the output synthetic point cloud image gradually approaches the point cloud image in the real domain; The point cloud image A in the simulation domain is input into the trained generative adversarial network to obtain a synthetic point cloud image A*, and the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain is assigned to the synthetic point cloud image A* to achieve automatic labeling of the synthetic point cloud.

2. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 1, characterized in that: The steps of obtaining a point cloud image of the workpiece to be processed in the virtual scene at the same viewing angle as the real domain are as follows: Shooting and acquiring a depth map of the workpiece to be processed at the same viewing angle as that of the real domain in the virtual scene; The depth map of each viewing angle is converted into a point cloud map.

3. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 2, characterized in that: The conversion of the depth map into a point cloud map is performed according to the following steps: The depth value of each pixel in the depth map is used to calculate the coordinates of each point in the point cloud map in the camera coordinate system according to a preset formula; By utilizing the conversion relationship between the world coordinate system in the virtual scene and the camera coordinate system of the virtual camera, the coordinates of each point in the point cloud image in the camera coordinate system are converted into coordinates in the world coordinate system, thereby obtaining the coordinates of each point in the point cloud image in the world coordinate system, that is, obtaining a point cloud image.

4. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 3, characterized in that: The preset formula is as follows: x c =(u-u0)·z c / f y c =(v-v0)·z c / f z c =d / s Among them, x c ,y c and z c are the X, Y and Z coordinates of the pixel in the camera coordinate system, u and v are the horizontal and vertical coordinate values ​​of the pixel, u0 and v0 are the horizontal and vertical difference in pixels between the pixel coordinates of the center point of the image and the pixel coordinates of the origin of the image coordinate system, d is the depth value of the pixel, s is the scaling factor of the depth map, and f is the focal length of the virtual camera.

5. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 4, characterized in that: The acquisition of the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain is performed according to the following steps: Obtaining coordinate information of the workpiece to be processed in the virtual scene; The coordinate information of the workpiece to be processed obtained in the virtual scene is assigned to each point cloud image in the simulation domain, thereby obtaining the coordinate information of the workpiece to be processed in each point cloud image in the simulation domain.

6. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 5, characterized in that: The conversion relationship between the world coordinate system in the virtual scene and the coordinate system in the point cloud image in the simulation image is obtained according to the following steps: Obtaining an external parameter matrix of a virtual camera for capturing a depth map of a workpiece to be processed in a virtual scene; The conversion relationship between the point cloud image in the world coordinate system and the camera coordinate system is obtained according to the following formula: Among them, Me is the external parameter matrix of the virtual camera, x w ,y w and z w are the coordinates of the point in the world coordinate system along the x, y, and z directions respectively.

7. The method for automatic annotation of synthetic point clouds based on mechanical CAD models according to claim 1, characterized in that: The coordinate information of the workpiece to be processed includes: the center coordinates of each component contained in the workpiece to be processed, the size of each component and the posture of each component.

8. A system for labeling using the method for automatically labeling a synthetic point cloud based on a mechanical CAD model according to any one of claims 1 to 7, characterized in that: The system includes a data acquisition module, a training module, and an automatic labeling module, wherein: The data acquisition module is used to acquire point cloud images in the real domain and the simulation domain, and to acquire coordinate information of the workpiece to be processed in the point cloud image in the simulation domain; The training module is used to train a generative adversarial network using point cloud images in the real domain and the simulated domain; The automatic labeling module is used to input the point cloud image A in the simulation domain into the trained generative adversarial network to obtain a synthetic point cloud image A*, and then assign the coordinate information of the workpiece to be processed in the point cloud image A in the simulation domain to the synthetic point cloud image A*, so as to realize the labeling of the synthetic point cloud image.

9. A synthetic point cloud automatic annotation system based on a mechanical CAD model, characterized in that: It includes a processor, which is used to execute the automatic annotation method of synthetic point cloud based on mechanical CAD model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically labeling a synthetic point cloud based on a mechanical CAD model as described in any one of claims 1 to 7 is implemented.

Images