Mechanical assembly three-dimensional target detection method and system based on CAD model domain point cloud data

Abstract

The invention belongs to the technical field of three-dimensional target detection, and particularly discloses a mechanical assembly three-dimensional target detection method and system based on CAD model domain point cloud data, and the method comprises the steps: obtaining model domain point cloud and corresponding labels based on a CAD three-dimensional model of a mechanical assembly, and enabling the labels to reflect the types and positions of parts; training a detector and a domain discriminator through the point cloud data of the model domain and the real domain, and taking a maximum domain discriminator error and a minimum detector error as loss functions; the detector comprises a feature extractor and a label predictor, the feature extractor extracts point cloud features, the domain discriminator discriminates whether the features are from the model domain or the reality domain, and the label predictor outputs corresponding labels according to the model features. And inputting the real domain point cloud into the trained detector, and outputting a corresponding label. According to the invention, the parts of the assembled assembly can be accurately identified and positioned.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional target detection, and more specifically, relates to a method and system for three-dimensional target detection of a mechanical assembly based on CAD model domain point cloud data. Background Art

[0002] In actual engineering, the detection of assembly components mainly uses 2D image cameras for data collection and object positioning and extraction. The quality of 2D image collection is easily affected by various factors such as lighting conditions, exposure, and light reflection on the object surface. In addition, for complex assembly scenes, multiple parts block each other and are projected on the same plane in the 2D image space, making it difficult to distinguish between devices. It is also inconvenient to disassemble these assembled components and take photos, scan, and other data collection operations for each single component.

[0003] In recent years, with the development of computer vision technology, 3D object detection technology has gradually emerged, and various neural networks for semantic segmentation and object detection of point clouds have emerged. Point cloud-based vision tasks have made great progress, which is largely due to the fact that the collected point cloud datasets are often finely labeled and have large data volumes. However, in real practice, obtaining such point cloud data and manually labeling them is highly dependent on professionals in this field, which makes data acquisition and labeling more difficult and time-consuming, and the accuracy is not high. In addition, when performing point cloud object detection in different domains in the real world, the differences between domains may lead to performance degradation. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a three-dimensional target detection method and system for mechanical assemblies based on CAD model domain point cloud data, the purpose of which is to accurately identify and locate parts of the assembled assembly.

[0005] To achieve the above object, according to one aspect of the present invention, a method for detecting three-dimensional objects of mechanical assemblies based on point cloud data in a CAD model domain is proposed, comprising the following steps:

[0006] Detector training phase:

[0007] Based on the CAD 3D model of the mechanical assembly, the model domain point cloud and corresponding labels are obtained, and the labels reflect the category and location information of the parts in the mechanical assembly; the real domain point cloud data of the mechanical assembly is collected;

[0008] The detector and the domain discriminator are trained by using the model domain point cloud data and the real domain point cloud data to obtain a trained detector;

[0009] The detector includes a feature extractor and a label predictor. During training, the feature extractor extracts features of the model domain point cloud and the real domain point cloud respectively to obtain model features and actual features. The model features and actual features are input into a domain discriminator, and the domain discriminator discriminates whether the features come from the model domain or the real domain. The label predictor outputs corresponding labels according to the model features. The loss function during training is constructed based on maximizing the domain discrimination result error and minimizing the label prediction result error.

[0010] Detector application stage:

[0011] The real-domain point cloud of the mechanical assembly is input into the trained detector, the feature extractor extracts the actual features of the real-domain point cloud, and the label predictor outputs the corresponding labels based on the actual features, thereby realizing the recognition and positioning of parts in the mechanical assembly.

[0012] As a further preferred example, the loss function E(θ f ,θ y ,θ d ) specifically:

[0013]

[0014] in, is the loss function for label prediction, is the loss function for domain discrimination, and the hyperparameter λ is used to control and The specific gravity of f is the network parameter of the feature extractor, θ y is the network parameter of the label predictor, θ d is the network parameter of the domain discriminator; i represents the sample number, and N represents the total number of samples.

[0015] As a further preferred embodiment, and The cross entropy loss function is used.

[0016] As further preferred, the step of obtaining a model domain point cloud based on a CAD three-dimensional model of a mechanical assembly includes:

[0017] A virtual depth map is obtained based on the CAD 3D model of the mechanical assembly, and then the virtual depth map is converted into a model domain point cloud.

[0018] As a further preferred embodiment, after obtaining a virtual depth map based on the CAD three-dimensional model of the mechanical assembly, a depth error is introduced to enhance the virtual depth map, and Gaussian blur processing is performed on the virtual depth map, and the processed virtual depth map is then converted into a model domain point cloud.

[0019] As a further preferred embodiment, the introducing of the depth error to enhance the virtual depth map is specifically as follows:

[0020] z a =(1+Shift R )×z s

[0021] Among them, z s is the depth value of the pixel in the original virtual depth map, z a To enhance the depth value of the pixel in the virtual depth map, Shift R is the amount of deviation introduced; for different virtual depth maps, Shift R Take a random value from -0.1 to 0.1, and for each pixel in a virtual depth map, Shift R The value remains unchanged.

[0022] As a further preferred method, the model domain point cloud is enhanced in advance, including transforming the model domain point cloud and introducing uniformly distributed random noise, and then changing the local point cloud distribution through an interpolation algorithm; the detector and the domain discriminator are trained through the enhanced model domain point cloud.

[0023] As a further preference, the transforming operation performed on the model domain point cloud includes performing one or more transformations of rotation, translation, and scaling on the model domain point cloud.

[0024] As a further preferred embodiment, the local point cloud distribution is changed by an interpolation algorithm, specifically:

[0025] According to the position of the sensor when acquiring the real domain point cloud, the part of the point cloud close to the sensor in the model domain point cloud is interpolated using local neighboring points to increase the density of the part of the point cloud.

[0026] According to another aspect of the present invention, a mechanical assembly three-dimensional object detection system based on CAD model domain point cloud data is provided, comprising a processor, wherein the processor is used to execute the above-mentioned mechanical assembly three-dimensional object detection method based on CAD model domain point cloud data.

[0027] In general, the above technical solution conceived by the present invention has the following technical advantages compared with the prior art:

[0028] 1. The present invention is based on the CAD three-dimensional model, collects point cloud information of parts in the model domain, and introduces a domain discriminator and related loss functions to compensate for the difference between the model domain and the real domain data, so as to realize the training of the detector, thereby accurately identifying and locating the parts of the assembled assembly in the real world.

[0029] 2. The present invention introduces a domain discriminator to make the features extracted by the point cloud feature extractor for the model domain point cloud data and the real domain point cloud data more similar, and uses domain randomization technology to bridge the gap between data in different domains, so that the final detection network has better detection effect in the point cloud data collected in the real world.

[0030] 3. The present invention utilizes point cloud data in the CAD model domain for training, which can reduce the cost and workload of labeling point cloud data collected in the real world; and compared with 2D image detection technology, the use of CAD three-dimensional models can use the information of the third dimension to achieve natural object background decoupling and object distinction in front and back overlapping areas.

[0031] 4. Based on the point cloud data obtained in the model domain, the present invention performs a series of data enhancements on the point cloud data, which can generate more diverse training samples, help improve the generalization ability of the model, adapt it to a wider range of real-world scenarios, and improve the robustness of the network model to point cloud data in different domains. At the same time, data enhancement of model domain data can expand the training data set and reduce the dependence on real-world collected data, which is particularly beneficial when large-scale data annotation is difficult or expensive, and can accelerate the model development and deployment process. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a method for detecting three-dimensional objects of a mechanical assembly based on point cloud data in a CAD model domain according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] The embodiment of the present invention provides a method for detecting three-dimensional objects of mechanical assemblies based on point cloud data of CAD model domain, such as Figure 1 As shown, the following steps are included:

[0035] S1. For a mechanical assembly composed of multiple parts, obtain its CAD three-dimensional model, and obtain the model domain point cloud and corresponding labels through the CAD model. The labels reflect the category and location information of the parts. Obtain the real domain point cloud of the mechanical assembly.

[0036] Furthermore, obtaining the model domain point cloud through the CAD model includes: obtaining a virtual depth map based on the CAD model of the mechanical assembly, and then converting the virtual depth map into point cloud data. Since the assembly model itself contains size, position, and posture information, it is relatively easy to obtain label information. The specific form of the label is the position box of the part. The box information includes the type of part, the spatial coordinates of the center point of the part, the length, width, height, and angle posture. The size of the mechanical assembly and the number of parts are not limited. The real domain point cloud of the mechanical assembly is obtained by collecting the mechanical assembly through sensors.

[0037] Furthermore, for the virtual depth map, the depth error is introduced by linearly varying the deviation with depth, and boundary blur is added, and then the processed virtual depth map is converted into a model domain point cloud; the details are as follows:

[0038] The depth error is introduced by superimposing a deviation that varies linearly with depth on the original virtual depth map. The calculation formula is as follows:

[0039] z a =(1+Shift R )×z s (1)

[0040] Among them, z s is the depth value of each pixel in the original virtual depth map, z s is the depth value of each pixel in the enhanced virtual depth map; Shift R is the deviation. For each pixel of a depth map, Shift R Take the same value to ensure that the semantic structure of a depth map is not destroyed, but for different virtual depth maps Shift R Take random values ​​ranging from -0.1 to 0.1 to expand the range of variation of training data.

[0041] For boundary blur, Gaussian blur is performed on the virtual depth map, and the size of the Gaussian convolution kernel varies randomly between 5 and 21.

[0042] S2. Transform the point cloud in the model domain and introduce some uniformly distributed random noise; change the local point cloud distribution through the interpolation algorithm and adjust the point cloud density in some areas; simulate the point cloud distribution in the real point cloud data set and enhance the point cloud data.

[0043] Furthermore, the transformation operation is one or more of rotation, translation, and scaling transformations performed on the model domain point cloud, and the combination is not limited. The transformation operations such as rotation, translation, and scaling are specifically implemented as follows.

[0044] The transformation matrix M is shown in formula (2):

[0045]

[0046] Among them, a 3×3 represents the rotation matrix, t 3×1 represents the translation vector, p 1×3 represents the perspective transformation vector and S represents the overall scaling factor.

[0047] Furthermore, in point cloud data, uniformly distributed random noise can be used to simulate the accuracy error of sensors or the influence of other unknown factors. Uniformly distributed random noise is a common type of random noise that generates random numbers in a uniformly distributed manner within a given range. This noise is usually used to simulate the accuracy error of sensors and environmental noise. The characteristic of uniformly distributed random noise is that it can be evenly distributed within a specified range without obvious bias. Therefore, uniformly distributed random noise can be added to the model domain point cloud to simulate the accuracy error of point cloud data collected by sensors in the real world.

[0048] Furthermore, the local point cloud distribution is changed through the interpolation algorithm, specifically: using local neighboring points to estimate missing points or increase the density of points, that is, according to the position of the sensor when acquiring the real-domain point cloud data, the local neighboring points are used to interpolate the part of the point cloud close to the sensor in the model domain point cloud to increase the density of this part of the point cloud. This interpolation algorithm can change the density of the local point cloud to simulate the different densities of point clouds collected by sensors at different distances in the real world, thus achieving data enhancement.

[0049] In addition, some parameters in the above data enhancement method can be changed, and data enhancement experiments can be conducted multiple times. The appropriate data enhancement strategy can be determined through a series of control experiments to improve the detection effect of the three-dimensional object detection network trained based on the model domain point cloud in the point cloud dataset collected in the real world.

[0050] S3. The three-dimensional object detection network (detector) is trained using the model domain point cloud and the corresponding labels as the training set; the detector includes a feature extractor and a label predictor, the feature extractor is used to extract point cloud features, and the label predictor is used to determine the label based on the features; and a domain discriminator is introduced to discriminate whether the features output by the feature extractor come from the model domain or the real domain.

[0051] Specifically, the detector and domain discriminator are jointly trained, and the loss function is to maximize the domain discriminator error and minimize the detector error: the model domain point cloud and the real domain point cloud are respectively input into the feature extractor to obtain model features and actual features accordingly; the model features are input into the label predictor to output the corresponding labels; the model features and the actual features are input into the domain discriminator, and the domain discriminator discriminates whether the features come from the model domain or the real domain, so that the features extracted by the feature extractor for the model domain point cloud data and the real domain point cloud data are more similar, so as to bridge the gap between data in different domains and improve the detection effect of the detector on the real point cloud data.

[0052] Furthermore, the purpose of introducing a domain discriminator is to input the features extracted from the model domain point cloud and the features extracted from the real domain point cloud into the domain discriminator to obtain domain classification results. The point-level features corresponding to the model domain data will be input into the label predictor to obtain semantic geometry results, while the point-level features corresponding to the model domain and real domain data will be input into the domain discriminator to obtain domain classification results. The point-level features that can be transferred between different domains should satisfy the requirements that the label predictor can correctly complete the semantic geometry task and the domain discriminator cannot distinguish whether the point is from the model domain or the real domain. Therefore, the goal of the network during training is to minimize the label prediction error and maximize the domain discrimination error, thereby constructing a loss function and minimizing the loss function as the training goal. The loss function of the entire network is shown in formula (3):

[0053]

[0054] Among them, L y (·,·) is the loss function for label prediction, L d (·,·) is the loss function for domain classification, and the hyperparameter λ is used to control the weight between the two objectives. y (·,·) and L d (·,·) can use the cross entropy loss function; θ y is the network parameter of the label predictor, θ f is the network parameter of the feature extractor, θ d is the network parameter of the domain discriminator, G f (·;θ f ) represents the feature extractor, G d (·;θ d ) represents the domain discriminator, G y (·;θ y ) represents the label predictor; i represents the sequence number of the sample, N represents the total number of samples; x i Represents the point cloud input of the i-th sample, y i Represents the label of the i-th sample, which includes the domain label and the position label.

[0055] Finally, the output features of the feature extractor are distinguishable for the original task in the model domain, but indistinguishable between different domains, thus making the feature distribution of the model domain and the real domain data similar at the feature representation level. According to the adversarial domain adaptation theory, this corresponds to the difference distance d between the probability distribution of the feature space of the model domain and the real domain. HΔH (D S ,D T ) becomes smaller, so the generalization error ε of the migration model from the source domain to the target domain S (h) The upper bound value becomes smaller and the generalization ability improves.

[0056] S4. The real-domain point cloud of the mechanical assembly is input into the trained detector. The feature extractor extracts the actual features of the real-domain point cloud. The label predictor outputs the corresponding label according to the actual features, that is, the category and position information of the components are obtained, thereby realizing the recognition and positioning of the components in the mechanical assembly.

[0057] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional target detection method for mechanical assemblies based on CAD model domain point cloud data, characterized in that: The steps include: Detector training phase: Based on the CAD 3D model of the mechanical assembly, the model domain point cloud and corresponding labels are obtained, and the labels reflect the category and location information of the parts in the mechanical assembly; the real domain point cloud data of the mechanical assembly is collected; The detector and the domain discriminator are trained by using the model domain point cloud data and the real domain point cloud data to obtain a trained detector; The detector includes a feature extractor and a label predictor. During training, the feature extractor extracts features of the model domain point cloud and the real domain point cloud respectively to obtain model features and actual features. The model features and the actual features are input into a domain discriminator, and the domain discriminator discriminates whether the features come from the model domain or the real domain; the label predictor outputs the corresponding label according to the model features; The loss function during training is constructed based on maximizing the domain discrimination result error and minimizing the label prediction result error; Detector application stage: The real-domain point cloud of the mechanical assembly is input into the trained detector, the feature extractor extracts the actual features of the real-domain point cloud, and the label predictor outputs the corresponding labels based on the actual features, thereby realizing the recognition and positioning of parts in the mechanical assembly.

2. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to claim 1, characterized in that: The loss function E(θ) during training f ,θ y ,θ d ) specifically: in, is the loss function for label prediction, is the loss function for domain discrimination, and the hyperparameter λ is used to control and The specific gravity of f is the network parameter of the feature extractor, θ y is the network parameter of the label predictor, θ d is the network parameter of the domain discriminator; i represents the sample number, and N represents the total number of samples.

3. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data as claimed in claim 2, characterized in that: and The cross entropy loss function is used.

4. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to claim 1, characterized in that: The method of obtaining a model domain point cloud based on a CAD three-dimensional model of a mechanical assembly includes: A virtual depth map is obtained based on the CAD 3D model of the mechanical assembly, and then the virtual depth map is converted into a model domain point cloud.

5. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to claim 4, characterized in that: After obtaining the virtual depth map based on the CAD 3D model of the mechanical assembly, the depth error is introduced to enhance the virtual depth map, and the virtual depth map is subjected to Gaussian blur processing. The processed virtual depth map is then converted into a model domain point cloud.

6. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data as claimed in claim 5, characterized in that: The introduction of depth error to enhance the virtual depth map is specifically as follows: z a =(1+Shift R )×z s Among them, z s is the depth value of the pixel in the original virtual depth map, z a Shift is the depth value of the pixel in the enhanced virtual depth map. R is the amount of deviation introduced; for different virtual depth maps, Shift R Take a random value from -0.1 to 0.1, and for each pixel in a virtual depth map, Shift R The value remains unchanged.

7. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to any one of claims 1 to 6, characterized in that: The model domain point cloud is enhanced in advance, including transforming the model domain point cloud and introducing uniformly distributed random noise, and then changing the local point cloud distribution through an interpolation algorithm; the detector and domain discriminator are trained through the enhanced model domain point cloud.

8. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to claim 7, characterized in that: The transformation operation performed on the model domain point cloud includes performing one or more transformations of rotation, translation, and scaling on the model domain point cloud.

9. The method for detecting three-dimensional objects of mechanical assemblies based on CAD model domain point cloud data according to claim 7, characterized in that: The local point cloud distribution is changed by the interpolation algorithm, specifically: According to the position of the sensor when acquiring the real domain point cloud, the part of the point cloud close to the sensor in the model domain point cloud is interpolated using local neighboring points to increase the density of the part of the point cloud.

10. A three-dimensional target detection system for mechanical assemblies based on CAD model domain point cloud data, characterized in that: It includes a processor, which is used to execute the three-dimensional target detection method for mechanical assembly based on CAD model domain point cloud data as described in any one of claims 1-9.

Images