A 6D pose estimation method for industrial parts based on incomplete geometry completion

Abstract

The present invention discloses a 6D pose estimation method for industrial parts based on incomplete geometry completion. The original RGB-D image containing the target industrial part is obtained, including the RGB image and the original point cloud converted from the depth map, and segmented and positioned; the complete geometric structure is restored from the original target point cloud containing the incomplete geometric structure through the point cloud completion network to obtain the completed target point cloud; features are extracted from the target ROI cropped image, the original target point cloud and the completed target point cloud respectively, and multi-scale splicing and fusion are performed to obtain enhanced RGB-D features; the 6D pose of the target industrial part is regressed in a confidence manner. The present invention addresses the problem of geometric incompleteness of depth information of industrial parts caused by various complex factors in actual scenarios. By completing the incomplete geometry and obtaining enhanced RGB-D features, accurate and robust 6D pose estimation is achieved, which has good engineering practical value.

Description

Technical Field

[0001] The present invention relates to a pose estimation method in the field of computer vision and robotics technology, and in particular to a 6D pose estimation method for industrial parts based on incomplete geometry completion. Background Art

[0002] Using visual sensors to estimate the 6-degree-of-freedom (6D) pose of target industrial parts in three-dimensional space, including a 3-degree-of-freedom translation vector and a 3-degree-of-freedom rotation matrix, is a basic computer vision technology that widely serves various downstream robotic operation tasks in actual industrial application scenarios, such as robotic assembly, obstacle avoidance planning, and robot digital twins.

[0003] Traditional methods rely on various manually designed image features and solve pose based on feature correspondence or voting, resulting in very limited accuracy, robustness, and flexibility. With the rapid development of deep learning technology and RGB-D cameras, data-driven RGB-D-based methods have gradually achieved better pose estimation performance and attracted attention from the industry. Common industrial parts often lack effective surface texture. Because the depth map in RGB-D images provides supplementary geometric information, these methods perform better than RGB-based methods alone.

[0004] However, in real-world applications, the 3D data obtained from depth sensors is often incomplete and contains significant noise. This is especially true for reflective metal surfaces like those found on industrial parts, where potential occlusion can further exacerbate the problem. In such cases, some visible areas in the image may be missing from the corresponding depth map. The incomplete 3D data lacks sufficient geometric information to accurately predict and infer the 2D image, leading to reduced performance in 6D pose estimation for industrial parts. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a 6D pose estimation method for industrial parts based on incomplete geometry completion. This method uses deep learning technology to realize 6D pose estimation of industrial parts under incomplete geometric information from RGB-D images, effectively improving the accuracy and robustness of pose estimation.

[0006] The specific technical solutions created by the present invention are as follows:

[0007] S1: Use an RGB-D camera to capture the original RGB-D image of the target industrial part, including the RGB image and depth map, and convert the depth map into the corresponding original point cloud;

[0008] S2: Segment and locate the target industrial part. Use the Mask R-CNN method to segment the instance mask and ROI cropped image of the target industrial part from the RGB image. At the same time, based on the alignment relationship between the pixels of the RGB image and the pixels of the depth map, segment the original target point cloud corresponding to the instance mask of the target industrial part from the original point cloud.

[0009] S3: A point cloud completion network is used to recover the complete geometric structure of the target industrial part from the original target point cloud containing incomplete geometric structure to obtain the corresponding completed target point cloud;

[0010] S4: Extract RGB features from the ROI cropped image obtained in S2, extract original geometric features from the original target point cloud obtained in S2, and extract multi-scale geometric features from the completed target point cloud obtained in S3;

[0011] S5: All features obtained in S4 are subjected to multi-scale splicing and fusion to obtain enhanced RGB-D features to enhance the original information of the RGB-D image;

[0012] S6: Through supervised training, the accurate 6D pose of the target industrial part is regressed and evaluated from the enhanced RGB-D features in a confidence scoring manner.

[0013] In step S1, the target industrial part area in the original point cloud may be incomplete and contain a lot of noise due to factors such as the reflective metal surface of the target industrial part, partial occlusion, and depth sensor noise of the RGB-D camera.

[0014] In step S3, the point cloud completion network mainly consists of an encoder and a decoder structure, and realizes point cloud completion by predicting the mapping from the incomplete geometric space to the complete geometric space, specifically:

[0015] The encoder consists of a double-layer stacked PointNet network, and the decoder consists of an MLP network;

[0016] First, the encoder uses the original target point cloud containing incomplete geometric structure obtained in S2 as input and uses a double-layer stacked PointNet network to extract an intermediate feature vector from it;

[0017] Then, the decoder takes the intermediate feature vector as input and combines it with the MLP network to generate dense completion point clouds in stages from coarse to fine.

[0018] Finally, the dense completed point cloud is adjusted and processed through a shape protection network layer, and the final completed target point cloud is output to avoid local shape distortion.

[0019] The adjustment process of the shape protection network layer is as follows:

[0020]

[0021]

[0022]

[0023] Among them, p (i) To densely complete the points in the point cloud; is the distance point p in the original target point cloud (i) The two nearest points; is a virtual target point used to guide point p (i) Adjustment direction; D is the virtual target point With point p (i) The distance between is the target point cloud point for the final adjustment output; σ is a learnable adjustment parameter, e represents the natural base, ||||2 represents the 2-norm, and i represents the index of any point.

[0024] The step S4 specifically includes the following steps:

[0025] S41: Extracting RGB features from ROI cropped images using CNN network Where H and W are the height and width of the ROI cropped image, respectively. rgb is the dimension of RGB features, represents the set of real numbers;

[0026] S42: Extracting original geometric features from the original target point cloud using the MLP network Where N is the number of points in the original target point cloud, d p is the dimension of the original geometric features;

[0027] S43: Multi-scale geometric features extracted from the completed target point cloud include region-level features F r With the global feature F g :

[0028] Regional level characteristics F r It is obtained by aggregating the completed target point cloud points in the vicinity of each original target point cloud point through the MLP network and average pooling method. where d r is the dimension of regional-level characteristics;

[0029] Global feature F g It is extracted and processed from the completed target point cloud through the PointNet++ network. where d g is the dimension of the global feature.

[0030] The step S5 specifically includes the following steps:

[0031] S51: Align the RGB features F according to the alignment relationship between the pixels of the RGB image and the pixels of the depth map rgb After downsampling, the original geometric features F p Splice at corresponding positions;

[0032] S52: The original geometric feature F p And the corresponding regional feature F r Splicing; Due to the original geometric features F p and regional-level features F r There is a one-to-one correspondence, so the two are directly spliced ​​together.

[0033] S53: The global feature F g After copying N-1 times, it is combined with the global feature F g It is integrated with itself to make the global feature F g Expanded to Extended Global Features Then the global features will be expanded and regional-level features F r Perform splicing;

[0034] S54: Through the splicing operation of steps S51-S53, the RGB feature F is realized rgb , original geometric features F p , regional level features F r And the global feature F g The multi-scale splicing fusion of the fused initial RGB-D features is obtained and d f =d rgb +d p +d r +d g , where d f is the dimension of the initial RGB-D feature, d rgb is the dimension of RGB features, d p is the dimension of the original geometric features, d r is the dimension of regional-level features, d g is the dimension of global features;

[0035] Finally, the initial RGB-D feature F rgb-d Feature enhancement is performed through the MLP network to obtain the final enhanced RGB-D features where d' f To enhance the dimension of RGB-D features.

[0036] In step S6, N 6D poses are regressed and predicted from the enhanced RGB-D features, and the confidence of each predicted pose is calculated. Then, the predicted pose with the highest confidence is used as the final 6D pose estimation result of the target industrial part.

[0037] The beneficial effects of the present invention are:

[0038] (1) This paper achieves accurate 6D pose estimation of industrial parts. Thanks to the supplementary geometric information provided by RGB-D images, it solves the problem of degraded pose estimation performance of traditional methods for weakly textured industrial parts.

[0039] (2) The present invention completes the incomplete geometric information in the original target point cloud, solving the problem of geometric incompleteness of depth information caused by multiple complex factors such as sensor noise, metal surface reflection, and occlusion, and further improving the robustness of pose estimation of industrial parts in actual industrial application scenarios;

[0040] (3) The pose estimation method provided by the present invention can be directly deployed in various downstream robotic application tasks, effectively improving the flexibility and intelligence of the robotic system.

[0041] In summary, the industrial part pose estimation method provided by the present invention addresses the low-texture characteristics of industrial parts and the geometric incompleteness of depth sensor information caused by various complex factors in actual industrial scenarios. By completing the incomplete geometry and obtaining enhanced RGB-D features, it achieves accurate and robust 6D pose estimation, and has good engineering practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of 6D pose estimation of industrial parts based on incomplete geometry completion in the present invention.

[0043] Figure 2 Schematic diagram of a target industrial part used for method testing in an embodiment of the present invention.

[0044] Figure 3 Schematic diagram of the adjustment process of completing the point cloud in the present invention.

[0045] Figure 4 It is a schematic diagram of the fusion process of enhancing RGB-D features in the present invention.

[0046] Figure 5 is a visualization result diagram of the pose estimation in an embodiment of the present invention.

[0047] Figure 6 This is a visualization effect diagram of the incomplete geometric completion of the target industrial part in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further illustrated below using a self-built industrial parts pose estimation dataset:

[0049] like Figure 1 As shown in FIG, in this embodiment, the method of the present invention is trained and tested on a self-built industrial parts dataset. The self-built industrial parts dataset includes a total of about 30,000 data samples, covering 8 different industrial parts (see Figure 2 ) and various challenging conditions such as occlusion, illumination change, depth loss, etc. Each data sample consists of the original RGB image, a depth map, and the mask, category, and pose label of each target industrial part in the image. The 6D pose estimation process of industrial parts in this invention is as follows: Figure 1 As shown. For this data set, the specific steps of the pose estimation method provided by the present invention are:

[0050] S1: Use an RGB-D camera to capture the original RGB-D image containing the target industrial part, including the RGB image and depth map, and convert the depth map into the corresponding original point cloud.

[0051] In the self-built industrial parts dataset of this embodiment, due to factors such as the reflective metal surface, occlusion, and depth sensor noise of the RGB-D camera that may exist on the target industrial parts, the target industrial parts area in the original point cloud may be incomplete and contain a lot of noise.

[0052] S2: Segment and locate the target industrial part. Use the Mask R-CNN method to segment the instance mask and ROI cropped image of the target industrial part from the RGB image. At the same time, based on the alignment relationship between RGB pixels and depth pixels, segment the original target point cloud corresponding to the instance mask of the target industrial part from the original point cloud.

[0053] S3: A point cloud completion network is used to recover the complete geometric structure of the target industrial part from the original target point cloud containing incomplete geometric structure, and obtain the corresponding completed target point cloud.

[0054] The point cloud completion network consists of an encoder-decoder structure and completes the point cloud by predicting the mapping from the incomplete geometric space to the complete geometric space. Specifically, first, the encoder takes the original target point cloud containing the incomplete geometric structure in S2 as input and extracts an intermediate feature vector from it using a double-layer stacked PointNet network. Then, the decoder takes the intermediate feature vector as input and combines the MLP network to generate a dense completed point cloud in a coarse-to-fine manner. Finally, to avoid local shape distortion, the dense completed point cloud is adjusted through a shape protection network layer (see Figure 3 ), thus obtaining the final completed target point cloud, the adjustment process is described as follows:

[0055]

[0056]

[0057]

[0058] Among them, p (i) To densely complete the points in the point cloud; is the distance p in the original target point cloud (i) The two nearest points; is a virtual target point used to guide p (i) The adjustment direction of With p (i) The distance between is the target point cloud point for the final adjustment output; σ is a learnable adjustment parameter, e represents the natural base, 2 represents the 2-norm, and i represents the index of any point.

[0059] In this embodiment, a series of incomplete depth point clouds of target industrial parts are pre-generated from different perspectives, and random occlusion and noise are added to these point clouds to provide supplementary synthetic training data for the training of the point cloud completion network.

[0060] S4: Extract RGB features from the ROI cropped image in S2, extract original geometric features from the original target point cloud in S2, and extract multi-scale geometric features from the completed target point cloud in S3.

[0061] Among them, CNN network is used to extract RGB features from ROI cropped images Where H and W are the height and width of the ROI cropped image, respectively. rgb is the dimension of RGB features, Represents a set of real numbers; uses the MLP network to extract original geometric features from the original target point cloud Where N is the number of points in the original target point cloud, d p is the dimension of the original geometric features; the multi-scale geometric features extracted from the completed target point cloud include regional features and global features, regional features Through the MLP network and average pooling method, the completed target point cloud points within a certain range near each original target point cloud point are aggregated, where d r is the dimension of regional features, global features It is extracted from the completed target point cloud through the PointNet++ network, where d g is the dimension of the global feature.

[0062] In this embodiment, take d rgb =128,dp =128,d r =256,d g =512

[0063] S5: All features obtained in S4 are fused at multiple scales to enhance the original information of the RGB-D image and obtain enhanced RGB-D features.

[0064] like Figure 4 As shown, first, according to the alignment relationship between RGB pixels and depth pixels, the RGB feature F rgb After downsampling, the original geometric features F p Splicing is performed at the corresponding positions. Due to the original geometric features F p and regional-level features F r There is a one-to-one correspondence between them, so the two are directly spliced. Then, the global feature F g After copying N-1 times, g It is integrated with itself to make the global feature F g Expand to And combine it with the regional level feature F r Through the above splicing operation, the RGB feature F rgb , original geometric features F p , regional level features F r And the global feature F g The multi-scale splicing fusion of the fused initial RGB-D features is obtained where d f =d rgb +d p +d r +d g is the dimension of the initial RGB-D feature. Finally, the initial RGB-D feature is enhanced through the MLP network to obtain the final enhanced RGB-D feature Among them, d' f To enhance the dimension of RGB-D features.

[0065] In this embodiment, take d' f =128.

[0066] S6: Through supervised training, the 6D pose of the target industrial part is accurately regressed from the enhanced RGB-D features using confidence scores. First, N 6D poses are regressed and predicted from the enhanced RGB-D features. The confidence score of each predicted pose is calculated, and the predicted pose with the highest confidence score is used as the final 6D pose estimate of the target industrial part.

[0067] In this embodiment, 70% of the data in the self-built industrial parts dataset is taken, and model training is performed based on the NVIDIA A100 graphics processor and the Pytorch framework.

[0068] After the model training is completed, 30% of the data in the self-built industrial parts dataset is taken for testing.

[0069] The quantitative test results are shown in Table 1. The ratio of average model distance less than 0.02m (ADD≤0.02) is used as an evaluation metric to measure the accuracy of the pose estimation results. It can be seen that the average success rate of pose estimation using the proposed method reaches over 90%.

[0070] Table 1 Success rate of pose estimation of the method of the present invention (%)

[0071] Target part Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 average Success rate 98.1 93.0 84.8 97.6 97.4 78.9 98.7 99.9 93.6

[0072] Qualitative test results such as Figure 5 As shown in Figure 2. The target industrial part is projected onto the image plane with the estimated pose to approximately measure the accuracy of the estimation result. It can be seen that the method of the present invention can successfully estimate the 6D pose of the target industrial part, which further verifies the effectiveness of the method of the present invention. In addition, the effects of some incomplete geometry completion are as follows: Figure 6 shown.

[0073] The present invention is not limited to the above-described embodiments and is intended only to facilitate understanding of the methods and core concepts of the present invention. It should be noted that, without departing from the principles of the present invention, a number of improvements and modifications may be made to the present application by a person skilled in the art, and such improvements and modifications fall within the scope of protection of the claims of the present invention. Any material not described in detail in this specification is prior art known to those skilled in the art.

Claims

1. A 6D pose estimation method for industrial parts based on incomplete geometry completion, characterized by: The following steps are involved: S1: Use an RGB-D camera to capture the original RGB-D image of the target industrial part, including the RGB image and depth map, and convert the depth map into the corresponding original point cloud; S2: Segment and locate the target industrial part. Use the Mask R-CNN method to segment the instance mask and ROI cropped image of the target industrial part from the RGB image. At the same time, based on the alignment relationship between the pixels of the RGB image and the pixels of the depth map, segment the original target point cloud corresponding to the instance mask of the target industrial part from the original point cloud. S3: A point cloud completion network is used to recover the complete geometric structure of the target industrial part from the original target point cloud containing incomplete geometric structure to obtain the corresponding completed target point cloud; S4: Extract RGB features from the ROI cropped image obtained in S2, extract original geometric features from the original target point cloud obtained in S2, and extract multi-scale geometric features from the completed target point cloud obtained in S3; S5: All features obtained in S4 are subjected to multi-scale splicing and fusion to obtain enhanced RGB-D features to enhance the original information of the RGB-D image; S6: Through supervised training, the 6D pose of the target industrial part is regressed and estimated from the enhanced RGB-D features in a confidence-based manner; In step S3, the point cloud completion network mainly consists of an encoder and a decoder structure, and realizes point cloud completion by predicting the mapping from the incomplete geometric space to the complete geometric space. Specifically: The encoder consists of a double-layer stacked PointNet network, and the decoder consists of an MLP network; First, the encoder uses the original target point cloud containing incomplete geometric structure obtained in S2 as input and uses a double-layer stacked PointNet network to extract the intermediate feature vector from it; Then, the decoder takes the intermediate feature vector as input and combines it with the MLP network to generate dense completion point clouds in stages from coarse to fine. Finally, the dense completed point cloud is adjusted and processed through a shape protection network layer to output the final completed target point cloud; The adjustment process of the shape protection network layer is as follows: Among them, p (i) To densely complete the points in the point cloud; is the distance point p in the original target point cloud (i) The two nearest points; is a virtual target point used to guide point p (i) Adjustment direction; D is the virtual target point With point p (i) The distance between is the target point cloud point for the final adjustment output; σ is a learnable adjustment parameter, e represents the natural base, || ||2 represents the 2-norm, and i represents the index of any point.

2. The method for 6D pose estimation of industrial parts based on incomplete geometry completion according to claim 1, characterized in that: In step S1, the target industrial part region in the original point cloud may be incomplete and contain noise.

3. The method for 6D pose estimation of industrial parts based on incomplete geometry completion according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: Extracting RGB features from ROI cropped images using CNN network Where H and W are the height and width of the ROI cropped image, respectively. rgb is the dimension of RGB features, represents the set of real numbers; S42: Extracting original geometric features from the original target point cloud using the MLP network Where N is the number of points in the original target point cloud, d p is the dimension of the original geometric features; S43: Multi-scale geometric features extracted from the completed target point cloud include region-level features F r With the global feature F g : Regional level characteristics F r It is obtained by aggregating the completed target point cloud points within the range of each original target point cloud point through the MLP network and average pooling. where d r is the dimension of regional-level characteristics; Global feature F g It is extracted and processed from the completed target point cloud through the PointNet++ network. where d g is the dimension of the global feature.

4. The method for 6D pose estimation of industrial parts based on incomplete geometry completion according to claim 1, characterized in that: Step S5 specifically includes the following steps: S51: Align the RGB features F according to the alignment relationship between the pixels of the RGB image and the pixels of the depth map rgb After downsampling, the original geometric features F p Splice at corresponding positions; S52: The original geometric feature F p And the corresponding regional feature F r Perform splicing; S53: The global feature F g After copying N-1 times, it is combined with the global feature F g It is integrated with itself to make the global feature F g Expanded to Extended Global Features Then the global features will be expanded and regional-level features F r Perform splicing; S54: Through the splicing operation of steps S51-S53, the RGB feature F is realized rgb , original geometric features F p , regional level features F r And the global feature F g The multi-scale splicing fusion of the fused initial RGB-D features is obtained and d f =d rgb +d p +d r +d g , where d f is the dimension of the initial RGB-D feature, d rgb is the dimension of RGB features, d p is the dimension of the original geometric features, d r is the dimension of regional-level features, d g is the dimension of global features; Finally, the initial RGB-D feature F rgb-d Feature enhancement is performed through the MLP network to obtain the final enhanced RGB-D features where d' f To enhance the dimension of RGB-D features.

5. The method for estimating 6D pose of industrial parts based on incomplete geometry completion according to claim 1, characterized in that: In step S6, N 6D poses are regressed and predicted from the enhanced RGB-D features, and the confidence of each predicted pose is calculated. Then, the predicted pose with the highest confidence is used as the final 6D pose estimation result of the target industrial part.

Images