A pose estimation method for weakly textured targets based on feature fusion

Abstract

The present invention discloses a method for estimating the pose of a weakly textured target based on feature fusion, comprising the steps of: 1. semantically segmenting the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target; 2. cropping the RGB image using the minimum bounding box; 3. extracting color features using a convolutional neural network; 4. acquiring a depth image; 5. segmenting the depth image and converting it into point cloud data; 6. acquiring point features, local geometric features and global geometric features in the point cloud data; 7. fusing the color features with the point features, local geometric features and global geometric features to obtain target fusion features; 8. inputting the target fusion features into a pose estimation network to output a pose estimation result. The present invention can be effectively applied to pose estimation of weakly textured targets, solves the problem of insufficient consideration of local features in pose estimation during feature fusion, improves the accuracy of pose estimation, and is convenient for popularization and use.

Description

Technical Field

[0001] The present invention belongs to the technical field of target pose estimation, and in particular relates to a method for estimating the pose of a weakly textured target based on feature fusion. Background Art

[0002] Target pose estimation can load computer-generated virtual objects into real image sequences, obtain the pose of objects, and help robotic arms grasp objects. It has wide applications in fields such as robotic arm grasping and augmented reality.

[0003] In the prior art, feature descriptors are used to extract target features to achieve pose estimation, such as "A pose estimation method combining SURF descriptors and autoencoders" (CN114037742A). This invention extracts SURF feature points of color images, and then uses the features extracted by the convolutional autoencoder and the corresponding pose information in the rendering data to form an offline feature template, and selects the K feature vectors with the smallest distance in the feature template to vote to obtain the 6D pose of the target. This invention reduces manual labeling, reduces environmental complexity, and reduces the amount of calculation. However, the use of feature descriptors such as SURF requires the target to have rich texture patterns. Therefore, this method is difficult to extract features for weak texture targets, and there is a problem of poor effect on weak texture targets. "A pose estimation method for weak texture objects" (CN113223181A) is optimized for weak texture targets. This invention obtains color embedded features and geometric embedded features through color images and point clouds respectively, and then uses the self-attention mechanism to extract position-dependent feature maps, and performs pose estimation after pixel-by-pixel fusion. This invention can enrich the information of each pixel feature and adaptively adjust the weights of different features to improve the recognition accuracy of each pixel. However, this invention uses a dense fusion method, ignoring the influence of local features between point clouds on pose estimation, thereby limiting the accuracy of pose estimation. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for weak texture target pose estimation based on feature fusion in view of the deficiencies in the above-mentioned prior art. The method has simple steps, reasonable design, convenient implementation, and can be effectively applied to weak texture target pose estimation. It solves the problem of insufficient consideration of local features in pose estimation during feature fusion, improves the accuracy of pose estimation, and has stronger model adaptability, good use effect, and is easy to promote and use. .

[0005] To solve the above technical problems, the technical solution adopted by the present invention is: a method for estimating the pose of a weakly textured target based on feature fusion, comprising the following steps:

[0006] Step 1: Perform semantic segmentation on the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target;

[0007] Step 2: Using the minimum bounding box to crop the RGB image to obtain a cropped RGB image;

[0008] Step 3: Use convolutional neural network to extract the color features of the target in the cropped RGB image;

[0009] Step 4: Obtain the depth image of the cropped RGB image;

[0010] Step 5: segmenting the depth image using the mask and converting it into point cloud data;

[0011] Step 6: Obtain point features, local geometric features and global geometric features in the point cloud data;

[0012] Step 7: Fusing the color feature with the point feature, the local geometric feature and the global geometric feature to obtain a target fusion feature;

[0013] Step 8: Input the target fusion features into the pose estimation network and output the pose estimation result.

[0014] In the above-mentioned method for weak texture target pose estimation based on feature fusion, the specific process of extracting the color features of the target in the cropped RGB image using a convolutional neural network in step 3 includes:

[0015] Step 301: downsample the cropped RGB image features using 18 convolutional layers to obtain features with a dimension of 512;

[0016] Step 302: Upsample the features using four upsampling layers to obtain 32-dimensional color features.

[0017] In the above-mentioned method for weak texture target pose estimation based on feature fusion, the specific process of obtaining point features, local geometric features and global geometric features in point cloud data in step 6 includes:

[0018] Step 601: extracting point features from point cloud data using the PointNet network;

[0019] Step 602: randomly select 256 position points, fuse the features of multiple position points, and reduce the impact of target occlusion and segmentation noise;

[0020] Step 603: Use the farthest point sampling method to find 128 points evenly distributed in the space;

[0021] Step 604: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area;

[0022] Step 605: In each local area, for point clouds in three spatial scales of 0.05 cm, 0.1 cm and 0.2 cm, PointNet is used to extract features, and connection aggregation is performed to form local geometric features;

[0023] Step 606: Use the farthest point sampling method to find 64 points evenly distributed in the space;

[0024] Step 607: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area;

[0025] Step 608: In each local area, feature extraction and clustering are performed on point clouds within two spatial scales of 0.2 cm and 0.3 cm;

[0026] Step 609: Use MLP to extract the global geometric features of the target based on the local geometric features.

[0027] In the above-mentioned method for estimating the pose of a weakly textured target based on feature fusion, the specific process of fusing the color feature with the point feature, the local geometric feature and the global geometric feature in step 7 to obtain the target fusion feature includes:

[0028] Step 701: Perform a 1D convolution operation on the color feature, and then fuse it with the point feature to form a point fusion feature;

[0029] Step 702: perform a 1D convolution operation on the point fusion feature again, and then fuse it with the local geometric feature to form a local fusion feature;

[0030] Step 703: Fuse the global geometric features in step 609, the point fusion features in step 701, and the local fusion features in step 702 to form a final target fusion feature.

[0031] In the above-mentioned method for weak texture target pose estimation based on feature fusion, the specific process of inputting the target fusion feature into the pose estimation network and outputting the pose estimation result in step eight includes:

[0032] Step 801: The target fusion features are used as a training set to train a pose estimation network;

[0033] Step 802: the pose estimation network predicts the rotation and translation of the target and the confidence of the pose prediction;

[0034] Step 803: taking the pose prediction made by the position point with the highest confidence as the initial pose;

[0035] Step 804: Use a four-layer fully connected network to optimize the initial pose to obtain the final pose estimation result.

[0036] In the above-mentioned method for weak texture target pose estimation based on feature fusion, the pose estimation network in step eight includes a loss function, and the loss function weights the pose loss by the confidence level, and the loss function Loss is:

[0037]

[0038] Among them, i represents the i-th point of N positions, represents the pose loss of the i-th point among N positions, c i represents the confidence of the predicted pose at the i-th point, and ω represents the balance parameter.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1. The method of the present invention has simple steps, reasonable design and is easy to implement.

[0041] 2. The present invention obtains higher quality target features by focusing on the fine-grained geometric features between local point cloud data.

[0042] 3. The present invention solves the problem of insufficient consideration of local features during feature fusion in pose estimation, thereby improving the accuracy of pose estimation.

[0043] 4. The present invention can be effectively applied to the pose estimation of weakly textured targets, with high accuracy, stronger model adaptability, good use effect, and easy promotion and use.

[0044] In summary, the method of the present invention has simple steps, reasonable design, and easy implementation. It can be effectively applied to the pose estimation of weakly textured targets, solves the problem of insufficient consideration of local features during feature fusion in pose estimation, improves the accuracy of pose estimation, has stronger model adaptability, good use effect, and is easy to promote and use.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 is a network structure diagram of the present invention;

[0048] Figure 3 This is a visualization diagram of the target pose estimation results of the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 and Figure 2As shown, the method for estimating the pose of a weakly textured target based on feature fusion of the present invention comprises the following steps:

[0050] Step 1: Perform semantic segmentation on the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target;

[0051] Step 2: Using the minimum bounding box to crop the RGB image to obtain a cropped RGB image;

[0052] Step 3: Use convolutional neural network to extract the color features of the target in the cropped RGB image;

[0053] Step 4: Obtain the depth image of the cropped RGB image;

[0054] Step 5: segmenting the depth image using the mask and converting it into point cloud data;

[0055] Step 6: Obtain point features, local geometric features and global geometric features in the point cloud data;

[0056] Step 7: Fusing the color feature with the point feature, the local geometric feature and the global geometric feature to obtain a target fusion feature;

[0057] Step 8: Input the target fusion features into the pose estimation network and output the pose estimation result.

[0058] In this embodiment, the specific process of extracting the color features of the target in the cropped RGB image using a convolutional neural network in step 3 includes:

[0059] Step 301: downsample the cropped RGB image features using 18 convolutional layers to obtain features with a dimension of 512;

[0060] Step 302: Upsample the features using four upsampling layers to obtain 32-dimensional color features.

[0061] In this embodiment, the specific process of obtaining point features, local geometric features and global geometric features in the point cloud data in step 6 includes:

[0062] Step 601: extracting point features from point cloud data using the PointNet network;

[0063] Step 602: randomly select 256 position points, fuse the features of multiple position points, and reduce the impact of target occlusion and segmentation noise;

[0064] Step 603: Use the farthest point sampling method to find 128 points evenly distributed in the space;

[0065] Step 604: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area;

[0066] Step 605: In each local area, for point clouds in three spatial scales of 0.05 cm, 0.1 cm and 0.2 cm, PointNet is used to extract features, and connection aggregation is performed to form local geometric features;

[0067] Step 606: Use the farthest point sampling method to find 64 points evenly distributed in the space;

[0068] Step 607: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area;

[0069] Step 608: In each local area, feature extraction and clustering are performed on point clouds within two spatial scales of 0.2 cm and 0.3 cm;

[0070] Step 609: Use MLP to extract the global geometric features of the target based on the local geometric features.

[0071] In this embodiment, the specific process of fusing the color feature with the point feature, the local geometric feature and the global geometric feature in step 7 to obtain the target fusion feature includes:

[0072] Step 701: Perform a 1D convolution operation on the color feature, and then fuse it with the point feature to form a point fusion feature;

[0073] Step 702: perform a 1D convolution operation on the point fusion feature again, and then fuse it with the local geometric feature to form a local fusion feature;

[0074] Step 703: Fuse the global geometric features in step 609, the point fusion features in step 701, and the local fusion features in step 702 to form a final target fusion feature.

[0075] In this embodiment, the specific process of inputting the target fusion feature into the pose estimation network and outputting the pose estimation result in step eight includes:

[0076] Step 801: The target fusion features are used as a training set to train a pose estimation network;

[0077] Step 802: the pose estimation network predicts the rotation and translation of the target and the confidence of the pose prediction;

[0078] Step 803: taking the pose prediction made by the position point with the highest confidence as the initial pose;

[0079] Step 804: Use a four-layer fully connected network to optimize the initial pose to obtain the final pose estimation result.

[0080] In this embodiment, the pose estimation network in step eight includes a loss function, and the loss function weights the pose loss by confidence. The loss function Loss is:

[0081]

[0082] Among them, i represents the i-th point of N positions, represents the pose loss of the i-th point among N positions, c i represents the confidence of the predicted pose at the i-th point, and ω represents the balance parameter.

[0083] When it comes to specific implementation, is the coordinate of the sampling point of the target 3D model, respectively through the Ground Truth pose matrix [R|t] and the estimated pose matrix The average distance between the coordinates of corresponding points after the transformation. The calculation formula is:

[0084]

[0085] Among them, M represents the set of sampling points of the target 3D model, i represents the i-th point of N position points, p is used as a marker to indicate that this is the pose loss, and x j represents the jth point in the sampling point set, (Rx j +t) represents the coordinates of the sampling point after the Ground Truth pose transformation. Represents the coordinates after the pose transformation estimated by the th point.

[0086] For targets with rotational symmetry, the pose loss is defined as: the coordinates of the target 3D model sampling points, by estimating the pose matrix The average distance between the transformed coordinates and the coordinates of the nearest point after transformation by the Ground Truth pose matrix [R|t] is calculated as:

[0087]

[0088] Among them, x k Represents x j The closest point.

[0089] The present invention is based on Intel(R) Xeon(R) CPU E5-2678 v3 2.50GHz processor, NVIDIA GeForceRTX 2080 graphics card, under Ubuntu16.04 system, using Pytorch0.4.1 and Python3.6 for development, and using CUDA9.0 and cuDNN7.6.4 for network acceleration training.

[0090] Table 1 shows the initial pose results and pose optimization results of the proposed method in the LineMOD dataset. The dataset contains 13 weak texture target sequences under complex backgrounds, each of which contains 1100-1300 RGB-D images. They are Ape, Benchvise, Camera, Can, Cat, Driller, Duck, Eggbox, Glue, Holepuncher, Iron, Lamp, and Phone. Among them, Eggbox and Glue have rotational symmetry, and the size of each target is different. The experiment divides 15% of the RGB-D images of each target into a training set, and the rest are test sets. The training set contains a total of 2372 RGB-D images of 13 targets, and the test set contains a total of 13406 RGB-D images of 13 targets.

[0091] The comparison index uses ADD accuracy, and the ADD calculation formula is as follows:

[0092]

[0093] Among them, Num pre Indicates the number of correct pose estimates, Num GT Indicates the number of all true poses. If ADD is less than 10% of the target maximum diameter, the pose estimation is considered correct. The ADD calculation formula is:

[0094]

[0095] Higher accuracy indicates better pose estimation method.

[0096] Table 1 Pose optimization results

[0097]

[0098] When extracting local geometric features of point clouds, the method of the present invention extracts point cloud features for point clouds of different radius spaces in the local area. Since different targets are trained at the same time, the multi-scale radius selected is the same. As a result, for smaller targets, the local geometric feature extraction is not fine enough, thus affecting the experimental accuracy. For large and medium-sized targets, the pose estimation accuracy is better.

[0099] In order to further verify the effect of the method of the present invention, the target pose estimation results are visualized, and the result is shown in the figure below. Figure 3 shown.

[0100] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for estimating the pose of a weakly textured target based on feature fusion, characterized in that: The following steps are involved: Step 1: Perform semantic segmentation on the RGB image where the target is located to obtain the target pixel mask and the minimum bounding box corresponding to the target; Step 2: Using the minimum bounding box to crop the RGB image to obtain a cropped RGB image; Step 3: Use a convolutional neural network to extract the color features of the target in the cropped RGB image. The specific process includes: Step 301: downsample the cropped RGB image features using 18 convolutional layers to obtain features with a dimension of 512; Step 302: Upsample the features using four upsampling layers to obtain 32-dimensional color features; Step 4: Obtain the depth image of the cropped RGB image; Step 5: segmenting the depth image using the mask and converting it into point cloud data; Step 6: Obtaining point features, local geometric features and global geometric features in the point cloud data. The specific process includes: Step 601: extracting point features from point cloud data using the PointNet network; Step 602: randomly select 256 position points, fuse the features of multiple position points, and reduce the impact of target occlusion and segmentation noise; Step 603: Use the farthest point sampling method to find 128 points evenly distributed in the space; Step 604: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area; Step 605: In each local area, for point clouds in three spatial scales of 0.05 cm, 0.1 cm and 0.2 cm, PointNet is used to extract features, and connection aggregation is performed to form local geometric features; Step 606: Use the farthest point sampling method to find 64 points evenly distributed in the space; Step 607: Taking each evenly distributed point as a center point, divide the sphere with a fixed radius into a local area; Step 608: In each local area, feature extraction and clustering are performed on point clouds within two spatial scales of 0.2 cm and 0.3 cm; Step 609: using MLP to extract the global geometric features of the target based on the local geometric features; Step 7: Fusing the color feature with the point feature, the local geometric feature and the global geometric feature to obtain a target fusion feature; Step 8: Input the target fusion feature into the pose estimation network and output the pose estimation result. The pose estimation network includes a loss function. The loss function weights the pose loss by the confidence level. The loss function Loss is: Among them, i represents the i-th point of N positions, represents the pose loss of the i-th point among N positions, c i represents the confidence of the predicted pose of the i-th point, and ω represents the balance parameter; is the coordinate of the sampling point of the target 3D model, respectively through the Ground Truth pose matrix [R|t] and the estimated pose matrix After the transformation, the average distance between the coordinates of corresponding points is The calculation formula is: Among them, M represents the set of sampling points of the target 3D model, i represents the i-th point of N position points, p is used as a marker to indicate that this is the pose loss, and x j represents the jth point in the sampling point set, (Rx j +t) represents the coordinates of the sampling point after the Ground Truth pose transformation. Represents the coordinates after the pose transformation estimated by the point; For targets with rotational symmetry, the pose loss is defined as: the coordinates of the target 3D model sampling points, by estimating the pose matrix The average distance between the transformed coordinates and the coordinates of the nearest point after transformation by the Ground Truth pose matrix [R|t] is calculated as: Among them, x k Represents x j The closest point.

2. A method for estimating a weakly textured target pose based on feature fusion according to claim 1, characterized in that: The specific process of fusing the color features with the point features, the local geometric features and the global geometric features to obtain the target fusion features described in step 7 includes: Step 701: Perform a 1D convolution operation on the color feature, and then fuse it with the point feature to form a point fusion feature; Step 702: perform a 1D convolution operation on the point fusion feature again, and then fuse it with the local geometric feature to form a local fusion feature; Step 703: Fuse the global geometric features in step 609, the point fusion features in step 701, and the local fusion features in step 702 to form a final target fusion feature.

3. A method for estimating a weakly textured target pose based on feature fusion according to claim 2, characterized in that: The specific process of inputting the target fusion features into the pose estimation network and outputting the pose estimation results in step 8 includes: Step 801: The target fusion features are used as a training set to train a pose estimation network; Step 802: the pose estimation network predicts the rotation and translation of the target and the confidence of the pose prediction; Step 803: taking the pose prediction made by the position point with the highest confidence as the initial pose; Step 804: Use a four-layer fully connected network to optimize the initial pose to obtain the final pose estimation result.

Images