6D target attitude estimation algorithm of improved feature extraction network

Abstract

The invention provides a 6D target attitude estimation algorithm for improving a feature extraction network, and the algorithm comprises the steps: S1, collecting an RGB-D image of a to-be-detected object, carrying out the semantic segmentation, and dividing a training set and a test set; s2, constructing an improved image feature extraction network by using a jump connection method and an attention mechanism module; s3, performing initial feature extraction of the point cloud by using Point Net, and then constructing an improved point cloud feature extraction network by using a K-nearest neighbor method and global pooling; s4, constructing a 6D target attitude estimation network on the basis that S1, S2 and S3 are not assembled; s5, inputting the RGB-D image of the object in the training set into the 6D target attitude estimation network and then training; and S6, inputting the RGB-D image of the object in the test set into the trained attitude estimation network, and directly predicting rotation transformation and translation transformation results.

Description

Technical Field

[0001] The present invention relates to the field of 6D target posture estimation, and in particular to a 6D target posture estimation algorithm with an improved feature extraction network. Background Art

[0002] 6D target pose estimation is to calculate the rotation matrix and translation vector of the target in the camera coordinate system, which has been widely used in fields such as machine grasping, autonomous driving, and augmented reality. These applications face the difficulty of achieving accurate and real-time pose estimation for occluded objects or objects with weak textures. However, early methods only used RGB images for pose estimation, and the performance was poor when the object was widely occluded or had weak textures. Until the emergence of RGB-D cameras, it provided additional depth information, providing new ideas for solving these problems.

[0003] For RGB-D images, the traditional method of pose estimation is to manually extract features from RGB-D and then perform feature matching, which is easily affected by factors such as target occlusion and illumination changes, and the actual application effect is poor. As Convolutional Neural Network (CNN) is widely used in image processing tasks, some researchers have used CNN on RGB-D images to solve the pose estimation problem. These methods are mainly divided into two categories. One is to use CNN to detect 2D key points in RGB images, and then use the perspective n-point algorithm (Perspective-n-Point, PNP) to calculate 6D pose parameters, which solves the problem that traditional methods are not suitable for key point detection of textureless objects. However, in the case of occlusion, the method of detecting 2D key points cannot accurately estimate the pose of the object. The second is to use CNN to directly obtain the 6D pose of the object, that is, to use color information and point cloud information for feature fusion, and use CNN to estimate the initial pose from the RGB-D image, but the pose accuracy is not high, and it is necessary to use the iterative closest point algorithm (Iterative Closest Point, ICP) or multi-view hypothesis verification method to refine the estimation results, which is very time-consuming.

[0004] In view of the shortcomings of the second type of method, researchers have tried to improve the color features and point cloud features to improve the accuracy of pose estimation. However, in addition to the quality of the feature fusion method, the richness and effectiveness of the color features and point cloud features extracted for feature fusion will also affect the pose estimation results of the object. Therefore, we propose a 6D target pose estimation algorithm with an improved feature extraction network. Summary of the invention

[0005] The purpose of this invention is to provide a 6D target pose estimation algorithm with an improved feature extraction network. It consists of three main parts. The first part performs semantic segmentation on the RGB image, obtains the color map and depth map cropped according to the mask bounding box, and converts the depth map of the mask position into a point cloud; the second part performs feature extraction and fusion, uses CNN to extract features from the color map of the first part, uses the improved Pointnet++ method to extract features from the point cloud of the first part, and then fuses the color features with the point cloud features at the pixel level; the third part estimates the pose of each sampling point and outputs the pose estimation result with the highest confidence. Finally, the pose iteration module is applied to further improve the accuracy of pose estimation.

[0006] The technical solution of the present invention to solve the above problem is as follows: a 6D target posture estimation algorithm of an improved feature extraction network comprises the following steps:

[0007] S1, collect RGB-D images of the object to be tested and perform semantic segmentation, dividing them into training set and test set;

[0008] S2, build an improved image feature extraction network;

[0009] S3, build an improved point cloud feature extraction network;

[0010] S4, build a 6D target pose estimation network;

[0011] S5, inputting the RGB-D images of the objects in the training set into the 6D object pose estimation network for training;

[0012] S6, input the RGB-D image of the object in the test set into the trained posture estimation network to directly predict the rotation transformation and translation transformation results.

[0013] The beneficial effects of the present invention are:

[0014] The present invention discloses a 6D target posture estimation algorithm with an improved feature extraction network. The Densefusion algorithm is used as the basic framework. A skip connection method and an attention mechanism module are added to the image feature extraction stage, so that deep features and shallow features are effectively integrated, and the richness and effectiveness of image feature information are improved. Then, the point cloud is processed using the improved PointNet++ to solve the problem that fine features cannot be collected by simply using PointNet and the point cloud lacks semantic information. The network's adaptability to posture estimation environments such as occlusion and complex object shapes is enhanced. At the same time, the improved feature extraction network can obtain richer and more effective semantic information, and the two can be effectively integrated. Whether used alone or in combination, the accuracy of posture estimation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the 6D target pose estimation network;

[0016] Figure 2 It is an optimized image feature extraction network;

[0017] Figure 3 to Figure 4 This is the structure diagram of the SE module and Attention module in the optimized image feature extraction network;

[0018] Figure 5 It is an optimized point cloud feature extraction network;

[0019] Figure 6 It is the feature fusion process. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and specific examples. The examples are only used to explain the present invention and are not used to limit the scope of the present invention.

[0021] The specific embodiments of the present invention and their implementation process are as follows:

[0022] S1, collect RGB-D images of the object to be tested and perform semantic segmentation, dividing them into training set and test set;

[0023] S2, build an improved image feature extraction network;

[0024] S3, build an improved point cloud feature extraction network;

[0025] S4, build a 6D target pose estimation network;

[0026] S5, inputting the RGB-D images of the objects in the training set into the 6D object pose estimation network for training;

[0027] S6, input the RGB-D image of the object in the test set into the trained posture estimation network to directly predict the rotation transformation and translation transformation results.

[0028] Furthermore, in step S1, the RGB-D image of the object is an RGB image and a depth image obtained by a depth camera at the same time for the same object.

[0029] Furthermore, in step S1, the target object is cropped from the RGB-D image through the semantic segmentation network of PoseCNN, and finally a mask image containing only the target object is generated, and then the depth map of the mask position is converted into a point cloud through the camera intrinsic parameter.

[0030] Furthermore, in step S2, an improved image feature extraction network framework is constructed as follows: Figure 2As shown in the figure, ResNet and PSPNet are used as the basic framework of the image feature extraction network, and then a SE module (Squeeze and Excitation, SE) is added after the pyramid pooling module (Pyramid Pooling Module, PPM), and then an Attention module is added to the connection channel between the encoding process and the decoding process.

[0031] Furthermore, in step S2, the SE module is as follows: Figure 3 As shown in the figure, the attention weighted graph with the same size as the input channel is obtained through global pooling, fully connected layer and sigmoid function, and then the weights are assigned to the input channels. The structure of the Attention module is as follows: Figure 4 As shown, with the current level feature C of ResNet i and the higher-level features P of the upsampling layer i+1 As input, through two convolution structures and sigmoid function, we get i The attention weighted graphs of the same size are assigned weights and connected to P through jump connections. i+1 When input to the Attention module C i and P i+1 When the size is different, P i+1 Double upsampling is achieved through bilinear interpolation algorithm. Finally, P1 is sampled into feature map P0 with the same size as the input image. The input of the image feature extraction network constructed in this example is h×w×3, and the output is h×w×32.

[0032] Furthermore, in step S3, an improved point cloud feature extraction network framework is constructed as follows: Figure 5 As shown in the figure, PointNet is used for initial feature extraction of point clouds. The network input is an N×3 point cloud, which is a point cloud converted from the depth map of the mask position. Among them, N represents the number of mask point clouds, and 3 represents the spatial position of the point cloud. The n×3 point cloud in the figure is n points randomly sampled from the mask point cloud N. Then the KNN method is used to obtain the index of the k points closest to each sampling point, and the corresponding feature set (n, k, C) is obtained through the index. Finally, the features around each sampling point are obtained through maximum pooling. The input of the image feature extraction network constructed in this example is n×3, and the output is n×32. PointNet is a one-dimensional convolution operation, n is 500, and k is 6.

[0033] Furthermore, in step S4, a 6D target pose estimation network framework is constructed based on the semantic segmentation network, the image feature extraction network and the point cloud feature extraction network, such as Figure 1 As shown in the figure, CNN and PointNet++ represent the improved image feature extraction network and the improved point cloud feature network respectively.

[0034] Furthermore, the specific feature fusion process in step S4, 6D target posture estimation network is as follows: Figure 6 In the specific implementation, CNN outputs n×32 color features, and PointNet++ outputs n×64 point cloud features. Then, the color features corresponding to the positions are fused with the point cloud features through one-dimensional convolution and maximum pooling operations, and finally n×1408 global fusion features are output.

[0035] Furthermore, in step S4, the posture estimation in the 6D target posture estimation network specifically inputs the global fusion features into the neural network for posture estimation, and regresses the rotation matrix R, the translation vector t and the confidence c through the fusion features. The regression network consists of three identical small networks, each of which contains four layers of one-dimensional convolution.

[0036] Furthermore, in step S4, the iterative refinement in the 6D target pose estimation network specifically uses a CNN-based optimization method, which consists of three fully connected layers, and the fused features are formed into global features through the maximum pooling layer for iterative pose estimation. The iterative network outputs a residual pose each time. After k iterations, the predicted residual pose is concatenated with the original pose to obtain the final pose estimation result:

[0037]

[0038] Among them, [R k |t k ] represents the residual pose result after the kth iteration, and [R0|t0] represents the initial pose output of the pose estimation network.

[0039] Furthermore, in step S5, the 6D target pose estimation network constructed by training the LineMOD dataset is adopted. The LineMOD dataset contains 13 video frame images of weakly textured objects, wherein eggbox and glue are objects with rotational symmetry. We divide 15% of the images into training sets and the rest of the images into test sets.

[0040] Further, in step S6, for the LineMOD dataset, our method is compared with the previous RGB method and R-GBD in Table 1, and the symmetric objects are indicated in bold in the table. For the RGB method, the results in Table 1 are the results after optimization using the optimization method. It can be seen from Table 1 that our method achieves an accuracy of 98.7% on the LineMOD dataset when using the iterative network. Compared with the RGB methods PoseCNN, CDPN and DPOD, they are improved by 10.1%, 8.8% and 3.5% respectively, and most of the R-GBD methods in the table have higher average accuracy than the RGB methods, indicating that the use of depth or point cloud information can effectively improve the accuracy of posture recognition. Compared with the baseline DenseFusion, the algorithm in this paper has improved the posture estimation results of 13 objects in the LineMOD dataset, and its average accuracy has increased by 4.4%. Compared with the other RGB-D methods NF6D, Uni6D, Uni6Dv2 and CA6D, they are improved by 2.3%, 1.7%, 1.5% and 1.2% respectively. The results show that the accuracy of our method exceeds that of the rest.

[0041] Table 1 Posture estimation results on LineMOD dataset / %

[0042]

[0043] Furthermore, in step S6, in order to verify the improved image feature extraction network and point cloud feature extraction network, we compare DenseFusion with only improving the image feature extraction network, only improving the point cloud feature extraction network, and improving both in a non-iterative manner, as shown in Table 2.

[0044] As can be seen from Table 2, both the improved image feature extraction network and the improved point cloud feature extraction network can improve the final pose estimation accuracy. The improved image feature extraction network is improved by 8.4% compared with DenseFusion, and the improved point cloud feature extraction network is improved by 4.5% compared with DenseFusion. The improved image feature extraction network has a better improvement effect than the improved point cloud feature extraction network. This is because the improved image feature extraction network can make full use of shallow features, so that the final deep feature semantic information is rich; while the improved point cloud feature extraction network enhances the information extraction ability of point cloud features, but its semantic information is still not as rich and effective as image information. When both are combined, the average accuracy is improved by 0.9% and 4.7% respectively compared with the single optimization, indicating that the two feature information can effectively complement each other and jointly improve the accuracy of pose estimation.

[0045] Table 2 Improved network evaluation

[0046]

[0047] Therefore, in order to address the problem that the information extracted by the feature extraction network is not rich and effective enough, our two improvements to the feature extraction network can obtain richer and more effective semantic information, and the two can be effectively integrated. Whether used alone or in combination, they can improve the accuracy of pose estimation.

Claims

1. A 6D target pose estimation algorithm with an improved feature extraction network, characterized in that: The following steps are involved: S1, collect RGB-D images of the object to be tested and perform semantic segmentation, dividing them into training set and test set; S2, build an improved image feature extraction network; S3, build an improved point cloud feature extraction network; S4, build a 6D target pose estimation network; S5, inputting the RGB-D images of the objects in the training set into the 6D object pose estimation network for training; S6, input the RGB-D image of the object in the test set into the trained posture estimation network to directly predict the rotation transformation and translation transformation results.

2. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: The RGB-D image of the object in S1 is an RGB image and a depth image obtained by a depth camera at the same time for the same object.

3. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 2, characterized in that: In S1, the target object is cropped from the RGB-D image through the semantic segmentation network of PoseCNN, and finally a mask image containing only the target object is generated. The depth map of the mask position is then converted into a point cloud through the camera intrinsic parameters.

4. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: The improved image feature extraction network in S2 uses ResNet and PSPNet as the basic framework of the image feature extraction network, then adds a SE module (Squeeze and Excitation, SE) after the Pyramid Pooling Module (PPM), and then adds an Attention module to the connection channel between the encoding process and the decoding process.

5. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 4, characterized in that: In the SE module, an attention weighted graph with the same size as the input channel is obtained through global pooling, a fully connected layer and a sigmoid function, and then the weights are assigned to the input channels.

6. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 4, characterized in that: In the Attention module, the current level feature C of ResNet i and the higher-level features P of the upsampling layer i+1 As input, through two convolution structures and sigmoid function, we get i The attention weighted graphs of the same size are assigned weights and connected to P through jump connections. i+1 Add.

7. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: The improved point cloud feature extraction network in S3 uses PointNet to perform initial feature extraction of the point cloud, then uses the KNN method to obtain the index of the k points closest to each sampling point, obtains the corresponding feature set through the index, and finally obtains the surrounding features of each sampling point through maximum pooling.

8. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: In S4, the 6D target pose estimation network includes feature fusion, pose estimation and iterative refinement modules in addition to the semantic segmentation network, the image feature extraction network and the point cloud feature extraction network.

9. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 8, characterized in that: The feature fusion module extracts color features from the image feature extraction network and point cloud features output by the point cloud feature extraction network, fuses the color features corresponding to the positions with the point cloud features through one-dimensional convolution and maximum pooling operations, and finally outputs the global fusion features.

10. The 6D target pose algorithm of the improved feature extraction network according to claim 8, characterized in that: The posture estimation module inputs the global fusion features into the neural network for posture estimation, and regresses the rotation matrix R, the translation vector t and the confidence c through the fusion features.

11. The 6D target pose algorithm of the improved feature extraction network according to claim 10, characterized in that: The regression network consists of three identical small networks, each of which contains four layers of one-dimensional convolution.

12. The 6D object pose algorithm of the improved feature extraction network according to claim 8, characterized in that: The iterative refinement module uses a CNN-based optimization method and consists of three fully connected layers. The fused features are passed through the maximum pooling layer to form global features for iterative pose estimation. The iterative network outputs a residual pose each time. After k iterations, the predicted residual pose is concatenated with the original pose to obtain the final pose estimation result: Among them, [R k |t k ] represents the residual pose result after the kth iteration, and [R0|t0] represents the initial pose output of the pose estimation network.

13. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: In the S5, 15% of the images in the LineMOD dataset are divided into a training set to train the constructed 6D target pose estimation network.

14. The 6D target pose estimation algorithm of the improved feature extraction network according to claim 1, characterized in that: In S6, 85% of the images in the LineMOD dataset are divided into a test set, and the network outputs the pose estimation result with the highest confidence.

Images