Object 6D pose estimation method and system based on multi-channel attention mechanism

Abstract

The invention relates to the technical field of object pose estimation, in particular to an object 6D pose estimation method and system based on a multi-channel attention mechanism, and the method comprises the steps: S1, semantic segmentation and feature extraction: for an input RGB-D image, cutting a color image and a depth image of a target object from an input RGB image and a depth image through semantic segmentation, and carrying out feature extraction; and converting the depth image into a point cloud image by using the internal reference of the camera. According to the method, a multi-channel attention mechanism DA2Net is designed by introducing a position attention module and a channel attention module, the method has advantages in the aspects of feature association and global information redistribution, and meanwhile, the capability of capturing local and global context information is enhanced. On the basis of DenseFusion, the attention mechanism is added in a feature fusion network and an iterative refinement network, the understanding of local details is enhanced, and the shielded part is deduced from global information.

Description

Technical Field

[0001] The present invention relates to the technical field of object pose estimation, and in particular to a method and system for object 6D pose estimation based on a multi-channel attention mechanism. Background Art

[0002] 6DOF pose estimation is a key task in computer vision and robotics. It can obtain the six-degree-of-freedom pose of an object, that is, to obtain the position (x, y, z) of the object in three-dimensional space and the 3D translation and 3D rotation relationship relative to the camera. It is widely used in robot grasping and manipulation [1, 2], autonomous driving [3, 4, 5], augmented reality (AR) and virtual reality (VR). However, ideally, the technology needs to be robust to severe occlusion, illumination changes and complex backgrounds, but existing methods are difficult to meet the needs of practical applications and still need continuous exploration and research.

[0003] Mainstream 6D pose estimation algorithms include traditional methods and deep learning methods. Traditional pose estimation algorithms mainly rely on geometry and feature matching, extracting and matching features such as points, lines, and circles of the target object, and calculating the position and pose through visual algorithms. Existing methods include: ICP, SIFT, SURF, ORB

[11] , EPnP, and PPF. However, these algorithms, whether it is the registration iteration between two point clouds, feature point extraction, or the proposed correspondence between 2D and 3D feature points, are overly dependent on manually designed features and geometric models, making it difficult to perform effective pose estimation in complex environments and large-scale data.

[0004] In recent years, with the emergence of new technologies such as deep learning and convolutional neural networks, algorithms based on deep learning have become a hot topic. Many scholars have applied them to 6D pose estimation. These methods can automatically learn high-level features in images through large-scale data training, thereby achieving more accurate and robust pose estimation in complex environments. The pose estimation of deep learning methods can be divided into three types of methods according to the input data type: RGB images, RGB-D images, and point clouds. The RGB image method only uses visible light images for estimation, which has the characteristics of low acquisition cost and low equipment requirements, but faces challenges in dealing with illumination changes and occlusion. The point cloud method directly uses three-dimensional point cloud data to obtain rich spatial information from it, but while achieving high precision, it is accompanied by problems such as high data acquisition cost and complex processing. The RGB-D image method combines the information of RGB images and depth images. Through these two types of data, it has stronger robustness to illumination changes and occlusion, and can more accurately estimate the pose of objects.

[0005] Although pose estimation algorithms based on RGB-D images have made significant progress in many aspects, there are still challenges in dealing with severe occlusion and complex environments. In order to further improve the performance of RGB-D pose estimation algorithms, this paper proposes an end-to-end pose estimation algorithm based on a multi-channel attention mechanism DA2Net, aiming to improve the estimation accuracy. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings existing in the prior art and propose a method and system for object 6D pose estimation based on a multi-channel attention mechanism.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme: A method for estimating 6D pose of an object based on a multi-channel attention mechanism includes:

[0008] S1, semantic segmentation and feature extraction, for the input RGB-D image, the color image of the target object is cut out from the input RGB image and depth image through semantic segmentation And depth map, and use the camera internal parameters to convert the depth map into a point cloud map Then use the color feature extraction network and the geometric feature extraction network to obtain color features respectively. and point cloud features

[0009] S2, feature fusion network, combines the color features and point cloud features As input, after convolution processing, the outputs of the first and second layers of convolution are fused through channels to generate fusion features. and fusion features Then, the fusion feature F 2 The input is sent to the DA2Net attention module to capture local and global context information, and the global features are assigned to each position on the feature map aggregated by position and channel attention to obtain the feature information in the image. The features enhanced by the DA2Net attention module are subjected to two layers of convolution operations and average pooling to output new fused features. Then F 1 、F 2 and F 3 Generate pixel-level fusion features by splicing in the channel dimension

[0010] S3, pose estimation network, inputs the above pixel-level fusion feature F into the neural network for pose estimation, outputs the angular offset parameter, translation parameter and confidence of each pixel point, and obtains the object pose.

[0011] As a further solution of the present invention, in step S1: the encoder of the color feature extraction network is Resnet18, the decoder is 4 upsampling layers, and the color image of size H×W×3 is mapped to a color image of size H×W×d rgb In the space of , each pixel is a d rgb dimensional vector, representing the color features of the target object's location.

[0012] As a further solution of the present invention, in step S2, the fusion feature F 2 Input into the DA2Net attention module to capture local and global context information, assign global features to each position on the feature map aggregated by position and channel attention, and obtain feature information in the image as follows:

[0013] Based on the fusion feature F 2 Add another dimension to get fusion features Where H = 1, W = N, the fused feature X is input into the DA2Net attention module;

[0014] The structure of the DA2Net attention module includes a key feature collection branch and a dual-channel feature allocation branch, and the inputs of the key feature collection branch and the dual-channel feature allocation branch are both fusion features X;

[0015] In the key feature collection branch, first, the fused feature X is reshaped into two feature maps:

[0016] Then, feature map A and feature map B are input into bilinear pooling, and the feature vector pair (a i ,b i ) do the outer product and add them, the key feature collection branch is defined as:

[0017] G gather (X) = G bilinear (A,B) = A softmax(B) T

[0018] Output variables is the feature set G;

[0019] In the dual-channel feature allocation branch, the PAM module and the CAM module are introduced, and the output features of the two modules are aggregated to obtain the feature map V. Finally, the features in the feature set G are assigned to the feature map V, and the feature map Z is output. The fused feature X is used as the input of the two modules. First, for the PAM module, the input fused feature X first passes through a convolutional layer and reshapes to obtain two new feature maps. and feature map Where N = H × W, N is the number of pixels, then the feature map C and the feature map D are transposed and matrix multiplied, and the spatial attention map is obtained through softmax The formula is:

[0020]

[0021] where s ji Describes i th Position pair th At the same time, feature X passes through another convolutional layer and reshapes to obtain a new feature map Then perform matrix multiplication of the feature map E and the transpose of the spatial attention map S, and finally multiply it by a scale parameter α and perform weighted summation with the feature X to obtain the output M. The formula is:

[0022]

[0023] Among them, α is initialized to 0 and gradually learns to assign more weights. For the CAM module, the channel attention map is directly calculated by inputting feature X The formula is:

[0024]

[0025] where k ji Describes th Channel pair j th In addition, the transpose of K and the feature X are matrix multiplied and reshaped into Then multiply it by a scaling factor β and perform a weighted summation with feature X to get the final result The formula is:

[0026]

[0027] The β is initialized to 0 and gradually learns to get a larger weight. Then the output features of the two modules are summed element by element to complete feature fusion and reshaped into features. The formula is:

[0028] v i =Sum(M j ,N j )

[0029] Combine the key feature collection branch and the dual-channel feature allocation branch, distribute the features in the feature set G to each position of the feature V, and output the final feature map Z of the module. The formula is:

[0030]

[0031] Get the feature map Z.

[0032] As a further aspect of the present invention, the DA2Net attention module uses the following formula:

[0033]

[0034] Among them, Sum is element-by-element addition.

[0035] As a further solution of the present invention, in step S3, the object pose is improved based on the iterative refinement of the original network by the DA2Net attention module, and the extracted color feature F in the main network is reused in each iteration. I , to obtain a high-precision 6D pose. After n iterations, the final pose is:

[0036]

[0037] Among them, T k is the pose of the kth iteration, ΔT k is the predicted pose correction, T represents the actual pose of the object, and T 0 represents the initial pose output by the pose estimation network.

[0038] As a further solution of the present invention, in step S1: the geometric feature extraction network uses the PointNet variant structure in DenseFusion to extract geometric features, inputs a point cloud of size N×3, where N represents the number of point clouds, and outputs a geometric feature F of size N×3. P .

[0039] The present invention provides a 6D pose estimation system for an object based on a multi-channel attention mechanism, comprising:

[0040] The semantic segmentation module uses semantic segmentation to crop the color image and depth image of the target object from the input RGB image and depth image, and converts the depth image into a point cloud image using the camera intrinsic parameters;

[0041] The feature extraction module uses a color feature extraction network and a geometric feature extraction network to obtain color features and point cloud features respectively;

[0042] The pose estimation module inputs color features and point cloud features into the DenseFuison-DA2 network to obtain the object pose.

[0043] Compared with the prior art, the advantages and positive effects of the present invention are:

[0044] In the present invention, by introducing two attention modules of position and channel, a multi-channel attention mechanism DA2Net is designed, which has advantages in correlating features and redistributing global information, while enhancing its ability to capture local and global contextual information. On the basis of DenseFusion, this attention mechanism is added to the feature fusion network and the iterative refinement network to enhance the understanding of local details and infer the occluded parts from the global information. The multi-channel attention mechanism DA2Net of the present invention can better associate features, redistribute global information, and enhance the ability to capture local and global contextual information. Compared with the prior art, this innovation in processing logic and participation can more accurately understand local details and infer occluded parts from global information, thereby improving the accuracy of detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the DenseFusion-DA2 network structure diagram in the present invention;

[0046] Figure 2 This is the structural diagram of the DA2Net attention module in the present invention;

[0047] Figure 3 This is a flowchart of the iterative refinement process in the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] See also Figure 1-3 The present invention provides a technical solution: a 6D pose estimation method for an object based on a multi-channel attention mechanism provides a DenseFuison-DA2 network, and the DenseFuison-DA2 network includes:

[0050] S1, semantic segmentation and feature extraction, for the input RGB-D image, the color image of the target object is cut out from the input RGB image and depth image through semantic segmentation And depth map, and use the camera internal parameters to convert the depth map into a point cloud map Then use the color feature extraction network and the geometric feature extraction network to obtain color features respectively. and point cloud features

[0051] S2, feature fusion network, combines color features and point cloud features As input, after convolution processing, the outputs of the first and second layers of convolution are fused through channels to generate fusion features. and fusion features Then, the fusion feature F 2 Input into the DA2Net attention module to capture local and global context information, assign global features to each position on the feature map aggregated by position and channel attention, and obtain feature information in the image;

[0052] The pixel-level feature fusion network performs multi-layer convolution on the two features and fuses them layer by layer to finally obtain the pixel features for subsequent pose estimation. In the multi-layer convolution process, the occluded area may appear outside the receptive field of the convolution kernel or the local information may be incomplete due to the occlusion problem, making the convolution operation unable to fully capture the features of the occluded area, resulting in the loss of local features. In order to improve the information aggregation ability of the network, the DA2Net attention mechanism is added after the second layer of feature fusion.

[0053] After the features enhanced by the DA2Net attention module undergo two layers of convolution and average pooling, new fusion features are output. Then F 1 、F 2 and F 3 Generate pixel-level fusion features by splicing in the channel dimension

[0054] S3, pose estimation network, inputs the above pixel-level fusion feature F into the neural network for pose estimation, outputs the angular offset parameter, translation parameter and confidence of each pixel point, and obtains the object pose.

[0055] In this embodiment, in step S1: the encoder of the color feature extraction network is Resnet18, the decoder is 4 upsampling layers, and the color image of size H×W×3 is mapped to a color image of size H×W×d rgb In the space of , each pixel is a d rgb dimensional vector, indicating the color characteristics of the target object’s location. The segmentation framework in PoseCNN is used.

[0056] In this embodiment, in step S2, the fusion feature F 2 Input into the DA2Net attention module to capture local and global context information, assign global features to each position on the feature map aggregated by position and channel attention, and obtain feature information in the image as follows:

[0057] Local features are rich in images, reflect details, and are not easily affected by occlusion. Global features are a holistic representation of image information, representing image content from a large scale. Both have their own advantages. In order to more efficiently obtain and utilize local and global features, we combine the advantages of both, take A2-Net as the main body, introduce the channel attention module CAM and the pose attention module PAM in the feature distribution step, and propose the DA2Net attention module.

[0058] Based on the fusion feature F 2 Add another dimension to get fusion features Where H = 1, W = N, the fused feature X is input into the DA2Net attention module;

[0059] The structure of the DA2Net attention module includes a key feature collection branch and a dual-channel feature allocation branch. The inputs of the key feature collection branch and the dual-channel feature allocation branch are both fused features X;

[0060] In the key feature collection branch, first, the fused feature X is reshaped into two feature maps:

[0061] Then, feature map A and feature map B are input into bilinear pooling, and the feature vector pair (a i ,b i ) do the outer product and add them, the key feature collection branch is defined as:

[0062] G gather (X) = G bilinear (A,B) = A softmax(B) T

[0063] Output variables is the feature set G;

[0064] In the dual-channel feature allocation branch, the PAM module and the CAM module are introduced, and the output features of the two modules are aggregated to obtain the feature map V. Finally, the features in the feature set G are assigned to the feature map V, and the feature map Z is output. The fused feature X is used as the input of the two modules. First, for the PAM module, the input fused feature X first passes through a convolutional layer and reshapes to obtain two new feature maps. and feature map Where N = H × W, N is the number of pixels, then the feature map C and the feature map D are transposed and matrix multiplied, and the spatial attention map is obtained through softmax The formula is:

[0065]

[0066] where s jiDescribes i th Position pair th At the same time, feature X passes through another convolutional layer and reshapes to obtain a new feature map Then perform matrix multiplication of the feature map E and the transpose of the spatial attention map S, and finally multiply it by a scale parameter α and perform weighted summation with the feature X to obtain the output M. The formula is:

[0067]

[0068] Among them, α is initialized to 0 and gradually learns to assign more weights. For the CAM module, the channel attention map is directly calculated by inputting feature X The formula is:

[0069]

[0070] where k ji Describes th Channel pair j th In addition, the transpose of K and the feature X are matrix multiplied and reshaped into Then multiply it by a scaling factor β and perform a weighted summation with feature X to get the final result The formula is:

[0071]

[0072] The β is initialized to 0 and gradually learns to get a larger weight. Then the output features of the two modules are summed element by element to complete feature fusion and reshaped into features. The formula is:

[0073] v i =Sum(M j ,N j )

[0074] Combine the key feature collection branch and the dual-channel feature allocation branch, distribute the features in the feature set G to each position of the feature V, and output the final feature map Z of the module. The formula is:

[0075]

[0076] Get the feature map Z.

[0077] In this embodiment, the DA2Net attention module uses the following formula:

[0078] Z=F distr (G gather (X),V)

[0079] =G gather (X)Sum(Mj ,N j )

[0080] =A softmax(B) T Sum(M j ,N j )

[0081] Among them, Sum is element-by-element addition.

[0082] In this embodiment, in step S3, the object pose is improved by iterative refinement of the original network based on the DA2Net attention module. In each iteration, the extracted color feature F in the main network is reused. I , to obtain a high-precision 6D pose. After n iterations, the final pose is:

[0083]

[0084] Among them, T k is the pose of the kth iteration, ΔT k is the predicted pose correction, T represents the actual pose of the object, and T 0 represents the initial pose output by the pose estimation network.

[0085] After obtaining the pose estimation result, the iterative refinement of the original network is improved based on the above-mentioned DA2Net attention module to optimize the initially obtained estimated pose. The CNN-based iterative refinement is used, but the local feature loss problem will reappear in the original iterative refinement due to the use of multi-layer convolution operations, thus limiting the effect of the pose iterative refinement. Therefore, before obtaining the global features, the introduction of the DA2Net attention mechanism can effectively deal with this problem and can quickly and robustly refine the pose.

[0086] In this embodiment, in step S1: the geometric feature extraction network uses the PointNet variant structure in DenseFusion to extract geometric features, inputs a point cloud of size N×3, where N represents the number of point clouds, and outputs a geometric feature F of size N×3. P .

[0087] The present invention provides a 6D pose estimation system for an object based on a multi-channel attention mechanism, comprising:

[0088] The semantic segmentation module uses semantic segmentation to crop the color image and depth image of the target object from the input RGB image and depth image, and converts the depth image into a point cloud image using the camera intrinsic parameters;

[0089] The feature extraction module uses a color feature extraction network and a geometric feature extraction network to obtain color features and point cloud features respectively;

[0090] The pose estimation module inputs color features and point cloud features into the DenseFuison-DA2 network to obtain the object pose.

Claims

1. Object 6D pose estimation method based on multi-channel attention mechanism, It is characterized in that The method comprises the following steps: S1, semantic segmentation and feature extraction, for the input RGB-D image, the color image of the target object is cut out from the input RGB image and depth image through semantic segmentation And depth map, and use the camera internal parameters to convert the depth map into a point cloud map Then use the color feature extraction network and the geometric feature extraction network to obtain color features respectively. and point cloud features S2, feature fusion network, combines the color features and point cloud features As input, after convolution processing, the outputs of the first and second layers of convolution are fused through channels to generate fusion features. and fusion features Then, the fusion feature F 2 The input is sent to the DA2Net attention module to capture local and global context information, and the global features are assigned to each position on the feature map aggregated by position and channel attention to obtain the feature information in the image. The features enhanced by the DA2Net attention module are subjected to two layers of convolution operations and average pooling to output new fused features. Then F 1 、F 2 and F 3 Generate pixel-level fusion features by splicing in the channel dimension S3, pose estimation network, inputs the above pixel-level fusion feature F into the neural network for pose estimation, outputs the angular offset parameter, translation parameter and confidence of each pixel point, and obtains the object pose.

2. The object 6D pose estimation method based on the multi-channel attention mechanism according to claim 1, It is characterized in that In step S1: the encoder of the color feature extraction network is Resnet18, and the decoder is 4 upsampling layers, which maps the color image of size H×W×3 to a size of H×W×d rgb In the space of , each pixel is a d rgb dimensional vector, representing the color features of the target object's location.

3. The object 6D pose estimation method based on the multi-channel attention mechanism according to claim 1, It is characterized in that In step S2, the fusion feature F 2 Input into the DA2Net attention module to capture local and global context information, assign global features to each position on the feature map aggregated by position and channel attention, and obtain feature information in the image as follows: Based on the fusion feature F 2 Add another dimension to get fusion features Where H = 1, W = N, the fused feature X is input into the DA2Net attention module; The structure of the DA2Net attention module includes a key feature collection branch and a dual-channel feature allocation branch, and the inputs of the key feature collection branch and the dual-channel feature allocation branch are both fusion features X; In the key feature collection branch, first, the fused feature X is reshaped into two feature maps: Then, feature map A and feature map B are input into bilinear pooling, and the feature vector pair (a i ,b i ) do the outer product and add them, the key feature collection branch is defined as: G gather (X)=G bilinear (A,B)=A·softmax(B) T Output variables is the feature set G; In the dual-channel feature allocation branch, the PAM module and the CAM module are introduced, and the output features of the two modules are aggregated to obtain the feature map V. Finally, the features in the feature set G are assigned to the feature map V, and the feature map Z is output. The fused feature X is used as the input of the two modules. First, for the PAM module, the input fused feature X first passes through a convolutional layer and reshapes to obtain two new feature maps. and feature map Where N = H × W, N is the number of pixels, then the feature map C and the feature map D are transposed and matrix multiplied, and the spatial attention map is obtained through softmax The formula is: where s ji Describes i th Position pair th At the same time, feature X passes through another convolutional layer and reshapes to obtain a new feature map Then perform matrix multiplication of the feature map E and the transpose of the spatial attention map S, and finally multiply it by a scale parameter α and perform weighted summation with the feature X to obtain the output M. The formula is: Among them, α is initialized to 0 and gradually learns to assign more weights. For the CAM module, the channel attention map is directly calculated by inputting feature X The formula is: where k ji Describes th Channel pair j th In addition, the transpose of K and the feature X are matrix multiplied and reshaped into Then multiply it by a scaling factor β and perform a weighted summation with feature X to get the final result The formula is: The β is initialized to 0 and gradually learns to get a larger weight. Then the output features of the two modules are summed element by element to complete feature fusion and reshaped into features. The formula is: v i =Sum(M j ,N j ) Combine the key feature collection branch and the dual-channel feature allocation branch, distribute the features in the feature set G to each position of the feature V, and output the final feature map Z of the module. The formula is: Get the feature map Z.

4. The object 6D pose estimation method based on the multi-channel attention mechanism according to claim 3, Features: The DA2Net attention module uses the following formula: Z=F distr (G gather (X),V) =G gather (X)Sum(M j ,N j ) <h2 style=";text-align:left;direction:ltr">=A·softmax(B)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> Sum(M<h2 style=";text-align:left;direction:ltr"> j <h2 style=";text-align:left;direction:ltr"> ,N<h2 style=";text-align:left;direction:ltr"> j <h2 style=";text-align:left;direction:ltr"> ) Among them, Sum is element-by-element addition.

5. The object 6D pose estimation method based on the multi-channel attention mechanism according to claim 1, It is characterized in that In step S3, the object pose is improved based on the iterative refinement of the original network by the DA2Net attention module. During each iteration, the extracted color features F in the main network are reused. I , to obtain a high-precision 6D pose. After n iterations, the final pose is: Among them, T k is the pose of the kth iteration, ΔT k is the predicted pose correction, T represents the actual pose of the object, and T 0 represents the initial pose output by the pose estimation network.

6. The object 6D pose estimation method based on the multi-channel attention mechanism according to claim 1, It is characterized in that In step S1: the geometric feature extraction network uses the PointNet variant structure in DenseFusion to extract geometric features, inputs a point cloud of size N×3, where N represents the number of point clouds, and outputs a geometric feature F of size N×3. P .

7. Object 6D pose estimation system based on multi-channel attention mechanism, It is characterized in that The method for estimating 6D pose of an object based on a multi-channel attention mechanism according to any one of claims 1 to 6 is performed, comprising: The semantic segmentation module uses semantic segmentation to crop the color image and depth image of the target object from the input RGB image and depth image, and converts the depth image into a point cloud image using the camera intrinsic parameters; The feature extraction module uses a color feature extraction network and a geometric feature extraction network to obtain color features and point cloud features respectively; The pose estimation module inputs color features and point cloud features into the DenseFuison-DA2 network to obtain the object pose.

Images