Rigid body 6d pose estimation method and system based on feature fusion and codec structure

Abstract

The application discloses a rigid body 6D pose estimation method based on feature fusion and codec structure, which comprises the following steps: using a deep learning method to extract color features and geometric features of an object (calculating the geometric features to directly obtain an offset T), and performing feature fusion after processing to obtain fused features; then, the fused features are sequentially sent into an attention encoder and a residual decoder to obtain a rotation vector of a key point; finally, a rotation matrix R is calculated through a Kabsch formula. The application also discloses a rigid body 6D pose estimation system based on feature fusion and codec structure. The application brings special feature extraction means, an attention mechanism, a residual structure and the codec thought into 6D pose estimation, improves the calculation efficiency while ensuring the accuracy, and meets the real-time requirement of a robot grabbing environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and is a rigid body 6D pose estimation method and system based on feature fusion and encoder-decoder structure. BACKGROUND

[0002] High-precision pose estimation of objects is crucial for robot applications and has attracted interest in the research community in recent years. Detecting objects and their 6D poses (3D position and orientation) using vision is an important task for many robot applications, including object manipulation (e.g., picking and placing), part assembly, and so on. In cluttered environments, especially when the target object is occluded, the ability of the machine to determine the centroid of the target object may decrease. Therefore, using a 6D pose estimation method will effectively overcome this difficulty. In this context, it is of great significance to study a method that helps robots quickly and accurately identify the 6D pose of the target object from a cluttered and stacked environment.

[0003] Classic 6D pose estimation methods are limited in their detection performance under severe occlusion and illumination changes due to their reliance on handcrafted features and template matching. In recent years, researchers' research results in the field of target detection have successfully inspired a series of data-driven methods. These methods use RGB-D as input for 6D pose estimation and achieve good results on related datasets.

[0004] The drawback is that most mainstream methods use methods such as PnP and ICP to refine the pose estimation matrix at the end of the neural network. Such methods increase the time of model inference, which is fatal for machine operating environments that require real-time detection and estimation. Therefore, seeking a real-time and fast 6D pose estimation method has become a problem that needs to be solved to improve the efficiency of robot operation. SUMMARY

[0005] The present application overcomes the above-mentioned shortcomings of the prior art and provides a rigid body 6D pose estimation method and system based on feature fusion and encoder-decoder structure. This method can greatly improve the grasping speed while ensuring the accuracy of pose estimation.

[0006] To achieve the above-mentioned purposes, the technical solution adopted by the present application is to increase input information through feature fusion and replace ordinary refinement modules with an encoder-decoder structure. The rigid body 6D pose estimation method based on feature fusion and encoder-decoder structure includes the following steps:

[0007] S1: Extracting color features and geometric features of the rigid object, specifically including:

[0008] S11: Design of rigid object color feature extraction part: use pyramid VGG network model to extract color feature F of rigid object color , which is specifically defined as:

[0009] F color = Pyramid(image) (1)

[0010] Wherein, Pyramid() represents the pyramid VGG network model, and image represents the input image data;

[0011] S12: Design of rigid object geometry feature extraction part: use the combination network of convolution and max pooling layer to extract the geometry feature F of rigid object geometry , which is specifically defined as:

[0012] F geometry = M p (CNNs f (points)) (2)

[0013] Wherein, M p () represents the Max pooling layer; CNNs f () represents the convolution layer for extracting point cloud feature, points represents the input point cloud data, and M p () has the same meaning as above;

[0014] S13: After extracting the geometry feature, the offset T of the rigid object can be directly estimated, which is specifically defined as:

[0015] T = CNNs t (F geometry ) (3)

[0016] Wherein, CNNs t () represents the convolution layer for calculating T;

[0017] S14: The definition of loss function L T about T is as follows:

[0018]

[0019] Wherein, MSE() represents the mean square error loss, represents the label offset of the object, and T represents the predicted offset; MSE() has the same meaning as above;

[0020] S2: Fusion of color feature and geometry feature of rigid object, specifically including:

[0021] S21: The extracted color feature and geometry feature are respectively processed through the max pooling layer to obtain the global feature F global, which are defined as follows:

[0022] F global = Concat(M p (F color ), M p (F geometry )) (5)

[0023] wherein, ConCat() represents concatenating the feature matrices in the parentheses in the channel dimension; ConCat() has the same meaning as follows;

[0024] S22: generating a fusion feature F fusion , which is defined as follows:

[0025] F fusion = Concat(F geometry , F global , F color ) (6)

[0026] S3: performing attention encoding on the feature information F fusion , specifically including:

[0027] S31: sequentially passing F fusion through a channel attention module Attention channel and a spatial attention module Attention spatial , to obtain a channel attention module calculation result F channel and a spatial attention module calculation result F spatial respectively, which are defined as follows:

[0028] F channel = F fusion x Attention channel (F fusion ) (7)

[0029] F spatial = F channel x Attention spatial (F channel ) (8)

[0030] S32: sequentially concatenating the results of the perception passed through the perception F spatial after passing through a plurality of perception modules MLPs and a max-pooling layer in the channel dimension, to obtain a concatenation result F attention .

[0031] S4: decoding F attention to obtain a predicted result point to key point direction vector and a key point rotation vector, specifically including:

[0032] S41: Fattention V is calculated by the key point vector prediction pipeline, and a loss function L related to V v is defined as follows:

[0033]

[0034] wherein, represents the vector of the point in the label to the key point direction, and V represents the predicted result point to the key point direction vector;

[0035] S42: V and F attention are input into the rotation prediction pipeline after being spliced in the channel dimension to obtain R p , and a corresponding rotation loss L p is defined as:

[0036]

[0037] wherein, represents the key point vector in the label, and R p represents the predicted key point rotation vector;

[0038] S43: F attention is input into the residual prediction pipeline, and the result is added to R p to obtain R r , and a corresponding residual loss L r is defined as:

[0039]

[0040] wherein, represents the result predicted by the residual pipeline;

[0041] S5: the initial position vector C of the known key point and the final key point vector R r predicted can be used to calculate the rigid object rotation matrix R by the Kabsch formula;

[0042] S6: the offset T and the rotation matrix R required to be predicted are obtained from S13 and S5, so as to finally obtain the 6D pose of the rigid body.

[0043] The system for implementing the rigid body 6D pose estimation method based on feature fusion and codec structure of the application comprises:

[0044] a color feature and geometric feature extraction module of a rigid object, configured to extract color features and geometric features of the rigid object;

[0045] a color feature and geometric feature fusion module of a rigid object, configured to fuse the color features and geometric features of the rigid object;

[0046] An attention encoding module is configured to encode the feature information F fusion .

[0047] A decoding module is configured to decode F attention to obtain a predicted point-to-keypoint direction vector and a keypoint rotation vector.

[0048] A rigid object rotation matrix module is configured to calculate a rigid object rotation matrix R by using the Kabsch formula based on the known keypoint initial position vector C and the predicted final keypoint vector R r .

[0049] A rigid 6D pose module is configured to obtain a predicted offset T and a rotation matrix R by using S13 and S5, so as to finally obtain a rigid 6D pose.

[0050] The present application brings the special feature extraction method, attention mechanism, residual structure and the idea of encoder-decoder into the 6D pose estimation, improves the calculation efficiency while ensuring the accuracy, and meets the real-time requirement of the robot grasping environment.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] 1. The image information and point cloud information of vision are fully utilized, and the model robustness is increased.

[0053] 2. The attention encoder and the residual decoder are combined, the former makes the feature information extraction more sufficient, and the latter replaces the ordinary subsequent refinement module, and the model inference speed is improved. DETAILED DESCRIPTION

[0054] Figure 1 The network structure diagram of the method of the present application is shown.

[0055] Figure 2 The network structure diagram of the pyramid VGG network in the method of the present application is shown.

[0056] Figure 3 The network structure diagram of the encoder-decoder in the method of the present application is shown. DETAILED DESCRIPTION

[0057] Embodiment 1

[0058] The present application will be further described below with reference to the accompanying drawings. Figures 1-3 The present application will be further described below with reference to the accompanying drawings.

[0059] The network structure of the method of the present application is shown in the accompanying drawings. Figure 1 The rigid 6D pose estimation method based on feature fusion and encoder-decoder structure comprises the following steps:

[0060] S1: Extract the color feature and the geometric feature of the rigid object, specifically including:

[0061] S11: Design of the color feature extraction part of the rigid object: use the pyramid VGG network model to extract the color feature F of the rigid object color , which is specifically defined as:

[0062] F color = Pyramid(image) (1)

[0063] Wherein, Pyramid() represents the pyramid VGG network model, and image represents the input image data;

[0064] S12: Design of the geometric feature extraction part of the rigid object: use the combination network of convolution and max pooling layer to extract the geometric feature F of the rigid object geometry , which is specifically defined as:

[0065] F geometry = M p (CNNs f (points)) (2)

[0066] Wherein, M p () represents the Max pooling layer. CNNs f () represents the convolution layer for extracting the point cloud feature, and points represents the input point cloud data; M p () has the same meaning as above;

[0067] S13: After the geometric feature is extracted, the offset T of the rigid object can be directly estimated, which is specifically defined as:

[0068] T = CNNs t (F geometry ) (3)

[0069] Wherein, CNNs t () represents the convolution layer for calculating T;

[0070] S14: The definition of the loss function L T of T is as follows:

[0071]

[0072] Wherein, MSE() represents the mean square error loss, represents the label offset of the object, and T represents the predicted offset; MSE() has the same meaning as above;

[0073] S2: Fusion of the color feature and the geometric feature of the rigid object, specifically including:

[0074] S21: the extracted color features and geometric features are respectively processed by a max-pooling layer to obtain global features F global , and are specifically defined as follows:

[0075] F global = Concat (M p (F color ), M p (F geometry )) (5)

[0076] wherein ConCat () represents concatenation of the feature matrices in the parentheses in the channel dimension; ConCat () has the same meaning;

[0077] S22: a fusion feature F fusion is generated, and is specifically defined as follows:

[0078] F fusion = Concat (F geometry , F global , F color ) (6)

[0079] S3: the feature information F fusion is encoded by attention;

[0080] S4: decoding is performed to obtain a prediction result;

[0081] S5: from the known key point initial position vector C and the predicted final key point vector R r , a rigid object rotation matrix R can be calculated by the Kabsch formula;

[0082] S6: from S13 and S5, the required predicted offset T and rotation matrix R are obtained, so as to finally obtain the 6D pose of the rigid body.

[0083] The pyramid VGG network structure is shown in Figure 2 . Among them, VGG represents the network module in the manner of extracting features by the VGG network. After the two branches are respectively up-sampled to the original size, they are added, and then a convolution and a normalization operation are performed to obtain the extracted feature information.

[0084] The network structure of the encoder is shown in Figure 3 . The fusion feature F fusion is sequentially subjected to a channel attention module Attention channel and a spatial attention module Attention spatialand build the encoder-decoder according to the network structure shown in the figure after corresponding calculation. Wherein, the "point-to-key point vector" refers to the direction vector of each point in the point cloud to the key point (8 corner points of the 3D bounding box) of the object, that is, the content represented by "V" in the figure. The content represented by "R" in the figure is the vector of the predicted key point, that is, the three-dimensional coordinates of the 8 corner points in the current coordinate system.

[0085] Embodiment 2

[0086] The system for implementing the rigid body 6D pose estimation method based on feature fusion and encoder-decoder structure of embodiment 1 comprises:

[0087] A color feature and geometric feature extraction module for rigid body object is configured to extract the color feature and geometric feature of the rigid body object.

[0088] A color feature and geometric feature fusion module for rigid body object is configured to fuse the color feature and geometric feature of the rigid body object.

[0089] An attention encoding module is configured to perform attention encoding on the feature information F fusion .

[0090] A decoding module is configured to decode F attention to obtain the predicted point-to-key point direction vector and the key point rotation vector.

[0091] A rigid body object rotation matrix module is configured to calculate the rigid body object rotation matrix R by the known key point initial position vector C and the predicted final key point vector R r through the Kabsch formula.

[0092] A rigid body 6D pose module is configured to obtain the required predicted offset T and rotation matrix R from S13 and S5, so as to finally obtain the 6D pose of the rigid body.

[0093] The content described in the specification examples is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as limited to the specific forms stated in the examples, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1.A rigid body 6D pose estimation method based on feature fusion and codec structure, comprising the following steps: S1: extracting color features and geometric features of a rigid object; S2: fusing the color features and the geometric features of the rigid object; S3: encode the feature information F fusion with attention; S4: to F attention are decoded to get the predicted point-to-keypoint direction vectors and keypoint rotation vectors; S5: initial position vector C of known key points and predicted final key point vector R r The rigid object rotation matrix R is calculated by the Kabsch formula; S6: obtaining the required predicted offset T and rotation matrix R from S13 and S5, thereby finally obtaining the 6D pose of the rigid body; Step S1 specifically comprises: S11: Design of rigid object color feature extraction part: use the pyramid VGG network model to extract the color feature F of the rigid object color , specifically defined as: F color = Pyramid(image) (1) Wherein, Pyramid() represents a pyramid VGG network model, and image represents input image data; S12: Design of the rigid object geometry feature extraction part: use a combination network of convolution and max pooling layer to extract the geometry feature F of the rigid object geometry , is specifically defined as: F geometry = M p (CNNs f (points)) (2) wherein M p () denotes a Max pooling layer; CNNs f () denotes a convolutional layer for extracting point cloud features, points represent the input point cloud data; S13: After the geometric features are extracted, the offset T of the rigid object can be directly estimated, which is specifically defined as: T = CNNs t (F geometry ) (3) where CNNs t () denotes a convolutional layer for computing T; S14: Loss function L with respect to T T is defined as follows: where MSE() represents a mean square error loss, represents a label offset of the object, and T represents a predicted offset. S2 specifically comprises: S21: the extracted color features and geometric features are respectively processed by a max-pooling layer to obtain global features F global are defined as follows: F global = Concat(M p (F color ),M p (F geometry )) (5) Wherein, ConCat() represents concatenating the feature matrices in the parentheses in the channel dimension; S22: generate the fusion feature F fusion , and is specifically defined as follows: F fusion = Concat(F geometry ,F global ,F color ) (6) Step S3 specifically comprises: S31: F fusion passing through the channel attention module Attention channel and the spatial attention module Attention spatial , respectively, to obtain the channel attention module calculation result F channel and the spatial attention module calculation result F spatial , which are specifically defined as follows: F channel = F fusion Attention channel (F fusion ) (7) F spatial = F channel Attention spatial (F channel ) (8) S32: let F spatial After passing through several perceptual MLPs and max-pooling layers, the results of the perceptual passing are sequentially concatenated in the channel dimension to obtain a concatenation result F attention ; Step S4 specifically comprises: S41: F attention Compute V by keypoint vector prediction pipeline v is defined as follows: wherein, V represents a vector of a point-to-keypoint direction in the label, and V represents a predicted point-to-keypoint direction vector. S42: V is concatenated with F attention After concatenation in the channel dimension, the input rotation prediction pipeline yields R p The corresponding rotation loss L p is defined as: wherein, represents a keypoint vector in a label, R p represents a predicted keypoint rotation vector; S43: F attention input residual prediction pipeline and the result is added to R p to obtain R r , the corresponding residual loss L r is defined as: wherein, represents the result of the residual pipeline prediction.

Images