Robust 6D pose estimation method based on bidirectional matching and global attention network

Abstract

The present invention discloses a robust 6D pose estimation method based on bidirectional matching and global attention network, comprising the following steps: S1, obtaining a panoramic image and a depth image, inputting them into a ConvNext network and a Pointnet network respectively to extract color features and depth point clouds, and extracting a feature matrix based on the color features, the depth point cloud and the model point cloud; S2, inputting the depth point cloud and its normal vector information and the model point cloud and its normal vector information into a global point cloud attention module to obtain a common attention map; S3, inputting the common attention map and the feature matrix into an attention-aware pose estimation module to obtain the final pose. The present invention introduces a simple and effective global point cloud attention module to find the similarity between model points and depth points, thereby obtaining robust attention correlation. Experiments show that the present invention has advantages over current methods, and the accuracy and robustness of the prediction results of the present invention are improved in 6D pose estimation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a robust 6D pose estimation method based on bidirectional matching and a global attention network. Background Art

[0002] The task of object 6D pose estimation is to calculate the 3D rotation and 3D translation of the target in the current scene relative to the fixed world coordinates of the object. It is an important basic problem in human-computer interaction applications such as augmented reality, autonomous driving, and robot grasping. The traditional target pose estimation method is to estimate the pose based on the point correspondence between the current image and the object model based on manual key point detection and feature matching. Its main disadvantage is that the feature design cannot learn to fit the data, which makes it very difficult to estimate the pose of featureless objects. In recent years, with the development of deep learning, network-based pose estimation has become more and more popular.

[0003] The one-stage method and the two-stage method are two commonly used methods in deep learning pose estimation. The one-stage method estimates the pose information of the object directly from the input image or point cloud, and usually uses an end-to-end method for training and reasoning. It has the advantages of high real-time performance and simple model, but due to the diversity of input data and the influence of factors such as noise, it has high requirements for the accuracy and robustness of pose estimation. The two-stage method decomposes the pose estimation task into two subtasks: object detection and pose regression. First, the position of the object is detected in the input image or point cloud, and then the pose of the object is regressed. It has the advantages of high accuracy and good robustness, but it takes more computing time and storage space, and requires multiple models to work together, so it is usually more complicated than the one-stage method.

[0004] In these works, the methods vary greatly according to the input data, which can be mainly divided into methods using only RGB, only using depth information, and using both RGB and depth information. Traditional methods usually use template matching to find sub-images similar to the template image in another image based on the known template image, and then use the PnP algorithm to regress the 6D pose. However, this method cannot handle objects without texture. With the development of deep learning, some methods based on RGB data have developed density predictions for 2D-3D correspondences and estimated poses through the PnP-RANSAC algorithm. Some methods also consider self-occluded multi-view information or multi-part information of the target to predict more robust pose estimates. However, most of these methods are based on 2D projections. In the real three-dimensional space, the projection error of the key points on the 2D image is small, but the error in the corresponding three-dimensional space may be large, ignoring the geometric information of the model. On the other hand, different key points in the three-dimensional space may overlap in the projected two-dimensional space, which makes it difficult for methods that use two-dimensional images to predict poses to distinguish these points.

[0005] Some methods based on depth information use 3D convolutional networks or point cloud networks for feature extraction and 3D bounding box prediction. The recent cloudAAE attempts to reconstruct point cloud segments with the desired 6D pose to learn the latent code encoding 6D pose information and use it as input to regress pose parameters. However, the sparsity and non-texture of point clouds limit the performance of these methods. Moreover, depth images cannot capture the color features of the object surface, which also limits the performance of networks using only deep images.

[0006] To solve these problems, some methods try to extract the initial pose from the RGB image and then use the ICP or MCN algorithm to perform local registration of the point cloud. However, these algorithms are very time-consuming and cannot be optimized end-to-end. Therefore, some works based on RGB-D data use CNN and point cloud networks to extract features from RGB images and point clouds respectively, and then fuse these features for pose estimation and achieve good results. Recent methods have further improved and optimized the DenseFusion network. However, these methods do not fully utilize the prior information of the model point cloud. The subsequent BiCo-Net network uses a bidirectional mapping module to add the model point cloud information to the network for training, but does not consider global constraints on the point pair matching stage, which may cause the network to be unable to effectively handle noise during local registration, because global modeling can give a general description of the object and thus improve the stability of the object recognition system. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the robust 6D pose estimation method based on bidirectional matching and global attention network provided by the present invention solves the problems that the feature fusion method of the existing 6D pose estimation scheme does not cover comprehensive information, the global features and local features are not fully utilized, and the local matching of the bidirectional mapping network is not globally optimized.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a robust 6D pose estimation method based on bidirectional matching and global attention network, comprising the following steps:

[0009] S1. Obtain a panoramic image and a depth image, input them into the ConvNext network and the Pointnet network respectively to extract color features and depth point clouds, and extract feature matrices based on color features, depth point clouds and model point clouds;

[0010] S2, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the global point cloud attention module to obtain a common attention map;

[0011] S3. Input the common attention map and feature matrix into the attention-aware pose estimation module to obtain the final pose.

[0012] Further: in said S1, the feature matrix includes a dense feature matrix and a model feature matrix;

[0013] The S1 comprises the following sub-steps:

[0014] S11, obtaining a panoramic image and a depth image, and cutting out a region of interest from the panoramic image and the depth image;

[0015] S12, converting the depth image into scene point cloud information according to the region of interest of the panoramic image and the camera internal parameters, and inputting the region of interest of the panoramic image into the ConvNext network to obtain color features;

[0016] S13, randomly extract N points from the depth point cloud, extract the geometric features of the N points through the Pointnet network, and connect the geometric features with the color features to obtain a dense feature matrix;

[0017] S14. Randomly extract N points from the model point cloud, and extract the model feature matrix of the N points through a multi-layer perceptron.

[0018] Further: in S2, the global point cloud attention module includes a pseudo twin network module, a dot product function layer and a softmax function layer connected in sequence;

[0019] The S2 comprises the following sub-steps:

[0020] S21, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the pseudo twin network module respectively to obtain the depth attention feature matrix and the model attention feature matrix;

[0021] S22. Input the deep attention feature matrix and the model attention feature matrix into the dot product function layer to obtain the correlation feature matrix, and input the correlation feature matrix into the softmax function layer to obtain the common attention map for constraining point pair matching.

[0022] Further: the S21 is specifically:

[0023] The depth point cloud and its normal vector information and the model point cloud and its normal vector information are input twice at different positions through the pseudo twin network module to obtain the depth attention feature matrix and the model attention feature matrix.

[0024] Among them, the method for processing input by the pseudo twin network module is the same, and the method for processing the depth point cloud and its normal vector information and the model point cloud and its normal vector information by the pseudo twin network module is specifically:

[0025] SA1, using a multi-layer perceptron to upgrade the 6-dimensional depth point cloud and its normal vector information and the model point cloud and its normal vector information to 64 dimensions and 128 dimensions, to obtain the 64-dimensional first to second local features and the 128-dimensional third to fourth local features;

[0026] SA2, locally fuse the first and second local features of 64 dimensions to obtain the fifth local feature of 128 dimensions;

[0027] SA3, locally fuse the third to fourth local features of 128 dimensions to obtain a sixth local feature of 256 dimensions, encode the sixth local feature to obtain a seventh local feature of 1024 dimensions;

[0028] SA4, average pool the 1024-dimensional seventh local feature to obtain the global feature, densely fuse the global feature with the 128-dimensional fifth local feature, and decode and normalize the dense fusion result to obtain a deep attention feature matrix with both local feature information and global feature information;

[0029] SA5. Use the pseudo-twin network to exchange the positions of the depth point cloud and its normal vector information with the model point cloud and its normal vector information input, repeat SA1 to SA4, and obtain the model attention feature matrix with both local feature information and global feature information.

[0030] Further: In S22, the relevant feature matrix R is obtained m,n The specific expression is:

[0031] R m,n =g(F sa ,F ma )

[0032] In the formula, g(·) represents the dot product function, F sa represents the deep attention feature matrix, F ma Represents the model attention feature matrix;

[0033] Get the common attention map M m*n The specific expression is:

[0034]

[0035] In the formula, R i,n Represents the i-th element in the n-th column of the correlation feature matrix, R k,n Represents the kth element of the nth column in the correlation feature matrix, m represents the number of rows of the matrix, and n represents the number of columns of the matrix.

[0036] Further: In S22, the obtained common attention map is constrained by attention loss, and the attention loss L attentionThe specific expression is:

[0037]

[0038] In the formula, w total represents the feature constraint weight of the point pair, M m*n represents the joint attention map, m represents the number of rows in the matrix, and n represents the number of columns in the matrix;

[0039] The expression of the point pair feature constraint weight is specifically:

[0040] w total =γ 1 d m,n +γ 2 θ d,N +γ 3 θ N

[0041] In the formula, γ 1 , γ 2 and γ 3 Both represent weight parameters, d m,n represents the point pair distance matrix, θ d,N Represents the angle between the distance vector and the normal vector, θ N Represents the normal vector angle.

[0042] Further: the point pair distance matrix d m,n The specific expression is:

[0043]

[0044] In the formula, Indicates the mth depth point cloud i Points, Respectively represent the mth depth point cloud i The x-axis, y-axis, and z-axis coordinate values ​​of the points. Represents the mth depth point cloud i The model point cloud that matches the nth point j Points, Respectively represent the nth point cloud of the model j The x-axis, y-axis and z-axis coordinate values ​​of each point;

[0045] The angle θ between the distance vector and the normal vector d,N The angle θ between the normal vector N The specific expression is:

[0046]

[0047]

[0048] Where arccos(·) represents the inverse cosine function, N m Indicates the mth depth point cloud i The normal vector of the point, N n Indicates the model point cloud nth j The normal vector of a point, ||·|| represents the L2 norm, (·) T Indicates transpose.

[0049] Further: in said S3, the attention-aware pose estimation module includes first to third regression branches;

[0050] The first regression branch and the third regression branch each include a multilayer perceptron, a connection operation and a multilayer perceptron regressor connected in sequence, and the second regression branch includes a multilayer perceptron and a connection operation connected to each other;

[0051] The S3 comprises the following sub-steps:

[0052] S31, inputting the dense feature matrix as input features into the first regression branch to obtain point pair matching from scene points to model points, and inputting the model feature matrix as input features into the third regression branch to obtain point pair matching from model points to scene points;

[0053] Wherein, the point pair matching from scene point to model point and the point pair matching from model point to scene point include a plurality of predicted point pairs;

[0054] S32, inputting the dense feature matrix into the second regression branch, decoding the dense feature matrix through a multi-layer perceptron, and obtaining a candidate pose according to the decoded feature prediction;

[0055] S33, establishing a final loss function according to the set prediction loss, pose loss and attention loss, and calculating the first pose and the second pose from all predicted point pairs according to the final loss function through BiCo-net;

[0056] S34, merging the candidate pose, the first pose and the second pose, taking the average value after merging, and obtaining the final pose.

[0057] Further: in S31, the first regression branch and the third regression branch process the input features in the same way, and the method for processing the input features by the first regression branch is specifically:

[0058] The input features are input to the first regression branch, the input features are decoded by the multi-layer perceptron, the features of the decoded output are connected with the common attention map, and the connected results are input to the multi-layer perceptron regressor for corresponding prediction to obtain the point pair matching from the scene point to the model point.

[0059] Further: In S33, the prediction loss includes the image prediction loss function L s And the model prediction loss function L m , its specific expression is as follows:

[0060]

[0061]

[0062] Where ε represents the hyperparameter, N, i, x i represents the ground truth coordinates of the i-th point in the depth point cloud, represents the ground truth normal vector of the i-th point in the depth point cloud, Represents the coordinates of the corresponding prediction points generated by feature encoding and decoding, They represent the normal vectors of the corresponding predicted points generated by feature encoding and decoding, respectively, j represents the ground truth coordinates of the jth point in the model point cloud, represents the ground truth normal vector of the jth point in the model point cloud, Represents the coordinates of the corresponding prediction points generated after feature encoding and decoding, Represents the normal vector of the corresponding prediction point generated by feature encoding and decoding;

[0063] The pose loss L pose The specific expression is:

[0064]

[0065] Where R, t represents the ground truth pose, represents the predicted point-by-point pose, x k represents the kth point among K points randomly selected from the model point cloud of the object, K represents K points randomly selected from the model point cloud of the object, j represents the closest point, ||·|| represents the L2 norm, min j∈[1,K] ||·|| respectively represent the calculation of the L2 norm of all K points and the nearest point j;

[0066] The final loss function L total The specific expression is:

[0067]

[0068] Where, L attention Indicates loss of attention, Represents the balance parameter between loss functions;

[0069] In S34, the final posture T is obtained. final The specific expression is:

[0070] T final =average(T d ∪T s ∪T m )

[0071] Where, T d represents the candidate pose, T s represents the first pose, T m represents the second pose, ∪ represents the union, and average(·) represents the average function.

[0072] The beneficial effects of the present invention are:

[0073] (1) The present invention proposes a robust 6D pose estimation method based on bidirectional matching and global attention network, and introduces a simple and effective global point cloud attention module to find the similarity between model points and depth points, thereby obtaining robust attention correlation. Experiments show that the method of the present invention has advantages over the current state-of-the-art methods, and the accuracy and robustness of the prediction results of the present invention are improved in 6D pose estimation.

[0074] (2) The feature matrix extracted by the present invention enables the attention mechanism to better weight and focus on processing different feature channels, capturing the global, local information and different features of the object. The common attention map of the attention feature is predicted by the global point cloud attention module, which can combine the relationship between two different input data when generating attention weights, which can help the model better understand the interaction between input data and maintain consistency when processing multiple input data.

[0075] (3) The present invention performs global modeling while generating a common attention map, and uses the relative feature differences between the point pairs in the two point clouds as the weight-constrained feature attention map to obtain a rough description of the object, thereby improving the robustness of the target pose prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flow chart of the robust 6D pose estimation method based on bidirectional matching and global attention network of the present invention.

[0077] Figure 2 It is the main structural diagram of the network of the present invention.

[0078] Figure 3 This is a network structure diagram of the joint attention module of the present invention.

[0079] Figure 4 A graph is generated for the point-to-feature weight constraints of the present invention.

[0080] Figure 5Schematic diagram of visual comparison of the attention map of the present invention.

[0081] Figure 6 It is a comparison chart of different occlusion levels on the YCB-Video dataset of the present invention. DETAILED DESCRIPTION

[0082] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0083] Embodiment 1:

[0084] like Figure 1 As shown, in one embodiment of the present invention, a robust 6D pose estimation method based on bidirectional matching and global attention network includes the following steps:

[0085] S1. Obtain a panoramic image and a depth image, input them into the ConvNext network and the Pointnet network respectively to extract color features and depth point clouds, and extract feature matrices based on color features, depth point clouds and model point clouds;

[0086] S2, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the global point cloud attention module to obtain a common attention map;

[0087] S3. Input the common attention map and feature matrix into the attention-aware pose estimation module to obtain the final pose.

[0088] like Figure 2 As shown, the network of the present invention is composed of a ConvNext network, a global point cloud attention module and an attention-aware pose estimation module.

[0089] In S1, the feature matrix includes a dense feature matrix and a model feature matrix;

[0090] The S1 comprises the following sub-steps:

[0091] S11, obtaining a panoramic image and a depth image, and cutting out a region of interest from the panoramic image and the depth image;

[0092] S12, converting the depth image into scene point cloud information according to the region of interest of the panoramic image and the camera internal parameters, and inputting the region of interest of the panoramic image into the ConvNext network to obtain color features;

[0093] S13, randomly extract N points from the depth point cloud, extract the geometric features of the N points through the Pointnet network, and connect the geometric features with the color features to obtain a dense feature matrix;

[0094] S14. Randomly extract N points from the model point cloud, and extract the model feature matrix of the N points through a multi-layer perceptron.

[0095] In this embodiment, in each data set for the experiment, they not only provide RGB panoramic images and corresponding depth images, but also provide model files of all objects involved in this data set that need to estimate the pose. The model files contain the 3D rendering model and point cloud data of the object. The present invention randomly selects N points from the point cloud data of these models to form our model point cloud, and predicts their normal vectors using the open3d package based on this point cloud information, and then inputs them into the subsequent feature extraction stage. Therefore, each panoramic image and the corresponding depth image contain several objects. In order to evaluate the accuracy of the pose parameter prediction, the present invention needs to use the model point cloud and use the predicted pose to convert it into the scene coordinate system, and then compare the difference between it and the depth point cloud as the final judgment standard.

[0096] The ConvNext network can extract surface texture features from the panoramic image and then obtain color features. Since the region of interest cropped out by the present invention is small, the kernel size of the first convolutional layer of the ConvNext network is halved to retain more information.

[0097] In S2, the global point cloud attention module includes a pseudo twin network module, a dot product function layer and a softmax function layer connected in sequence;

[0098] The S2 comprises the following sub-steps:

[0099] S21, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the pseudo twin network module respectively to obtain the depth attention feature matrix and the model attention feature matrix;

[0100] S22. Input the deep attention feature matrix and the model attention feature matrix into the dot product function layer to obtain the correlation feature matrix, and input the correlation feature matrix into the softmax function layer to obtain the common attention map for constraining point pair matching.

[0101] In this embodiment, the present invention processes the depth point cloud and its normal vector information and the model point cloud and its normal vector information through a common attention module, so that the output common attention map can reflect the geometric similarity between point pairs without being affected by distribution differences.

[0102] like Figure 3As shown, in this embodiment, the S21 is specifically:

[0103] The depth point cloud and its normal vector information and the model point cloud and its normal vector information are input twice at different positions through the pseudo twin network module to obtain the depth attention feature matrix and the model attention feature matrix.

[0104] Among them, the method for processing input by the pseudo twin network module is the same, and the method for processing the depth point cloud and its normal vector information and the model point cloud and its normal vector information by the pseudo twin network module is specifically:

[0105] SA1, using a multi-layer perceptron to upgrade the 6-dimensional depth point cloud and its normal vector information and the model point cloud and its normal vector information to 64 dimensions and 128 dimensions, to obtain the 64-dimensional first to second local features and the 128-dimensional third to fourth local features;

[0106] SA2, locally fuse the first and second local features of 64 dimensions to obtain the fifth local feature of 128 dimensions;

[0107] SA3, locally fuse the third to fourth local features of 128 dimensions to obtain a sixth local feature of 256 dimensions, encode the sixth local feature to obtain a seventh local feature of 1024 dimensions;

[0108] SA4, average pool the 1024-dimensional seventh local feature to obtain the global feature, densely fuse the global feature with the 128-dimensional fifth local feature, and decode and normalize the dense fusion result to obtain a deep attention feature matrix with both local feature information and global feature information;

[0109] SA5. Use the pseudo-twin network to exchange the positions of the depth point cloud and its normal vector information with the model point cloud and its normal vector information input, repeat SA1 to SA4, and obtain the model attention feature matrix with both local feature information and global feature information.

[0110] The present invention uses a multi-layer perceptron to densely fuse point clouds and their normal vector information through a pseudo-twin network module to generate a deep attention feature matrix with both local feature information and global feature information. Similarly, a model attention feature matrix of dense fusion of model point clouds and normal vectors is generated.

[0111] In S22, the relevant feature matrix R is obtained. m,n The specific expression is:

[0112] R m,n =g(F sa ,F ma )

[0113] In the formula, g(·) represents the dot product function, Fsa represents the deep attention feature matrix, F ma Represents the model attention feature matrix;

[0114] Get the common attention map M m*n The specific expression is:

[0115]

[0116] In the formula, R i,n Represents the i-th element in the n-th column of the correlation feature matrix, R k,n Represents the kth element of the nth column in the correlation feature matrix, m represents the number of rows of the matrix, and n represents the number of columns of the matrix.

[0117] In S22, the obtained common attention map is constrained by the attention loss, and the attention loss L attemtion The specific expression is:

[0118]

[0119] In the formula, w total represents the feature constraint weight of the point pair, M m*n represents the joint attention map, m represents the number of rows in the matrix, and n represents the number of columns in the matrix;

[0120] like Figure 4 As shown, in this embodiment, in order to improve the accuracy and effectiveness of the generated joint attention map focusing on geometric features, attention loss is used as supervision to guide the joint attention map.

[0121] The principle of attention loss is as follows:

[0122] All points in the depth point cloud are converted to standard coordinates using the ground truth pose, and the mth point in the depth point cloud is converted to the standard coordinates. i points and all n points in the model point cloud j The points are taken as point pairs to calculate the features of each point pair. The present invention obtains three features of the point pair, namely, the distance, the normal vector angle, and the angle between the distance vector and the normal vector.

[0123] The expression of the point pair feature constraint weight is specifically:

[0124] w total =γ 1 d m,n +γ 2 θ d,N +γ 3 θ N

[0125] In the formula, γ 1 , γ 2 and γ3 Both represent weight parameters, d m,n represents the point pair distance matrix, θ d,N Represents the angle between the distance vector and the normal vector, θ N Represents the normal vector angle.

[0126] The weight parameters can well describe the positional relationship between the depth image and the model point cloud. Using these three items as weights to constrain feature extraction can greatly reduce the distribution differences between the points in the model point cloud and the depth point cloud. When the network extracts models and geometric features, the present invention also applies a common attention map to constrain the feature extraction of the bidirectional corresponding network. On the one hand, this method allows the network to further reduce the fault tolerance of point pair matching during point pair feature extraction. On the other hand, in the process of sampling feature point pairs in the network, the accuracy of each point pair is also improved. Therefore, the present invention believes that these three features improve the accuracy and robustness of the network's predicted posture to a certain extent.

[0127] For each point of the model point cloud that matches one of the points in the depth point cloud, the distance of each point pair is calculated by Euclidean distance to obtain a point pair distance matrix. The point pair distance matrix d m,n The specific expression is:

[0128]

[0129] In the formula, Indicates the mth depth point cloud i Points, Respectively represent the mth depth point cloud i The x-axis, y-axis, and z-axis coordinate values ​​of the points. Represents the mth depth point cloud i The model point cloud that matches the nth point j Points, Respectively represent the nth point cloud of the model j The x-axis, y-axis and z-axis coordinate values ​​of each point;

[0130] In this embodiment, the distance and the normal vector angle between the two points are calculated based on the coordinates and normal vectors of the two points. Therefore, the present invention multiplies the unit vectors of the two normal vectors and then takes the distance vector and the normal vector angle θ between them and the distance vector. d,N , for the normal vector N of the two input point features m and N n , and the normal vector angle between them is calculated as follows: the angle θ between the distance vector and the normal vector d,N The angle θ between the normal vector N The specific expression is:

[0131]

[0132]

[0133] Where arccos(·) represents the inverse cosine function, N m Indicates the mth depth point cloud i The normal vector of the point, N n Indicates the model point cloud nth j The normal vector of a point, ||·|| represents the L2 norm, (·) T Indicates transpose.

[0134] The key idea of ​​S3 of the present invention is to connect the common attention map with the feature matrix to guide the process of corresponding matching and pose estimation. Specifically, Figure 1 As shown, the present invention uses two different branches to respectively predict the point pair matching C from the scene point to the model point s→m And the point pair matching C from model point to scene point m→s , a multi-layer perceptron is used in each branch to decode the feature matrix F s or F m , and then connect the output features with the common attention map and input them into the multi-layer perceptron regressor for corresponding prediction. In order to encode more features to better predict the pose, the present invention also effectively connects the feature matrices between each branch.

[0135] In S3, the attention-aware pose estimation module includes first to third regression branches;

[0136] The first regression branch and the third regression branch each include a multilayer perceptron, a connection operation and a multilayer perceptron regressor connected in sequence, and the second regression branch includes a multilayer perceptron and a connection operation connected to each other;

[0137] The S3 comprises the following sub-steps:

[0138] S31, inputting the dense feature matrix as input features into the first regression branch to obtain point pair matching from scene points to model points, and inputting the model feature matrix as input features into the third regression branch to obtain point pair matching from model points to scene points;

[0139] Wherein, the point pair matching from scene point to model point and the point pair matching from model point to scene point include a plurality of predicted point pairs;

[0140] S32, inputting the dense feature matrix into the second regression branch, decoding the dense feature matrix through a multi-layer perceptron, and obtaining a candidate pose according to the decoded feature prediction;

[0141] S33, establishing a final loss function according to the set prediction loss, pose loss and attention loss, and calculating the first pose and the second pose from all predicted point pairs according to the final loss function through BiCo-net;

[0142] S34, merging the candidate pose, the first pose and the second pose, taking the average value after merging, and obtaining the final pose.

[0143] In S31, the first regression branch and the third regression branch process the input features in the same way. The method for processing the input features by the first regression branch is specifically:

[0144] The input features are input to the first regression branch, the input features are decoded by the multi-layer perceptron, the features of the decoded output are connected with the common attention map, and the connected results are input to the multi-layer perceptron regressor for corresponding prediction to obtain the point pair matching from the scene point to the model point.

[0145] In this embodiment, given the predicted scene to model match (x i ,n i ), the present invention supervises the corresponding prediction by prediction loss, in said S33, the prediction loss includes the image prediction loss function L s And the model prediction loss function L m , its specific expression is as follows:

[0146]

[0147]

[0148] Where ε represents the hyperparameter, N, i, x i represents the ground truth coordinates of the i-th point in the depth point cloud, represents the ground truth normal vector of the i-th point in the depth point cloud, Represents the coordinates of the corresponding prediction points generated by feature encoding and decoding, They represent the normal vectors of the corresponding predicted points generated by feature encoding and decoding, respectively, j represents the ground truth coordinates of the jth point in the model point cloud, represents the ground truth normal vector of the jth point in the model point cloud, Represents the coordinates of the corresponding prediction points generated after feature encoding and decoding, Represents the normal vector of the corresponding prediction point generated by feature encoding and decoding;

[0149] In this embodiment, the posture of the asymmetric target is supervised by the ground truth posture with ADD loss, and the posture of the symmetric target is supervised by the ground truth posture with ADD-S loss. pose The specific expression is:

[0150]

[0151] Where R, t represents the ground truth pose, represents the predicted point-by-point pose, x k represents the kth point among K points randomly selected from the model point cloud of the object, K represents K points randomly selected from the model point cloud of the object, j represents the closest point, ||·|| represents the L2 norm, min j∈[1,K] ||·|| respectively represent the calculation of the L2 norm of all K points and the nearest point j;

[0152] The final loss function L total The specific expression is:

[0153]

[0154] Where, L attention Indicates loss of attention, Represents the balance parameter between loss functions;

[0155] In S34, the final posture T is obtained. final The specific expression is:

[0156] T final =average(T d ∪T s ∪T m )

[0157] Where, T d represents the candidate pose, T s represents the first pose, T m represents the second pose, ∪ represents the union, and average(·) represents the average function.

[0158] Embodiment 2:

[0159] This embodiment is an ablation experiment for Embodiment 1.

[0160] In order to verify the influence of the global point cloud attention module and the ConvNext network on the robustness of predicted pose, the present invention rearranges and combines these two modules and conducts experiments on the Occ-LineMod dataset. Not only that, the present invention also independently constrains the bidirectional mapping and direct regression branches. As shown in Table 1, the improved CNN feature extraction module greatly improves the robustness of the subsequent common attention module to constrain each branch, and the average accuracy of the ADD(-S) indicator is improved by 2.2%. At the same time, the simultaneous constraints of the three branches by the common attention module make them related to each other, which significantly improves the subsequent pose prediction, and further improves the accuracy by 2.7% on the basis of improving the color feature extraction module.

[0161] Table 1 The impact of ConvNext module and attention perception module on pose prediction

[0162]

[0163] In order to verify the influence of the feature weights of the constrained common attention map on the robustness of the predicted pose, the present invention recombines these three weights and conducts experiments on the Occ-LineMod dataset. As shown in Table 2, if the common attention map with constrained features is generated based on GCCN using only the angle between the distance vector of two points and the normal, a final result of 73.3% can be obtained, which is 3.8% higher than the original network without constraint weights. However, based on the feature items in the PPF global modeling, the present invention uses the angle between the normal and the distance vector as another feature constraint to jointly guide the acquisition of the common attention map with other features, and the final result is 4.9% higher than the biasline of BiCo. This result shows that using the PPF point pair features as constraint items for global modeling can establish a closer connection between the two relatively independent branches in the bidirectional mapping network, thereby more effectively improving the network's ability to discern targets in robust scenes.

[0164] Table 2 The influence of PPF weight constraint on pose prediction

[0165] <![CDATA[d m,n ]]> <![CDATA[θ d,N ]]> <![CDATA[θ N ]]> ADD(-S) √ - - 73.4 - √ - 73.5 - - √ 73.9 √ - √ 73.3 √ √ - 73.5 - √ √ 74.0 √ √ √ 74.4

[0166] In order to verify the effectiveness of the point-by-point attention mechanism proposed in the present invention compared with GCCN, the present invention conducted a series of ablation experiments on the Occ-LineMOD and YCB-Video datasets. As shown in Table 6, the present invention first replaces the pseudo twin network (PSN) with the feature extraction network in GCCN. The experimental results show that the posture prediction performance of this method is significantly reduced compared with that using PSN. Then, the present invention applies the improved PPF weight constraint term to obtain a greater improvement. Finally, as shown in Table 6, Figure 5 As shown, the present invention obtains point clouds from different viewpoints of the "can" object in the Occ-LineMOD dataset to intuitively compare the results, selects a point in the depth point cloud, and projects the corresponding attention map in the model point cloud into the 2D image plane for visualization. The dotted line connects the scene point and the corresponding point with the maximum attention value. The first line is the attention map learned by the attention module of the present invention, and the second line is the attention map learned using the GCCN method. It can be seen that the point-by-point attention map of the present invention can better simulate the weight distribution that reflects the correlation between the model and the scene, thereby obtaining higher matching accuracy.

[0167] Table 3 Impact of Pseudo Twin Network Module (PSN)

[0168]

[0169] In order to verify the robustness of the network of the present invention to occlusion, the present invention draws the accuracy curve of the occlusion level when the occlusion level increases on the YCB-Video dataset. According to DenseFusion, the present invention calculates the occlusion level by calculating the percentage of the invisible surface of the model point in the image frame, and uses the ADD-S (<2cm) evaluation standard to measure its accuracy. The final accuracy curve is shown in Figure 6 As shown, it can be seen that the network of the present invention is still very stable when the occlusion is more serious, maintaining the same level of accuracy, while the accuracy of other methods decreases when the occlusion is more serious.

[0170] The beneficial effects of the present invention are as follows: the present invention proposes a robust 6D pose estimation method based on bidirectional matching and global attention network, and introduces a simple and effective global point cloud attention module to find the similarity between model points and depth points, thereby obtaining robust attention correlation. Experiments show that the method of the present invention has advantages over the most advanced methods, and the accuracy and robustness of the prediction results of the present invention are improved in 6D pose estimation.

[0171] The feature matrix extracted by the present invention enables the attention mechanism to better weight and focus on processing different feature channels, capturing the global, local information and different features of the object. The common attention map of the attention feature is predicted by the global point cloud attention module, which can combine the relationship between two different input data when generating attention weights, which can help the model better understand the interaction between input data and maintain consistency when processing multiple input data.

[0172] The present invention performs global modeling while generating a common attention map, and uses the relative feature differences between point pairs in the two point clouds as the weight-constrained feature attention map to obtain a rough description of the object, thereby improving the robustness of the target pose prediction.

[0173] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims

1. A robust 6D pose estimation method based on bidirectional matching and global attention network, characterized in that: The following steps are involved: S1. Obtain a panoramic image and a depth image, input them into the ConvNext network and the Pointnet network respectively to extract color features and depth point clouds, and extract feature matrices based on color features, depth point clouds and model point clouds; S2, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the global point cloud attention module to obtain a common attention map; S3, input the common attention map and feature matrix into the attention-aware pose estimation module to obtain the final pose; In S2, the global point cloud attention module includes a pseudo twin network module, a dot product function layer and a softmax function layer connected in sequence; The S2 comprises the following sub-steps: S21, input the depth point cloud and its normal vector information and the model point cloud and its normal vector information into the pseudo twin network module respectively to obtain the depth attention feature matrix and the model attention feature matrix; S22, input the deep attention feature matrix and the model attention feature matrix into the dot product function layer to obtain the correlation feature matrix, and input the correlation feature matrix into the softmax function layer to obtain a common attention map for constraining point pair matching; The S21 is specifically: The depth point cloud and its normal vector information and the model point cloud and its normal vector information are input twice at different positions through the pseudo twin network module to obtain the depth attention feature matrix and the model attention feature matrix. Among them, the method for processing input by the pseudo twin network module is the same, and the method for processing the depth point cloud and its normal vector information and the model point cloud and its normal vector information by the pseudo twin network module is specifically: SA1, using a multi-layer perceptron to upgrade the 6-dimensional depth point cloud and its normal vector information and the model point cloud and its normal vector information to 64 dimensions and 128 dimensions, to obtain the 64-dimensional first to second local features and the 128-dimensional third to fourth local features; SA2, locally fuse the first and second local features of 64 dimensions to obtain the fifth local feature of 128 dimensions; SA3, locally fuse the third to fourth local features of 128 dimensions to obtain a sixth local feature of 256 dimensions, encode the sixth local feature to obtain a seventh local feature of 1024 dimensions; SA4, average pool the 1024-dimensional seventh local feature to obtain the global feature, densely fuse the global feature with the 128-dimensional fifth local feature, and decode and normalize the dense fusion result to obtain a deep attention feature matrix with both local feature information and global feature information; SA5. Use the pseudo-twin network to exchange the positions of the depth point cloud and its normal vector information with the model point cloud and its normal vector information input, repeat SA1 to SA4, and obtain the model attention feature matrix with both local feature information and global feature information.

2. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 1, characterized in that: In S1, the feature matrix includes a dense feature matrix and a model feature matrix; The S1 comprises the following sub-steps: S11, obtaining a panoramic image and a depth image, and cutting out a region of interest from the panoramic image and the depth image; S12, converting the depth image into scene point cloud information according to the region of interest of the panoramic image and the camera internal parameters, and inputting the region of interest of the panoramic image into the ConvNext network to obtain color features; S13, randomly extract N points from the depth point cloud, extract the geometric features of the N points through the Pointnet network, and connect the geometric features with the color features to obtain a dense feature matrix; S14. Randomly extract N points from the model point cloud, and extract the model feature matrix of the N points through a multi-layer perceptron.

3. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 1, characterized in that: In S22, the relevant feature matrix R is obtained. m,n The specific expression is: R m,n =g(F sa ,F ma ) In the formula, g(·) represents the dot product function, F sa represents the deep attention feature matrix, F ma Represents the model attention feature matrix; Get the common attention map M m*n The specific expression is: In the formula, R i,n Represents the i-th element in the n-th column of the correlation feature matrix, R k,n Represents the kth element of the nth column in the correlation feature matrix, m represents the number of rows of the matrix, and n represents the number of columns of the matrix.

4. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 1, characterized in that: In S22, the obtained common attention map is constrained by the attention loss, and the attention loss L attention The specific expression is: In the formula, w total represents the feature constraint weight of the point pair, M m*n represents the joint attention map, m represents the number of rows in the matrix, and n represents the number of columns in the matrix; The expression of the point pair feature constraint weight is specifically: w total =γ1d m,n +γ2θ d,N +γ3θ N In the formula, γ1, γ2 and γ3 all represent weight parameters, d m,n represents the point pair distance matrix, θ d,N Represents the angle between the distance vector and the normal vector, θ N Represents the normal vector angle.

5. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 4, characterized in that: The point pair distance matrix d m,n The specific expression is: In the formula, Indicates the mth depth point cloud i Points, Respectively represent the mth depth point cloud i The x-axis, y-axis, and z-axis coordinate values ​​of the points. Represents the mth depth point cloud i The model point cloud that matches the nth point j Points, Respectively represent the nth point cloud of the model j The x-axis, y-axis and z-axis coordinate values ​​of each point; The angle θ between the distance vector and the normal vector d,N The angle θ between the normal vector N The specific expression is: Where arccos(·) represents the inverse cosine function, N m Indicates the mth depth point cloud i The normal vector of the point, N n Indicates the model point cloud nth j The normal vector of a point, ‖·‖ represents the L2 norm, (·) T Indicates transpose.

6. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 2, characterized in that: In S3, the attention-aware pose estimation module includes a first regression branch, a second regression branch and a third regression branch; The first regression branch and the third regression branch each include a multilayer perceptron, a connection operation and a multilayer perceptron regressor connected in sequence, and the second regression branch includes a multilayer perceptron and a connection operation connected to each other; The S3 comprises the following sub-steps: S31, inputting the dense feature matrix as input features into the first regression branch to obtain point pair matching from scene points to model points, and inputting the model feature matrix as input features into the third regression branch to obtain point pair matching from model points to scene points; Wherein, the point pair matching from scene point to model point and the point pair matching from model point to scene point include a plurality of predicted point pairs; S32, inputting the dense feature matrix into the second regression branch, decoding the dense feature matrix through a multi-layer perceptron, and obtaining a candidate pose according to the decoded feature prediction; S33, establishing a final loss function according to the set prediction loss, pose loss and attention loss, and calculating the first pose and the second pose from all predicted point pairs according to the final loss function through BiCo-net; S34, merging the candidate pose, the first pose and the second pose, taking the average value after merging, and obtaining the final pose.

7. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 6, characterized in that: In S31, the first regression branch and the third regression branch process the input features in the same way. The method for processing the input features by the first regression branch is specifically: The input features are input to the first regression branch, the input features are decoded by the multi-layer perceptron, the features of the decoded output are connected with the common attention map, and the connected results are input to the multi-layer perceptron regressor for corresponding prediction to obtain the point pair matching from the scene point to the model point.

8. The robust 6D pose estimation method based on bidirectional matching and global attention network according to claim 6, characterized in that: In S33, the prediction loss includes the image prediction loss function L s And the model prediction loss function L m , its specific expression is as follows: Where ε represents the hyperparameter, N, i, x i represents the ground truth coordinates of the i-th point in the depth point cloud, represents the ground truth normal vector of the i-th point in the depth point cloud, Represents the coordinates of the corresponding prediction points generated by feature encoding and decoding, They represent the normal vectors of the corresponding predicted points generated by feature encoding and decoding, respectively, j represents the ground truth coordinates of the jth point in the model point cloud, represents the ground truth normal vector of the jth point in the model point cloud, Represents the coordinates of the corresponding prediction points generated after feature encoding and decoding, Represents the normal vector of the corresponding prediction point generated by feature encoding and decoding; The pose loss L pose The specific expression is: Where R, t represents the ground truth pose, represents the predicted point-by-point pose, x k represents the kth point among K points randomly selected from the model point cloud of the object, K represents K points randomly selected from the model point cloud of the object, j represents the closest point, ‖·‖ represents the L2 norm, min j∈[1,K] ‖·‖ respectively represent the calculation of the L2 norm of all K points and the nearest point j; The final loss function L total The specific expression is: Where, L attention Indicates loss of attention, Represents the balance parameter between loss functions; In S34, the final posture T is obtained. final The specific expression is: T final =average(T d ∪T s ∪T m ) Where, T d represents the candidate pose, T s represents the first pose, T m represents the second pose, ∪ represents the union, and average(·) represents the average function.

Images