A Transformer-based 6D pose estimation method

Abstract

This invention relates to a Transformer-based 6D pose estimation method. It employs a pose estimation network comprising a Transformer-based 2D keypoint feature extraction module, a keypoint structure modeling module, and a pose inference module. The PnP algorithm is used to calculate the 6D pose from the correspondence between 2D keypoints and the target object's 3D keypoints. The Transformer-based 2D keypoint feature extraction module extracts 2D keypoint features of the target object from an RGB image. The keypoint structure modeling module uses self-attention to learn the structural relationships and contextual information of the extracted keypoint features and predicts keypoint coordinates. The pose inference module uses the PnP algorithm to calculate the target object's pose by comparing the predicted 2D keypoints with the target object's 3D keypoints. This method leverages the superiority and efficiency of the Transformer structure to implement a pure Transformer-based pose estimation network, fully utilizing the geometric invariance of the keypoint structure to improve the accuracy of 6D pose estimation, exceeding that of CNN-based pose estimation networks.

Description

Technical Field

[0001] This invention belongs to the field of image detection, specifically relating to a 6D pose estimation method based on Transformer, which can efficiently and accurately estimate the 3D pose of objects in RGB images. Background Technology

[0002] 6D pose estimation of objects refers to detecting objects in an image and estimating their 3D position and orientation. Object pose estimation of a single image has always been a very important research topic in the field of computer vision. In addition, 6D pose estimation of objects is crucial for augmented reality, virtual reality, robot grasping, and autonomous driving technologies.

[0003] Currently, 6D pose estimation algorithms based on depth images have achieved good results. However, the acquisition of depth images depends on RGB-D cameras. The acquisition of photos by RGB-D cameras is affected by factors such as resolution, field of view and frame rate, and the images are large in size, making it impossible to integrate them into wearable devices for real-time pose estimation of moving objects.

[0004] Therefore, RGB image-based schemes have been widely studied. Traditional 6D pose estimation methods are mainly divided into feature point-based methods and template-based methods. However, these schemes still have many limitations. For example, feature point-based methods cannot handle pose estimation of objects without texture, while template-based methods cannot handle pose estimation of occluded objects.

[0005] With the emergence of deep learning, especially the rise of convolutional neural networks (CNNs), the accuracy and robustness of monocular 6D target pose estimation have been continuously improving, sometimes even surpassing methods that rely on depth data. Most current pose estimation algorithms use CNNs to directly or indirectly regress and predict object pose, but they still face many challenges. While some algorithms can directly regress object pose, they require extensive parameter training, resulting in slightly lower accuracy. Other two-stage pose estimation algorithms rely on object keypoints and camera parameters to calculate the object pose; while their accuracy is higher than direct prediction methods, they do not establish the structural relationships between object keypoints. Summary of the Invention

[0006] In view of the defects or deficiencies of the existing technology, the purpose of this invention is to provide a 6D pose estimation method based on Transformer to efficiently and accurately estimate the 3D pose of objects in RGB images.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A 6D pose estimation method based on Transformer is characterized by employing a pose estimation network comprising a Transformer-based 2D keypoint feature extraction module, a keypoint structure modeling module, and a pose inference module. The method uses the PnP algorithm to calculate the 6D pose from the correspondence between 2D keypoints and 3D keypoints of the target object; wherein:

[0009] The Transformer-based 2D keypoint feature extraction module extracts 2D keypoint features of a target object from an RGB image. For an input target object image, the module outputs a set of keypoint feature vectors, representing eight keypoint features extracted from the image.

[0010] The keypoint structure modeling module includes a self-attention layer and a multilayer perceptron layer. The self-attention layer learns the structural relationships and contextual information of the predicted keypoint features, while the multilayer perceptron layer predicts the feature vectors after relationship modeling as two-dimensional coordinate points on the image, which are also called two-dimensional keypoints.

[0011] The pose inference module uses the PNP algorithm to calculate the pose of the target object by combining the predicted 2D keypoints in the image with the 3D keypoints of the target object, and outputs rotation and translation matrices.

[0012] According to the present invention, the following steps are specifically performed:

[0013] Step 1: Obtain multiple sets of two-dimensional images containing the target to be identified. Each set of two-dimensional images contains fifteen groups of target objects; and each target object contains more than a thousand RGB images, the only difference being the image acquisition angle. This will give you a training image set.

[0014] Step 2: Data preprocessing. Cropping the images in the original dataset ensures that the objects appear completely in the images. The cropped image size is 256*256.

[0015] Step 3: Input the training image set into the Transformer-based 2D object key point feature extraction module; obtain the key point features in the image;

[0016] Step 4: Input the obtained keypoint features into the keypoint structure modeling module. The 2D keypoint structure modeling module utilizes the geometric relationships between keypoints of the same object from different perspectives. It employs a self-attention structure to interactively learn the eight keypoint features extracted from the same object, thereby extracting 2D keypoint features with geometric consistency for a single object. The interactive features are then input into a multilayer perceptron layer to obtain the 2D keypoint coordinates.

[0017] Step 5: The pose reasoning module uses the extracted 2D keypoint coordinates and the 3D keypoints of the object model to reason about the final pose of the object through the PNP algorithm.

[0018] Specifically, the construction of the Transformer-based key point feature extraction module includes the following steps:

[0019] Step 101: Input the training images into the network;

[0020] Step 102: Segment the target image into patches and serialize them;

[0021] Step 103: Add position embedding to the image patch;

[0022] Step 104: Predefine J learnable d-dimensional keypoint embedding vectors;

[0023] Step 105: Input the image patch and the d-dimensional keypoint embedding vector into the Transformer encoder structure.

[0024] Step 106: The Transformer encoder structure outputs the learned two-dimensional key point features.

[0025] Specifically, the construction of the key point structure modeling module includes the following steps:

[0026] Step 201: In the keypoint structure modeling module, a self-attention mechanism is applied to integrate the structural relationships between keypoints. The self-attention mechanism essentially adds some learnable parameters. During inference, these parameters are used to obtain a series of attention weights to simulate the strength of the correlation between keypoints, thereby learning the structural relationships between object keypoints. Keypoint features are then fed into the self-attention module to calculate the similarity between keypoint features, thus learning the correlation between them. For the input keypoint feature sequence, the similarity between the query and each key is calculated to obtain weights.

[0027] Step 202: Normalize the weights using the Softmax function;

[0028] Step 203: Perform a weighted summation of the weights and their corresponding values ​​to obtain the final key feature with correlations;

[0029] Step 204: Input the interactive key point features into the multilayer perceptron layer to obtain the two-dimensional key point coordinates.

[0030] Furthermore, the construction of the Transformer-based keypoint feature extraction module also includes image serialization and two-dimensional keypoint feature extraction. The specific construction method is as follows:

[0031] (1) Serialize the input 2D image: Serialize the image... Processed into a series of flat 2D image patches Where (H, W) is the resolution of the input image, C is the number of input channels, and (P, P) is the resolution of each image patch. This is the number of image patches obtained, which are then used with a trainable linear projection. x PATCH Mapping to D dimensions, the output of this projection is called patch embedding;

[0032] (2) Add positional information to the patch embedding, that is, add positional encoding P to the sequence block, and use the resulting sequence Z0 as the image feature sequence.

[0033]

[0034] in, This represents the number of image patches obtained.

[0035] (3) Predefine J learnable d-dimensional keypoint embedding vectors. J Before training begins, embedding vectors (keypoints) are generated for these J learnable d-dimensional keypoints. J Perform random initialization, where J represents the number of key points.

[0036] (4) The obtained image feature sequence Z0 and the keypoint embedding vector keypoints J As input, it is fed into the Transformer encoder to learn the relationships between image patches and to embed the image feature sequence Z0 with the keypoint embedding vector. J Global interaction is performed in the Transformer encoder; each Transformer encoder layer consists of a multi-head self-attention (MSA) block and an MLP block; Layer Norm (LN) is applied before each block and residual connections are applied after each block; the MLP contains a hidden layer with an intermediate GELU nonlinear activation function; finally, the Transformer encoder outputs keypoint features after interacting with the image and feeds them into the keypoint structure modeling module.

[0037] Specifically, the construction of the key point structure modeling module includes a self-attention layer and a multilayer perceptron layer, and the specific construction method is as follows:

[0038] (1) The key point features extracted by the Transformer-based key point feature extraction module are fed into the self-attention module to perform interactive learning of the correlation between key point features. In the self-attention module, the structural relationship between key points is integrated by using the attention mechanism on the input feature sequence. The self-attention mechanism is actually to add some learnable parameters. During the inference process, a series of attention weights are obtained through these parameters to simulate the strength of the correlation between key points, thereby achieving the learning of the structural relationship between key points of the object.

[0039] (2) In self-attention, each keypoint feature generates three different vectors: the Query vector (Q), the Key vector (K), and the Value vector (V). These vectors are generated by multiplying the embedding vector X by three different weight matrices W. Q W K W V The query vector (Q) is then multiplied with the key vector (K) to calculate a score for each keypoint vector. A softmax activation function is applied to the score. This softmax score determines the "contribution" of each keypoint to encoding the current position. Keypoints already at this position will receive the highest softmax score. The result is then multiplied with the value vector (V) to obtain the output vector.

[0040] (3) The output vector is fed into a multilayer perceptron for dense prediction and regression of key point coordinates; the key point coordinate regression head is implemented by eight MLPs with individual parameters; the MLP is used to generate a heat map, and the softmax function is used to convert the heat map into a probability distribution map to obtain the key point coordinates;

[0041] (4) Subsequently, the distance between the predicted keypoints and the actual keypoint coordinates is calculated, and the loss is defined as follows:

[0042]

[0043] In the formula, N is the number of key points. These are the predicted coordinates of the key points. These are the actual coordinates of the key points;

[0044] smooth L1 The definition is as follows:

[0045]

[0046] In the formula, |x| represents the absolute distance between the predicted coordinates of the keypoint and the actual value of the keypoint.

[0047] Furthermore, the specific implementation method of the pose reasoning module described in step 5, which uses the extracted 2D keypoint coordinates and the 3D keypoints of the object model to reason about the final pose of the object through the PNP algorithm, involves calculating the rotation matrix and translation matrix of the target object in the RGB image and executing the following steps:

[0048] Step 1: Obtain a two-dimensional image containing the target to be identified, thus obtaining the image to be identified;

[0049] Step II: Use the Transformer-based 2D keypoint feature extraction module as described in claim 2 or claim 4 to obtain the 2D keypoint features of the target in the image to be identified. The predicted 2D keypoints are projections of the 3D keypoints of a predefined object;

[0050] Step III: Using the key point structure modeling module described in claim 3 or claim 5, interactive learning is performed on the key point features to extract two-dimensional key point features with geometric consistency for a single object. The interactive features are then input into a multilayer perceptron layer to obtain a set of two-dimensional key point coordinates in the RGB image. The set of two-dimensional key points includes Q two-dimensional key points, where Q is a positive integer.

[0051] Step IV: Use the PnP algorithm to calculate the 6D pose from the correspondence between the 2D keypoints and the 3D keypoints of the target object; the 3D keypoints are eight 3D coordinate points obtained on the object model using the farthest point sampling algorithm (FPS). PnP only uses the correspondence between the 8 2D keypoints and the 3D keypoints to estimate the 3D rotation and 3D translation of the target object in camera coordinates.

[0052] The 6D pose estimation method based on Transformer of the present invention has the following advantages compared with the prior art:

[0053] 1. Leveraging the advantages and efficiency of the transformer structure, a pure transformer-based pose estimation network was implemented, which can effectively learn the relationships between features and model global feature relationships. Furthermore, it was demonstrated that the 6D pose estimation algorithm can be implemented from a pure sequence to a sequence approach.

[0054] 2. Introducing a self-attention mechanism can model the correlation between keypoint features to learn the inherent geometric connections between object keypoints. This leverages the invariance of structural features, thereby improving the accuracy of pose prediction. The accuracy surpasses that of CNN-based pose estimation networks. Attached Figure Description

[0055] Figure 1 This is an overview diagram of the Transformer-based 6D pose estimation method of the present invention.

[0056] Figure 2 This is a schematic diagram of the structure of the Transformer-based 2D key point feature extraction module for objects.

[0057] Figure 3 This is a schematic diagram of the Transformer encoder structure;

[0058] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. Detailed Implementation

[0059] Regarding the problem of modeling keypoint structural relationships, the applicant noted in their research that the Transformer can learn the relationships between patches. Furthermore, the applicant observed that in Natural Language Processing (NLP), the main approach is to first pre-train the Transformer on a large, general-purpose corpus, and then fine-tune the model to achieve different downstream tasks. The Transformer model is also increasingly being used in computer vision. Vit, which has made contributions to the vision field, aims to model global contextual information, not just region-level relationships. The emergence of Vit has demonstrated that the Transformer can significantly improve the performance of dense recognition tasks.

[0060] Therefore, based on the general model of Transformer, image sequences and key point feature sequences are extracted and fed together as input into the Transformer encoder structure for interaction, thereby extracting two-dimensional key point features of RGB images. Self-attention layers and multilayer perceptrons are added to model the structural relationships of the extracted two-dimensional key point features and implement dense coordinate prediction. Combined with the PnP algorithm, the pose recognition problem in the field of 6D pose estimation is solved.

[0061] like Figure 1 As shown, this embodiment presents a Transformer-based 6D pose estimation method. It employs a pose estimation network including a Transformer-based 2D keypoint feature extraction module, a keypoint structure modeling module, and a pose inference module. The PnP algorithm is used to calculate the 6D pose from the correspondence between 2D keypoints and the target object's 3D keypoints; wherein:

[0062] The Transformer-based 2D keypoint feature extraction module extracts 2D keypoint features of a target object from an RGB image. For an input target object image, the module outputs a set of keypoint feature vectors, representing eight keypoint features extracted from the image.

[0063] The keypoint structure modeling module includes a self-attention layer and a multilayer perceptron layer. The self-attention layer learns the structural relationships and contextual information of the predicted keypoint features, while the multilayer perceptron layer predicts the feature vectors after relationship modeling as two-dimensional coordinate points on the image, which are also called two-dimensional keypoints.

[0064] The pose inference module uses the PNP algorithm to calculate the pose of the target object by combining the predicted 2D key points of the image with the 3D key points of the target object, and outputs rotation and translation matrices.

[0065] The construction of the Transformer-based 2D object key point feature extraction module includes the following steps:

[0066] Step 101: Input the training images into the network;

[0067] Step 102: Segment the target image into patches and serialize them;

[0068] Step 103: Add position embedding to the image patch;

[0069] Step 104: Predefine J learnable d-dimensional keypoint embedding vectors;

[0070] Step 105: Input the image patch and the d-dimensional keypoint embedding vector into the Transformer encoder structure.

[0071] Step 106: The Transformer encoder structure outputs the learned two-dimensional key point features.

[0072] The construction of the key point structure modeling module includes the following steps:

[0073] Step 201: In the keypoint structure modeling module, a self-attention mechanism is applied to integrate the structural relationships between keypoints. The self-attention mechanism essentially adds some learnable parameters. During inference, these parameters are used to obtain a series of attention weights to simulate the strength of the correlation between keypoints, thereby learning the structural relationships between keypoints of an object. Keypoint features are fed into the self-attention module to calculate the similarity between keypoint features, thus learning the correlation between them. For the input keypoint feature sequence, the similarity between the query and each key is calculated to obtain weights. Common similarity calculations include dot product, concatenation, and perceptron.

[0074] Step 202: Normalize the weights using the Softmax function.

[0075] Step 203: Sum the weights and corresponding values ​​to obtain the final key point features with correlation.

[0076] Step 204: Input the interactive key point features into the multilayer perceptron layer to obtain the two-dimensional key point coordinates.

[0077] Pose reasoning module construction:

[0078] Step 301: Use the PnP algorithm to calculate the final 6D pose from the correspondence between the 2D keypoints and the 3D keypoints of the target object; the 3D keypoints are eight 3D coordinate points obtained on the object model using the farthest point sampling (FPS) algorithm. Since PnP only uses the correspondence between the 8 2D keypoints and the 3D keypoints to estimate the 3D rotation and 3D translation of the target object in the camera coordinates, the final predicted 2D keypoints and the corresponding 3D keypoints of the object are input into the PnP algorithm to calculate the pose of the object.

[0079] Figure 2 The structure for key point feature extraction during the network training phase is demonstrated, and the output is the two-dimensional key point features of the predicted target object.

[0080] Step 401: The standard Transformer takes a one-dimensional token embedding as input. To process a two-dimensional image, the image is first... Processed into a series of flat 2D image patches Here, (H, W) is the resolution of the input image, C is the number of input channels, and (P, P) is the resolution of each image patch. This is the number of image patches obtained, which are then used with a trainable linear projection. x PATCH Mapping to D dimensions, the output of this projection is called patch embedding.

[0081] Step 402: Add the position code P to all input patch embeddings to preserve positional information, and use the resulting sequence Z0 as the input to the encoder:

[0082]

[0083] in, This represents the number of image patches obtained.

[0084] Step 403: Predefine J learnable d-dimensional keypoint embedding vectors. J Before training begins, embedding vectors (keypoints) are generated for these J learnable d-dimensional keypoints. J Perform random initialization, where J represents the number of key points.

[0085] Step 404: Combine the obtained image feature sequence Z0 with the keypoint embedding vectors. J As input, it is fed into the Transformer encoder to learn the relationships between image patches and to embed the image feature sequence Z0 with the keypoint embedding vector. J Perform global interactions in the Transformer encoder.

[0086] Step 405: Figure 3 The structure of the Transformer encoder is shown. Each Transformer encoder layer consists of a multi-head self-attention (MSA) block and an MLP block. Layer Norm (LN) is applied before each block, and residual connections are applied after each block. The MLP contains a hidden layer with an intermediate GELU nonlinear activation function. The input to the multi-head attention layer is based on... The computed triple (query, key, value):

[0087] query=Z l-1 W Q

[0088] key = Z l-1 W K

[0089] value = Z l-1 W V

[0090] in, These are the learning parameters for three linear projection layers, where d is the dimension of (Query, Key, Value).

[0091] Self-attention is represented as:

[0092]

[0093] Multi-head attention mechanisms refer to mechanisms with m independent attention operations and multiple sets of (q, k, v) matrices. Each set of (q, k, v) matrices represents one operation of the attention mechanism. By concatenating these matrices and multiplying them by a parameter matrix W0, the final output of the multi-head attention layer can be obtained.

[0094] MSA(Z l-1 )=[SA1(Z l-1 SA2(Z) l-1 );…;SA m (Z l-1 )]W O

[0095] d is typically set to C / m. Then, the output of the MSA is transformed by an MLP block with skip connections. The input of the l-th coding layer of the Transformer is represented as:

[0096]

[0097] Step 406: Finally, the Transformer encoder outputs the key point features after interacting with the image and sends them to the key point structure modeling module.

[0098] Figure 1 The key point structure modeling module is implemented using the following steps:

[0099] Step 501: The keypoint features extracted by the Transformer-based keypoint feature extraction module are fed into the self-attention module for interactive learning of the correlation between keypoint features. The input matrix I∈R(d, N) is multiplied by three different parameter matrices W. q W k W v Given three intermediate matrices Q, K, and V ∈ R(d, d), with the same dimensions, transpose K and multiply it by Q to obtain the Attention matrix A ∈ R(N, N), representing the pairwise attention between each position. Applying a softmax operation to A yields... Finally, multiply it by V to obtain the output vector O∈R(d, N).

[0100]

[0101]

[0102] Step 502: The keypoint coordinate regression head is implemented using eight MLPs with individual parameters; the MLPs are used to generate heatmaps, and the softmax function is used to convert the heatmaps into probability distribution maps P(u, v), where (u, v) is the two-dimensional location in the heatmap, so the coordinates x of the i-th keypoint are... i y i Represented as;

[0103]

[0104]

[0105] In the formula, P(u, v) is the probability distribution map of the key points. It is floating-point multiplication.

[0106] Step 503: Next, calculate the distance between the predicted keypoints and the actual keypoint coordinates, with the loss defined as follows:

[0107]

[0108] In the formula, N is the number of key points. These are the predicted coordinates of the key points. These are the actual coordinates of the key points.

[0109] smooth L1 The definition is as follows:

[0110]

[0111] Figure 1 The specific implementation steps of the attitude inference module are as follows:

[0112] Step 601: Input the test image into the network and use the trained network to predict the two-dimensional key points of the target object on the RGB image. The predicted two-dimensional key points are the projections of the three-dimensional key points of the predefined object.

[0113] Step 602: Use the PnP algorithm to calculate the 6D pose from the correspondence between the 2D key points and the 3D key points of the target object; the 3D key points are eight 3D coordinate points obtained on the object model using the farthest point sampling algorithm (FPS). PnP only uses the correspondence between the 8 2D key points and the 3D key points to estimate the 3D rotation and 3D translation of the target object in the camera coordinates.

[0114] Specific experimental examples:

[0115] To demonstrate the effectiveness of the Transformer-based 6D pose estimation method presented in this embodiment, the inventors trained and tested the method on the LINEMOD dataset, which contains fifteen target objects, and calculated the poses for thirteen of them.

[0116] The RGB image of the target object used for training and its corresponding real 2D coordinates are input into a Transformer-based 2D keypoint extraction network for training, and the corresponding training model is obtained. Then, the test image is input into the trained model to predict the 2D keypoints of the RGB image. The 2D keypoints and 3D keypoints are then combined using the PnP algorithm to obtain the final predicted pose matrix. The ADD evaluation metric is used to evaluate the accuracy of the predicted pose matrix. As shown in Table 1 below, this embodiment of the Transformer-based 6D pose estimation method can realize the pose calculation of the target object.

[0117] Table 1

[0118]

[0119]

Claims

1. A 6D pose estimation method based on Transformer, characterized in that, This method employs a pose estimation network comprising a Transformer-based 2D keypoint feature extraction module, a keypoint structure modeling module, and a pose inference module. It uses the PnP algorithm to calculate the 6D pose from the correspondence between 2D keypoints and the target object's 3D keypoints; where: The Transformer-based object 2D key point feature extraction module is used to extract the 2D key point features of the target object in the RGB image; for the input target object image, the 2D key point feature extraction module will output a set of key point feature vectors, representing the eight key point features extracted from the image. The key point structure modeling module includes a self-attention layer and a multilayer perceptron layer; wherein, the self-attention layer learns the structural relationship and contextual information of the predicted key point features, and the multilayer perceptron layer predicts the feature vector after relationship modeling as two-dimensional coordinate points on the image, which are also called two-dimensional key points. The pose inference module uses the pip algorithm to calculate the pose of the target object by combining the predicted two-dimensional key points of the image with the three-dimensional key points of the target object, and outputs a rotation matrix and a translation matrix. The construction of the Transformer-based keypoint feature extraction module also includes image serialization and two-dimensional keypoint feature extraction. The specific construction method is as follows: (1) Serialize the input 2D image: Serialize the image... Processed into a series of flat 2D image patches Where (H,W) is the resolution of the input image, C is the number of input channels, and (P,P) is the resolution of each image patch. This is the number of image patches obtained, which are then used with a trainable linear projection. x PATCH Mapping to D dimensions, the output of this projection is called patch embedding; (2) Add positional information to the patch embedding, that is, add positional encoding P to the sequence block, and use the resulting sequence Z0 as the image feature sequence: in, It is the number of image patches obtained; (3) Predefine J learnable d-dimensional keypoint embedding vectors. J Before training begins, embedding vectors (keypoints) are generated for these J learnable d-dimensional keypoints. J Perform random initialization, where J represents the number of key points; (4) The obtained image feature sequence Z0 and the keypoint embedding vector keypoints J As input, it is fed into the Transformer encoder to learn the relationships between image patches and to embed the image feature sequence Z0 with the keypoint embedding vector. J Global interactions are performed in the Transformer encoder; each Transformer encoder layer consists of a multi-head self-attention (MSA) block and an MLP block; Layer Norm (LN) is applied before each block and residual connections are applied after each block; the MLP contains a hidden layer with an intermediate GELU nonlinear activation function; finally, the Transformer encoder outputs keypoint features after interacting with the image and feeds them into the keypoint structure modeling module. The specific construction method of the key point structure modeling module is as follows: (1) The key point features extracted by the Transformer-based key point feature extraction module are fed into the self-attention module to perform interactive learning of the correlation between key point features. In the self-attention module, the structural relationship between key points is integrated by using the attention mechanism on the input feature sequence. The self-attention mechanism is actually to add some learnable parameters. During the inference process, a series of attention weights are obtained through these parameters to simulate the strength of the correlation between key points, thereby achieving the learning of the structural relationship between key points of the object. (2) In self-attention, each keypoint feature generates three different vectors: the Query vector (Q), the Key vector (K), and the Value vector (V). These vectors are generated by multiplying the embedding vector X by three different weight matrices W. Q W K W V The query vector (Q) is then multiplied with the key vector (K) to calculate a score for each keypoint vector. A softmax activation function is applied to the score. This softmax score determines the "contribution" of each keypoint to encoding the current position. Keypoints already at this position will receive the highest softmax score. The result is then multiplied with the value vector (V) to obtain the output vector. (3) The output vector is fed into a multilayer perceptron for dense prediction and regression of key point coordinates; the key point coordinate regression head is implemented by eight MLPs with individual parameters; the MLP is used to generate a heat map, and the softmax function is used to convert the heat map into a probability distribution map to obtain the key point coordinates; (4) Subsequently, the distance between the predicted keypoints and the actual keypoint coordinates is calculated, and the loss is defined as follows: In the formula, N is the number of key points. These are the predicted coordinates of the key points. These are the actual coordinates of the key points; smooth L1 The definition is as follows: In the formula, |x| represents the absolute distance between the predicted coordinates of the keypoint and the actual value of the keypoint.

2. The method as described in claim 1, characterized in that, Please follow these steps: Step 1: Obtain multiple sets of two-dimensional images containing the target to be identified. Each set of two-dimensional images contains fifteen groups of target objects; and each target object contains more than a thousand RGB images, the only difference being the image acquisition angle. This will give you a training image set. Step 2: Data preprocessing. Cropping the images in the original dataset ensures that the objects appear completely in the images. The cropped image size is 256*256. Step 3: Input the training image set into the Transformer-based 2D object key point feature extraction module; Obtain key point features in an image; Step 4: Input the obtained keypoint features into the keypoint structure modeling module; The 2D keypoint structure modeling module utilizes the geometric relationship between keypoints of the same object from different perspectives. It employs a self-attention structure to interactively learn the features of eight keypoints extracted from the same object, thereby extracting 2D keypoint features with geometric consistency for a single object. The interactive features are then input into a multilayer perceptron layer to obtain the 2D keypoint coordinates. Step 5: The pose reasoning module uses the extracted 2D keypoint coordinates and the 3D keypoints of the object model to reason about the final pose of the object through the PNP algorithm.

3. The method as described in claim 1, characterized in that, The construction of the Transformer-based 2D keypoint feature extraction module includes the following steps: Step 201: Input the training images into the network; Step 202: Segment the target image into patches and serialize them; Step 203: Add location embedding to the image patch; Step 204: Predefine J learnable d-dimensional keypoint embedding vectors; Step 205: Input the image patch and the d-dimensional keypoint embedding vector into the Transformer encoder structure; Step 206: The Transformer encoder structure outputs the learned two-dimensional key point features.

4. The method as described in claim 1, characterized in that, The construction of the key point structure modeling module includes the following steps: Step 301: Apply a self-attention mechanism to the key point structure modeling module to integrate the structural relationships between key points. Feed the key point features into the self-attention module, calculate the similarity between key point features, and learn the association between key point features. For the input key point feature sequence, calculate the weight by performing similarity calculation between query and each key. Step 302: Normalize the weights using the Softmax function; Step 303: Calculate the weights and corresponding values ​​using a weighted sum to obtain the final key feature with correlations; Step 304: Input the interactive key point features into the multilayer perceptron layer to obtain the two-dimensional key point coordinates.

5. The method as described in claim 2, characterized in that, Step 5 describes a pose inference module that uses the extracted 2D keypoint coordinates and the 3D keypoints of the object model to infer the final pose of the object through a PNP algorithm. The specific implementation method involves calculating the rotation and translation matrices of the target object in the RGB image and executing the following steps: Step 1: Obtain a two-dimensional image containing the target to be identified, thus obtaining the image to be identified; Step II: Use a Transformer-based 2D key point feature extraction module to obtain the 2D key point features of the target in the image to be identified; The predicted 2D keypoints are projections of the 3D keypoints of a predefined object; Step III: The key point structure modeling module is used to interactively learn the key point features, extract the two-dimensional key point features with geometric consistency of a single object, and input the interactive features into the multilayer perceptron layer to obtain the set of two-dimensional key point coordinates in the RGB image; the set of two-dimensional key points includes Q two-dimensional key points, where Q is a positive integer; Step IV: Calculate the 6D pose using the PnP algorithm to establish the correspondence between 2D keypoints and 3D keypoints of the target object; The 3D keypoints are eight 3D coordinate points obtained on the object model using the farthest point sampling algorithm FPS. PnP only uses the correspondence between the 8 2D keypoints and the 3D keypoints to estimate the 3D rotation and 3D translation of the target object in camera coordinates.

Images