6D object attitude estimation method based on prior guidance and adaptive weighted fusion

Abstract

The invention discloses a 6D object attitude estimation method based on prior guidance and adaptive weighted fusion, and belongs to the technical field of attitude estimation, and the method comprises the following steps: S1, collecting RGBD image pairs, and generating a pair of cost matrixes; s2, according to the pair of cost matrixes, generating fusion features of the anchor point image, querying fusion features of the image and coded joint features; s3, obtaining a first relative attitude and a second relative attitude of the target object according to the fusion feature of the anchor point image and the fusion feature of the query image; and S4, obtaining a final pose according to the coded joint features and the first relative pose and the second relative pose of the target object. According to the method, various new objects can be processed more flexibly, and additional data acquisition or complex preprocessing steps are not needed; the generalization ability enables the model to have better adaptability in practical application, and the model can be quickly applied to new tasks and scenes.

Description

Technical Field

[0001] This invention belongs to the field of attitude estimation technology, specifically relating to a 6D object attitude estimation method based on prior guidance and adaptive weighted fusion. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, 6D object pose estimation has become one of the key technologies in fields such as robotics, autonomous driving, and augmented reality. 6D pose estimation refers to extracting the position and orientation information of an object from an image, typically including the object's 3D coordinates and its rotation angle relative to the camera. Traditional 6D pose estimation methods usually rely on pre-created object models, such as CAD models or video sequences. These models may have limitations in practical applications, especially when dealing with unknown or unseen objects, where their applicability is greatly reduced. In recent years, deep learning-based models have been continuously innovated, and these methods can learn the geometric features of objects from images to perform efficient pose estimation. However, most methods still rely on specific object models or training sets for instance-level pose estimation and face significant challenges when handling multiple scenes or new objects.

[0003] While existing 6D object pose estimation methods have achieved some success on known objects, they still face numerous challenges, especially when dealing with unknown objects. Traditional methods rely on predefined object models or video sequences for pose estimation, which limits their generalization ability. For example, CAD model-based estimation methods require a pre-prepared model for each object during testing, and these methods cannot provide effective support for new objects without models or video sequences. Another type of image-based model-free method, while not requiring a pre-defined object model, still needs multiple reference views or video sequences, and for unknown objects, it usually needs to learn from object-specific visual features, further limiting the methods. Therefore, overcoming these limitations and adopting more flexible and adaptable techniques has become an urgent need for current technological development.

[0004] Existing technologies, such as general 6D object pose estimation methods based on depth images, decompose the 6D object pose estimation task into three sub-tasks: viewpoint encoding, in-plane rotation regression, and position refinement, and employ a cascaded structure for efficient inference. Specifically, this method first trains a lightweight viewpoint encoding network using a large amount of synthetic depth map data generated from ShapeNet, mapping the object's depth image to an invariant feature space with viewpoint information, thereby constructing a viewpoint dictionary containing candidate viewpoints and their corresponding rotation matrices. Next, in the inference stage, the input depth image and the target segmentation mask are preprocessed to obtain preliminary pose estimates. Then, the viewpoint encoding network is used to extract viewpoint features from the input depth image, and the best-matching candidate viewpoint is retrieved from the viewpoint dictionary. The in-plane rotation of the input image relative to the candidate viewpoint is then estimated through a regression network. Finally, the position refinement module, combined with consistency verification, corrects the preliminary translation estimate to obtain the complete 6D object pose. The main steps of this method are as follows: 1) Train a lightweight view coding network using a large amount of synthetic depth map data, and construct a view dictionary based on the 3D mesh model of the target object; 2) Preprocess the input depth image and segmentation mask to calculate a preliminary object position estimate; 3) Extract features from the input depth map using the view coding network, and retrieve multiple candidate viewpoints from the view dictionary; 4) For each candidate viewpoint, estimate the relative rotation angle using an in-plane rotation regression module, and sort and filter multiple candidate poses using a consistency verification module; 5) Update the preliminary translation estimate using a position refinement module, and output the final 6D object pose; 6) Further optimize the output pose using ICP refinement to obtain a higher accuracy pose estimation result. The drawback of this technical solution is that detailed model information of the target object must be pre-built or collected before testing. This reliance not only increases the initial construction cost and hardware investment of the system, but also leads to matching errors due to insufficient model information when processing new or unknown objects, resulting in a high bit error rate.

[0005] In existing technologies, such as one-time object pose estimation methods, the object pose estimation problem is transformed into a visual localization problem. This method does not rely on CAD models but only requires constructing a sparse SfM model of the object through a simple video scan. Then, a graph attention network is used for 2D-3D feature matching to obtain an accurate 6D pose. Specifically, first, a video scan of the target object is acquired, and a sparse point cloud model of the object is reconstructed using the Structured Bundles Method (SfM). Next, in the inference phase, 2D local features are extracted from the query image and directly matched with 3D points in the SfM model. A graph attention mechanism is used to aggregate multiple 2D features corresponding to the same 3D point, thereby generating a stable and globally consistent 2D-3D correspondence. Finally, the 6D pose of the object is solved using the PnP algorithm combined with RANSAC, and a feature tracking module is used for real-time updates. The main process of this method is as follows: 1) Initialization phase: a) Acquire a video scan of the target object and reconstruct a sparse 3D point cloud model of the object using the SfM method; b) Construct a correspondence graph of the object using the images captured in the video and the camera pose, laying the foundation for subsequent matching. 2) Inference Stage: a) Extract 2D local features for each query image; b) Use a graph attention network to adaptively aggregate multiple 2D features corresponding to the same 3D point, directly achieving 2D-3D feature matching; c) Utilize the PnP algorithm combined with RANSAC to solve for the 6D pose of the object from the matched 2D-3D correspondence, while updating the pose of consecutive frames through a feature tracking module; 3) Final Output: The above process yields real-time and accurate 6D pose estimation results for objects without requiring additional training for specific objects or categories, thus achieving one-time, universal object pose estimation. The drawback of this technical solution is that the 6D pose estimation process often relies on multiple viewpoints or continuous video sequences to extract global and local features of the object. This not only complicates the system processing flow and increases computational resource requirements but also results in significant lag in real-time response. Because it requires simultaneous matching and alignment of multi-view data, processing time increases significantly, leading to slow response speeds in dynamic scenes and failing to meet high real-time requirements.

[0006] Existing methods typically rely on pre-built 3D object models or require video sequences as references, and have high requirements for obtaining object instances, increasing the barrier to entry and data preparation costs. There is a need to investigate a technical solution that uses only user-provided text descriptions to specify the target object and combines visual-language fusion technology to achieve 6D pose estimation under cross-scene conditions, thereby reducing the dependence on object models and video data and improving universality and convenience.

[0007] When dealing with scenes featuring complex backgrounds, varying lighting conditions, partial object occlusion, and challenging appearances (e.g., low texture, symmetrical structures), existing technologies fall short in terms of depth and robustness in multimodal information fusion. Many methods fail to achieve sufficient and efficient synergistic optimization between textual semantics, visual features, and 3D geometric constraints, often treating object segmentation, feature extraction, and pose estimation as separate processes. This makes it difficult for the rich semantic information contained in the text description to effectively guide the extraction of visual features and the accurate establishment of subsequent geometric relationships, thus affecting the quality of feature matching. Especially when facing the aforementioned challenging objects, the accuracy and stability of pose estimation are difficult to guarantee. A technical solution is needed that involves end-to-end joint training of object segmentation, feature extraction, and pose estimation tasks, and fully integrates textual and image information through a vision-language fusion module to enhance the recognition ability and matching robustness of novel concept objects. Summary of the Invention

[0008] To address the above problems, this invention proposes a 6D object pose estimation method based on prior guidance and adaptive weighted fusion.

[0009] The technical solution of this invention is: a 6D object pose estimation method based on prior guidance and adaptive weighted fusion, comprising the following steps:

[0010] S1. Acquire RGBD image pairs and generate a pair of cost matrices;

[0011] S2. Based on a pair of cost matrices, generate the fusion features of the anchor image, the fusion features of the query image, and the encoded joint features;

[0012] S3. Based on the fusion features of the anchor point image and the fusion features of the query image, obtain the first relative pose and the second relative pose of the target object;

[0013] S4. Based on the encoded joint features, the first relative pose and the second relative pose of the target object, the final pose is obtained.

[0014] Furthermore, S1 includes the following sub-steps:

[0015] S11. Acquire RGBD image pairs, extract anchor point images and query images;

[0016] S12. Input the anchor image and the query image into the visual model, and extract the global feature representation of the anchor image and the global feature representation of the query image;

[0017] S13. Input the anchor image and the query image into the LoFTR feature extraction network to extract the local feature representation of the anchor image and the local feature representation of the query image;

[0018] S14. Generate several text prompts and extract text features using a language model;

[0019] S15. Generate a text feature matrix based on the text features;

[0020] S16. Calculate a pair of cost matrices based on the global feature representation of the anchor point image, the global feature representation of the query image, and the text feature matrix.

[0021] Furthermore, in S13, the local feature representation of the anchor point image The expression is:

[0022] ;

[0023] in, Represents the RGB values ​​of the anchor point image. This represents the LoFTR feature extraction network;

[0024] In S13, the local feature representation of the query image is... The expression is:

[0025] ;

[0026] in, This indicates that the query image contains RGB values;

[0027] In S16, the expression for a pair of cost matrices is:

[0028] ;

[0029] ;

[0030] in, This represents the cost matrix between the anchor image and the text prompt. This represents the cost matrix between the query image and the text prompt. It represents The first in One element, It represents The first in One element, Indicates the first Textual features of a text prompt This represents the global feature representation of the anchor point image. This represents the global feature representation of the queried image. This represents the cosine similarity function.

[0031] Furthermore, S2 includes the following sub-steps:

[0032] S21. Perform cost aggregation on a pair of cost matrices to obtain the cost features of the anchor image and the cost features of the query image;

[0033] S22. Process the local feature representations of the anchor point image and the query image to obtain joint features;

[0034] S23. Encode the joint features;

[0035] S24. Using the Swint Transformer network as the guiding network, extract the guiding features of the anchor image and the guiding features of the query image;

[0036] S25. The cost features of the anchor image and the cost features of the query image, as well as the guiding features of the anchor image and the guiding features of the query image, are concatenated to generate the fusion features of the anchor image and the fusion features of the query image.

[0037] and Representing from the image and Local feature maps extracted from them.

[0038] Furthermore, in S21, the cost characteristics of the anchor point image Cost features of query images The expression is:

[0039] ;

[0040] ;

[0041] in, This represents the cost feature of the anchor image after the first Transformer layer. This represents the cost features of the query image after the first Transformer layer. This indicates the first layer of Transformer processing. This indicates the second layer of Transformer processing;

[0042] In S22, joint features The expression is:

[0043] ;

[0044] ;

[0045] ;

[0046] in, This represents the local features of the anchor image after being encoded by the self-attention layer of the LoFTR network. This represents the local features of the query image after being encoded by the self-attention layer of the LoFTR network. This represents the local feature representation of the anchor point image. This represents the local feature representation of the query image. This represents the cross-attention layer of the LoFTR feature extraction network. This represents the self-attention layer of the LoFTR feature extraction network;

[0047] In S23, the expression for encoding the joint features is:

[0048] ;

[0049] in, Represents the encoded joint features. This refers to the feature encoding method that combines the 8-Point algorithm with the Vision Transformer (ViT).

[0050] In S24, the guiding features of the anchor point image The expression is:

[0051] ;

[0052] in, Indicates guiding the network, Represents the RGB values ​​of the anchor point image;

[0053] In S24, the guiding features of the query image are... The expression is:

[0054] ;

[0055] in, This indicates that the query image contains RGB values;

[0056] In S25, the fusion features of the anchor point images The expression is:

[0057] ;

[0058] in, Indicates a splicing operation;

[0059] In S25, the fusion features of the query image are... The expression is:

[0060] .

[0061] Furthermore, S3 includes the following sub-steps:

[0062] S31. Input the fusion features of the anchor image and the fusion features of the query image into the decoder to obtain the feature maps of the anchor image and the query image at each layer; wherein, the decoder includes several upsampling layers;

[0063] S32. Based on the guiding features of the anchor point image and the feature maps at each layer, obtain the final high-resolution feature map of the anchor point image; based on the guiding features of the query image and the feature maps at each layer, obtain the final high-resolution feature map of the query image.

[0064] S33. Add segmentation heads to the final high-resolution feature map of the anchor point image and the final high-resolution feature map of the query image to obtain the segmentation mask of the anchor point image and the segmentation mask of the query image.

[0065] S34. Based on the segmentation mask of the anchor point image and the segmentation mask of the query image, extract the final feature map of the anchor point image and the final feature map of the query image respectively.

[0066] S35. Based on each feature point in the final feature map of the anchor point image, find the nearest neighbor feature point in the final feature map of the query image to obtain several matching point pairs.

[0067] S36. Use a distance threshold to filter several matching point pairs;

[0068] S37. Backproject the filtered matching point pairs onto 3D space to obtain the point cloud of the anchor point image and the point cloud of the query image, and generate the matching point set.

[0069] S38. Optimize the matching point set using the total loss function and generate the first relative pose of the target object;

[0070] S39. Randomly select the minimum point set from the matching point set and estimate the pose using the 8-point algorithm;

[0071] S310. Based on the pose, verify all point clouds of the matching point set to obtain the number of interior points;

[0072] S311. Determine the pose with the maximum number of interior points as the second relative pose of the target object.

[0073] Furthermore, in S31, the anchor point image is in the... Feature map of layer The expression is:

[0074] ;

[0075] in, Indicates the anchor point image at the 1st Feature map of the layer This represents the convolution operation. Indicates an upsampling operation;

[0076] In S31, the queried image is in the... Feature map of layer The expression is:

[0077] ;

[0078] in, Indicates the query image is in the 1st position. Feature map of the layer;

[0079] In S32, the final high-resolution feature map of the anchor point image. The expression is:

[0080] ;

[0081] in, The guiding features of the anchor point image;

[0082] In S32, the final high-resolution feature map of the query image. The expression is:

[0083] ;

[0084] in, Indicates guiding features for the query image;

[0085] In S33, the segmentation mask of the anchor point image. The expression is:

[0086] ;

[0087] in, This represents the Sigmoid activation function;

[0088] In S33, the segmentation mask of the query image is used. The expression is:

[0089] ;

[0090] In S34, the final feature map of the anchor point image. The expression is:

[0091] ;

[0092] In S34, the final feature map of the query image is... The expression is:

[0093] ;

[0094] In S35, the expression for finding the nearest neighbor feature point in the final feature map of the query image is:

[0095] ;

[0096] in, The feature points represent the final feature map of the anchor point image. This represents the feature points of the final feature map of the queried image. Represents the normalized cosine similarity. Indicates cosine similarity;

[0097] In S36, the expression for filtering several pairs of matching points using a distance threshold is:

[0098] ;

[0099] in, Indicates the distance threshold;

[0100] In S38, the total loss function The expression is:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] in, This represents the segmentation weight hyperparameter. This represents the loss function for the segmentation task. This represents the final feature loss function. The segmentation mask representing the anchor point image. This represents the true segmentation mask of the queried image. Indicates the smoothing term. Indicates the negative sample weight. Indicates the weight of positive samples. Indicates positive sample loss. Indicates negative sample loss. This represents the negative sample loss of the anchor point image. This represents the negative sample loss of the query image. Represents the set of matching point pairs. The index variable represents the index of the currently matched feature pair in the anchor point image. This represents the index variable of the currently matching feature pair in the query image. Represents the boundary values ​​of negative samples. Represents the boundary values ​​of positive samples. Indicates the local index of the negative sample used within each image. This indicates the th negative sample in the current image set. Feature vectors of feature points This represents the set of negative samples for the anchor point image. This represents the set of negative samples for the query image. The feature vector representing the feature map of the anchor point image. The feature vector representing the feature map of the query image. Represents the normalized cosine similarity. This indicates taking the maximum value. This indicates taking the minimum value.

[0109] Furthermore, S4 includes the following sub-steps:

[0110] S41. Based on the encoded joint features and the number of inliers corresponding to the second relative pose, obtain the prediction weights;

[0111] S42. Based on the predicted weights, obtain the weights after rotation and the weights after translation;

[0112] S43. Perform weighted fusion of the first and second relative poses of the target object to obtain the prior pose.

[0113] S44. Determine the prior function based on the prior pose;

[0114] S45. Use prior functions to determine the solver's scoring function;

[0115] S46. Based on the solver's scoring function, the prior pose is updated using the solver to determine the updated pose.

[0116] S47. Extract the rotation matrix from the updated pose, and use the rotated weights to perform weighted fusion of the rotation matrix to obtain the final rotation matrix.

[0117] S48. Extract the translation vector from the updated pose, and use the translated weights to perform weighted fusion of the translation vector to obtain the final translation vector.

[0118] S49. Combine the final rotation matrix and the final translation vector to obtain the final pose.

[0119] Furthermore, in S41, the expression for the prediction weights is:

[0120] ;

[0121] in, This represents the original rotation prediction weights that have not been normalized by sigmoid. This represents the original translation prediction weights that have not been normalized by sigmoid. Represents the encoded joint features. This represents the second relative attitude of the target object. This indicates the number of interior points corresponding to the second relative attitude. This represents a multilayer perceptron network with adaptive learning weights, which dynamically generates rotation and translation weight coefficients by taking input features, pose, and number of interior points.

[0122] In S42, the rotated weights The expression is:

[0123] ;

[0124] in, Represents the sigmoid function;

[0125] In S42, the translated weights The expression is:

[0126] ;

[0127] In S43, the prior pose The expression is:

[0128] ;

[0129] in, This represents the first relative pose of the target object. , The weights are dynamically calculated to balance the contributions of deep learning and traditional geometric methods, thereby optimizing the accuracy and robustness of camera pose estimation.

[0130] In S44, the prior function The expression is:

[0131] ;

[0132] in, Indicates the hypothetical model, Indicates the number of fixed grid points. Indicates the first Grid points, This represents the transformation result of the prior pose. This represents the transformation result of the assumed model on the grid points;

[0133] In S45, the scoring function The expression is:

[0134] ;

[0135] in, Indicates prior weights, This represents a feature point in the final feature map of the anchor point image. This represents a feature point in the final feature map of the query image; two points together form a matching point pair. ,in It is a set of matching points. Represents the energy function, measuring the energy of point pairs. in pose Matching quality below, Indicates an indicator function, when the condition is met. The function value is 1 when the condition is true, and 0 when the condition is false.

[0136] In S46, the updated pose The expression is:

[0137] ;

[0138] in, Represents sparse corresponding points. This indicates a solver based on prior guidance;

[0139] In S47, sampling weights The expression is:

[0140] ;

[0141] in, Represents an exponential function. This indicates temperature hyperparameters. Indicates hyperparameters, Denotes the first sparse corresponding point set For matching, This represents an element that describes the sub-difference set. express;

[0142] In S47, the final rotation matrix The expression is:

[0143] ;

[0144] in, This represents the rotation matrix extracted from the first relative pose of the target object. This represents the rotation matrix extracted from the updated pose;

[0145] In S48, the final translation vector The expression is:

[0146] ;

[0147] in, This represents the translation vector extracted from the first relative pose of the target object. This represents the translation vector extracted from the updated pose.

[0148] The beneficial effects of this invention are:

[0149] (1) The present invention adopts an open vocabulary setting, which allows the model to perform pose estimation on new objects that have not been seen during the training phase. Compared with methods that rely on predefined object models or video sequences, the present invention can handle various new objects more flexibly without additional data collection or complex preprocessing steps. This generalization ability makes the model more adaptable in practical applications and can be quickly applied to new tasks and scenarios.

[0150] (2) This invention achieves the collaborative optimization of text semantics, visual features and geometric constraints by constructing a hierarchical feature interaction network. At the bottom layer, multi-scale geometric features are adaptively extracted through the dynamic sampling mechanism of deformable convolution, accurately focusing on the key local structure of the target object. In the middle layer feature fusion stage, cross-modal attention units are innovatively deployed, and a bidirectional modulation relationship between text description and visual features is established by using a multi-head attention mechanism to achieve semantic-guided feature enhancement and geometric alignment. In the top layer optimization stage, a geometric consistency gating module is introduced to transform high-level semantic information into differentiable spatial constraints, and semantic-driven reweighting strategy effectively suppresses ambiguity in the pose of symmetrical objects. Through end-to-end training, a complete technical chain from fine-grained geometric feature extraction to cross-modal semantic-geometric joint optimization is formed, showing good accuracy improvement and robustness advantages in complex scenarios.

[0151] (3) This invention improves the accuracy and robustness of 6D pose estimation by adaptively weighting and fusing PointDSC prediction with RANSAC solver. Specifically, the PointDSC algorithm is first used to perform preliminary matching of the point cloud to generate an initial sparse corresponding point set. Then, the improved RANSAC algorithm is used to optimize these matching points, remove outliers, and improve matching accuracy. Based on this, a priori guidance mechanism is introduced, and the weights of the two are dynamically adjusted by combining the outputs of PointDSC prediction and solver through an adaptive weighted fusion strategy. This fusion method not only makes full use of the advantages of PointDSC in local feature matching, but also leverages the strengths of RANSAC in global geometric constraints, effectively solving the matching problem of low-texture regions and symmetrical objects. Attached Figure Description

[0152] Figure 1 This is a flowchart of a 6D object pose estimation method based on prior guidance and adaptive weighted fusion. Detailed Implementation

[0153] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0154] like Figure 1 As shown, this invention provides a 6D object pose estimation method based on prior guidance and adaptive weighted fusion, comprising the following steps:

[0155] S1. Acquire RGBD image pairs and generate a pair of cost matrices;

[0156] S2. Based on a pair of cost matrices, generate the fusion features of the anchor image, the fusion features of the query image, and the encoded joint features;

[0157] S3. Based on the fusion features of the anchor point image and the fusion features of the query image, obtain the first relative pose and the second relative pose of the target object;

[0158] S4. Based on the encoded joint features, the first relative pose and the second relative pose of the target object, the final pose is obtained.

[0159] : Parameters are The pre-trained language model (CLIP text encoder) is defined as a language model that uses a vocabulary... Mapping the token sequence to the feature space ,Right now .in, Indicates the vocabulary list The set of all possible sequences of all tokens in the set. It represents the set of real numbers.

[0160] : Parameters are A pre-trained visual model (CLIP image encoder) is defined as mapping an input image to a feature space. ,Right now .in, This indicates the spatial dimension of the output feature map.

[0161] Text hints are textual information used to describe the target object, such as "a brown mug".

[0162] Anchor image: an image pair containing RGB and depth information ( ),in It is an RGB image. It is a depth image.

[0163] Query images, image pairs containing RGB and depth information ( ),in It is an RGB image. It is a depth image.

[0164] The data space of a depth image represents a set of depth information.

[0165] RGB image data space, representing a set of color images.

[0166] In this embodiment of the invention, S1 includes the following sub-steps:

[0167] S11. Acquire RGBD image pairs, extract anchor point images and query images;

[0168] S12. Input the anchor image and the query image into the visual model, and extract the global feature representation of the anchor image and the global feature representation of the query image;

[0169] S13. Input the anchor image and the query image into the LoFTR feature extraction network to extract the local feature representation of the anchor image and the local feature representation of the query image;

[0170] S14. Generate several text prompts and extract text features using a language model;

[0171] S15. Generate a text feature matrix based on the text features;

[0172] S16. Calculate a pair of cost matrices based on the global feature representation of the anchor point image, the global feature representation of the query image, and the text feature matrix.

[0173] To enrich the semantic representation of the text, a set of templates is used to generate... Each version of the text prompt { , ,…, Text prompts for each version Each text cue is encoded into a feature vector; for each text cue, a language model is used to extract text features; finally, the text features are represented as a matrix.

[0174] In this embodiment of the invention, in S13, the local feature representation of the anchor point image... The expression is:

[0175] ;

[0176] in, Represents the RGB values ​​of the anchor point image. This represents the LoFTR feature extraction network;

[0177] In S13, the local feature representation of the query image is... The expression is:

[0178] ;

[0179] in, This indicates that the query image contains RGB values;

[0180] In S16, the expression for a pair of cost matrices is:

[0181] ;

[0182] ;

[0183] in, This represents the cost matrix between the anchor image and the text prompt. This represents the cost matrix between the query image and the text prompt. It represents The first in One element, It represents The first in One element, Indicates the first Textual features of a text prompt This represents the global feature representation of the anchor point image. This represents the global feature representation of the queried image. This represents the cosine similarity function.

[0184] In this embodiment of the invention, S2 includes the following sub-steps:

[0185] S21. Perform cost aggregation on a pair of cost matrices to obtain the cost features of the anchor image and the cost features of the query image;

[0186] S22. Process the local feature representations of the anchor point image and the query image to obtain joint features;

[0187] S23. Encode the joint features;

[0188] S24. Using the Swint Transformer network as the guiding network, extract the guiding features of the anchor image and the guiding features of the query image;

[0189] S25. The cost features of the anchor image and the cost features of the query image, as well as the guiding features of the anchor image and the guiding features of the query image, are concatenated to generate the fusion features of the anchor image and the fusion features of the query image.

[0190] To further enrich the local details of the feature map, guiding features are added to each upsampling layer. After processing through three upsampling layers, the final high-resolution feature map is obtained.

[0191] and Representing from the image and Local feature maps extracted from them.

[0192] In this embodiment of the invention, in S21, the cost features of the anchor point image... Cost features of query images The expression is:

[0193] ;

[0194] ;

[0195] in, This represents the cost feature of the anchor image after the first Transformer layer. This represents the cost features of the query image after the first Transformer layer. This indicates the first layer of Transformer processing. This indicates the second layer of Transformer processing;

[0196] In S22, joint features The expression is:

[0197] ;

[0198] ;

[0199] ;

[0200] in, This represents the local features of the anchor image after being encoded by the self-attention layer of the LoFTR network. This represents the local features of the query image after being encoded by the self-attention layer of the LoFTR network. This represents the local feature representation of the anchor point image. This represents the local feature representation of the query image. This represents the cross-attention layer of the LoFTR feature extraction network. This represents the self-attention layer of the LoFTR feature extraction network;

[0201] In S23, the expression for encoding the joint features is:

[0202] ;

[0203] in, Represents the encoded joint features. This refers to the feature encoding method that combines the 8-Point algorithm with the Vision Transformer (ViT).

[0204] In S24, the guiding features of the anchor point image The expression is:

[0205] ;

[0206] in, Indicates guiding the network, Represents the RGB values ​​of the anchor point image;

[0207] In S24, the guiding features of the query image are... The expression is:

[0208] ;

[0209] in, This indicates that the query image contains RGB values;

[0210] In S25, the fusion features of the anchor point images The expression is:

[0211] ;

[0212] in, Indicates a splicing operation;

[0213] In S25, the fusion features of the query image are... The expression is:

[0214] .

[0215] In this embodiment of the invention, S3 includes the following sub-steps:

[0216] S31. Input the fusion features of the anchor image and the fusion features of the query image into the decoder to obtain the feature maps of the anchor image and the query image at each layer; wherein, the decoder includes several upsampling layers;

[0217] S32. Based on the guiding features of the anchor point image and the feature maps at each layer, obtain the final high-resolution feature map of the anchor point image; based on the guiding features of the query image and the feature maps at each layer, obtain the final high-resolution feature map of the query image.

[0218] S33. Add segmentation heads to the final high-resolution feature map of the anchor point image and the final high-resolution feature map of the query image to obtain the segmentation mask of the anchor point image and the segmentation mask of the query image.

[0219] S34. Based on the segmentation mask of the anchor point image and the segmentation mask of the query image, extract the final feature map of the anchor point image and the final feature map of the query image respectively.

[0220] S35. Based on each feature point in the final feature map of the anchor point image, find the nearest neighbor feature point in the final feature map of the query image to obtain several matching point pairs.

[0221] S36. Use a distance threshold to filter several matching point pairs;

[0222] S37. Backproject the filtered matching point pairs onto 3D space to obtain the point cloud of the anchor point image and the point cloud of the query image, and generate the matching point set.

[0223] S38. Optimize the matching point set using the total loss function and generate the first relative pose of the target object;

[0224] S39. Randomly select the minimum point set from the matching point set and estimate the pose using the 8-point algorithm;

[0225] S310. Based on the pose, verify all point clouds of the matching point set to obtain the number of interior points;

[0226] S311. Determine the pose with the maximum number of interior points as the second relative pose of the target object.

[0227] A segmentation head is added to the high-resolution feature map output by the decoder to predict the segmentation mask of the target object. The segmentation head consists of convolutional layers and activation functions.

[0228] The input point cloud has undergone preliminary matching by the segmentation and matching module, but noise and outliers may exist. PointDSC further optimizes the matching point pairs by learning the geometric relationships between the point clouds, outputting the first relative pose of the target object. A contrastive loss function is used to optimize the feature matching process. The solver employs the RANSAC solver.

[0229] In this embodiment of the invention, in S31, the anchor point image is in the... Feature map of layer The expression is:

[0230] ;

[0231] in, Indicates the anchor point image at the 1st Feature map of the layer This represents the convolution operation. Indicates an upsampling operation;

[0232] In S31, the queried image is in the... Feature map of layer The expression is:

[0233] ;

[0234] in, Indicates the query image is in the 1st position. Feature map of the layer;

[0235] In S32, the final high-resolution feature map of the anchor point image. The expression is:

[0236] ;

[0237] in, The guiding features of the anchor point image;

[0238] In S32, the final high-resolution feature map of the query image. The expression is:

[0239] ;

[0240] in, Indicates guiding features for the query image;

[0241] In S33, the segmentation mask of the anchor point image. The expression is:

[0242] ;

[0243] in, This represents the Sigmoid activation function;

[0244] In S33, the segmentation mask of the query image is used. The expression is:

[0245] ;

[0246] In S34, the final feature map of the anchor point image. The expression is:

[0247] ;

[0248] In S34, the final feature map of the query image is... The expression is:

[0249] ;

[0250] In S35, the expression for finding the nearest neighbor feature point in the final feature map of the query image is:

[0251] ;

[0252] in, The feature points represent the final feature map of the anchor point image. This represents the feature points of the final feature map of the queried image. Represents the normalized cosine similarity. Indicates cosine similarity;

[0253] In S36, the expression for filtering several pairs of matching points using a distance threshold is:

[0254] ;

[0255] in, Indicates the distance threshold;

[0256] In S38, the total loss function The expression is:

[0257] ;

[0258] ;

[0259] ;

[0260] ;

[0261] ;

[0262] ;

[0263] ;

[0264] in, This represents the segmentation weight hyperparameter. This represents the loss function for the segmentation task. This represents the final feature loss function. The segmentation mask representing the anchor point image. This represents the true segmentation mask of the queried image. Indicates the smoothing term. Indicates the negative sample weight. Indicates the weight of positive samples. Indicates positive sample loss. Indicates negative sample loss. This represents the negative sample loss of the anchor point image. This represents the negative sample loss of the query image. Represents the set of matching point pairs. The index variable represents the index of the currently matched feature pair in the anchor point image. This represents the index variable of the currently matching feature pair in the query image. Represents the boundary values ​​of negative samples. Represents the boundary values ​​of positive samples. Indicates the local index of the negative sample used within each image. This indicates the th negative sample in the current image set. Feature vectors of feature points This represents the set of negative samples for the anchor point image. This represents the set of negative samples for the query image. The feature vector representing the feature map of the anchor point image. The feature vector representing the feature map of the query image. Represents the normalized cosine similarity. This indicates taking the maximum value. This indicates taking the minimum value.

[0265] Positive sample loss is used to shorten the distance between matching feature pairs. Negative sample loss is used to widen the distance between mismatched feature pairs. Combining positive and negative sample losses yields the final feature loss function. To simultaneously optimize segmentation and matching tasks, the feature loss and segmentation loss are combined to obtain the total loss function. The boundary values ​​of positive samples are used to control the minimum distance between feature pairs.

[0266] In this embodiment of the invention, S4 includes the following sub-steps:

[0267] S41. Based on the encoded joint features and the number of inliers corresponding to the second relative pose, obtain the prediction weights;

[0268] S42. Based on the predicted weights, obtain the weights after rotation and the weights after translation;

[0269] S43. Perform weighted fusion of the first and second relative poses of the target object to obtain the prior pose.

[0270] S44. Determine the prior function based on the prior pose;

[0271] S45. Use prior functions to determine the solver's scoring function;

[0272] S46. Based on the solver's scoring function, the prior pose is updated using the solver to determine the updated pose.

[0273] S47. Extract the rotation matrix from the updated pose, and use the rotated weights to perform weighted fusion of the rotation matrix to obtain the final rotation matrix.

[0274] S48. Extract the translation vector from the updated pose, and use the translated weights to perform weighted fusion of the translation vector to obtain the final translation vector.

[0275] S49. Combine the final rotation matrix and the final translation vector to obtain the final pose.

[0276] In this embodiment of the invention, in S41, the expression for the prediction weight is:

[0277] ;

[0278] in, This represents the original rotation prediction weights that have not been normalized by sigmoid. This represents the original translation prediction weights that have not been normalized by sigmoid. Represents the encoded joint features. This represents the second relative attitude of the target object. This indicates the number of interior points corresponding to the second relative attitude. This represents a multilayer perceptron network with adaptive learning weights, which dynamically generates rotation and translation weight coefficients by taking input features, pose, and number of interior points.

[0279] In S42, the rotated weights The expression is:

[0280] ;

[0281] in, Represents the sigmoid function;

[0282] In S42, the translated weights The expression is:

[0283] ;

[0284] In S43, the prior pose The expression is:

[0285] ;

[0286] in, This represents the first relative pose of the target object. , The weights are dynamically calculated to balance the contributions of deep learning and traditional geometric methods, thereby optimizing the accuracy and robustness of camera pose estimation.

[0287] In S44, the prior function The expression is:

[0288] ;

[0289] in, Indicates the hypothetical model, Indicates the number of fixed grid points. Indicates the first Grid points, This represents the transformation result of the prior pose. This represents the transformation result of the assumed model on the grid points;

[0290] In S45, the scoring function The expression is:

[0291] ;

[0292] in, Indicates prior weights, This represents a feature point in the final feature map of the anchor point image. This represents a feature point in the final feature map of the query image; two points together form a matching point pair. ,in It is a set of matching points. Represents the energy function, measuring the energy of point pairs. in pose Matching quality below, Indicates an indicator function, when the condition is met. The function value is 1 when the condition is true, and 0 when the condition is false.

[0293] In S46, the updated pose The expression is:

[0294] ;

[0295] in, Represents sparse corresponding points. This indicates a solver based on prior guidance;

[0296] Sampling is weighted according to sampling weights to update the pose estimate of the prior-guided solver. Sampling weights The expression is:

[0297] ;

[0298] in, Represents an exponential function. This represents the temperature hyperparameter, used to control the smoothness or "sharpness" of the weight distribution. Denotes the first sparse corresponding point set For matching, Indicates hyperparameters, This represents an element that describes the sub-difference set. Indicates the Sampson distance;

[0299] In S47, the final rotation matrix The expression is:

[0300] ;

[0301] in, This represents the rotation matrix extracted from the first relative pose of the target object. This represents the rotation matrix extracted from the updated pose;

[0302] In S48, the final translation vector The expression is:

[0303] ;

[0304] in, This represents the translation vector extracted from the first relative pose of the target object. This represents the translation vector extracted from the updated pose.

[0305] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A 6D object pose estimation method based on prior guidance and adaptive weighted fusion, characterized in that, Includes the following steps: S1. Acquire RGBD image pairs and generate a pair of cost matrices; S2. Based on a pair of cost matrices, generate the fusion features of the anchor image, the fusion features of the query image, and the encoded joint features; S3. Based on the fusion features of the anchor point image and the fusion features of the query image, obtain the first relative pose and the second relative pose of the target object; S4. Based on the encoded joint features, the first relative pose and the second relative pose of the target object, the final pose is obtained.

2. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 1, characterized in that, S1 includes the following sub-steps: S11. Acquire RGBD image pairs, extract anchor point images and query images; S12. Input the anchor image and the query image into the visual model, and extract the global feature representation of the anchor image and the global feature representation of the query image; S13. Input the anchor image and the query image into the LoFTR feature extraction network to extract the local feature representation of the anchor image and the local feature representation of the query image; S14. Generate several text prompts and extract text features using a language model; S15. Generate a text feature matrix based on the text features; S16. Calculate a pair of cost matrices based on the global feature representation of the anchor point image, the global feature representation of the query image, and the text feature matrix.

3. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 2, characterized in that, In S13, the local feature representation of the anchor point image The expression is: ； in, Represents the RGB values ​​of the anchor point image. This represents the LoFTR feature extraction network; In step S13, the local feature representation of the query image is... The expression is: ； in, This indicates that the query image contains RGB values; In S16, the expression for a pair of cost matrices is: ； ； in, This represents the cost matrix between the anchor image and the text prompt. This represents the cost matrix between the query image and the text prompt. It represents The first in One element, It represents The first in One element, Indicates the first Textual features of a text prompt This represents the global feature representation of the anchor point image. This represents the global feature representation of the queried image. This represents the cosine similarity function.

4. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 1, characterized in that, S2 includes the following sub-steps: S21. Perform cost aggregation on a pair of cost matrices to obtain the cost features of the anchor image and the cost features of the query image; S22. Process the local feature representations of the anchor point image and the query image to obtain joint features; S23. Encode the joint features; S24. Using the Swint Transformer network as the guiding network, extract the guiding features of the anchor image and the guiding features of the query image; S25. The cost features of the anchor image and the cost features of the query image, as well as the guiding features of the anchor image and the guiding features of the query image, are concatenated to generate the fusion features of the anchor image and the fusion features of the query image.

5. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 4, characterized in that, In S21, the cost features of the anchor point image Cost features of query images The expression is: ； ； in, This represents the cost feature of the anchor image after the first Transformer layer. This represents the cost features of the query image after the first Transformer layer. This indicates the first layer of Transformer processing. This indicates the second layer of Transformer processing; In S22, the joint feature The expression is: ； ； ； in, This represents the local features of the anchor image after being encoded by the self-attention layer of the LoFTR network. This represents the local features of the query image after being encoded by the self-attention layer of the LoFTR network. This represents the local feature representation of the anchor point image. This represents the local feature representation of the query image. This represents the cross-attention layer of the LoFTR feature extraction network. This represents the self-attention layer of the LoFTR feature extraction network; In step S23, the expression for encoding the joint features is: ； in, Represents the encoded joint features. This represents a feature encoding method combining the 8-Point algorithm with a visual transformer; In S24, the guiding features of the anchor point image The expression is: ； in, Indicates guiding the network, Represents the RGB values ​​of the anchor point image; In step S24, the guiding features of the query image are... The expression is: ； in, This indicates that the query image contains RGB values; In S25, the fusion features of the anchor point images The expression is: ； in, Indicates a splicing operation; In step S25, the fusion features of the query image are... The expression is: 。 6. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Input the fusion features of the anchor image and the fusion features of the query image into the decoder to obtain the feature maps of the anchor image and the query image at each layer; wherein, the decoder includes several upsampling layers; S32. Based on the guiding features of the anchor point image and the feature maps at each layer, obtain the final high-resolution feature map of the anchor point image; based on the guiding features of the query image and the feature maps at each layer, obtain the final high-resolution feature map of the query image. S33. Add segmentation heads to the final high-resolution feature map of the anchor point image and the final high-resolution feature map of the query image to obtain the segmentation mask of the anchor point image and the segmentation mask of the query image. S34. Based on the segmentation mask of the anchor point image and the segmentation mask of the query image, extract the final feature map of the anchor point image and the final feature map of the query image respectively. S35. Based on each feature point in the final feature map of the anchor point image, find the nearest neighbor feature point in the final feature map of the query image to obtain several matching point pairs. S36. Use a distance threshold to filter several matching point pairs; S37. Backproject the filtered matching point pairs onto 3D space to obtain the point cloud of the anchor point image and the point cloud of the query image, and generate the matching point set. S38. Optimize the matching point set using the total loss function and generate the first relative pose of the target object; S39. Randomly select the minimum point set from the matching point set and estimate the pose using the 8-point algorithm; S310. Based on the pose, verify all point clouds of the matching point set to obtain the number of interior points; S311. Determine the pose with the maximum number of interior points as the second relative pose of the target object.

7. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 6, characterized in that, In S31, the anchor point image is in the... Feature map of layer The expression is: ； in, Indicates the anchor point image at the 1st Feature map of the layer This represents the convolution operation. Indicates an upsampling operation; In S31, the query image is in the... Feature map of layer The expression is: ； in, Indicates the query image is in the 1st position. Feature map of the layer; In step S32, the final high-resolution feature map of the anchor point image The expression is: ； in, The guiding features of the anchor point image; In step S32, the final high-resolution feature map of the query image is obtained. The expression is: ； in, Indicates guiding features for the query image; In S33, the segmentation mask of the anchor point image The expression is: ； in, This represents the Sigmoid activation function; In step S33, the segmentation mask of the query image is... The expression is: ； In step S34, the final feature map of the anchor point image The expression is: ； In step S34, the final feature map of the query image is... The expression is: ； In step S35, the expression for finding the nearest neighbor feature point in the final feature map of the query image is: ； in, The feature vector representing the final feature map of the anchor point image. This represents the feature vector of the final feature map of the query image. Represents the normalized cosine similarity. Indicates cosine similarity; In step S36, the expression for filtering several pairs of matching points using a distance threshold is as follows: ； in, Indicates the distance threshold; In S38, the total loss function The expression is: ； ； ； ； ； ； ； in, This represents the segmentation weight hyperparameter. This represents the loss function for the segmentation task. This represents the final feature loss function. The segmentation mask representing the anchor point image. This represents the true segmentation mask of the queried image. Indicates the smoothing term. Indicates the negative sample weight. Indicates the weight of positive samples. Indicates positive sample loss. Indicates negative sample loss. This represents the negative sample loss of the anchor point image. This represents the negative sample loss of the query image. Represents the set of matching point pairs. The index variable represents the index of the currently matched feature pair in the anchor point image. This represents the index variable of the currently matching feature pair in the query image. Represents the boundary values ​​of negative samples. Represents the boundary values ​​of positive samples. Indicates the local index of the negative sample used within each image. This indicates the th negative sample in the current image set. Feature vectors of feature points This represents the set of negative samples for the anchor point image. This represents the set of negative samples for the query image. The feature vector representing the feature map of the anchor point image. The feature vector representing the feature map of the query image. Represents the normalized cosine similarity. This indicates taking the maximum value. This indicates taking the minimum value.

8. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 1, characterized in that, S4 includes the following sub-steps: S41. Based on the encoded joint features and the number of inliers corresponding to the second relative pose, obtain the prediction weights; S42. Based on the predicted weights, obtain the weights after rotation and the weights after translation; S43. Perform weighted fusion of the first and second relative poses of the target object to obtain the prior pose. S44. Determine the prior function based on the prior pose; S45. Use prior functions to determine the solver's scoring function; S46. Based on the solver's scoring function, the prior pose is updated using the solver to determine the updated pose. S47. Extract the rotation matrix from the updated pose, and use the rotated weights to perform weighted fusion of the rotation matrix to obtain the final rotation matrix. S48. Extract the translation vector from the updated pose, and use the translated weights to perform weighted fusion of the translation vector to obtain the final translation vector. S49. Combine the final rotation matrix and the final translation vector to obtain the final pose.

9. The 6D object pose estimation method based on prior guidance and adaptive weighted fusion according to claim 8, characterized in that, In step S41, the expression for the prediction weight is: ； in, This represents the original rotation prediction weights that have not been normalized by sigmoid. This represents the original translation prediction weights that have not been normalized by sigmoid. Represents the encoded joint features. This represents the second relative attitude of the target object. This indicates the number of interior points corresponding to the second relative attitude. This represents a multilayer perceptron network with adaptively learned weights. In S42, the rotated weights The expression is: ； in, Represents the sigmoid function; In S42, the translated weights The expression is: ； In S43, the prior pose The expression is: ； in, This represents the first relative pose of the target object. Indicates dynamically calculated weights; In S44, the prior function The expression is: ； in, Indicates the hypothetical model, Indicates the number of fixed grid points. Indicates the first Grid points, This represents the transformation result of the prior pose. This represents the transformation result of the assumed model on the grid points; In S45, the scoring function The expression is: ； in, Indicates prior weights, This represents a feature point in the final feature map of the anchor point image. This represents a feature point in the final feature map of the queried image. Represents the energy function. Indicates an indicator function; In S46, the updated pose The expression is: ； in, Represents sparse corresponding points. This indicates a solver based on prior guidance; In S47, the final rotation matrix The expression is: ； in, This represents the rotation matrix extracted from the first relative pose of the target object. This represents the rotation matrix extracted from the updated pose; In S48, the final translation vector The expression is: ； in, This represents the translation vector extracted from the first relative pose of the target object. This represents the translation vector extracted from the updated pose.

Images