A six-degree-of-freedom pose estimation system and method based on a mamba network

Abstract

The application belongs to the technical field of augmented reality, and particularly relates to a six-degree-of-freedom pose estimation system and method based on a Mamba network, which comprises an image feature extraction and context modeling module, a scene coordinate regression and coarse pose estimation module, a pose sequence construction module, a bidirectional Mamba encoding module of a pose sequence, and a pose residual prediction and refining module; a first stage coarse pose is obtained in the first stage, and the second stage realizes the refinement of the first stage pose. The method is completely based on monocular RGB images, does not depend on depth maps, point clouds, IMU and other additional sensor data, and has the characteristics of simple implementation, light model, fast reasoning speed and adaptation to indoor and outdoor scenes.

Description

Technical Field

[0001] This application belongs to the field of augmented reality technology, specifically relating to a six-degree-of-freedom pose estimation system and method based on Mamba network. Background Technology

[0002] With the development of intelligent robots, AR / VR, and unmanned systems, high-precision camera pose estimation based on single RGB images has become a key task in computer vision. Traditional end-to-end methods mainly extract image features and directly regress pose through convolutional neural networks or Transformers. However, due to limitations in local receptive fields or global modeling efficiency, they often suffer from unstable convergence, sensitivity to environmental changes, and insufficient prediction accuracy.

[0003] Scene coordinate regression methods (such as ACE and GLACE) obtain initial poses by predicting dense pixel-level scene coordinates and combining them with PnP optimization, exhibiting strong geometric consistency and thus receiving widespread attention in recent years. However, these methods lack the ability to model the global context and cannot further refine the coarse pose. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a six-DOF pose estimation system and method based on a Mamba network. This method uses a single RGB image as input and an optimized two-stage network structure as its core to achieve high-precision pose calculation for indoor and outdoor scenes. The technical solution is as follows:

[0005] A six-DOF pose estimation system based on bidirectional Mamba includes an image feature extraction and context modeling module, a scene coordinate regression and coarse pose estimation module, a pose sequence construction module, a pose sequence bidirectional Mamba encoding module, and a pose residual prediction and refinement module.

[0006] The image feature extraction and context modeling module encodes the RGB image input image feature extraction network model into a local feature map, and performs global context modeling on the feature map through the bidirectional Mamba module to obtain context-enhanced features;

[0007] The scene coordinate regression and coarse pose estimation module concatenates local features and context enhancement features in the channel dimension, predicts the three-dimensional scene coordinates corresponding to each pixel on the image plane through the scene coordinate regression head, and estimates the first-stage coarse pose by combining intrinsic parameters and using RANSAC-based PnP pose estimation.

[0008] The pose sequence construction module constructs a pose sequence based on the coarse pose of the current frame and several previous frames, and projects the pose parameters of each frame to a high-dimensional embedding space of the same dimension to obtain a pose embedding sequence.

[0009] The pose sequence bidirectional Mamba encoding module performs bidirectional Mamba encoding on the pose embedding sequence, captures the forward and backward dependencies of the pose in the time dimension, and outputs the global context features of the pose.

[0010] The pose residual prediction and refinement module inputs the global context features of pose into the multilayer perceptron residual prediction network, outputs the pose residual vector under the three-dimensional special Euclidean group SE(3) Lie algebra, and obtains the residual transformation matrix through exponential mapping. The residual transformation matrix is ​​combined with the first-stage coarse pose to obtain the second-stage refined pose.

[0011] Preferably, it also includes a noise scheduling training module, which applies gradually increasing simulated noise to the coarse pose output in the first stage during the training process to construct training samples under multiple error distributions and improve the generalization ability of the pose residual prediction network in the second stage in the test scenario.

[0012] Preferably, the bidirectional Mamba structure in the image feature extraction and context modeling module is specifically as follows:

[0013] (1) Local feature map Rearranged to a length of Feature sequences ;in , These are the original heights of the images. Original width of , Number of channels;

[0014] (2) For the characteristic sequence Enter the forward Mamba branch and the backward Mamba branch respectively. The forward branch is set to... Update the hidden state and output the forward encoded features Backward branch Update the hidden state and output the backward encoded features ;

[0015] (3) For each position i, forward-encoded features With backward encoding features Perform splicing or weighted fusion to obtain fused features. Then Rearranged into context-enhanced feature maps according to their original spatial order. .

[0016] Preferably, during the training phase, the scene coordinate regression and coarse pose estimation module includes:

[0017] (1) Construct training samples: For each RGB image in the pose estimation dataset Preprocessing is performed to create a two-dimensional table annotation for the image, resulting in a pixel coordinate supervised label map. ;

[0018] (2) Image feature extraction: Using the ACE encoder with ResNet or a lightweight convolutional network as the backbone, the RGB image is subjected to multi-layer convolution, normalization and non-linear mapping, and the downsampled local feature map is output. ,in , These are the original heights of the images. Original width of , Number of channels;

[0019] (3) Bidirectional Mamba context modeling: Flattened into a length using either row-major or column-major order. The feature sequence is input into the bidirectional Mamba module, where forward and backward state updates and outputs are performed respectively to obtain the context-enhanced feature sequence. This sequence is then restored to a feature map and projected onto the given data. , To increase the number of channels for features;

[0020] (4) Feature stitching and scene coordinate regression: and By splicing along the channel dimension, we obtain... It predicts the 3D scene coordinates of each pixel through several layers of convolution and upsampling modules. ;

[0021] (5) Calculation of reprojection loss: The predicted scene coordinates Transformed into predicted pixel coordinates using extrinsic and intrinsic parameter matrices. Use L1 loss or loss metric and The difference in these factors leads to the supervision loss in the first stage of scene coordinate regression. This refers to the reprojection loss;

[0022] (6) Supervision of reprojection loss: In the first stage of training, the pose solver is not backpropagated directly. Instead, the scene coordinates predicted by the network are projected based on the real pose and camera intrinsic parameters, and the reprojection error on the image plane is calculated. This reprojection loss is then used to calculate the reprojection error. As a supervisory signal, the scene coordinate regression network is backpropagated and updated. By minimizing the reprojection error, the predicted scene coordinates satisfy the geometric consistency constraint, thereby indirectly improving the accuracy and stability of subsequent coarse pose solving.

[0023] Preferably, the coarse pose training module for the second stage of pose residual prediction and refinement based on the output of the first stage includes:

[0024] (1) Coarse pose generation: Using the first-stage image feature extraction network that has been trained or has fixed parameters, inference is performed on each RGB image in the training set to obtain the corresponding coarse pose. ;

[0025] (2) Calculation of true residual: Calculate the true pose for each sample. Coarse pose relative transformation And through the three-dimensional special Euclidean group SE(3) logarithmic mapping log(·), Mapped to a six-dimensional Lie algebra vector , as an ideal residual monitoring signal; This represents the rotation difference between the true pose and the coarse pose. This represents the translation difference between the true pose and the coarse pose.

[0026] (3) Pose sequence construction: In the sequence training scenario, the coarse poses of the current image and its previous N frames are selected from the time dimension to form a sequence. The pose parameters of each frame (e.g., quaternions + translation, a total of 7 dimensions) are projected onto a unified D-dimensional embedding space through a fully connected layer to obtain the pose embedding sequence. ;

[0027] (4) Bidirectional Mamba timing coding: Input a bidirectional Mamba pose sequence encoder, perform forward and backward state updates separately, and fuse the outputs to obtain the sequence encoding features. ;

[0028] (5) Global pooling and residual prediction: for Global average pooling or attention-weighted pooling is performed in the time dimension to obtain global pose context features. ,Will Input a multilayer perceptron residual prediction network and output a six-dimensional prediction residual. , To test the difference in rotation prediction between the true pose and the coarse pose in the test set, To measure the translation prediction difference between the true pose and the coarse pose in the test set;

[0029] (6) Residual supervision and optimization: The Smooth L1 loss function is used to measure the predicted residuals. Compared with the true residual The difference can be used to introduce different loss weight coefficients to the rotation and translation components, thus obtaining the loss for the second-stage pose residual prediction. The parameters of the pose sequence construction module, the bidirectional Mamba encoding module, and the residual prediction network are backpropagated and updated.

[0030] Preferably, the noise scheduling training module adopts a learning strategy that gradually increases the noise range, with the specific steps as follows:

[0031] First, the coarse pose is calculated using the target dataset. Compared with the true pose Statistical analysis of its rotational error With translation error The noise level is determined by the distribution and the upper limit of the noise level is determined by the quantiles.

[0032] coarse pose after noise injection = The "current coarse pose" serves as the input to the second-stage network, and the true residual is recalculated. It is then mapped to the Lie algebra space to construct a new supervision signal, ensuring that the second-stage network learns stably, converges, and has excellent generalization ability under a wider error distribution.

[0033] Preferably, the pose residual prediction and refinement module employs a three-dimensional special Euclidean group SE(3) Lie algebra residual supervision and combination strategy, specifically including:

[0034] (1) The prediction residual vector obtained from the second-stage network prediction Compared with the true residual Compare;

[0035] (2) Define the weighted residual loss function:

[0036] ;

[0037] in and These are the rotation and translation weight hyperparameters, respectively.

[0038] (3) By mapping the three-dimensional special Euclidean group SE(3) exponent, Convert to residual pose matrix and the first stage coarse pose Combine to obtain refined pose During the verification and testing process, rotational geodesic distance and translational error were used to evaluate the refined pose quality.

[0039] A six-DOF pose estimation method based on Mamba networks includes the following steps during the training phase:

[0040] S1. Using an RGB image dataset with camera pose annotations, the real scene coordinate labels of each pixel are obtained based on a pre-known 3D scene model or through offline optimization, and training data for the first stage of scene coordinate regression and coarse pose estimation is constructed.

[0041] S2. The image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module are jointly trained using training data, so that they can predict the three-dimensional scene coordinates of each pixel from a single frame of RGB image. Then, the scene coordinates are converted into pixel coordinates and the real pixel coordinates are used to calculate the loss for indirect supervision.

[0042] S3. Using the coarse pose output and corresponding real pose of the first-stage network on the training set, calculate the real pose residual in the three-dimensional special Euclidean group SE(3) Lie algebra space, and combine the noise scheduling training module to generate training samples with multiple error scales, and train the pose sequence construction module, the pose sequence bidirectional Mamba encoding module and the pose residual prediction and refinement module.

[0043] Reasoning stage:

[0044] Input a single-frame RGB image of the scene to be estimated. The first-stage coarse pose is obtained through the image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module. The pose sequence at the current moment is constructed, and the sequence is input into the pose sequence bidirectional Mamba encoding module. Combined with the pose residual prediction and refinement module, the second-stage refined pose is obtained, realizing pose estimation under single-frame or short sequence conditions.

[0045] Preferably, the steps for obtaining the first-stage coarse pose during the inference phase are as follows:

[0046] (1) For the input RGB image After normalization and scaling, the sampled local feature map is fed into the trained ACE encoder to obtain the downsampled local feature map. ;

[0047] (2) Flattening the feature sequence and inputting it into the bidirectional Mamba context modeling module yields an enhanced feature map with global contextual information. ;

[0048] (3) In the channel dimension and The images are stitched together and fed into a scene coordinate regression head to predict the corresponding 3D scene coordinate map pixel by pixel. ;

[0049] (4) Select using confidence level or reprojection error A subset of pixels is used to construct a 2D-3D matching set. Combined with camera intrinsics, the first-stage coarse pose of the image is solved using a RANSAC-based PnP pose solving algorithm. ;

[0050] (5) It can be used as one of the inputs for the second stage of pose refinement, or it can be used together with the coarse poses of the previous few frames in a video stream to form a pose sequence.

[0051] The preferred second-stage refined pose reasoning process:

[0052] (2) In sequence mode, collect the current frame and its preceding frames. coarse pose of frames These are respectively mapped to high-dimensional embedding vectors. Forming a length of pose embedding sequence ;

[0053] (3) Input the trained bidirectional Mamba pose sequence encoder to obtain the encoded sequence features. The global pose context vector is obtained by global pooling in the time dimension. ;

[0054] (4) Input a 3D Euclidean group SE(3) pose residual prediction network and output a 6D pose residual vector. ;

[0055] (5) By mapping the three-dimensional special Euclidean group SE(3) exponent, Convert to residual pose matrix and the first stage coarse pose By combining the positions, the final refined pose is obtained. = ;

[0056] (6) It serves as a reference for constructing the pose sequence at the next moment, and is used in the second-stage refined pose inference process of subsequent frames.

[0057] Compared with the prior art, the beneficial effects of this application are as follows:

[0058] This method uses a single RGB image as input and an optimized two-stage network structure as its core. It achieves high-precision pose calculation for indoor and outdoor scenes through a process of "scene coordinate regression + coarse pose solving + pose sequence residual refinement." This invention requires no additional information such as depth maps or point clouds, relying solely on the visual features of the image itself for inference. Attached Figure Description

[0059] Figure 1 This is a flowchart of the application process;

[0060] Figure 2 Technical framework diagram;

[0061] Figure 3 This is a simulation experiment diagram. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0063] A six-DOF pose estimation system based on Mamba network includes an image feature extraction and context modeling module, a scene coordinate regression and coarse pose estimation module, a pose sequence construction module, a pose sequence bidirectional Mamba encoding module, and a pose residual prediction and refinement module.

[0064] The image feature extraction and context modeling module encodes the RGB image input image feature extraction network model into a local feature map, and performs global context modeling on the feature map through the bidirectional Mamba module to obtain context-enhanced features;

[0065] The scene coordinate regression and coarse pose estimation module concatenates local features and context enhancement features in the channel dimension, predicts the three-dimensional scene coordinates corresponding to each pixel on the image plane through the scene coordinate regression head, and estimates the first-stage coarse pose by combining intrinsic parameters and using RANSAC-based PnP pose estimation.

[0066] The pose sequence construction module constructs a pose sequence based on the coarse pose of the current frame and several previous frames, and projects the pose parameters of each frame to a high-dimensional embedding space of the same dimension to obtain a pose embedding sequence.

[0067] The pose sequence bidirectional Mamba encoding module performs bidirectional Mamba encoding on the pose embedding sequence, captures the forward and backward dependencies of the pose in the time dimension, and outputs the global context features of the pose.

[0068] The pose residual prediction and refinement module inputs the global context features of the pose into the multilayer perceptron residual prediction network, outputs the pose residual vector under the six-dimensional three-dimensional special Euclidean group SE(3) Lie algebra, and obtains the residual transformation matrix through exponential mapping. The residual transformation matrix is ​​combined with the first-stage coarse pose to obtain the second-stage refined pose.

[0069] The method of the present invention includes two core stages in the overall process: (1) the first stage: scene coordinate prediction and coarse pose solution based on image features and Mamba global modeling; (2) the second stage: pose sequence residual prediction and final pose refinement based on bidirectional Mamba.

[0070] The first-stage coarse pose regression module directly regresses the initial six-degree-of-freedom pose of the current camera based on the above high-dimensional features, and obtains the coarse pose estimation result. The second-stage pose residual optimization module uses the intermediate features of the first stage and the coarse pose result to learn the residual between the coarse pose and the true pose in the three-dimensional special Euclidean group SE(3) Lie group-Lie algebra space, outputs the pose correction amount, refines the coarse pose, and thus obtains the final high-precision pose estimation result.

[0071] The specific methods and procedures are as follows: Figure 2 As shown, it includes the following two main stages: the model training stage and the inference stage.

[0072] A six-DOF pose estimation method based on Mamba networks includes the following steps during the training phase:

[0073] S1. Using an RGB image dataset with camera pose annotations, the real scene coordinate labels of each pixel are obtained based on a pre-known 3D scene model or through offline optimization, and training data for the first stage of scene coordinate regression and coarse pose estimation is constructed.

[0074] S2. The image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module are jointly trained using training data, so that they can predict the three-dimensional scene coordinates of each pixel from a single frame of RGB image. Then, the scene coordinates are converted into pixel coordinates and the real pixel coordinates are used to calculate the loss for indirect supervision.

[0075] S3. Using the coarse pose output and corresponding real pose of the first-stage network on the training set, calculate the real pose residual in the three-dimensional special Euclidean group SE(3) Lie algebra space, and combine the noise scheduling training module to generate training samples with multiple error scales, and train the pose sequence construction module, the pose sequence bidirectional Mamba encoding module and the pose residual prediction and refinement module.

[0076] Reasoning stage:

[0077] Input a single-frame RGB image of the scene to be estimated. The first-stage coarse pose is obtained through the image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module. The pose sequence at the current moment is constructed, and the sequence is input into the pose sequence bidirectional Mamba encoding module. Combined with the pose residual prediction and refinement module, the second-stage refined pose is obtained, realizing pose estimation under single-frame or short sequence conditions.

[0078] I. Model Training Phase:

[0079] The model training phase includes the first phase, which is the scene coordinate regression and coarse pose estimation module training, and the second phase, which is the pose residual prediction and refinement module training. Through this two-stage training process, the network first learns the global mapping from the image to the coarse pose, and then learns to perform residual correction on the pose in the three-dimensional special Euclidean group SE(3) space.

[0080] Training of the first-order scene coordinate regression and coarse pose estimation module:

[0081] 1. Training data construction:

[0082] (1) Select the pose estimation dataset, and denote each image as . Size is The corresponding real camera pose is denoted as (2) The actual pose use Homogeneous transformation matrix representation: = ,in for Rotation matrix, for Translation vector;

[0083] (3) For each RGB image in the pose estimation dataset Preprocessing is performed to create a two-dimensional table annotation for the image, resulting in a pixel coordinate supervised label map. (4) Construct training sample pairs The training dataset that constitutes the coarse pose regression network.

[0084] 2. Image preprocessing:

[0085] (1) Input image (1) Perform normalization processing to scale the pixel values ​​of each channel to [0,1] or a standard normal distribution; (2) Scale or fill the image to a fixed size, for example (3) Further data augmentation, such as random flipping and color jittering, can be performed on the training set to enhance the generalization ability of the network and obtain images. .

[0086] 3. Image feature extraction and bidirectional Mamba coding:

[0087] (1) Local convolution feature extraction: for the preprocessed image Inputting multiple convolutional layers and residual blocks yields local feature maps. , dimension .

[0088] (2) Feature serialization: converting feature maps Expanded by rows or columns, it becomes a feature sequence of length L. .

[0089] (3) Forward Mamba encoding: Encodes the sequence by The sequential inputs to the forward Mamba module yield the forward encoded feature sequence. .

[0090] (4) Reverse Mamba encoding: converting the sequence into a sequence by The sequential inputs to the reverse Mamba module yield the reverse encoded feature sequence. .

[0091] (5) Two-way feature fusion: for each location The forward and backward features are concatenated along the channel dimension to obtain... ,in ;Will Rearrange back into a two-dimensional format to obtain the fused features. .

[0092] 4. Feature fusion and global pose vector construction:

[0093] (1) Channel splicing: and By stitching the channels in one dimension, a fused feature map is obtained. ;

[0094] (2) Fully connected mapping: By inputting multiple fully connected layers, the predicted scene coordinates are obtained, resulting in the coarse pose vector predicted by the first-stage network. .

[0095] 5. Calculation of reprojection loss:

[0096] The first stage of pose loss function design: This stage does not directly optimize the pose, but rather trains the network to predict the 3D scene coordinates corresponding to pixels. The loss function measures the reprojection error between the predicted 3D coordinates projected back onto the image plane and the actual pixel positions. To achieve robust training without RANSAC, the following strategy is adopted:

[0097] Tanh Robust Truncation: Instead of directly using L2 or L1 distance, it uses the hyperbolic tangent function (Tanh) to handle reprojection errors. This effectively limits the gradient influence of outliers, preventing the network from being dominated by incorrect predictions.

[0098] Dynamic Curriculum Schedule: Introduces a threshold that decays over training time. (For example, reducing it from 50px to 1px). In the early stages of training, the threshold is relatively large, allowing the network to learn the general structure of the scene; as training progresses, the threshold is gradually tightened, forcing the network to focus on high-precision predictions.

[0099] Brief description of the mathematical form:

[0100] ;

[0101] ;

[0102] in These are the predicted scene coordinates. It is the true pose. It's the pixel position. It is a projection function. As a threshold, the relative training time t is dynamically adjusted during training, maintaining a close value at the beginning of training. And dropped to at the end of training. , and These are all hyperparameters.

[0103] Predicted scene coordinates Transformed into predicted pixel coordinates using extrinsic and intrinsic parameter matrices. Use L1 loss or loss metric and The difference in these factors leads to the supervision loss in the first stage of scene coordinate regression. This refers to the reprojection loss.

[0104] Throughout the training phase, the parameters of the image feature extraction network, the bidirectional Mamba encoder, and the fully connected regression layer are continuously updated via backpropagation until... convergence.

[0105] 6. Monitoring of reprojection loss:

[0106] Supervision of reprojection loss: During the first stage of training, instead of directly backpropagating to the pose solver, the scene coordinates predicted by the network are projected based on the real pose and camera intrinsic parameters. The reprojection error on the image plane is calculated, and this reprojection loss is applied. The scene coordinate regression network is updated via backpropagation using the supervision signal. By minimizing the reprojection error, the predicted scene coordinates satisfy the geometric consistency constraint, thereby indirectly improving the accuracy and stability of the subsequent coarse pose solution based on RANSAC-based PnP.

[0107] The noise scheduling training module applies gradually increasing simulated noise to the coarse pose output in the first stage during training to construct training samples under multiple error distributions, thereby improving the generalization ability of the pose residual prediction network in the second stage in the test scenario.

[0108] First, the first-stage method is run on the target dataset to obtain the coarse pose. Compared with the true pose Statistical analysis of its rotational error (Geometric angle, degrees) and translation error (Euclidean distance, meters) distribution, and the upper limit of noise is determined by quantiles, taking ,in Represents the 90th quantile, calculated during training. Inject rotational noise (in degrees), according to Injected translational noise (unit: meters, or converted to...) (cm), ensuring that the training noise is on the same order of magnitude as the actual coarse pose error, avoiding training-test mismatch. Simultaneously, a curriculum-based noise enhancement strategy is employed: a smaller upper limit is used in the early stages of training (e.g., cm), to ensure that the training noise is of the same order of magnitude as the actual coarse pose error, avoiding training-test mismatch. A lesson-based noise enhancement strategy is also adopted: a smaller upper limit is used in the early stages of training (e.g., cm ), and then gradually increased to This aims to improve robustness and generalization ability to larger initial value deviations while ensuring convergence stability.

[0109] coarse pose after noise injection = The "current coarse pose" serves as the input to the second-stage network, and the true residual is recalculated. It is then mapped to the Lie algebra space to construct a new supervision signal, ensuring that the second-stage network learns stably, converges, and has excellent generalization ability under a wider error distribution.

[0110] Training of the pose residual prediction and refinement module in the second stage:

[0111] (1) Coarse pose generation: Using the first-stage image feature extraction network that has been trained or has fixed parameters, inference is performed on each RGB image in the training set to obtain the corresponding coarse pose. .

[0112] (2) Calculation of true residual: Calculate the true pose for each sample. Coarse pose relative transformation And through the three-dimensional special Euclidean group SE(3) logarithmic mapping log(·), Mapped to a six-dimensional Lie algebra vector , as an ideal residual monitoring signal; For rotational residuals, This is the translation residual.

[0113] (3) Pose sequence construction: In the sequence training scenario, the coarse poses of the current image and its previous N frames are selected from the time dimension to form a sequence. The pose parameters of each frame (e.g., quaternions + translation, a total of 7 dimensions) are projected onto a unified D-dimensional embedding space through a fully connected layer to obtain the pose embedding sequence. .

[0114] (4) Bidirectional Mamba timing coding: Input a bidirectional Mamba pose sequence encoder, perform forward and backward state updates separately, and fuse the outputs to obtain the sequence encoding features. .

[0115] (5) Global pooling and residual prediction: for Global average pooling or attention-weighted pooling is performed in the time dimension to obtain global pose context features. ,Will Input a multilayer perceptron residual prediction network and output a six-dimensional prediction residual. .

[0116] (6) Residual supervision and optimization: The Smooth L1 loss function is used to measure the predicted residuals. Compared with the true residual The difference can be used to introduce different loss weight coefficients to the rotation and translation components, thus obtaining the loss for the second-stage pose residual prediction. The parameters of the pose sequence construction module, the bidirectional Mamba encoding module, and the residual prediction network are backpropagated and updated.

[0117] II. Reasoning process:

[0118] Input a single-frame RGB image of the scene to be estimated. After image feature extraction and context modeling module and scene coordinate regression and coarse pose estimation module, the first-stage coarse pose is obtained. Construct the pose sequence at the current time. Input the sequence into the pose sequence bidirectional Mamba encoding module. Combine with the pose residual prediction and refinement module to obtain the second-stage refined pose, realizing high-precision pose estimation under single-frame or short sequence conditions.

[0119] The image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module in the inference phase perform the following steps:

[0120] (1) For the input RGB image After normalization and scaling, the sampled local feature map is fed into a trained convolutional neural network (CNN) encoder to obtain the downsampled local feature map. .

[0121] (2) Flattening the feature sequence and inputting it into the bidirectional Mamba context modeling module yields an enhanced feature map with global contextual information. .

[0122] (3) In the channel dimension and The images are stitched together and fed into a scene coordinate regression head to predict the corresponding 3D scene coordinate map pixel by pixel. .

[0123] (4) Select using confidence level or reprojection error A subset of pixels is used to construct a 2D-3D matching set. Combined with camera intrinsics, the first-stage coarse pose of the image is solved using a RANSAC-based PnP pose solving algorithm. This application employs perspective-n-point (PnP) pose estimation based on Random Sample Consensus (RANSAC) to perform consistency screening on candidate 2D–3D corresponding points and solve for coarse pose. As an example, this application uses the following parameter settings during testing: Input a 3D scene coordinate graph. The input image has a baseline resolution of 480; the RANSAC iteration count (number of hypotheses) is set to 64; points with a reprojection error of less than 10 pixels are identified as inliers and used to construct the 2D–3D matching set; the smoothing coefficient for soft inlier counting is set to 100 to control the smoothness of the hypothesis score distribution; to ensure numerical stability, the single-point reprojection error is truncated to within 100 pixels during the consistency check, meaning that errors exceeding 100 pixels are counted as 100 pixels in the calculation. This yields the first-stage coarse pose. .

[0124] (5) It can be used as one of the inputs for the second stage of pose refinement, or it can be used together with the coarse poses of the previous few frames in a video stream to form a pose sequence.

[0125] The execution steps of the pose sequence construction module, the pose sequence bidirectional Mamba encoding module, and the pose residual prediction and refinement module in the inference phase include:

[0126] (1) In sequence mode, collect the current frame and its preceding frames. coarse pose of frames These are respectively mapped to high-dimensional embedding vectors. Forming a length of pose embedding sequence .

[0127] (2) Input the trained bidirectional Mamba pose sequence encoder to obtain the encoded sequence features. The global pose context vector is obtained by global pooling in the time dimension. .

[0128] (3) Input a 3D Euclidean group SE(3) pose residual prediction network and output a 6D pose residual vector. .

[0129] (4) Through the three-dimensional special Euclidean group SE(3) exponent mapping, Convert to Residual transformation matrix and the first stage coarse pose By combining the positions, the final refined pose is obtained. .

[0130] (5) In scenarios requiring long-term continuous positioning, from Read from Rotation matrix and The translation vector represents the six-DOF pose estimation result of the camera output by the method of this invention. Simultaneously, It serves as a reference for constructing the pose sequence at the next moment, and is used in the second-stage refinement process of subsequent frames.

[0131] The pose residual prediction and refinement module employs a three-dimensional special Euclidean group SE(3) Lie algebra residual supervision and combination strategy, specifically including:

[0132] (1) The residual vector obtained from the second-stage network prediction With the true residual vector Comparison, among which For rotational residuals, For translation residuals;

[0133] (2) Define the weighted residual loss function:

[0134] ;

[0135] in and These are the rotation and translation weight hyperparameters, respectively.

[0136] (3) By mapping the three-dimensional special Euclidean group SE(3) exponent, Convert to residual pose matrix and the first stage coarse pose Combine to obtain refined pose During the verification and testing process, rotational geodesic distance and translational error were used to evaluate the refined pose quality.

[0137] The above two-stage Mamba pose estimation method fully utilizes the ACE and bidirectional Mamba structure to extract global contextual information in a single frame image in the first stage to obtain a relatively reliable coarse pose. In the second stage, the residual between the coarse pose and the true pose is explicitly learned in the three-dimensional special Euclidean group SE(3) space, and a progressive perturbation enhancement strategy is introduced to enable the system to output a more accurate and stable final pose estimation result under the condition of different initial errors, thereby significantly improving the overall pose regression accuracy and robustness.

[0138] Table 1 Comparison of Experimental Results

[0139]

[0140] Table 1 shows the publicly available indoor camera relocalization dataset 7-Scenes (7Scenes) used for verification in this embodiment. This dataset consists of multiple indoor sequences, providing the ground truth camera pose for each frame and the corresponding image data (although the dataset contains depth information, this application only inputs monocular RGB images during the inference stage and does not rely on additional information such as depth maps and point clouds). Table 1 presents the comparison results of localization accuracy on the 7Scenes dataset, where translation error is measured in centimeters (cm), rotation error in degrees (deg), and the median is used as the statistical indicator. The comparison methods include hLoc(SP+SG) and DenseVLAD+R2D2: the former is a hierarchical localization framework, which typically uses SuperPoint to extract local features and combines SuperGlue to complete matching before solving for geometric pose; the latter uses DenseVLAD for global retrieval and combines R2D2 for local feature matching to complete pose estimation. Both are typical visual localization schemes based on retrieval and feature matching. As shown in Table 1, under the same RGB inference input conditions, the median translation error of the method in this application is 1.22 cm and the median rotation error is 0.36°, which are significantly lower than those of the comparison methods (approximately 3.26–3.31 cm and 1.07–1.08°). This indicates that the two-stage scheme of "scene coordinate regression + coarse pose solution + pose sequence residual refinement" described in this application can achieve higher accuracy and stronger robustness of pose calculation in indoor scenes, and the inference stage does not require additional information such as depth maps and point clouds.

Claims

1. A six-DOF pose estimation system based on a Mamba network, characterized in that, It includes modules for image feature extraction and context modeling, scene coordinate regression and coarse pose estimation, pose sequence construction, bidirectional Mamba coding of pose sequences, and pose residual prediction and refinement. The image feature extraction and context modeling module encodes the RGB image input image feature extraction network model into a local feature map, and performs global context modeling on the feature map through the bidirectional Mamba module to obtain context-enhanced features; The scene coordinate regression and coarse pose estimation module concatenates local features and context enhancement features in the channel dimension, predicts the three-dimensional scene coordinates corresponding to each pixel on the image plane through the scene coordinate regression head, and estimates the first-stage coarse pose by combining intrinsic parameters and using RANSAC-based PnP pose estimation. The pose sequence construction module constructs a pose sequence based on the coarse pose of the current frame and several previous frames, and projects the pose parameters of each frame to a high-dimensional embedding space of the same dimension to obtain a pose embedding sequence. The pose sequence bidirectional Mamba encoding module performs bidirectional Mamba encoding on the pose embedding sequence, captures the forward and backward dependencies of the pose in the time dimension, and outputs the global context features of the pose. The pose residual prediction and refinement module inputs the global context features of the pose into the multilayer perceptron residual prediction network, outputs the pose residual vector under the three-dimensional special Euclidean group SE(3) Lie algebra, and obtains the residual transformation matrix through exponential mapping. The residual transformation matrix is ​​combined with the first-stage coarse pose to obtain the second-stage refined pose.

2. The six-DOF pose estimation system based on Mamba network according to claim 1, characterized in that, It also includes a noise scheduling training module, which applies gradually increasing simulated noise to the coarse pose output in the first stage during training to construct training samples under multiple error distributions and improve the generalization ability of the pose residual prediction network in the second stage in the test scenario.

3. The six-DOF pose estimation system based on Mamba network according to claim 1, characterized in that, During the training phase, the scene coordinate regression and coarse pose estimation module includes: (1) Construct training samples: For each RGB image in the pose estimation dataset Preprocessing is performed to create a two-dimensional table annotation for the image, resulting in a pixel coordinate supervised label map. ; (2) Image feature extraction: Using the ACE encoder with ResNet or a lightweight convolutional network as the backbone, the RGB image is subjected to multi-layer convolution, normalization and non-linear mapping, and the downsampled local feature map is output. ,in , The original height of the input RGB image is shown below. Original width of , Number of channels; (3) Bidirectional Mamba context modeling: Flattened into a length of either row-major or column-major order. The feature sequence is input into the bidirectional Mamba module, where forward and backward state updates and outputs are performed respectively to obtain the context-enhanced feature sequence. This sequence is then restored to a feature map and projected onto the given data. , To increase the number of channels for features; (4) Feature stitching and scene coordinate regression: and By splicing along the channel dimension, we obtain... It predicts the 3D scene coordinates of each pixel through several layers of convolution and upsampling modules. ; (5) Calculation of reprojection loss: The predicted scene coordinates Transformed into predicted pixel coordinates using extrinsic and intrinsic parameter matrices. Use L1 loss or loss metric and The difference in these factors leads to the supervision loss in the first stage of scene coordinate regression. This refers to the reprojection loss; (6) Supervision of reprojection loss: In the first stage of training, the pose solver is not backpropagated directly. Instead, the scene coordinates predicted by the network are projected based on the real pose and camera intrinsic parameters, and the reprojection error on the image plane is calculated. This reprojection loss is then used to calculate the reprojection error. As a supervisory signal, the scene coordinate regression network is backpropagated and updated.

4. The six-DOF pose estimation system based on Mamba network according to claim 1, characterized in that, The bidirectional Mamba structure in the image feature extraction and context modeling module is specifically as follows: (1) Local feature map Rearranged to a length of Feature sequences ;in , The original height of the input RGB image is shown below. Original width of , Number of channels; (2) For the characteristic sequence Enter the forward Mamba branch and the backward Mamba branch respectively. The forward branch is set to... Update the hidden state and output the forward encoded features Backward branch Update the hidden state and output the backward encoded features ; (3) For each position , forward encoded features With backward encoding features Perform splicing or weighted fusion to obtain fused features. Then Rearranged into context-enhanced feature maps according to their original spatial order. .

5. The six-DOF pose estimation system based on Mamba network according to claim 1, characterized in that, The coarse pose training module based on the first-stage output is used to train the second-stage pose residual prediction and refinement module, which includes: (1) Coarse pose generation: Using the first-stage image feature extraction network that has been trained or has fixed parameters, inference is performed on each RGB image in the training set to obtain the corresponding coarse pose. ; (2) Calculation of true residual: Calculate the true pose for each sample. Coarse pose relative transformation And through the three-dimensional special Euclidean group SE(3) logarithmic mapping log(·), Mapped to a six-dimensional Lie algebra vector , as an ideal residual monitoring signal; This represents the rotation difference between the true pose and the coarse pose. This represents the translation difference between the true pose and the coarse pose. (3) Pose sequence construction: In the sequence training scenario, the coarse poses of the current image and its previous N frames are selected from the time dimension to form a sequence. The pose parameters of each frame are projected onto a unified D-dimensional embedding space through a fully connected layer to obtain the pose embedding sequence. ; (4) Bidirectional Mamba timing coding: Input a bidirectional Mamba pose sequence encoder, perform forward and backward state updates separately, and fuse the outputs to obtain the sequence encoding features. ; (5) Global pooling and residual prediction: for Global average pooling or attention-weighted pooling is performed in the time dimension to obtain global pose context features. ,Will Input a multilayer perceptron residual prediction network and output a six-dimensional prediction residual. , To test the difference in rotation prediction between the true pose and the coarse pose in the test set, To measure the translation prediction difference between the true pose and the coarse pose in the test set; (6) Residual supervision and optimization: The smoothed L1 loss function is used to measure the predicted residuals. Compared with the true residual The difference can be used to introduce different loss weight coefficients to the rotation and translation components, thus obtaining the loss for the second-stage pose residual prediction. The parameters of the pose sequence construction module, the bidirectional Mamba encoding module, and the residual prediction network are backpropagated and updated.

6. The six-DOF pose estimation system based on Mamba network according to claim 2, characterized in that, The noise dispatching training module employs a learning strategy that gradually increases the noise range. The specific steps are as follows: First, the coarse pose is calculated using the target dataset. Compared with the true pose Statistical analysis of its rotational error With translation error The noise level is determined by the distribution and the upper limit of the noise level is determined by the quantiles. Then, the coarse pose after noise injection = The "current coarse pose" serves as the input to the second-stage network, and the true residual is recalculated. It is then mapped to the Lie algebra space to construct a new monitoring signal.

7. The six-DOF pose estimation system based on Mamba network according to claim 4, characterized in that, The pose residual prediction and refinement module employs a three-dimensional special Euclidean group SE(3) Lie algebra residual supervision and combination strategy, specifically including: (1) The prediction residual vector obtained from the second-stage network prediction Compared with the true residual Compare; (2) Define the weighted residual loss function: ； in and These are the rotation and translation weight hyperparameters, respectively. (3) The three-dimensional special Euclidean group SE(3) exponent mapping is used to... Convert to residual pose matrix and the first stage coarse pose Combine to obtain refined pose During the verification and testing process, rotational geodesic distance and translational error were used to evaluate the refined pose quality.

8. A six-DOF pose estimation method based on Mamba networks, characterized in that, The training phase includes the following steps: S1. Using an RGB image dataset with camera pose annotations, the training data for the first stage of scene coordinate regression and coarse pose estimation is constructed based on a pre-known 3D scene model or by obtaining the real scene coordinate label of each pixel through offline optimization. S2. Jointly train the image feature extraction and context modeling module, as well as the scene coordinate regression and coarse pose estimation module, using training data to enable them to predict the 3D scene coordinates of each pixel from a single frame of RGB image, and apply reprojection loss. Backpropagation of gradients indirectly supervises scene coordinate regression. S3. Using the coarse pose output and corresponding real pose of the first-stage network on the training set, calculate the real pose residual in the three-dimensional special Euclidean group SE(3) Lie algebra space, and generate training samples with multiple error scales by combining the noise scheduling training module. Train the pose sequence construction module, the pose sequence bidirectional Mamba encoding module, and the pose residual prediction and refinement module. Reasoning stage: Input a single-frame RGB image of the scene to be estimated. The first-stage coarse pose is obtained through the image feature extraction and context modeling module and the scene coordinate regression and coarse pose estimation module. The pose sequence at the current moment is constructed, and the sequence is input into the pose sequence bidirectional Mamba encoding module. Combined with the pose residual prediction and refinement module, the second-stage refined pose is obtained, realizing pose estimation under single-frame or short sequence conditions.

9. The six-DOF pose estimation method based on Mamba network according to claim 8, characterized in that, The steps for obtaining the first-stage coarse pose during the inference phase are as follows: (1) For the input RGB image After normalization and scaling, the sampled local feature map is fed into the trained ACE encoder to obtain the downsampled local feature map. ; (2) Flattening the feature sequence and inputting it into the bidirectional Mamba context modeling module yields an enhanced feature map with global contextual information. ; (3) In the channel dimension and The images are stitched together and fed into a scene coordinate regression head to predict the corresponding 3D scene coordinate map pixel by pixel. ; (4) Select using confidence level or reprojection error A subset of pixels is used to construct a 2D-3D matching set. Combined with camera intrinsics, the first-stage coarse pose of the image is solved using a RANSAC-based PnP pose solving algorithm. ; (5) It can be used as one of the inputs for the second stage of pose refinement, or it can be used together with the coarse poses of the previous few frames in a video stream to form a pose sequence.

10. The six-DOF pose estimation method based on Mamba network according to claim 8, characterized in that, The second stage of refining the pose reasoning process: (1) In sequence mode, collect the current frame and the previous frame. coarse pose of frames These are respectively mapped to high-dimensional embedding vectors. This forms a pose embedding sequence of length N. ; (2) Input the trained bidirectional Mamba pose sequence encoder to obtain the encoded sequence features. And perform global pooling in the time dimension to obtain the global pose context vector. ; (3) Input a 3D Euclidean group SE(3) pose residual prediction network and output a 6D pose residual vector. ; (4) By mapping the three-dimensional special Euclidean group SE(3) exponent, Convert to residual pose matrix and the first stage coarse pose Combine to obtain refined pose ; (5) It serves as a reference for constructing the pose sequence at the next moment, and is used in the second-stage refined pose inference process of subsequent frames.

Images