Camera six-degree-of-freedom pose estimation system and method based on three-dimensional gaussian optimization

Abstract

This application belongs to the field of computer vision technology, specifically relating to a camera six-DOF pose estimation system and method based on 3D Gaussian optimization. The invention generates initial network weights through an initial scene coordinate regression module, regresses 3D Gaussian attribute information through a 3D Gaussian optimization module, optimizes the initial network weights through differentiable rendering, renders the image corresponding to the optimized pose using a pre-trained 3D Gaussian model through an image rendering module, and completes camera coordinate residual regression using the original and rendered images through a camera coordinate residual regression module. This invention combines 3D Gaussian with object pose estimation tasks, further optimizing camera relocalization results by introducing pixel features from the image. The entire process conforms to geometric disorder, further improving the algorithm's adaptability and robustness.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, specifically relating to a six-degree-of-freedom camera pose estimation system and method based on three-dimensional Gaussian optimization. Background Technology

[0002] Camera six-DOF pose estimation refers to calculating the position and orientation of a camera in the world coordinate system. It is a core technology for applications such as augmented reality, autonomous driving, industrial robot grasping, and medical assisted navigation.

[0003] Existing visual pose estimation methods are mainly divided into three categories: feature-based methods and learning-based methods. Feature-based methods (such as SIFT+PnP) are prone to failure in weak or repetitive texture regions, and the computational complexity increases significantly with the number of feature points. Among the learning-based methods, there are absolute camera pose regression (such as PoseNet, DFNet) and scene coordinate regression methods (such as DSAC, ACE). However, existing methods usually have the following limitations: (1) Lack of closed-loop verification: Traditional pose estimation is often a feedforward process, lacking a "synthesis-contrast" mechanism to verify whether the estimated pose is visually consistent with the real scene. (2) Insufficient utilization of geometric consistency: Although 3D Gaussian sputtering (3DGS) provides high-quality real-time rendering capabilities, there are currently few methods that can effectively combine the explicit scene representation of 3DGS with the pose regression task to use rendering errors to correct the pose in reverse.

[0004] Therefore, there is an urgent need for a method that can combine the high-fidelity rendering capabilities of 3D Gaussian with the regression capabilities of deep learning to further improve the accuracy and robustness of pose estimation. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a camera six-DOF pose estimation system and method based on three-dimensional Gaussian optimization, aiming to improve the accuracy and robustness of camera six-DOF pose estimation and provide strong support for fields such as industrial automation, robot vision, AR / VR, autonomous driving, and medical robots.

[0006] A camera six-DOF pose estimation system based on 3D Gaussian optimization includes an initial scene coordinate regression module, a 3D Gaussian optimization module, an image rendering module, and a camera coordinate residual regression module.

[0007] Initial scene coordinate regression module: Obtain the original query image to be located. The dataset is divided into training and testing sets. The training set images are processed by the initial scene coordinate regression module to generate the initial network weights. ;

[0008] 3D Gaussian optimization module: based on the initial network weights and original image information Regression of 3D Gaussian property information A loss function is constructed using differentiable rendering technology, and the initial network weights are optimized through backpropagation to obtain the optimized network weights. The pose is then obtained by selecting reliable Gaussian spheres and performing the PNP+RANSAC algorithm. ;

[0009] Image rendering module: Renders the pose obtained from the 3D Gaussian optimization module. Using a pre-trained 3D Gaussian model Perform view composition to generate a rendered image corresponding to the current estimated pose. ;

[0010] Camera coordinate residual regression module: This module returns the original query image to its original coordinates. With the rendered image Simultaneously inputting the camera coordinate residual regression module extracts the feature differences between the two, and regresses the camera coordinate residual. .

[0011] Preferably, the initial coordinate scene regression module is designed based on the ACE method, adopts the ResNet architecture, and only trains the decoder during training to construct a 3D Gaussian attribute regression network, including:

[0012] (1) Constructing an image containing RGB Real camera pose and camera internal reference The dataset is augmented during training by rotating and scaling the training data;

[0013] (2) Load the pre-trained weights into the encoder and set its requirements_grad=False. Randomly initialize the decoder parameters. After each photo passes through the encoder, high-dimensional features are generated. Then, a portion of the high-dimensional features are randomly selected and placed into the buffer pool. After processing all the data, the training process begins.

[0014] (3) The selected high-dimensional features are fed into the decoder and the predicted scene coordinates are output. ;

[0015] (4) Using reprojection error loss, Using real camera pose and camera internal reference Projecting back onto the image plane yields pixel coordinates Calculate its coordinates relative to the actual pixel coordinates. distance Backpropagation updates the decoder weights;

[0016] .

[0017] Preferably, the 3D Gaussian optimization module establishes a mapping between image features and 3D Gaussian properties, and optimizes the initial network, including:

[0018] (1) Extract the scene coordinate features from the two CNN networks respectively With RGB image features Concatenate along the channel dimension as input;

[0019] (2) The network outputs the rotation, scaling and opacity of a 3D Gaussian.

[0020] (3) Using the currently predicted Gaussian properties and scene coordinates as the center of Gaussian, combined with the actual camera pose. and Images are rendered using a miniaturized rasterizer. ;

[0021] (4) Construct the total loss function ;

[0022] in, The reprojection error is the scene coordinate. Constrain the Gaussian radius to be proportional to the reprojection error; , , To adjust the parameters;

[0023] (5) By minimizing Simultaneously, the parameters of the decoders for the 3D Gaussian attribute regression network and the scene coordinate regression network are updated.

[0024] Preferably, the image rendering module includes:

[0025] (1) Load the pre-trained 3D Gaussian model for this scene. ;

[0026] (2) Receive the coarse pose calculated by the PNP+Ransac algorithm. Perform 3D Gaussian rendering to generate a rendered image. This is to prepare for the camera coordinate residual regression module.

[0027] Preferably, the camera coordinate residual regression module outputs coordinate residuals, specifically including:

[0028] (1) Input the original query image Rendering images and predicted scene coordinates ;

[0029] (2) Two CNNs are used to extract the feature difference map of the two images, and then the residual Mr is regressed through a fully connected layer;

[0030] (3) The supervised value is calculated using the coordinate residual calculation formula based on pose difference, as follows:

[0031] ;

[0032] in, It predicts the scene coordinates in the camera coordinate system. and These are rotations and translations of the coarse pose. and These are rotation and translation of the actual pose. These are the supervisory values, and E is the identity matrix;

[0033] (4) By minimizing the network output value and To mitigate the losses between these factors, we optimize the camera coordinate residual regression network.

[0034] A six-DOF camera pose estimation method based on 3D Gaussian optimization includes the following steps:

[0035] S1. Construct a training dataset containing RGB images and their corresponding real camera poses; Initial network training: Use the training dataset to train the decoder of the initial scene coordinate regression module so that it outputs the scene coordinates corresponding to the pixels;

[0036] S2. Joint optimization training, 3D Gaussian attribute regression network, integrates image features with scene coordinate features, and outputs 3D Gaussian rotation, scaling and opacity attributes;

[0037] S3. Use differentiable rendering to obtain a rendered image, combine the reprojection error and the Gaussian radius constrained by the reprojection error to obtain the total loss function, and jointly update the weights of the decoder and the 3D Gaussian attribute regression network by minimizing the total loss function;

[0038] S4. Camera coordinate residual network training: Using the original image, rendered image and camera coordinates as input, the residual in the camera coordinate system is calculated using the true pose and estimated pose as the supervision value to train the camera coordinate residual regression module.

[0039] Preferably, step S1 uses the ACE method, and the network architecture uses ResNet, which includes an encoder and a decoder. The specific steps of the training phase are as follows:

[0040] (1) Construct a training set based on the pose estimation dataset, which includes RGB images and real camera poses. and camera internal reference ;

[0041] (2) Network initialization settings: load pre-trained weights as encoder and keep them frozen during training; only initialize and train the parameters of decoder.

[0042] (3) Forward propagation feature extraction: Input the RGB images of the training set into the encoder to extract high-dimensional features. Downsampling is performed, and training is accelerated by randomly selecting features. The selected features are then decoded by a decoder.

[0043] (4) Scene coordinate regression: The decoder outputs the predicted scene coordinates for each pixel in the image. ;

[0044] (5) Loss calculation and backpropagation, predicting scene coordinates Using real camera pose and camera internal reference Reprojection, calculating the error between pixels The network weights of the decoder are updated by minimizing this error.

[0045] Preferably, a 3D Gaussian attribute regression network is constructed and its 3D Gaussian attribute is combined with the object pose estimation task. The specific steps in the training phase are as follows:

[0046] (1) Feature fusion: High-dimensional features of scene coordinates and RGB images are extracted by two CNN networks, respectively, and denoted as follows: and Then, the inputs are fused to construct the input of the Gaussian regression network. ;

[0047] (2) Attribute regression: The three-dimensional Gaussian attribute regression network outputs the attribute information of the three-dimensional Gaussian. Specifically, this includes rotation, scaling, and opacity properties;

[0048] (3) Differentiable rendering, utilizing the regressed Gaussian properties and predicted scene coordinates Irender generates the rendered image (Irender) from the current viewpoint using differentiable rendering technology;

[0049] (4) Construct the total loss function and calculate the photometric error between the rendered image and the original RGB image. and structural similarity error And combined with reprojection error Error in calculating Gaussian radius The total loss function is obtained. ;

[0050] (5) Joint optimization, by minimizing the total loss function The decoder of the initial scene coordinate regression module and the weights of the 3D Gaussian attribute regression network are jointly updated via backpropagation.

[0051] Preferably, a pre-trained 3D Gaussian model is used. The specific steps for view composition during the training phase are as follows:

[0052] (1) Obtain the pre-trained model and load the pre-trained 3D Gaussian model for the current scene;

[0053] (2) Obtain the coarse camera pose. In the training process, the reliable Gaussian sphere obtained in the previous step is selected, and the coarse camera pose Tcoarse is calculated using the PNP+Ransac algorithm.

[0054] (3) Image rendering, based on the coarse pose Tcoarse and camera intrinsic parameters The three-dimensional Gaussian model G is projected onto a two-dimensional plane and then rasterized.

[0055] (4) Generate rendering images and synthesize rendering images that strictly correspond to the current input pose;

[0056] (5) Data output: output the generated rendered image. and the original query image The paired training data is passed to the camera coordinate residual regression module.

[0057] Preferably, step S4 uses the original image and the rendered image to complete the camera coordinate residual regression. The specific steps of the training phase are as follows:

[0058] (1) Data preparation, obtaining the original query images Rendered images generated by the image rendering module and the current predicted scene coordinates ;

[0059] (2) Supervision signal calculation, based on the real camera pose in the dataset. Calculate the true residual value in the camera coordinate system using the current coarse pose Tcoarse. ;

[0060] (3) Feature difference extraction: extract the original query image With rendering images Simultaneously input the residual regression network to extract the feature differences between the two. ;

[0061] (4) Residual regression, combined with the input predicted scene coordinates The network outputs the predicted camera coordinate residuals. ;

[0062] (5) Loss calculation and backpropagation, with As a supervisor, a loss function is constructed. The weights of the camera coordinate residual regression module are updated through backpropagation to achieve the training of the camera coordinate residual regression module.

[0063] Compared with the prior art, the beneficial effects of this application are as follows:

[0064] This invention generates initial network weights through an initial scene coordinate regression module, regresses 3D Gaussian attribute information through a 3D Gaussian optimization module, optimizes the initial network weights through differentiable rendering, renders the image corresponding to the optimized pose using a pre-trained 3D Gaussian model through an image rendering module, and completes camera coordinate residual regression using the original and rendered images through a camera coordinate residual regression module. This invention combines 3D Gaussian with object pose estimation, further optimizing camera relocalization results by introducing pixel features from the image. The entire process conforms to geometric disorder, further improving the algorithm's adaptability and robustness. This invention can achieve accurate estimation of the six-DOF camera pose, providing strong support for fields such as industrial automation, robot vision, AR / VR, autonomous driving, and medical robotics. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0066] Figure 1 This is a schematic diagram of the framework disclosed in the embodiments of the present invention;

[0067] Figure 2 This is a schematic diagram of the process disclosed in the embodiments of the present invention;

[0068] Figure 3 To downsample real images;

[0069] Figure 4 Render the image for downsampling;

[0070] Figure 5 To render the image;

[0071] Figure 6 To output realistic images. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0073] like Figure 1 As shown, a camera six-DOF pose estimation system based on 3D Gaussian optimization includes an initial scene coordinate regression module, a 3D Gaussian optimization module, an image rendering module, and a camera coordinate residual regression module.

[0074] Initial scene coordinate regression module: Obtain the original query image to be located. The dataset is divided into training and testing sets. The training set images are processed by the initial scene coordinate regression module to generate the initial network weights. ;

[0075] 3D Gaussian optimization module: based on the initial network weights and original image information Regression of 3D Gaussian property information A loss function is constructed using differentiable rendering technology, and the initial network weights are optimized through backpropagation to obtain the optimized network weights. The pose is then obtained by selecting reliable Gaussian spheres and performing the PNP+RANSAC algorithm. .

[0076] Image rendering module: Renders the pose obtained from the 3D Gaussian optimization module. Using a pre-trained 3D Gaussian model Perform view composition to generate a rendered image corresponding to the current estimated pose. .

[0077] Camera coordinate residual regression module: This module returns the original query image to its original coordinates. With the rendered image Simultaneously inputting the camera coordinate residual regression module extracts the feature differences between the two, and regresses the camera coordinate residual. .

[0078] I. Model Training Phase:

[0079] 1. Initial coordinate scene regression module:

[0080] Designed based on the ACE method and using the ResNet architecture, only the decoder needs to be trained during training.

[0081] (1) Constructing an image containing RGB Real camera pose and camera internal reference The dataset is augmented during training by rotating and scaling the training data.

[0082] (2) Load the pre-trained weights into the encoder and set its requirements_grad=False. Randomly initialize the decoder parameters. For each image, after passing through the encoder, 4800 high-dimensional features are generated, and then 1024 features are randomly selected and placed into the buffer pool. After processing all the data, the formal training process begins.

[0083] (3) Input the high-dimensional features into the decoder and output the predicted scene coordinates. .

[0084] (4) Using reprojection error loss, Using real camera pose and Projecting back onto the image plane yields pixel coordinates Calculate its coordinates relative to the actual pixel coordinates. distance Backpropagation updates the decoder weights.

[0085] 2. 3D Gaussian Optimization Module:

[0086] This module aims to establish image feature and 3D Gaussian attribute information. Mapping between them and optimizing the initial network.

[0087] (1) Extract the scene coordinate features from the two CNN networks respectively With RGB image features Concatenate along the channel dimension as input.

[0088] (2) The network outputs the rotation, scaling and opacity of a 3D Gaussian.

[0089] (3) Using the currently predicted Gaussian properties and scene coordinates (as the center of Gaussian), combined with the actual pose, render the image using a differentiable rasterizer. .

[0090] (4) Construct the total loss function ;

[0091] in, Include , The reprojection error is the scene coordinate. Constrain the Gaussian radius to be proportional to the reprojection error (a larger error results in a larger radius, indicating uncertainty).

[0092] (5) By minimizing Simultaneously, the Decoder parameters of the 3D Gaussian attribute regression network and the scene coordinate regression network are updated.

[0093] 3. Image rendering module: This module is mainly used for view composition.

[0094] (1) Load the pre-trained standard 3D Gaussian model for this scene. (Contains millions of Gaussian spheres).

[0095] (2) Receive the coarse pose calculated by the PNP+Ransac algorithm. Perform 3D Gaussian rendering to generate a rendered image from this viewpoint. This is to prepare for subsequent modules.

[0096] 4. Camera coordinate residual regression module: This module is used for fine-tuning pose.

[0097] (1) Input: Original query image Rendering images And the current estimated camera coordinates.

[0098] (2) Two CNNs are used to extract the feature difference map of the two images, and then the residual Mr is regressed through a fully connected layer.

[0099] (3) The supervised value is calculated using the coordinate residual calculation formula based on pose difference, as follows:

[0100] ;

[0101] in, It predicts the scene coordinates in the camera coordinate system. and These are rotations and translations of the coarse pose. and These are rotation and translation of the actual pose. is the supervisory value, and E is the identity matrix.

[0102] (4) By minimizing the network output value and To mitigate the losses between these factors, we optimize the camera coordinate residual regression network.

[0103] Figure 2 As shown, a six-DOF camera pose estimation method based on 3D Gaussian optimization includes the following steps:

[0104] S1. Construct a training dataset containing RGB images and their corresponding real camera poses; Initial network training: Use the training dataset to train the decoder of the initial scene coordinate regression module so that it outputs the scene coordinates corresponding to the pixels.

[0105] Step S1 uses the ACE method, and the network architecture uses ResNet, which includes an encoder and a decoder. The specific steps in the training phase are as follows:

[0106] (1) Construct a training set based on the pose estimation dataset, which includes RGB images and real camera poses. and camera internal reference ;

[0107] (2) Network initialization settings: load pre-trained weights as encoder and keep them frozen during training; only initialize and train the parameters of decoder.

[0108] (3) Forward propagation feature extraction: Input the RGB images of the training set into the encoder to extract high-dimensional features. Downsampling is performed, and training is accelerated by randomly selecting features. The selected features are then decoded by a decoder.

[0109] (4) Scene coordinate regression: The decoder outputs the predicted scene coordinates for each pixel in the image. ;

[0110] (5) Loss calculation and backpropagation, predicting scene coordinates Using real camera pose and camera internal reference Reprojection, calculating the error between pixels The network weights of the decoder are updated by minimizing this error.

[0111] S2. Joint optimization training, 3D Gaussian attribute regression network, integrates image features with scene coordinate features, and outputs 3D Gaussian rotation, scaling and opacity attributes.

[0112] The following are the specific steps during the training phase of constructing a 3D Gaussian attribute regression network and combining it with the object pose estimation task:

[0113] (1) Feature fusion: High-dimensional features of scene coordinates and RGB images are extracted by two CNN networks, respectively, and denoted as follows: and Then, the inputs are fused to construct the input of the Gaussian regression network. ;

[0114] (2) Attribute regression: The three-dimensional Gaussian attribute regression network outputs the attribute information of the three-dimensional Gaussian. Specifically, this includes rotation, scaling, and opacity properties;

[0115] (3) Differentiable rendering, utilizing the regressed Gaussian properties and scene coordinates Irender generates the rendered image (Irender) from the current viewpoint using differentiable rendering technology;

[0116] (4) Construct the total loss function and calculate the photometric error between the rendered image and the original RGB image. and structural similarity error And combined with reprojection error Error in calculating Gaussian radius The total loss function is obtained. ;

[0117] (5) Joint optimization, by minimizing the total loss function The decoder of the initial scene coordinate regression module and the weights of the 3D Gaussian attribute regression network are jointly updated via backpropagation.

[0118] S3. Use differentiable rendering to obtain a rendered image, combine the reprojection error and the Gaussian radius constrained by the reprojection error to obtain the total loss function, and jointly update the weights of the decoder and the 3D Gaussian attribute regression network by minimizing the total loss function.

[0119] Using a pre-trained 3D Gaussian model The specific steps for view composition during the training phase are as follows:

[0120] (1) Obtain the pre-trained model and load the pre-trained 3D Gaussian model G for the current scene;

[0121] (2) Obtain the coarse camera pose. In the training process, the reliable Gaussian sphere obtained in the previous step is selected, and the coarse camera pose Tcoarse is calculated using the PNP+Ransac algorithm.

[0122] (3) Image rendering, based on the coarse pose Tcoarse and camera intrinsic parameters The three-dimensional Gaussian model is projected onto a two-dimensional plane and then rasterized.

[0123] (4) Generate rendering images and synthesize rendering images that strictly correspond to the current input pose;

[0124] (5) Data output: output the generated rendered image. and The paired training data is passed to the camera coordinate residual regression module.

[0125] S4. Camera coordinate residual network training: Using the original image, rendered image and camera coordinates as input, the residual in the camera coordinate system is calculated using the true pose and estimated pose as the supervision value to train the camera coordinate residual regression module.

[0126] Step S4 uses the original image and the rendered image to perform camera coordinate residual regression. The specific steps during the training phase are as follows:

[0127] (1) Data preparation, obtaining the original query images Rendered images generated by the image rendering module and the current predicted scene coordinates ;

[0128] (2) Supervision signal calculation, based on the real camera pose in the dataset. Calculate the true residual value in the camera coordinate system using the current coarse camera pose Tcoarse. ;

[0129] (3) Feature difference extraction: extract the original query image With rendering images Simultaneously input the residual regression network to extract the feature differences between the two. ;

[0130] (4) Residual regression, combined with the input predicted scene coordinates The network outputs the predicted camera coordinate residuals. ;

[0131] (5) Loss calculation and backpropagation, with As a supervisor, a loss function is constructed. The weights of the camera coordinate residual regression module are updated through backpropagation to achieve the training of the camera coordinate residual regression module.

[0132] II. Model Inference Stage:

[0133] (1) Obtain the RGB image to be tested.

[0134] (2) Input the initial scene coordinate regression module to obtain pixel-level predicted scene coordinates. .

[0135] (3) Input the Gaussian optimization network to obtain the rotation, scale and octane of each pixel.

[0136] (4) Filtering and solving: The rotation and scaling are converted into a 3x3 covariance matrix, and then the projection radius of the Gaussian sphere in screen space is calculated. High-quality 2D-3D corresponding points are selected based on Opacity and Radius. The PNP+RANSAC algorithm is then used to calculate the corresponding points. .

[0137] (5) Using a pre-trained scene 3DGS model and Render the image .Will and Input camera coordinate residual regression module, output camera coordinate residual .

[0138] (6) Obtain the camera coordinate residuals Then, add to get ,pass And the rigid body transformation solution method based on SVD is obtained and This refers to the final optimized camera pose.

[0139] Figure 3 , Figure 4 , Figure 5 , Figure 6 These are, respectively, the downsampled real image, the downsampled rendered image, the rendered image, and the output real image for this application.

[0140] Table 1 Comparison Results Data Table

[0141] .

[0142] Experimental data show that the proposed method significantly outperforms current industry benchmark algorithms such as ACE and HLoc in terms of localization accuracy and recall (e.g., Average Recall reaches 100% for 5cm and 5°) on standard datasets. While maintaining high-frequency operation, it demonstrates excellent environmental adaptability and system stability, providing a more reliable and accurate six-degree-of-freedom pose estimation scheme for complex application scenarios such as autonomous driving, robot collaboration, and high-precision AR navigation.

[0143] Note that the above are merely preferred embodiments and technical principles of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above, and various modifications, reconfigurations, combinations, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the foregoing embodiments, the present invention is not limited to these embodiments; more equivalent embodiments may be included without departing from the design concept of the present invention, and their scope of protection is determined by the appended claims.

Claims

1. A six-DOF camera pose estimation system based on three-dimensional Gaussian optimization, characterized in that, It includes an initial scene coordinate regression module, a 3D Gaussian optimization module, an image rendering module, and a camera coordinate residual regression module; Initial scene coordinate regression module: Obtain the original query image to be located. The dataset is divided into training and testing sets. The training set images are processed by the initial scene coordinate regression module to generate the initial network weights. ; 3D Gaussian optimization module: based on the initial network weights and original image information Regression of 3D Gaussian property information A loss function is constructed using differentiable rendering technology, and the initial network weights are optimized through backpropagation to obtain the optimized network weights. The pose is then obtained by selecting reliable Gaussian spheres and performing the PNP+RANSAC algorithm. ; Image rendering module: Renders the pose obtained from the 3D Gaussian optimization module. Using a pre-trained 3D Gaussian model Perform view composition to generate a rendered image corresponding to the current estimated pose. ; Camera coordinate residual regression module: This module returns the original query image to its original coordinates. With the rendered image Simultaneously inputting the camera coordinate residual regression module extracts the feature differences between the two, and regresses the camera coordinate residual. ; The camera coordinate residual regression module outputs coordinate residuals, specifically including: (1) Input the original query image Rendering images and predicted scene coordinates ; (2) Two CNNs are used to extract the feature difference map of the two images, and then the residual Mr is regressed through a fully connected layer; (3) The supervised value is calculated using the coordinate residual calculation formula based on pose difference, as follows: ； in, It predicts the scene coordinates in the camera coordinate system. and These are rotations and translations of the coarse pose. and These are rotation and translation of the actual pose. These are the supervisory values, and E is the identity matrix; (4) By minimizing the network output value and To mitigate the losses between these factors, we optimize the camera coordinate residual regression network.

2. The camera six-DOF pose estimation system based on three-dimensional Gaussian optimization according to claim 1, characterized in that, The initial coordinate scene regression module is designed based on the ACE method and adopts the ResNet architecture. During training, only the decoder is trained to construct a 3D Gaussian attribute regression network, including: (1) Constructing an image containing RGB Real camera pose and camera internal reference The dataset is augmented during training by rotating and scaling the training data; (2) Load the pre-trained weights into the encoder and set its requirements_grad=False. Randomly initialize the decoder parameters. After each photo passes through the encoder, high-dimensional features are generated. Then, a portion of the high-dimensional features are randomly selected and placed into the buffer pool. After processing all the data, the training process begins. (3) The selected high-dimensional features are fed into the decoder and the predicted scene coordinates are output. ; (4) Using reprojection error loss, Using real camera pose and camera internal reference Projecting back onto the image plane yields pixel coordinates Calculate its coordinates relative to the actual pixel coordinates. distance Backpropagation updates the decoder weights; 。 3. The camera six-DOF pose estimation system based on three-dimensional Gaussian optimization according to claim 2, characterized in that, The 3D Gaussian optimization module establishes a mapping between image features and 3D Gaussian properties, and optimizes the initial network, including: (1) The scene coordinate features extracted by the two CNN networks respectively With RGB image features Concatenate along the channel dimension as input; (2) The network outputs the rotation, scaling and opacity of a 3D Gaussian. (3) Using the currently predicted Gaussian properties and scene coordinates as the center of Gaussian, combined with the actual camera pose. and Images are rendered using a miniaturized rasterizer. ; (4) Construct the total loss function ; in, The reprojection error is the scene coordinate. Constrain the Gaussian radius to be proportional to the reprojection error; , , To adjust the parameters; (5) By minimizing Simultaneously, the parameters of the decoders for the 3D Gaussian attribute regression network and the scene coordinate regression network are updated.

4. The camera six-DOF pose estimation system based on three-dimensional Gaussian optimization according to claim 3, characterized in that, The image rendering module includes: (1) Load the pre-trained 3D Gaussian model for this scene. ; (2) Receive the coarse pose calculated by the PNP+Ransac algorithm. Perform 3D Gaussian rendering to generate a rendered image. This is to prepare for the camera coordinate residual regression module.

5. A six-DOF camera pose estimation method based on three-dimensional Gaussian optimization, adapted to the system described in any one of claims 1-4, characterized in that, Includes the following steps: S1. Construct a training dataset containing RGB images and their corresponding real camera poses; Initial network training: Use the training dataset to train the decoder of the initial scene coordinate regression module so that it outputs the scene coordinates corresponding to the pixels; S2. Joint optimization training, 3D Gaussian attribute regression network, integrates image features with scene coordinate features, and outputs 3D Gaussian rotation, scaling and opacity attributes; S3. Use differentiable rendering to obtain a rendered image, combine the reprojection error and the Gaussian radius constrained by the reprojection error to obtain the total loss function, and jointly update the weights of the decoder and the 3D Gaussian attribute regression network by minimizing the total loss function; S4. Camera coordinate residual network training: Using the original image, rendered image and camera coordinates as input, the residual in the camera coordinate system is calculated using the true pose and estimated pose as the supervision value to train the camera coordinate residual regression module.

6. The camera six-DOF pose estimation method based on three-dimensional Gaussian optimization according to claim 5, characterized in that, Step S1 uses the ACE method, and the network architecture uses ResNet, which includes an encoder and a decoder. The specific steps in the training phase are as follows: (1) Construct a training set based on the pose estimation dataset, which includes RGB images and real camera poses. and camera internal reference ; (2) Network initialization settings: load pre-trained weights as encoder and keep them frozen during training; only initialize and train the parameters of decoder. (3) Forward propagation feature extraction: Input the RGB images of the training set into the encoder to extract high-dimensional features. Downsampling is performed, and training is accelerated by randomly selecting features. The selected features are then decoded by a decoder. (4) Scene coordinate regression: The decoder outputs the predicted scene coordinates for each pixel in the image. ; (5) Loss calculation and backpropagation, predicting scene coordinates Using real camera pose and camera internal reference Reprojection, calculating the error between pixels The network weights of the decoder are updated by minimizing this error.

7. The camera six-DOF pose estimation method based on three-dimensional Gaussian optimization according to claim 5, characterized in that, The following are the specific steps during the training phase of constructing a 3D Gaussian attribute regression network and combining it with the object pose estimation task: (1) Feature fusion: High-dimensional features of scene coordinates and RGB images are extracted by two CNN networks, respectively, and denoted as follows: and Then, the inputs are fused to construct the input of the Gaussian regression network. ; (2) Attribute regression: The three-dimensional Gaussian attribute regression network outputs the attribute information of the three-dimensional Gaussian. Specifically, this includes rotation, scaling, and opacity properties; (3) Differentiable rendering, utilizing the regressed Gaussian properties and predicted scene coordinates Irender generates the rendered image (Irender) from the current viewpoint using differentiable rendering technology; (4) Construct the total loss function and calculate the photometric error between the rendered image and the original RGB image. and structural similarity error And combined with reprojection error Error in calculating Gaussian radius The total loss function is obtained. ; (5) Joint optimization, by minimizing the total loss function The decoder of the initial scene coordinate regression module and the weights of the 3D Gaussian attribute regression network are jointly updated via backpropagation.

8. The camera six-DOF pose estimation method based on three-dimensional Gaussian optimization according to claim 7, characterized in that, Using a pre-trained 3D Gaussian model The specific steps for view composition during the training phase are as follows: (1) Obtain the pre-trained model and load the pre-trained 3D Gaussian model for the current scene; (2) Obtain the coarse camera pose. In the training process, the reliable Gaussian sphere obtained in the previous step is selected, and the coarse camera pose Tcoarse is calculated using the PNP+Ransac algorithm. (3) Image rendering, based on the coarse pose Tcoarse and camera intrinsic parameters The three-dimensional Gaussian model G is projected onto a two-dimensional plane and then rasterized. (4) Generate rendering images and synthesize rendering images that strictly correspond to the current input pose; (5) Data output: output the generated rendered image. and the original query image The paired training data is passed to the camera coordinate residual regression module.

9. The camera six-DOF pose estimation method based on three-dimensional Gaussian optimization according to claim 7, characterized in that, Step S4 uses the original image and the rendered image to perform camera coordinate residual regression. The specific steps during the training phase are as follows: (1) Data preparation, obtaining the original query images Rendered images generated by the image rendering module and the current predicted scene coordinates ; (2) Supervision signal calculation, based on the real camera pose in the dataset. Calculate the true residual value in the camera coordinate system using the current coarse pose Tcoarse. ; (3) Feature difference extraction: extract the original query image With rendering images Simultaneously input the residual regression network to extract the feature differences between the two. ; (4) Residual regression, combined with the input predicted scene coordinates The network outputs the predicted camera coordinate residuals. ; (5) Loss calculation and backpropagation, with As a supervisor, a loss function is constructed. The weights of the camera coordinate residual regression module are updated through backpropagation to achieve the training of the camera coordinate residual regression module.

Images