Camera pose estimation method based on 2D Gaussian splashing

Abstract

The invention provides a camera pose estimation method based on 2D Gaussian splash. The camera pose estimation method comprises the following steps: acquiring a depth map and a normal map from a training image pose based on a pre-trained 2D Gaussian splash model; acquiring a grid ray starting point based on the depth map; generating a main ray and a hemispherical ray for each sampling point based on the normal diagram and the ray starting point; constructing a ray and image matching network and a lightweight convolutional neural network, and performing network training through a loss function; estimating the position of the camera through the trained ray and image matching network and the lightweight convolutional neural network based on the ray features of the main ray and the hemispherical ray and the image features of the query image, and constructing a rotation matrix to realize initial camera pose estimation; and optimizing the initial camera pose by adopting a depth-guided pose optimization method to obtain a final optimized camera 6D pose estimation result. The method gets rid of the dependence of an initial value, and has the advantages of high precision, strong robustness, efficient calculation and wide application.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and in particular to a camera pose estimation method based on 2D Gaussian splashing. Background Technology

[0002] Camera pose estimation aims to recover the camera's translation and rotation in three-dimensional space from an image, and is the foundation for advanced applications such as augmented reality, robotics, and autonomous driving.

[0003] Traditional methods are mainly divided into two categories: methods that rely on pre-deployed markers, which fail in unstructured natural scenes; and feature matching-based localization and mapping methods, such as visual odometry (VO) and simultaneous localization and mapping (SLAM) systems (representative works include ORB-SLAM). Although these methods do not require markers, they are prone to losing tracking in environments with drastic changes in lighting, motion blur, or weak texture, and they also suffer from cumulative errors.

[0004] With the development of neural rendering technology, methods such as Neural Radiation Field (NeRF) perform pose estimation through an analysis-by-synthesis paradigm, finding the camera pose that best matches the rendered image with the query image through backpropagation optimization. However, its implicit representation leads to the optimization process being heavily dependent on initial values, prone to getting trapped in local optima, and computationally expensive, making it difficult to apply in real time. The recently proposed 2D Gaussian Splash (2DGS) generates higher quality depth and normal maps than 3D Gaussian Splash (3DGS) by constraining Gaussian primitives to the object surface, providing a better foundation for geometric inference. However, existing 2DGS-based methods (such as 2DGS-SLAM) still rely heavily on external depth information, failing to fully utilize its ability to generate high-quality geometric information (such as depth and normals), and lack a complete 6D pose estimation framework that can fully exploit the multi-view geometry and rendering characteristics of 2DGS.

[0005] In summary, existing pose estimation methods still have the following shortcomings:

[0006] (1) The strong dependence on the initial value leads to the pose estimation result deviating significantly from the true value when there is no good initial value or prior knowledge, thus resulting in estimation failure;

[0007] (2) Insufficient robustness in complex scenes, especially under conditions of occlusion, weak texture, and changing lighting;

[0008] (3) The dilemma of balancing accuracy and efficiency: high-precision estimation methods (such as NeRF variants) are slow, while the accuracy and robustness of fast estimation methods (such as 3DGS applications) need to be improved.

[0009] (4) The potential of emerging efficient representations (such as 2DGS) has not been fully explored, and a high-performance pose estimation process adapted to them has not been constructed. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a camera pose estimation method based on 2D Gaussian splashing.

[0011] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0012] A camera pose estimation method based on 2D Gaussian splashing includes:

[0013] Step 1: Based on the pre-trained 2D Gaussian splash model, obtain the depth map and normal map from the pose of the training image;

[0014] Step 2: Perform geometric alignment sampling based on the depth map to obtain grid sampling points and ray origins;

[0015] Step 3: Perform multi-viewpoint ray generation based on the normal map and ray origin, generating a principal ray and a hemispherical ray for each sampling point;

[0016] Step 4: Construct a ray-image matching network and a lightweight convolutional neural network, and train the matching network and convolutional neural network by constructing loss functions through geometric score alignment loss and orientation consistency loss;

[0017] Step 5: Based on the ray features of the main ray and the hemispherical ray and the image features of the query image, estimate the camera position and construct the rotation matrix through the trained ray-image matching network and the trained lightweight convolutional neural network, and combine the rotation matrix and the camera position to realize the initial camera pose estimation.

[0018] Step 6: Optimize the initial camera pose using a depth-guided pose optimization method to obtain the final optimized 6D camera pose estimation result.

[0019] Furthermore, the process of obtaining the grid sampling points and ray origins is as follows:

[0020] The training images are divided into One grid;

[0021] For each grid cell, calculate the cumulative opacity of each pixel within that grid cell. ,in, It is the first A Gaussian in a pixel Alpha value, select Largest pixel As the sampling point of this grid;

[0022] For each sampling point ,in, Represents the column coordinates of a pixel. Represents the row coordinates of a pixel; using its corresponding camera extrinsic parameters. Camera internal parameters and depth map The 3D world coordinates of the sampling points are calculated based on the camera projection geometry and used as the ray origin. , The calculation formula is as follows:

[0023]

[0024] in, Indicates sampling point The depth value at that location.

[0025] Furthermore, the generation process of the main ray and the hemispherical ray is specifically as follows:

[0026] The direction of the main ray is the normalized vector from the ray origin to the camera optical center;

[0027] Based on normal map A local tangent space is constructed. On the hemisphere defined by the normal axis in this local coordinate system, 128 uniformly distributed direction vectors are generated using the Sobol low-difference sequence. After these direction vectors are projected back to the world coordinate system, they constitute the set of hemispherical rays originating from the ray origin.

[0028] Furthermore, the core of the ray-image matching network is a multi-head attention module.

[0029] Furthermore, the structure of the lightweight convolutional neural network consists of two convolutional dimensionality reduction modules cascaded with an MLP regression head.

[0030] Furthermore, the calculation process for the camera position is as follows:

[0031] The principal ray and hemispherical ray are processed by a 2-layer residual MLP, and a 128-dimensional ray feature is output for each ray. The image features of the query image are extracted using a pre-trained visual Transformer model.

[0032] The matching score between ray features and image features of the query image is obtained by using a trained ray-image matching network.

[0033] For each principal ray and hemispherical ray generated from a training image, select the one with the highest matching score against the query image. Rays; these rays are analyzed based on their origin. Clustering and deduplication are performed, and the ray with the highest score in each cluster is used to form a ray set.

[0034] Through the objective function Find a point in three-dimensional space As camera position The estimation of the objective function Represented as:

[0035]

[0036] in, Let M be the identity matrix, and M be the number of rays in the ray set. As the starting point of the ray, Let be the direction vector of the ray. The weight of the ray.

[0037] Furthermore, the construction process of the rotation matrix is ​​as follows:

[0038] Calculate the weighted average direction of the rays in the ray set and use it as the camera observation direction. The calculation formula is:

[0039]

[0040] A trained lightweight convolutional neural network is used to predict the camera orientation vector from the image features of the query image. Using the camera's upward direction vector and camera observation direction Construct the rotation matrix, rotation matrix Its forward vector Right vector upper vector .

[0041] Furthermore, the depth-guided pose optimization method is specifically as follows:

[0042] First, use the initial camera pose. The synthesized image is obtained by rendering a pre-trained 2D Gaussian splash model. and the corresponding depth image Local feature descriptors are used to perform feature matching between the query image and the synthesized image. Then, a grid-based motion statistical filtering mechanism and Lowe ratio test are introduced to obtain a set of high-quality 2D-2D feature matching pairs.

[0043] Then, spatial nonmaximum suppression is used to homogenize the matching pairs; and for each feature point on the synthetic image, the matching pairs are further filtered by calculating the mean and standard deviation of the effective depth in its small neighborhood.

[0044] Finally, based on the filtered matching pairs, through depth images Establish a 2D-3D corresponding point set; input the 2D-3D corresponding points into a solver composed of the EPnP algorithm and the RANSAC framework to obtain the preliminarily optimized camera pose. ;by Using the initial values, iterative weighted nonlinear least squares fine-tuning based on reprojection error is performed to obtain the camera's 6D pose. .

[0045] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the camera pose estimation method based on 2D Gaussian splashing.

[0046] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the camera pose estimation method based on 2D Gaussian splashing.

[0047] The beneficial effects of this invention are:

[0048] (1) Get rid of initial value dependence: By reverse analysis of the 2DGS rendering process and the establishment of ray-image matching mechanism, this invention can directly estimate 6D pose without any initial pose guessing, solving the core problem of the traditional analysis-synthesis method being sensitive to initial values.

[0049] (2) High precision and strong robustness: The innovative geometric alignment sampling ensures that the sampling points are derived from well reconstructed areas; multi-viewpoint ray generation provides rich view constraints; the depth-guided optimization module ensures the data quality for PnP through multi-level filtering (GMS, depth detection, spatial NMS), so that the method can still maintain high precision and stability in challenging scenarios such as occlusion and weak texture.

[0050] (3) High computational efficiency: Thanks to the high-speed rendering capability of 2DGS itself, as well as the efficient implementation of feature matching, PnP and other processes, this invention has higher computational efficiency than many NeRF-based methods while maintaining high accuracy.

[0051] (4) Wide applicability and leading performance: This invention has been fully validated on several representative public datasets, such as Mip-NeRF 360 (indoor-outdoor mixed scene) and Tanks & Temples (large-scale real scene). Experimental results show that its performance is significantly better than existing methods based on fixed / random pose priors, as well as state-of-the-art methods (such as 6DGS) under pose-free prior conditions, and it has achieved leading results in both mean angle error (MAE) and mean translation error (MTE). Attached Figure Description

[0052] Figure 1 This is a flowchart of the camera pose estimation method described in this invention.

[0053] Figure 2 This is a schematic diagram of the multi-viewpoint ray generation described in this invention.

[0054] Figure 3 This is a flowchart of the initial camera pose estimation process of the method described in this invention.

[0055] Figure 4 This is a flowchart illustrating the depth-guided camera pose optimization method described in this invention.

[0056] Figure 5 This is a visualization of the camera pose estimation results described in this invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0058] A camera pose estimation method based on 2D Gaussian splashing, referring to Figure 1 It includes the following steps:

[0059] Step 1: 2DGS Scene Model Pre-training: Train a 2DGS model using multi-view RGB images of the target scene and their corresponding camera poses, optimizing the parameters of the Gaussian ellipse (position, scaling, rotation, opacity, and spherical harmonic coefficients). Train until convergence to obtain a 2DGS model capable of rendering the scene's color, depth, and normals with high quality.

[0060] Step 2, ray matching with the image and camera orientation vector prediction (training phase), specifically includes the following steps:

[0061] Step 2.1: Obtaining the depth map and normal map

[0062] For each training image The pose is input into the pre-trained 2DGS model to obtain the rendered depth map. and normal diagram These two geometric information maps will be used for subsequent geometric alignment sampling and multi-view ray generation.

[0063] Step 2.2, Geometric Alignment Sampling

[0064] Will Divided into A grid, which in this embodiment is divided into 1 grid. A grid. For each grid, iterate through each pixel. According to the formula Calculate its cumulative opacity ,in, It is the first A Gaussian in a pixel Alpha value. Select Largest pixel This serves as the sampling point for the grid.

[0065] For each sampling point ,in, Represents the column coordinates (horizontal pixel index) of a pixel. Represents the row coordinates (vertical pixel index) of a pixel. Use its corresponding camera extrinsic parameters. Camera internal parameters and rendering depth map The 3D world coordinates of the sampling points are calculated based on the camera projection geometry and used as the ray origin. The calculation formula is as follows:

[0066]

[0067] in, Indicates sampling point The rendering depth value at that location.

[0068] Step 2.3: Multi-viewpoint ray generation

[0069] At each ray origin Ray generation is performed on the sampled point. Specifically, the principal ray direction is a normalized vector pointing from the ray origin to the camera's optical center. Then, to simulate observations of the sampled point from other perspectives, a surface normal map is rendered based on the 2DGS model. A local tangent space is constructed. On a hemisphere defined in this local coordinate system with the normal as the axis, 128 uniformly distributed direction vectors are generated using the Sobol low-difference sequence. After these direction vectors are projected back to the world coordinate system, they constitute the set of hemispherical rays originating from the ray's starting point.

[0070] Each sampling point generates a total of 129 rays (1 principal ray + 128 hemispherical rays), each ray originating from its starting point. The direction vector and color value are defined. The color value is taken from the corresponding training image. At the sampling point The RGB value at that location. (Refer to...) Figure 2 .

[0071] Step 2.4: Matching Rays with Images

[0072] For each ray generated at each sampling point, the ray's origin, direction, color, and other information are encoded and processed through a 2-layer residual MLP (Multilayer Perceptron) to output a 128-dimensional ray feature for each ray. Simultaneously, image features of the training images were extracted using a pre-trained visual Transformer model (a frozen DINOv2 ViT-S / 14 model). .

[0073] A ray-image matching network is constructed, the core of which is a multi-head attention module. Its purpose is to calculate the overall relevance of each ray to the query image and output a quantified matching score. The multi-head attention module incorporates ray features... As a query, image features As the key, attention weights are calculated by scaling the dot product to generate an attention score map. The most relevant spatial rays in the image are identified and trained. To transform this multi-dimensional relevance assessment into a unified scalar score that can be used for subsequent geometric computation, this embodiment sums and aggregates the attention score map along the image feature dimension to obtain the matching score. :

[0074]

[0075] in, Indicates the first The image features and the first Correlation weights between rays. Matching score. The higher the value, the more the appearance features of the 3D sampling point corresponding to the origin of the ray match the features "observed" from the perspective of the current training image. In other words, the geometric and appearance information represented by the ray is more consistent with the current image. Therefore, the ray will play a more important role in the subsequent camera pose estimation through weighting.

[0076] Align loss with geometric score Match score The formula for supervised learning is:

[0077]

[0078] Among them, geometric target score By the The projected distance from the ray to the current camera optical center is obtained by mapping using the hyperbolic tangent function.

[0079] Step 2.5: Predicting the upward direction vector of the camera.

[0080] Image features are regressed into unit vectors pointing upwards from the camera using a lightweight convolutional neural network. The network uses the image features of the training images output by the aforementioned DINOv2 model. As input, its structure consists of a two-level convolutional dimensionality reduction module cascaded with a multilayer perceptron (MLP) regression head.

[0081] Using directional consistency loss Make the predicted upward unit vector The average value of the orientation of all camera poses in the training set The formula for the alignment and orientation consistency loss is as follows:

[0082]

[0083] Step 2.6: Network training for ray-to-image matching and camera orientation vector prediction.

[0084] The overall training loss function combines geometric score alignment and orientation consistency constraints. The geometric score alignment loss ensures that the predicted ray matching score conforms to the true camera imaging geometry, while the orientation consistency loss constrains the predicted camera orientation to align with the average scene orientation through cosine similarity. Therefore, the overall training loss function... for:

[0085]

[0086] During training, each training image and the rays it generates constitute an independent training sample. The Adam optimizer is used, with a total of 1500 iterations and a batch size of 32. The learning rate is set to 1e-3, the weight decay is 1e-3, and a gradient clipping strategy is employed. The maximum norm is limited to 1.0. The OneCycleLR scheduling strategy is used for the learning rate, and linear warming and cosine annealing are employed for the first 300 iterations.

[0087] Step 3, Initial Camera Pose Estimation (Inference Stage), refer to Figure 3 Specifically, it includes the following steps:

[0088] Enter a query image Processed using the frozen DINOv2 ViT-S / 14 model Extract its feature map and pool it into global features, that is, query the image features of the image. The ray-image matching network trained in step 2 is used to calculate the ray features generated from all training images. Image features of the query image Match scores between them.

[0089] For each training image generated ray (principal ray and hemispherical ray), select and query images. The highest score match ray ( =25). These rays are analyzed based on the ray origin. Clustering and deduplication are performed, and the rays with the highest scores in each cluster are combined to form a ray set. The camera position is then determined using the clustered and deduplicated ray sets. estimation, rotation matrix The construction of the initial camera pose estimation is achieved.

[0090] Camera position The estimation is constructed as a weighted least squares optimization problem: finding a point in three-dimensional space. The objective function is to minimize the sum of the squared weighted perpendicular distances from the point to all rays in the ray set, thus achieving the desired result. It can be represented as:

[0091]

[0092] in, Let M be the identity matrix, and M be the number of rays in the ray set. As the starting point of the ray, Let be the direction vector of the ray. The weight of each ray. That is its corresponding matching score. Is it projected onto and The projection matrix of a vertical plane.

[0093] The weighted average direction of the rays in the ray set is calculated using the following formula and used as the camera's observation direction. :

[0094]

[0095] The lightweight convolutional neural network trained in step 2 extracts image features from the query image. Predicting the upward direction vector of the camera Using the camera's upward direction vector and camera observation direction Construct the rotation matrix, rotation matrix Its forward vector Right vector upper vector Rotation matrix Combined with camera position Get camera pose .

[0096] Step 4, Depth-guided camera pose optimization method, refer to Figure 4 Specifically, it includes the following steps:

[0097] Step 4.1, 2D-2D Feature Matching

[0098] Using the estimated initial camera pose The synthesized image is obtained by rendering a pre-trained 2D Gaussian splash model. and the corresponding depth image Local feature descriptors (AKAZE, Accelerated-KAZE) are used to perform feature matching between the query image and the synthesized image. Then, a grid-based motion statistics (GMS) filtering mechanism and Lowe ratio test are introduced to eliminate false matches, resulting in a high-quality set of 2D-2D feature matching pairs. ,in, To query feature points on an image, For feature points on the synthesized image, This represents the confidence score for feature matching calculated based on GMS.

[0099] Furthermore, for the match pairs after filtering out the aforementioned mismatches, to avoid their excessive clustering in local areas of the image, in The upper-space NMS (Non-Maximum Suppression) is used to achieve uniform filtering of matching pairs: the query image is then processed. Divided into The grid is configured such that only the confidence score is retained within each grid cell. The top 4 feature points (or all feature points if there are fewer than 4 feature points in the grid) are selected to ensure that the 2D-3D matching feature points used for subsequent pose solving are evenly distributed in the image space.

[0100] Furthermore, regarding Each feature point on Calculate its small neighborhood (e.g. Mean of effective depth of neighborhood and standard deviation .like If a feature point is found to be in a depth-instable region, it is considered to be eliminated, thereby removing feature points and their corresponding matching pairs located in depth-discontinuous regions.

[0101] Step 4.2, Depth-guided pose optimization

[0102] For the matching pairs composed of query image feature points and synthetic image feature points retained through the above filtering... In the rendered depth map Using bilinear interpolation to obtain Subpixel precision depth value and utilize Back projection yields its 3D world coordinates Thus, a 2D-3D corresponding point set is established. .

[0103] Furthermore, the corresponding 2D-3D points are input into a solver consisting of the Efficient Perspective-n-Point (EPnP) algorithm and the Random Sample Consensus (RANSAC) framework to obtain the preliminarily optimized camera pose. .

[0104] Furthermore, with Using the initial values, iterative weighted nonlinear least squares fine-tuning based on reprojection error is performed to obtain the camera's 6D pose. This serves as the final estimation result of the present invention.

[0105] The fine-tuning process is specifically achieved by solving the following weighted nonlinear least squares problem:

[0106]

[0107] in, For camera projection matrix, For the camera intrinsic parameter matrix, Given the camera pose in the current iteration, For adaptive weights, their values ​​are determined by the matching confidence. and current reprojection error The decision was made jointly, and the specific calculations are as follows:

[0108]

[0109] in, The confidence score for feature matching, based on The reciprocal of the feature descriptor matching distance is calculated. This represents the reprojection error in the current iteration. This is a scaling parameter used to control the sensitivity of the weights to errors.

[0110] This weighted nonlinear least squares problem is solved iteratively using the Levenberg-Marquardt algorithm. In each iteration, the reprojection error is calculated based on the current pose. And update the weights Then, the pose estimate is updated by solving the weighted nonlinear least squares equation. Iterate until convergence.

[0111] Step 5: Test performance. For example... Figure 5 The image shown is a visualization of the camera pose estimation results described in this invention. When evaluating the performance of 6D camera pose estimation, this invention uses Mean Angular Error (MAE) and Mean Translation Error (MTE) as core metrics. MAE measures the angular difference between the estimated camera rotation pose and the true value, in degrees (°); MTE measures the Euclidean distance difference between the estimated camera translation position and the true value, with the unit (u) representing the normalized scale of the scene, typically taken as the maximum side length of the scene bounding box.

[0112] Evaluations were conducted on the Mip-NeRF 360 dataset for a 6D camera pose estimation task: the mean angle error (MAE) of this invention was 4.3° and the mean translation error (MTE) was 0.004u, compared with the existing best method 6DGS (MAE 24.3°, MTE 0.022u), achieving error reduction rates of 82.3% and 81.8%, respectively. Under three comparison settings—fixed pose prior, random pose prior, and no pose prior—this invention achieved the best results, demonstrating that by combining geometrically consistent surface point sampling and multi-view ray generation strategies with an attention matching mechanism, discriminative geometric features across viewpoints can be effectively extracted, thereby achieving high-precision pose initialization.

[0113] Evaluation results on the large-scale real-world scene dataset Tanks & Temples, which includes complex geometry and occlusion, demonstrate that the proposed invention maintains robust performance even under significant occlusion and complex structures. Across five object categories, the average MAE is 2.2° and the average MTE is 0.043u, representing error reductions of 91.1% and 87.6% respectively compared to the best method with fixed pose prior (MAE 24.7°, MTE 0.346u). The performance improvement is particularly significant compared to 6DGS, another method without pose prior (MAE 21.7°, MTE 0.268u). This result strongly demonstrates that the depth-guided pose optimization module proposed in this invention, combining robust 2D-2D matching based on mesh motion statistics and 2D-3D correspondence filtering with spatial nonmaximum suppression, can effectively overcome interference from occlusion and complex geometry, achieving stable and accurate pose optimization.

[0114] In more challenging low-texture, highly symmetrical indoor scenes (such as the "Room" scene in Mip-NeRF 360), the performance advantages of this invention remain significant. Its MAE and MTE are further reduced by 9.0% and 12.5% ​​respectively compared to the baseline without the optimization module, indicating that the rendering depth-guided optimization process designed in this invention has stronger geometric inference and completion capabilities for texture-poor regions.

[0115] The initial design goal of this invention was to achieve highly robust 6D pose estimation that does not rely on pose priors, and experimental results validate this. On two authoritative datasets with different characteristics and varying levels of challenge, this invention achieved significantly better performance than the comparison methods under the stringent setting of "no pose prior." This demonstrates that the ray-image matching framework constructed through the inverted 2D Gaussian splash rendering pipeline can learn a geometrically consistent representation with strong generalization capabilities, rather than overfitting to a specific scene or viewpoint. Furthermore, this invention significantly narrows the performance gap between methods with "known accurate pose priors" and methods with "no pose priors," highlighting the crucial role of the proposed multi-view ray generation and hierarchical feature matching mechanism in improving the model's generalization and robustness.

[0116] Applications of this invention:

[0117] The pose estimation method based on 2D Gaussian splashing provided in this invention, due to its high accuracy, lack of pose prior dependence, and robustness to occlusion and low-texture scenes, can be widely applied in various intelligent vision systems that require fast and accurate recovery of camera or object spatial pose from a single image. Specific application areas include, but are not limited to:

[0118] In the field of robot autonomous navigation and scene reconstruction, it can be used for visual SLAM and real-time localization: providing real-time, high-precision self-6D pose estimation for mobile robots, drones, or autonomous vehicles relying solely on a monocular RGB camera, supporting simultaneous localization and mapping (SLAM) in unknown or dynamic environments, especially suitable for challenging scenarios lacking obvious textures or exhibiting motion blur. Furthermore, in tasks such as industrial inspection and warehouse inventory management, it can be combined with pre-trained 2DGS scene models to achieve rapid camera relocalization and scene understanding.

[0119] In the fields of augmented reality and mixed reality, it enables real-time device pose tracking and virtual registration: providing stable and accurate 6D camera pose output for AR glasses, smartphones or head-mounted displays, enabling virtual objects to fit into real scenes with high fidelity. Even when users move quickly or there is partial occlusion, it can still maintain visual consistency and immersion between virtual and real, and is suitable for various scenarios such as interactive entertainment, remote collaboration, and industrial guidance.

[0120] In the fields of autonomous driving and high-precision positioning, it supports vehicle visual positioning and pose estimation: in areas without GPS or with limited signals (such as tunnels, underground parking garages, and urban canyons), the vehicle's precise position and heading angle can be calculated in real time through an onboard monocular camera and a pre-built 2DGS high-precision scene model, providing key state inputs for path planning, lane keeping and environmental perception, and improving the safety redundancy of the autonomous driving system under complex road conditions.

[0121] In the field of video analysis and surveillance, it enables the recovery of mobile camera trajectories and scene analysis: for video sequences captured by fixed or mobile surveillance cameras, it recovers the camera pose corresponding to each frame, and then reconstructs the walking path or scanning trajectory of the photographer. This can be used for applications such as security patrol analysis, sports event shooting trajectory optimization, and virtual camera assembly in film and television production.

[0122] In the fields of remote sensing and mapping and 3D reconstruction, it assists in aerial image positioning and multi-view fusion: for oblique images taken by UAVs or satellites, in the absence of accurate extrinsic parameter calibration, it can quickly estimate the spatial pose of the camera by matching a single image with an existing 2DGS model of the scene, supporting efficient urban modeling, terrain analysis and disaster assessment.

[0123] In summary, the method proposed in this invention, as a camera 6D pose estimation technique that does not rely on initial pose and has strong generalization capabilities, serves as a crucial bridge connecting two-dimensional visual observation and three-dimensional spatial perception. Its technological achievements can be widely applied in multiple cutting-edge fields such as autonomous systems, mixed reality, intelligent transportation, and surveying and remote sensing, demonstrating clear engineering practical value and broad application prospects.

[0124] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the aforementioned camera pose estimation method based on 2D Gaussian splashing.

[0125] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the aforementioned camera pose estimation method based on 2D Gaussian splashing.

[0126] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A camera pose estimation method based on 2D Gaussian splashing, characterized in that, include: Step 1: Based on the pre-trained 2D Gaussian splash model, obtain the depth map and normal map from the pose of the training image; Step 2: Perform geometric alignment sampling based on the depth map to obtain grid sampling points and ray origins; Step 3: Perform multi-viewpoint ray generation based on the normal map and ray origin, generating a principal ray and a hemispherical ray for each sampling point; Step 4: Construct a ray-image matching network and a lightweight convolutional neural network, and train the matching network and convolutional neural network by constructing loss functions through geometric score alignment loss and orientation consistency loss; Step 5: Based on the ray features of the main ray and the hemispherical ray and the image features of the query image, estimate the camera position and construct the rotation matrix through the trained ray-image matching network and the trained lightweight convolutional neural network, and combine the rotation matrix and the camera position to realize the initial camera pose estimation. Step 6: Optimize the initial camera pose using a depth-guided pose optimization method to obtain the final optimized 6D camera pose estimation result.

2. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The specific process for obtaining the grid sampling points and ray starting points is as follows: The training images are divided into One grid; For each grid cell, calculate the cumulative opacity of each pixel within that grid cell. ,in, It is the first A Gaussian in a pixel Alpha value, select Largest pixel As the sampling point of this grid; For each sampling point ,in, Represents the column coordinates of a pixel. Represents the row coordinates of a pixel; using its corresponding camera extrinsic parameters. Camera internal parameters and depth map The 3D world coordinates of the sampling points are calculated based on the camera projection geometry and used as the ray origin. , The calculation formula is as follows: ， in, Indicates sampling point The depth value at that location.

3. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The generation process of the main ray and hemispherical ray is as follows: The direction of the main ray is the normalized vector from the ray origin to the camera optical center; Based on normal map A local tangent space is constructed. On the hemisphere defined by the normal axis in this local coordinate system, 128 uniformly distributed direction vectors are generated using the Sobol low-difference sequence. After these direction vectors are projected back to the world coordinate system, they constitute the set of hemispherical rays originating from the ray origin.

4. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The core of the ray-image matching network is the multi-head attention module.

5. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The lightweight convolutional neural network consists of a cascaded structure of two convolutional dimensionality reduction modules and an MLP regression head.

6. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The specific process for calculating the camera position is as follows: The principal ray and hemispherical ray are processed by a 2-layer residual MLP, and a 128-dimensional ray feature is output for each ray. The image features of the query image are extracted using a pre-trained visual Transformer model. The matching score between ray features and image features of the query image is obtained by using a trained ray-image matching network. For each principal ray and hemispherical ray generated from a training image, select the one with the highest matching score against the query image. Rays; these rays are analyzed based on their origin. Clustering and deduplication are performed, and the ray with the highest score in each cluster is used to form a ray set. Through the objective function Find a point in three-dimensional space As camera position The estimation of the objective function Represented as: ， in, Let M be the identity matrix, and M be the number of rays in the ray set. As the starting point of the ray, Let be the direction vector of the ray. The weight of the ray.

7. The camera pose estimation method based on 2D Gaussian splashing according to claim 6, characterized in that, The construction process of the rotation matrix is ​​as follows: Calculate the weighted average direction of the rays in the ray set and use it as the camera observation direction. The calculation formula is: ， A trained lightweight convolutional neural network is used to predict the camera orientation vector from the image features of the query image. Using the camera's upward direction vector and camera observation direction Construct the rotation matrix, rotation matrix Its forward vector Right vector upper vector .

8. The camera pose estimation method based on 2D Gaussian splashing according to claim 1, characterized in that, The depth-guided pose optimization method is specifically as follows: First, use the initial camera pose. The synthesized image is obtained by rendering a pre-trained 2D Gaussian splash model. and the corresponding depth image Local feature descriptors are used to perform feature matching between the query image and the synthesized image. Then, a grid-based motion statistical filtering mechanism and Lowe ratio test are introduced to obtain a set of high-quality 2D-2D feature matching pairs. Then, spatial nonmaximum suppression is used to homogenize the matching pairs; and for each feature point on the synthetic image, the matching pairs are further filtered by calculating the mean and standard deviation of the effective depth in its small neighborhood. Finally, based on the filtered matching pairs, through depth images Establish a 2D-3D corresponding point set; input the 2D-3D corresponding points into a solver composed of the EPnP algorithm and the RANSAC framework to obtain the preliminarily optimized camera pose. ;by Using the initial values, iterative weighted nonlinear least squares fine-tuning based on reprojection error is performed to obtain the camera's 6D pose. .

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the camera pose estimation method based on 2D Gaussian splash as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the camera pose estimation method based on 2D Gaussian splash as described in any one of claims 1 to 8.

Images