A Visual Relocalization Method for 3D Gaussian Splash Based on a Visual Base Model

Abstract

The application discloses a 3D Gaussian splash visual repositioning method based on a visual basic model driver, and comprises the following steps: S1, 3D Gaussian splash space representation construction and high-level feature extraction; S2, 2D-3D feature matching and pose optimization, the initial estimated pose is adjusted by using a 2D-3D feature matching method to obtain a more accurate pose, and key point sparsification is adopted to reduce space and calculation resource consumption so as to improve processing efficiency and real-time performance; and S3, accurate positioning result acquisition and robustness and calculation burden evaluation, accurate positioning results are finally obtained, high robustness under the condition of environment change is achieved, and the application overcomes the problems of poor adaptability, low repositioning accuracy and large resource occupation of the existing 3D visual repositioning framework, and through fully utilizing the global feature description capability of the visual basic model and combining the 3D Gaussian splash technology, high-precision and real-time visual repositioning with robustness, scalability and high environmental adaptability in a dynamic environment is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a 3D Gaussian splash visual relocalization method driven by a visual basic model. Background Technology

[0002] With the continuous innovation of autonomous driving technology, the increasing popularity of augmented reality applications, and the rapid development of robotics, visual relocation technology has become a key approach to achieving high-precision, low-cost positioning. This technology can accurately determine its own position through visual perception and analysis of the surrounding environment, playing a vital role in improving the safety of autonomous vehicles, enhancing the immersiveness of augmented reality experiences, and increasing the efficiency of robotic operations.

[0003] However, traditional visual relocation methods often face numerous challenges in dynamic environments. Common factors such as changes in lighting and occlusion can interfere with the acquisition and processing of visual information, thus affecting the accuracy and stability of localization. For example, in the event of a sudden change in lighting conditions, traditional visual algorithms may struggle to accurately identify environmental features, leading to localization failure or increased errors. Similarly, when a target object is occluded by other objects, traditional visual relocation methods may also fail to acquire sufficient information to determine the target's location.

[0004] To address these issues, researchers began exploring relocalization models based on Neural Radiation Field (NeRF) and 3D Gaussian Splash (3DGS). These models, by constructing complex environmental representations and feature extraction mechanisms, can overcome the limitations of traditional methods to some extent, improving localization accuracy and robustness. However, despite their achievements, these models still have some shortcomings in practical applications. For example, the NeRF model has a slow rendering speed, making it difficult to meet the demands of real-time applications; while the 3DGS model, although possessing high localization accuracy, has a large footprint and relatively high computational resource requirements. Furthermore, the adaptability of these models to environmental changes needs improvement, especially in complex and dynamic environments, where their performance may be significantly affected.

[0005] The technical problem this invention aims to solve is to provide a 3D Gaussian splash visual relocalization method driven by a visual fundamental model that can effectively improve positioning accuracy and real-time performance in complex environments. This method overcomes the problems of poor adaptability, low relocalization accuracy, and high resource consumption in existing 3D visual relocalization frameworks. By fully utilizing the global feature description capabilities of the visual fundamental model and combining it with 3D Gaussian splashing technology, this method achieves high-precision, real-time visual relocalization with robustness, scalability, and high environmental adaptability in dynamic environments. Summary of the Invention

[0006] The purpose of this invention is to provide a 3D Gaussian splash visual relocalization method driven by a visual basic model, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a 3D Gaussian splash visual relocalization method driven by a visual basic model, comprising the following steps:

[0008] S1. 3D Gaussian splash spatial representation construction and high-level feature extraction: 3D Gaussian splash is used to construct a spatial representation, generate a 3D Gaussian point cloud map that conforms to the scene, and then use the visual basic model to extract high-level features from the query image and compare them with the features of the reference image to obtain the initial estimated pose of the observation.

[0009] S2. Based on the 3D Gaussian splash (3DGS) in step 1, represent the scene using a set of three-dimensional Gaussian distributions, where each point is associated with a point P. i Correlated, the characteristic of each point is the average value Covariance matrix Σ∈R 3×3 Transparency (o) and color (c). These parameters define the contribution of a point to 3D space. Through a tile-based rasterization process, 3DGS achieves real-time rendering by effectively mixing Gaussian functions using α-mixing, where each point in the scene is modeled with a Gaussian distribution in 3D space.

[0010]

[0011] In the formula: x∈R 3 Let μ be a point in three-dimensional space, where μ∈R^3 is the mean (or location) of a Gaussian distribution. Let be the covariance matrix, controlling the shape and orientation of the Gaussian distribution, (·). T This represents the transpose operation, (·). -1 This represents the matrix inverse operation;

[0012] S3, 2D-3D Feature Matching and Pose Optimization: The initial estimated pose is adjusted using the 2D-3D feature matching method to obtain an accurate pose. Keypoint sparsification is adopted to reduce space and computational resource consumption and improve processing efficiency and real-time performance.

[0013] S4. Given a trained 3DGS map, a sparsification process is applied to the map to achieve more efficient and robust relocalization, given the camera pose T. r The reference frame is used to input the pose into the trained 3DGS model to obtain the rendered image I. r , from I r In the process, SuperPoint is used to extract a set of 2D key points. For each key point Generate a Centered patch mask Its size is λ×λ;

[0014] Then, retrieve the camera coordinates O. c The Gaussian distribution in the image is projected onto the image coordinates O. i In, and only retain those falling within the patch mask. The Gaussian distribution within the pixel range is used to achieve sparsification of the 3DGS map. The entire map sparsification process is described by the following equation:

[0015]

[0016] In the formula f 3dgs f keypoint f p f mask These respectively represent: a parameterized 3DGS function, a key point extraction function, a mask generation function, and a function for extracting new 3DGS parameters;

[0017] S5. Accurate positioning results acquisition and robustness and computational burden assessment, ultimately obtaining accurate positioning results, achieving high robustness and low computational burden under environmental change conditions.

[0018] Preferably, each 3D Gaussian is defined in 3D space by its mean and covariance matrix. By transforming its mean and adjusting its covariance matrix accordingly, it is projected onto a 2D image plane. After projection, the covariance matrix of the 3D Gaussian becomes the camera space covariance matrix, as shown in the following equation:

[0019] ∑′=JW∑W T J T

[0020] Where W represents the transformation matrix from the world to the camera, and J is the Jacobian matrix of the affine approximation of the projection transformation. For optimization of the covariance matrix, it is usually decomposed into Σ = RSS. T R T ,in Let S ∈ R represent the direction of Gauss in space. 3×3 This indicates its scaling along different axes.

[0021] Preferably, by sorting the depths of the Gaussian distribution in camera space, the present invention can query the attributes of each two-dimensional Gaussian distribution, facilitating the determination of the color of each pixel in the subsequent volume rendering process.

[0022]

[0023] Here, α is derived from the product of the two-dimensional covariance contribution calculated by transparency o and Σ′ and the corresponding pixel coordinates in the image space. This formula makes the influence of each Gaussian on the image appropriately weighted by the transparency and spatial distribution in the rendering scene.

[0024] Preferably, for the 3D map representation, the Gaussian point cloud is initialized using LiDAR data. For the RGB color representation of each Gaussian, the 0th order spherical harmonic (SH) is used instead of the full SH decomposition. Although the SH decomposition better interprets lighting effects, the reliance on the global descriptor ensures robustness to significant lighting changes. This approach helps to greatly reduce the memory overhead typically associated with 3DGS.

[0025] Preferably, after constructing the 3DGS map and establishing the sparse database, SelaVPR is used as the global locator G. loc From reference image database B ref Obtain the global descriptor F global SelaVPR leverages DINOv2’s powerful visual feature extraction capabilities and enhances them with a lightweight adapter, enabling it to capture large-scale static landmarks under a variety of lighting conditions without modifying the core model.

[0026] Preferably, the step 4 is performed on the query image I. query It is input into G loc In, its global descriptor F query With F ref The descriptor matching in the image is used to filter out the best candidate image frames. Next, input I into SuperPoint (keypoint extractor). query Extract key points Using LightGlue, at key points in candidate frames and Perform 2D-2D feature matching between them, and the key points of the matching When the confidence score s i It is only retained when the threshold τ is exceeded.

[0027] Preferably, according to step 4, for the selected key points, the pose corresponding to the reference image is input into the 3DGS model, and the depth map D of that pose is retrieved. ref Using a camera model, this invention calculates the three-dimensional coordinates of matching key points. The 2D-2D matching is converted to 2D-3D matching, which eliminates the scale ambiguity present in 2D-2D pose estimation. Finally, the PnP algorithm combined with RANSAC is used to recover the absolute pose of the reference frame. The pose recovery process can be described as follows:

[0028]

[0029] Compared with existing technologies, the technical effects and advantages of this invention are as follows: This 3D Gaussian splash visual relocalization method based on a visual fundamental model,

[0030] 1. Higher Accuracy: Traditional visual relocalization methods often rely on local feature matching and localization, which limits the accuracy of localization to some extent. The visual base model used in this invention, through deep learning and other techniques, can extract and describe global features of the environment. These global features not only contain rich spatial information but also reflect the correlation and consistency between different parts of the environment, thus significantly improving the accuracy of the initial pose.

[0031] 2. Storage Optimization: In 3D Gaussian splash models, each spatial point requires storing its corresponding splash information, which often leads to a large space footprint. To address this issue, this invention proposes a keypoint sparsification method. This method analyzes environmental features to select representative and crucial spatial points as keypoints, and only stores splash information for these keypoints. This not only significantly reduces the model's space footprint but also retains sufficient information to support subsequent visual relocalization. This storage optimization strategy makes the 3D Gaussian splash model of this invention more efficient and practical in real-world applications.

[0032] 3. Strong Robustness: In complex and dynamic environments, traditional visual relocalization methods are often susceptible to various interference factors, such as changes in lighting, occlusion, and noise. The method proposed in this invention, by combining a visual basic model and 3D Gaussian splashing technology, constructs a visual relocalization framework with strong adaptability to environmental changes. This framework can fully utilize the advantages of global feature description and keypoint sparsity, and can maintain stable localization performance even in complex and ever-changing dynamic environments. In addition, the method of this invention further improves the robustness and reliability of localization through a series of optimization algorithms and strategies, providing a strong guarantee for high-precision, real-time visual relocalization in practical applications. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the 3D Gaussian splash visual relocalization framework driven by the visual basic model of the present invention.

[0034] Figure 2 This is a flowchart of the 2D-3D feature matching process in the pose optimization process of this invention;

[0035] Figure 3 This is a schematic diagram illustrating the scene representation optimization for key point sparsity in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Please see Figure 1-3 This invention provides a technical solution: a 3D Gaussian splash visual relocalization method driven by a visual basic model, comprising the following steps:

[0038] S1. 3D Gaussian Splash Spatial Representation Construction and High-Level Feature Extraction. A spatial representation is constructed using 3D Gaussian splashes to generate a 3D Gaussian point cloud map that conforms to the scene. Then, a visual baseline model is used to extract high-level features from the query image and compare them with features from a reference image to obtain the initial estimated pose of the observation.

[0039] S2. Represent the scene using a set of three-dimensional Gaussian distributions based on the 3D Gaussian splash (3DGS) from step 1. Each point is associated with a point P. i Correlated, the characteristic of each point is the average value covariance matrix Transparency (o) and color (c). These parameters define the contribution of a point to 3D space. Through a tile-based rasterization process, 3DGS achieves real-time rendering by effectively mixing Gaussian functions using α-mixing, thus providing high-fidelity scene representation. Each point in the scene is modeled using a Gaussian distribution in 3D space:

[0040]

[0041] In the formula: It is a point in three-dimensional space. It is the mean (or location) of a Gaussian distribution. The covariance matrix controls the shape and orientation of the Gaussian distribution. (·) T This represents the transpose operation, (·). -1 This represents the matrix inverse operation;

[0042] S3, 2D-3D Feature Matching and Pose Optimization: The initial estimated pose is adjusted using the 2D-3D feature matching method to obtain an accurate pose. Keypoint sparsification is adopted to reduce space and computational resource consumption and improve processing efficiency and real-time performance.

[0043] S4. Given a trained 3DGS map, this invention applies a sparsification process to the map to achieve more efficient and robust relocalization. Specifically, for the camera pose T... rUsing the reference frame, this invention inputs the pose into the trained 3DGS model to obtain the rendered image I. r . From I r In this invention, SuperPoint is used to extract a set of 2D key points. For each key point This invention generates a Centered patch mask Its size is λ×λ;

[0044] Then, the present invention retrieves the camera coordinates O. c The Gaussian distribution in the image is projected onto the image coordinates O. i In, and only retain those falling within the patch mask. A Gaussian distribution within the pixel range. This process achieves sparsification of the 3DGS map. The entire map sparsification process can be described by the following equation:

[0045]

[0046] In the formula f 3dgs f keypoint f p f mask These respectively represent: a parameterized 3DGS function, a key point extraction function, a mask generation function, and a function for extracting new 3DGS parameters;

[0047] S5. Accurate positioning results acquisition and robustness and computational burden assessment, ultimately obtaining accurate positioning results, achieving high robustness and low computational burden under environmental change conditions.

[0048] Each 3D Gaussian is defined in 3D space by its mean and covariance matrix. By transforming its mean and adjusting its covariance matrix accordingly, it is projected onto a 2D image plane. After projection, the covariance matrix of the 3D Gaussian becomes the camera space covariance matrix, as shown below:

[0049] ∑′=JM∑W T J T

[0050] Where W represents the transformation matrix from the world to the camera, and J is the Jacobian matrix of the affine approximation of the projection transformation. For optimization of the covariance matrix, it is typically decomposed into Σ = RSS. T R T ,in Indicates the direction of Gauss in space. This indicates its scaling along different axes.

[0051] By sorting the depths of the Gaussian distribution in camera space, this invention can query the attributes of each two-dimensional Gaussian distribution, facilitating the determination of the color of each pixel in the subsequent volume rendering process.

[0052]

[0053] Here, α is derived from the product of the two-dimensional covariance contribution calculated from transparency o and Σ′ and the corresponding pixel coordinates in the image space. This formula ensures that the influence of each Gaussian on the image is appropriately weighted by the transparency and spatial distribution in the rendered scene.

[0054] For the 3D map representation, the Gaussian point cloud is initialized using LiDAR data. For the RGB color representation of each Gaussian, the 0th-order spherical harmonic (SH) is used instead of the full SH decomposition. While SH decomposition can better interpret lighting effects, the invention's reliance on a global descriptor ensures robustness to significant lighting changes. This approach also helps to significantly reduce the memory overhead typically associated with 3DGS.

[0055] After constructing the 3DGS map and establishing the sparse database, this invention uses SelaVPR as the global locator G. loc From reference image database B ref Obtain the global descriptor F global SelaVPR leverages DINOv2's powerful visual feature extraction capabilities and enhances them with a lightweight adapter, enabling it to capture large-scale static landmarks under a variety of lighting conditions without modifying the core model.

[0056] For query image I query It is input into G loc In, its global descriptor F query With F ref The descriptor matching in the image is used to filter out the best candidate image frames. Next, this invention inputs I into the SuperPoint (keypoint extractor). query Extract key points Using LightGlue (matcher), at key points in candidate frames and Perform 2D-2D feature matching between them. Key points of matching. Only when their confidence scores s i It is only retained when the threshold τ is exceeded.

[0057] For the selected keypoints, the pose corresponding to the reference image is input into the 3DGS model, and the depth map D of that pose is retrieved. ref Using a camera model, this invention calculates the three-dimensional coordinates of matching key points. The 2D-2D matching is converted to 2D-3D matching, which eliminates the scaling ambiguity present in 2D-2D pose estimation. Finally, this invention uses the PnP algorithm combined with RANSAC to recover the absolute pose of the reference frame. The pose recovery process can be described as follows:

[0058]

[0059] like Figure 1 The diagram illustrates a 3D Gaussian splash visual relocalization framework driven by a visual baseline model. It utilizes images, pose data, and LiDAR point clouds to generate a 3D Gaussian point cloud map. Through rendering, the corresponding images and depth maps are combined with a keypoint extractor to sparsify the Gaussian distribution in the 3DGS map, thereby generating a sparse 3D Gaussian point cloud map. These sparse 3D Gaussian point cloud maps, along with local features of the keypoints, are then used to construct a sparse database.

[0060] like Figure 2 The flowchart shown illustrates the 2D-3D feature matching process in pose optimization. In this process, the global locator performs global descriptor matching between the query image and the reference image, selecting the best candidate image and its initial pose. The best candidate image is queried from a sparse database to retrieve depth information and local features corresponding to the initial pose. Keypoints in the query image are extracted and matched with keypoints in the candidate frame during 2D-to-2D feature matching. Utilizing depth information, this invention performs 2D-to-3D pose optimization using the PnP+RANSAC algorithm. The optimized pose can be re-inserted into the sparse database for iterative refinement.

[0061] like Figure 3 The diagram illustrates scene representation optimization using keypoint sparsity. Given a camera pose input to a 3D geometry (3DGS) model, this invention obtains a rendered image and uses a learning-based keypoint extractor to identify keypoint coordinates within the rendered image. Centered on the pixel coordinates of each keypoint, this invention selects a region of size λ×λ. Then, the 3D Gaussian distribution in the 3DGS image is projected onto the pixel plane, retaining only those Gaussian distributions falling within the region. Importantly, to simplify the visualization process, this invention uses a single keypoint to demonstrate the filtering out of redundant Gaussian distributions.

[0062] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D Gaussian splash visual relocalization method based on visual basis model driving, characterized in that, Comprise the following steps: S1, using 3D Gaussian splash to build spatial representation, generate 3D Gaussian point cloud map conforming to the scene, and then use visual basic model to extract high-level features from the query image, and compare with the features of the reference image, obtain the initial estimated pose of observation; S2. Represent the scene using a set of three-dimensional Gaussian distributions, each point with a point P i associated, the feature of each point is the mean value covariance matrix∈R 3×3 , transparency o and color c, by a tile-based rasterization process, each point in the scene is modeled in three-dimensional space with a Gaussian distribution: where x ∈ R 3 is a point in three-dimensional space, μ ∈ R^3 is the mean of the Gaussian distribution, is the covariance matrix, which controls the shape and orientation of the Gaussian distribution, (·) T denotes the transpose operation, (·) -1 denotes the matrix inverse operation; S3, 2D-3D feature matching and pose optimization: adjust the initial estimated pose by using 2D-3D feature matching method; S4. Given a trained 3DGS map, apply a sparsity reduction process to the map, considering the camera pose T. r The reference frame is used to input the pose into the trained 3DGS model to obtain the rendered image I. r , from I r In the process, SuperPoint is used to extract a set of 2D key points. For each key point Generate a Centered patch mask Its size is λ×λ; The camera coordinates O c are then retrieved from the Gaussians, projected into image coordinates O i and only the Gaussians falling within the patch mask P pixel range are kept, the whole map sparsification process is described by the following equation: where f 3dgs , f keypoint , f p , f mask respectively represent a 3DGS function with parameters, a key point extraction function, a mask generation function, a function for extracting new 3DGS parameters; S5, accurate positioning result acquisition and robustness, calculation burden evaluation, finally obtain accurate positioning result.

2. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 1, characterized in that: Each three-dimensional Gaussian is defined by its mean and covariance matrix in three-dimensional space, by transforming its mean and adjusting its covariance matrix accordingly, it is projected onto the two-dimensional image plane, the covariance matrix of the projected three-dimensional Gaussian becomes the camera space covariance matrix, as follows: ∑' = JW∑W T J T where W represents the world-to-camera transformation matrix, J is the Jacobian matrix of the affine approximation of the projection transformation, and for the optimization of the covariance matrix, the decomposition is T R T where denotes the direction of the Gaussian in space, S e R 3×3 denotes its scaling along the different axes.

3. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 2, characterized in that: By sorting the depth of the Gaussian distribution in the camera space, the properties of each two-dimensional Gaussian distribution are queried: Where alpha is derived from the product of the transparency o, the two-dimensional covariance contribution calculated from Σ', and the corresponding pixel coordinates in the image space.

4. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 3, characterized in that: For 3D map representation, Gaussian point cloud is initialized using LiDAR data, and for RGB color representation of each Gaussian, 0-order spherical harmonic is used.

5. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 1, characterized in that: After the 3DGS map is constructed and the sparse database is built, Sela VPR is used as a global localizer G loc The global descriptors F ref are obtained from the reference image database B global .

6. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 1, characterized in that: According to step 4, for query image I query It is input into G loc In the process, the best candidate image frames are filtered out, and I is input into SuperPoint. query Extract key points Using LightGlue, at key points in candidate frames and Perform 2D-2D feature matching between them, and the key points of the matching When the confidence score s i It is only retained when the threshold τ is exceeded.

7. The 3D Gaussian splash visual relocalization method based on visual basis model driving according to claim 1, characterized in that: According to step 4, for the selected key points, the pose corresponding to the reference image is input into the 3D GS model, and the depth map D of the pose is retrieved ref , using the camera model, the three-dimensional coordinates of the matching key points are calculated Convert 2D-2D matching to 2D-3D matching, use PnP algorithm combined with RANSAC to recover the absolute pose of the reference system, the process of pose recovery is described as:

Images