A three-dimensional gaussian slam method and system with direct insertion and pruning optimization
By employing full keyframe modeling and adaptive pruning strategies, the problems of mapping quality and computational overhead in the 3D Gaussian SLAM method under different motion modes are solved, achieving efficient and stable 3D scene reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-04-28
- Publication Date
- 2026-07-14
AI Technical Summary
Existing 3D Gaussian SLAM methods suffer from degraded mapping quality, increased computational overhead, and limited scalability under different camera motion modes, especially in FOE motion or across motion sequences where Gaussian initialization is inefficient and redundant Gaussians accumulate.
A full keyframe modeling approach combined with a multi-feature-guided adaptive pruning strategy is adopted. Through a Gaussian point initializer and an adaptive Gaussian pruner, efficient Gaussian generation and dynamic redundancy suppression are achieved to adapt to different motion modes.
Achieve high-quality 3D scene representation under different motion modes, reduce computation and storage costs, and improve system stability and generalization ability.
Smart Images
Figure CN122391494A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and robotics, and in particular to a three-dimensional Gaussian SLAM method and system with direct insertion and pruning optimization. Background Technology
[0002] In fields such as 3D reconstruction, augmented reality, autonomous driving, and robot perception, systems often need to build high-quality 3D scene representations under real-time conditions to obtain accurate geometric structures and consistent visual effects.
[0003] In existing technologies, Simultaneous Localization and Mapping (SLAM) refers to the process by which a mobile device (such as a camera or robot) in an unknown environment constructs an environmental map while simultaneously estimating its own pose (position and orientation) in real time through continuous sensing data (such as images, depth, or lasers).
[0004] 3D Gaussian Splatting (3DGS) refers to representing a 3D scene as a set of differentiable, Gaussian-distributed voxels, achieving high-fidelity, real-time view compositing through differentiable sputtering operations. This representation method can achieve efficient rendering and optimization while preserving detailed geometric and appearance information.
[0005] In the fields of modern computer vision and robotics, real-time 3D SLAM technology has become an important means of understanding the environment and enhancing scene perception capabilities. 3D Gaussian-based SLAM methods can represent 3D scenes with high fidelity and efficiency, while providing rich geometric and appearance information.
[0006] In existing 3D Gaussian SLAM methods, most studies have proposed Gaussian insertion and pruning strategies, local depth priors, and isotropic regularization techniques for incremental scene reconstruction to improve mapping accuracy and optimization stability. For example, GS-SLAM achieves high-fidelity mapping under scanning motion by combining adaptive Gaussian generation and pruning with coarse-to-fine pose optimization; SplaTAM enhances local keyframe updates using depth priors and contour constraints; and MonoGS reduces rendering artifacts through depth backprojection and isotropic regularization. However, these methods generally rely on motion pattern assumptions or novel region separation strategies. Under focus expansion (FOE) motion or cross-motion sequences, high line-of-sight overlap makes it difficult to effectively suppress redundant Gaussian lines between consecutive keyframes, resulting in low initialization efficiency, degraded mapping quality, and limited optimization scalability.
[0007] On the other hand, some vision-prior-based solutions achieve large-scale scene reconstruction and keyframe tracking by introducing large-scale pre-trained models, but their high computational and memory overhead is significant, and Gaussian redundancy and optimization instability are still difficult to avoid in FOE motion or across motion sequences. Hierarchical or multi-scale representation methods, such as Giga-SLAM, rely on predefined scene ranges and fixed resolutions, making it difficult to balance local details and overall structure in different motion modes, further limiting the system's flexibility and versatility.
[0008] In summary, although existing 3D Gaussian SLAM methods have made some progress in specific scenarios, several technical bottlenecks remain. First, incremental mapping requires continuously adding new Gaussians to represent the continuously observed scene. Some methods control the scale by restricting Gaussian initialization to a finite subspace, but this strategy relies on scanning motion. In practical applications, cameras often involve FOE motion or cross-motion sequences, causing a significant decrease in mapping accuracy under non-scanning motion, limiting their applicability in real-world environments. Second, existing methods employ fixed-rule Gaussian pruning, removing inefficient Gaussians through thresholding. However, redundant Gaussians generated across motion sequences may not fall within the threshold and cannot be effectively removed. This directly increases computational burden, degrades mapping quality, and limits the system's flexibility and scalability under multiple motion modes.
[0009] Therefore, designing a unified 3D Gaussian SLAM framework that can efficiently and stably generate high-quality scene representations under different camera motion modes (including scanning motion (where the camera's motion direction and the line of sight have a significant angle, often resulting in clear distinction of new regions observed between consecutive frames, which is beneficial for local incremental Gaussian modeling) and focus-of-expansion motion (FOE, where the camera's motion direction and the line of sight are basically the same, and new observed regions highly overlap with existing structures, making it difficult for Gaussians to distinguish in local incremental mapping, increasing redundancy and optimization difficulty) while solving the problems of Gaussian initialization and redundancy suppression has become a core technical challenge that urgently needs to be addressed. Summary of the Invention
[0010] This invention addresses the limitations of existing technologies across various motion modes, particularly in FOE motion or cross-viewpoint continuous observation scenarios. These limitations include low Gaussian initialization efficiency and the accumulation of redundant Gaussians leading to decreased mapping quality and increased computational overhead. The invention proposes a direct insertion and pruning-optimized 3D Gaussian SLAM method and system. It replaces the new region-driven local update mechanism with a full keyframe modeling approach, fundamentally improving the system's robustness and mapping quality across different motion modes by performing global Gaussian modeling on each keyframe. Simultaneously, it incorporates a multi-feature-guided adaptive pruning strategy to dynamically control the number of Gaussians during mapping, achieving efficient Gaussian generation and dynamic redundancy suppression. This effectively controls the growth of Gaussian size and improves the system's computational efficiency and scalability, enabling high-quality reconstruction across different motion modes. This method effectively solves the performance degradation problem of existing methods under complex motion conditions and significantly reduces computational and storage costs.
[0011] To achieve the above objectives, the present invention provides the following technical solution:
[0012] In a first aspect, the present invention provides a three-dimensional Gaussian SLAM method with direct insertion and pruning optimization, comprising the following steps:
[0013] S1. Use a general depth estimator to predict the corresponding metric depth map for each frame of input RGB image sequence, thereby constructing RGB-D observation data;
[0014] S2. Combine the RGB-D observation data of the current frame with the existing 3D Gaussian map, estimate the camera pose through a differentiable rendering optimization method, and realize motion tracking of continuous frames.
[0015] S3. After obtaining the camera pose, evaluate the input frames to determine whether they should be used as keyframes for mapping.
[0016] S4. For the selected keyframe, a new Gaussian representation is generated using a Gaussian point initializer, and then optimized to fit the current observation data. An adaptive Gaussian pruning tool is used to perform redundancy detection and deletion on the optimized Gaussian set to obtain a compact 3D Gaussian scene representation.
[0017] Furthermore, in step S4, the Gaussian point initializer adopts a distilled dual-branch network structure, including a dual-head initializer and a back-projection module. The dual-head initializer is used to decompose the input image into two types of structural features, which are formalized into a structure-aware representation learning problem and solved by the distillation learning method. The back-projection module is used to map the pixel coordinates in the two-dimensional image space to the three-dimensional space, thereby providing a precise spatial location for Gaussian generation.
[0018] Furthermore, the specific process of the dual-head initializer includes:
[0019] Given an input image The network first extracts low-level features through a shared backbone network, and then feeds them into two branches to model different structural information:
[0020] Point-level saliency branches output high-frequency feature maps, which are used to characterize textured regions;
[0021] The region consistency branch outputs a low-frequency feature map, which is used to characterize weak textures or uniform regions.
[0022] The network is trained using a teacher-student distillation strategy, where the teacher network consists of a point feature extraction model and a region segmentation model; the student network achieves a compressed representation of the teacher's capabilities by minimizing feature differences, and its optimization objective is:
[0023]
[0024] in, and These represent the feature outputs of the student network and the teacher network, respectively. For student network parameters;
[0025] After training, the dual-head initializer directly outputs structure-aware features during the inference phase to guide subsequent Gaussian generation.
[0026] Furthermore, the specific process of the back projection module includes:
[0027] Given pixel coordinates in the input image and its corresponding depth value The camera's intrinsic and extrinsic parameters are back-projected into three-dimensional point coordinates; let the camera intrinsic parameter matrix be... The extrinsic parameter matrix is Where R represents the rotation matrix, If we denote the translation vector, then the three-dimensional coordinates of the pixel are represented as follows:
[0028]
[0029] in, The homogeneous coordinates of the pixels are used for representation. This is the depth value corresponding to that pixel.
[0030] Further, in step S4, the specific process of the Gaussian point initializer includes: in the first... In each keyframe processing step, the initializer receives the current RGB-D input and the existing Gaussian map, and generates a new Gaussian set. It is then combined with the historical Gaussian set to participate in optimization, resulting in an updated scene representation.
[0031] Furthermore, in step S4, the adaptive Gaussian pruner achieves adaptive identification and suppression of redundant Gaussians by uniformly modeling multimodal features. The multimodal features include the temporal evolution information of the Gaussians, motion consistency, and reconstruction residuals.
[0032] Further, in step S4, the specific process of the adaptive Gaussian pruning tool includes: adaptively modulating the Gaussian opacity based on the Gaussian's lifespan, optical flow information, and color and depth reconstruction errors; deleting Gaussians with opacity below a set threshold to simplify the Gaussian set.
[0033]
[0034] in, This is the updated Gaussian opacity. This is the output value of the Gaussian pruning tool. It is the initial Gaussian opacity.
[0035] Furthermore, step S2 specifically includes:
[0036] First, a constant velocity model is used for pose initialization:
[0037]
[0038] in This represents the camera extrinsic parameters for frame t.
[0039] Subsequently, under the condition of fixed Gaussian parameters, pose optimization is performed through differentiable rendering to minimize the error between the rendering result and the observation.
[0040] Secondly, this invention proposes a direct insertion and pruning optimized three-dimensional Gaussian SLAM system, comprising the following modules to implement the method described in any of the above-mentioned embodiments:
[0041] The camera tracking module is used to estimate the camera pose of the current frame in order to achieve Gaussian projection and mapping optimization.
[0042] A Gaussian point cloud initializer is used to generate a new Gaussian representation in the current 3D space and optimize it to fit the current observation data.
[0043] An adaptive Gaussian pruner is used to perform redundancy detection and deletion on the optimized Gaussian set, resulting in a compact 3D Gaussian scene representation.
[0044] Furthermore, the Gaussian point cloud initializer includes:
[0045] A dual-head initializer is used to perform structure-aware feature decomposition on an image and generate two complementary structural feature representations.
[0046] The back projection module is used to map pixel coordinates in a two-dimensional image space to a three-dimensional space.
[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0048] This invention addresses the limitations of existing 3D Gaussian splash SLAM technology in terms of mapping quality, computational overhead, and scalability by proposing a direct insertion and pruning optimization 3D Gaussian SLAM method, which effectively overcomes the aforementioned shortcomings through a novel modeling approach.
[0049] Firstly, regarding mapping quality, this invention employs a structure-aware Gaussian initialization strategy. It adaptively generates Gaussian representations of different types and densities based on the structural complexity of image regions, enabling richly textured regions to be finely depicted, while weakly textured regions are covered in a compact manner, thus balancing structural detail with overall integrity. Because this representation maintains consistency across different motion modes (such as scan motion and FOE motion), it avoids the structural aliasing and artifact problems that occur in traditional methods under conditions of high overlap observation or line-of-sight alignment, thereby achieving more stable and accurate mapping results.
[0050] Secondly, regarding computational cost and storage overhead, this invention employs a strategy combining single-keyframe Gaussian generation with adaptive pruning. It only requires Gaussian initialization once per keyframe and dynamically adjusts the Gaussian contribution through a learning-driven redundancy suppression mechanism, eliminating the need for multi-round iterative densification or complex region selection processes. This "single-generation, dynamic optimization" design effectively reduces the disordered growth of the number of Gaussians and avoids the redundancy accumulation problem caused by repeated observations in traditional methods, thereby significantly reducing computational burden and storage requirements.
[0051] Finally, regarding scalability, this invention, through a unified modeling Gaussian initialization and redundancy suppression mechanism, eliminates the dependence on new region identification and specific motion patterns, enabling the system to naturally adapt to various observation modes such as scan motion, FOE motion, and cross-motion sequences. Since modeling and optimization are performed only on complete keyframes at each step, strong assumptions about observation separability are avoided, thereby improving the system's stability and generalization ability in complex real-world environments.
[0052] In summary, this invention systematically improves the key issues of existing technologies from three aspects: mapping quality, computational efficiency, and adaptability of pruning after direct insertion by introducing structure-aware initialization and adaptive pruning mechanisms. This significantly enhances the practicality and performance of the 3DGS-SLAM method, which can be adapted to real-world scenes and motion patterns after training on simulation datasets. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0054] Figure 1 The flowchart illustrates the three-dimensional Gaussian SLAM method with direct insertion and pruning optimization provided in this embodiment of the invention. Detailed Implementation
[0055] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.
[0056] This invention proposes a direct insertion and pruning optimization method for 3D Gaussian SLAM. The overall system architecture and process are as follows: Figure 1 As shown, the specific steps include:
[0057] For a given input RGB image sequence, this invention first uses a general depth estimator to predict the corresponding metric depth map for each frame, thereby constructing RGB-D observation data. Subsequently, the RGB-D data of the current frame is combined with an existing 3D Gaussian map, and the camera pose is estimated through a differentiable optimization method to achieve motion tracking of consecutive frames.
[0058] After obtaining the camera pose, the input frames are evaluated to determine whether they should be used as keyframes for mapping. For selected keyframes, a new Gaussian representation is generated using a Gaussian Points Initializer, and optimized to fit the current observation data. Specifically, in the... In each keyframe processing step, the initializer receives the current RGB-D input and the existing Gaussian map, and generates a new Gaussian set. The updated scene representation is then optimized together with the historical Gaussian set. The entire process starts from the initial keyframe, proceeds through incremental mapping of consecutive keyframes, and finally yields a complete 3D Gaussian scene representation.
[0059] The key components of this method include the Gaussian point initializer and the Gaussian pruner. The Gaussian point initializer (GPI) employs a distilled, two-branch network structure (e.g., Figure 1 As shown, the Gaussian pruning tool is used to efficiently generate high-quality Gaussian scalars; the Gaussian pruning tool is used to perform redundancy detection and deletion on the optimized Gaussian set to obtain a compact scene representation. Through the above process, this invention directly models Gaussian initialization, redundancy suppression, and scalability within a unified full keyframe modeling framework, thereby achieving high-quality and high-efficiency 3D mapping.
[0060] 1. Gaussian point initializer
[0061] Given an input image and its corresponding depth map This invention first performs structure-aware feature decomposition on the image, generating two complementary structural feature representations. Then, based on this structural information, a set of Gaussian parameters is generated in the current three-dimensional space. This data is then inserted into the global Gaussian map for subsequent optimization and mapping. The process consists of two core components: a dual-head initialization network and a back-projection module.
[0062] 1.1 Dual-headed initializer
[0063] This module decomposes the input image into two classes of structural features, formalizing it as a structure-aware representation learning problem, which is solved using a distillation learning method. Specifically, as... Figure 1 As shown, given the input image The network first extracts low-level features through a shared backbone network, and then feeds them into two branches to model different structural information:
[0064] Point-level saliency branches output high-frequency feature maps, which are used to characterize textured regions;
[0065] The region consistency branch outputs a low-frequency feature map, which is used to characterize weak textures or uniform regions.
[0066] The network is trained using a teacher-student distillation strategy, where the teacher network consists of a point feature extraction model and a region segmentation model. The student network achieves a compressed representation of the teacher's capabilities by minimizing feature differences, and its optimization objective is:
[0067]
[0068] in, and These represent the feature outputs of the student network and the teacher network, respectively. These are the parameters for the student network. After training, the dual-head network directly outputs structure-aware features during the inference phase to guide subsequent Gaussian generation.
[0069] 1.2 Back Projection Module
[0070] This module maps pixel coordinates in a two-dimensional image space to a three-dimensional space, thus providing a precise spatial location for Gaussian generation. Specifically, given pixel coordinates in the input image... and its corresponding depth value The camera's intrinsic and extrinsic parameters are used to back-project the data into three-dimensional point coordinates. Let the camera intrinsic parameter matrix be... The extrinsic parameter matrix is Where R represents the rotation matrix,
[0071] Let the translation vector be represented as: Then the three-dimensional coordinates of a pixel can be expressed as:
[0072]
[0073] in, The homogeneous coordinates of the pixels are used for representation. This is the depth value corresponding to that pixel.
[0074] 3. Adaptive Gaussian Pruner
[0075] To address the Gaussian redundancy problem caused by multiple observations, this invention proposes an Adaptive Gaussian Pruner. This method does not rely on traditional rule-based thresholding strategies; instead, it achieves adaptive identification and suppression of redundant Gaussians by uniformly modeling multimodal features. Specifically, this pruner comprehensively utilizes multiple features such as the temporal evolution information of Gaussians, motion consistency, and reconstruction residuals to discriminate Gaussians that appear reasonable individually but exhibit representational conflicts as a whole.
[0076] In implementation, this invention designs a lightweight prediction network to predict the redundancy level of each Gaussian based on its lifetime, optical flow information, and color and depth reconstruction errors. Compared to traditional methods that rely on only a single attribute for judgment, this method can effectively model the representation conflict relationship between multi-scale Gaussians, thereby achieving more accurate redundancy detection.
[0077] Furthermore, this invention gradually reduces the contribution of redundant Gaussians by adaptively modulating the Gaussian opacity, thereby ensuring the stability of the optimization process. Finally, Gaussians with opacity below a set threshold are deleted, achieving a simplification of the Gaussian set.
[0078]
[0079] in, This is the updated Gaussian opacity. This is the output value of the Gaussian pruning tool. It is the initial Gaussian opacity.
[0080] Through the above method, the present invention can explicitly model and eliminate redundancy and conflict between multi-scale Gaussians, effectively control the growth of the number of Gaussians, and thus obtain a more compact, accurate and scalable 3D Gaussian scene representation.
[0081] 4. Camera Tracking Module
[0082] This module is used to estimate the camera pose of the current frame to achieve Gaussian projection and mapping optimization. First, a constant-velocity model is used for pose initialization:
[0083]
[0084] Where Et represents the camera extrinsic parameters of frame t. Subsequently, under the condition of fixed Gaussian parameters, pose optimization is performed through differentiable rendering to minimize the error between the rendered result and the observation.
[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A three-dimensional Gaussian SLAM method with direct insertion and pruning optimization, characterized in that, Includes the following steps: S1. Use a general depth estimator to predict the corresponding metric depth map for each frame of input RGB image sequence, thereby constructing RGB-D observation data; S2. Combine the RGB-D observation data of the current frame with the existing 3D Gaussian map, estimate the camera pose through a differentiable rendering optimization method, and realize motion tracking of continuous frames. S3. After obtaining the camera pose, evaluate the input frames to determine whether they should be used as keyframes for mapping. S4. For the selected keyframe, a new Gaussian representation is generated using a Gaussian point initializer, and then optimized to fit the current observation data. The optimized Gaussian set is then redundancy-detected and deleted using an adaptive Gaussian pruning tool to obtain a compact 3D Gaussian scene representation.
2. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 1, characterized in that, In step S4, the Gaussian point initializer adopts a distilled dual-branch network structure, including a dual-head initializer and a back-projection module. The dual-head initializer is used to decompose the input image into two types of structural features, which is formalized into a structure-aware representation learning problem and solved by the distillation learning method. The back-projection module is used to map the pixel coordinates in the two-dimensional image space to the three-dimensional space, thereby providing a precise spatial location for Gaussian generation.
3. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 2, characterized in that, The specific process of the dual-head initializer includes: Given an input image The network first extracts low-level features through a shared backbone network, and then feeds them into two branches to model different structural information: Point-level saliency branches output high-frequency feature maps, which are used to characterize textured regions; The region consistency branch outputs a low-frequency feature map, which is used to characterize weak textures or uniform regions. The network is trained using a teacher-student distillation strategy, where the teacher network consists of a point feature extraction model and a region segmentation model; the student network achieves a compressed representation of the teacher's capabilities by minimizing feature differences, and its optimization objective is: ， in, and These represent the feature outputs of the student network and the teacher network, respectively. For student network parameters; After training, the dual-head initializer directly outputs structure-aware features during the inference phase to guide subsequent Gaussian generation.
4. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 2, characterized in that, The specific process of the back projection module includes: Given pixel coordinates in the input image and its corresponding depth value The camera's intrinsic and extrinsic parameters are back-projected into three-dimensional point coordinates; let the camera intrinsic parameter matrix be... The extrinsic parameter matrix is Where R represents the rotation matrix, If we denote the translation vector, then the three-dimensional coordinates of the pixel are represented as follows: ， in, The homogeneous coordinates of the pixels are used for representation. This is the depth value corresponding to that pixel.
5. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 1, characterized in that, In step S4, the specific process of the Gaussian point initializer includes: in the first step... In each keyframe processing step, the initializer receives the current RGB-D input and the existing Gaussian map, and generates a new Gaussian set. It is then combined with the historical Gaussian set to participate in the optimization, resulting in an updated scene representation.
6. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 1, characterized in that, In step S4, the adaptive Gaussian pruner achieves adaptive identification and suppression of redundant Gaussians by uniformly modeling multimodal features. The multimodal features include the temporal evolution information of the Gaussians, motion consistency, and reconstruction residuals.
7. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 1, characterized in that, In step S4, the specific process of the adaptive Gaussian pruning tool includes: adaptively modulating the Gaussian opacity based on the Gaussian's lifespan, optical flow information, and color and depth reconstruction errors; deleting Gaussians with opacity below a set threshold to simplify the Gaussian set. ， in, This is the updated Gaussian opacity. This is the output value of the Gaussian pruning tool. It is the initial Gaussian opacity.
8. The three-dimensional Gaussian SLAM method with direct insertion and pruning optimization according to claim 1, characterized in that, Step S2 specifically includes: First, a constant velocity model is used for pose initialization: ， in This represents the camera extrinsic parameters for frame t. Subsequently, under the condition of fixed Gaussian parameters, pose optimization is performed through differentiable rendering to minimize the error between the rendering result and the observation.
9. A three-dimensional Gaussian SLAM system with direct insertion and pruning optimization, characterized in that, For performing the method as described in any one of claims 1 to 8, comprising: The camera tracking module is used to estimate the camera pose of the current frame in order to achieve Gaussian projection and mapping optimization. A Gaussian point cloud initializer is used to generate a new Gaussian representation in the current 3D space and optimize it to fit the current observation data. An adaptive Gaussian pruner is used to perform redundancy detection and deletion on the optimized Gaussian set, resulting in a compact 3D Gaussian scene representation.
10. The three-dimensional Gaussian SLAM system with direct insertion and pruning optimization according to claim 9, characterized in that, The Gaussian point cloud initializer includes: A dual-head initializer is used to perform structure-aware feature decomposition on an image and generate two complementary structural feature representations. The back projection module is used to map pixel coordinates in a two-dimensional image space to a three-dimensional space.