gaussian splash dense color depth-slam method based on expert-gated color regularization

Abstract

The Gauss splash dense color depth-SLAM method based on expert gating color regularization belongs to the technical field of computer vision. By introducing an expert gating color regularization module, the color regularization intensity is adaptively adjusted according to the local reconstruction state of different Gauss primitives. An error-aware Gauss densification strategy is proposed, which identifies the insufficient reconstruction area according to the contour coverage, color residual error and depth residual error, and selectively inserts new Gauss primitives to reduce local rendering errors and improve the completeness of the map expression. Finally, the accumulated key frames are revisited through the Gauss map refinement stage to further optimize the Gauss map and alleviate the sequence dependence bias caused by incremental updating. The application achieves good rendering quality, geometric reconstruction accuracy and camera tracking performance on multiple commonly used SLAM datasets and unmanned aerial vehicle perspective datasets, and is suitable for fields such as robot autonomous navigation, unmanned aerial vehicle intelligent perception, augmented reality and intelligent traffic environment perception.

Description

Technical Field

[0001] This invention relates to a Gaussian splatter dense color depth-SLAM method based on expert-gated color regularization, belonging to the fields of computer vision and Gaussian splatter technology. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a key technology for intelligent systems such as robots, drones, autonomous driving, and augmented reality to achieve autonomous perception and environmental interaction. Traditional visual SLAM methods typically use sparse feature points, semi-dense point clouds, or voxel maps as environmental representations, achieving high-precision camera localization. However, the maps they construct often lack complete geometric details and realistic appearance representation, making it difficult to meet the requirements of tasks such as realistic rendering, 3D interaction, semantic understanding, and high-fidelity environment modeling. In recent years, NeRF-based SLAM methods have improved the quality of dense mapping and new perspective rendering through implicit neural representations, but they usually require high computational overhead and are prone to problems such as forgetting, uneven convergence, and insufficient real-time performance in long-sequence incremental optimization. 3D Gaussian splashing technology utilizes explicit Gaussian primitives and rasterization rendering mechanisms to achieve a better balance between rendering efficiency and visual quality, providing a new scene representation for dense color depth-SLAM (RGB-D SLAM). However, existing Gaussian splash-based SLAM systems still have the following shortcomings: First, fixed color regularization weights are often used in mapping optimization, which cannot adaptively adjust the appearance constraint strength according to the observation count, stability, and color sensitivity of different Gaussian primitives, leading to unstable color updates or insufficient adaptation in some areas. Second, existing Gaussian compaction strategies mostly rely on depth error or visibility statistics, making it difficult to reliably identify under-represented areas under sparse viewpoints, depth noise, and complex geometries. Third, the incremental optimization process of SLAM itself has a sequential dependency; early or short-term observation areas may be under-optimized, while frequently revisited areas may overfit the latest observations, thus affecting the global consistency and reconstruction quality of the Gaussian map. Therefore, there is an urgent need for a method that can simultaneously improve the stability of appearance optimization, the integrity of local mapping, and the global consistency of the Gaussian map. Figure 1 A consistent Gaussian splash dense RGB-D SLAM method. Summary of the Invention

[0003] This invention proposes a Gaussian splatter dense color depth-SLAM method based on expert-gated color regularization, aiming to solve the technical problems of existing Gaussian splatter SLAM in the incremental optimization process, such as fixed color regularization weights, inaccurate identification of insufficiently reconstructed local areas, and unbalanced global map convergence.

[0004] The Gaussian splatter dense color depth-SLAM method based on expert-gated color regularization is characterized by including the following steps: (1) RGB-D image sequence acquisition and data preprocessing: acquire the RGB-D image sequence of the scene to be processed, which includes synchronously acquired RGB images and depth images; read the RGB images, depth images and camera intrinsic parameters to provide input data for subsequent camera tracking and 3D Gaussian map construction; (2) Initialization of three-dimensional Gaussian map: Initialize the three-dimensional Gaussian map using the first frame RGB-D image; the three-dimensional Gaussian map is composed of several Gaussian elements, each Gaussian element includes the three-dimensional center position, scale, color parameters and opacity; and set the camera pose of the first frame to the preset initial pose; (3) Starting from the second frame, read each frame in the RGB-D image sequence in sequence, and perform the following steps in a loop on the current frame: camera tracking based on differentiable rendering, error-aware Gaussian compaction, and expert-gated color regularization-guided Gaussian map update until all input frames have been processed. (4) Gaussian map refinement: After all frames in the input RGB-D image sequence have completed the processing of steps (2) and (3) above, the Gaussian map refinement stage is entered; key frames are randomly selected from the accumulated key frame database, and the Gaussian map is optimized based on the RGB-D image corresponding to the selected key frames; the optimization is to use the optimization function to further adjust the Gaussian meta-parameters; the optimization is repeated multiple times; so that the early observation area and sparse observation area can be further supervised to alleviate the local overfitting and global convergence imbalance caused by the sequential dependency in the incremental mapping process; (5) Camera pose and 3D map output: In practical applications, the input is the RGB-D image sequence to be processed, and the above steps of 3D Gaussian map initialization, camera tracking based on differentiable rendering, error-aware Gaussian compaction, expert-gated color regularization-guided Gaussian map update and Gaussian map refinement are executed; finally, the camera trajectory and 3D Gaussian map are output.

[0005] The method further includes step (6) rendering a three-dimensional Gaussian map, in which the three-dimensional Gaussian map generates color images, depth images, and contour images from any viewpoint through a differentiable Gaussian splash rendering method.

[0006] The method is characterized in that it is applied to robot autonomous navigation, UAV intelligent perception, augmented reality, digital twins, or intelligent transportation environment perception.

[0007] In step (2), the initialization of the three-dimensional Gaussian map includes: back-projecting two-dimensional pixels into three-dimensional space based on the effective depth pixels and camera intrinsic parameters in the initial RGB-D image frame to generate the three-dimensional center position of the initial Gaussian primitive; using the corresponding RGB pixel values ​​as the initial color parameters of the Gaussian primitive; determining the initial scale of the Gaussian primitive based on the depth value and the camera projection relationship; and setting the initial opacity for each Gaussian primitive to form the initial three-dimensional Gaussian map.

[0008] In step (3), the camera tracking based on differentiable rendering is as follows: predict the initial pose of the current frame based on the historical camera pose and using a constant velocity motion model; render the current 3D Gaussian map into a color image, a depth image, and a contour image according to the initial pose, and compare the rendering results with the RGB-D image shown in the current frame; iteratively optimize the camera pose of the current frame by minimizing the color residual and the depth residual; calculate the cumulative opacity of the current 3D Gaussian map, and calculate the tracking loss only in reliable pixel areas where the cumulative opacity is greater than or equal to a preset threshold in order to reduce the impact of new observation areas and outliers on the tracking results.

[0009] In step (3), the error-aware Gaussian densification is specifically as follows: After completing the camera tracking of the current frame, it is determined whether new Gaussian primitives need to be inserted based on the densification mask of the current 3D Gaussian map in the current frame; the densification mask is constructed by integrating the error-aware Gaussian densification process of contour coverage, color residual and depth residual; wherein, pixel areas with cumulative opacity lower than a preset threshold are determined as insufficient Gaussian coverage areas, pixel areas with color residual greater than a preset threshold are determined as insufficient appearance modeling areas, and pixel areas with depth residual greater than a preset threshold are determined as geometrically missing areas; the insufficient Gaussian coverage areas, insufficient appearance modeling areas or geometrically missing areas in the 3D Gaussian map are marked as selected areas of the densification mask, and new Gaussian primitives are inserted in these areas.

[0010] In step (3), the expert-gated color regularization-guided Gaussian map update is as follows: After completing the camera tracking and Gaussian compaction of the current frame, the 3D Gaussian map is updated based on the current frame and the keyframe database in the SLAM architecture. The keyframe database selects keyframes according to the view overlap. The 5 keyframes with the highest view overlap with the current frame are selected from the keyframe database. If there are less than 5, the actual number is taken. These keyframes are then combined with the current frame to form a local mapping window. Mapping optimization iteration is performed through the mapping optimization function to update each parameter of each Gaussian element. In each iteration, a frame is randomly sampled from the local mapping window as a local mapping window. The supervised frame optimizes the 3D Gaussian map. During the optimization process, for each Gaussian element, a state description feature is constructed, which includes opacity, observation count, spatial location, scale information, and color importance weights. The state description feature is input into a lightweight hybrid expert gating module, which generates different expert-assigned weights for each feature through a routing network. Multiple expert networks jointly output the color regularization gating weights for the corresponding Gaussian elements. The expert gating weights are used to minimize the residual between the current Gaussian color parameters and the color parameters cached in the previous optimized state or initialization state, thereby adaptively adjusting the color regularization intensity of different Gaussian elements.

[0011] Each Gaussian cell is set to an initial opacity value of 0.5.

[0012] For pixel regions where the depth residual is greater than a preset threshold, a new Gaussian element is generated. The new Gaussian element inherits the RGB color information and depth geometry information of the corresponding pixel and is inserted into the current 3D Gaussian map to enhance the map representation capability and rendering consistency of the local area.

[0013] The state description features include at least the opacity of the Gaussian unit, the cumulative number of observations, spatial location, scale information, and color importance weight; the lightweight hybrid expert gating module includes a routing network and multiple expert networks, wherein the routing network generates the assigned weights of each expert network according to the state description features, the multiple expert networks output candidate gating responses respectively, and the color regularization gating weight of the current Gaussian unit is obtained by weighted fusion.

[0014] In the optimization of the 3D Gaussian map, the optimization objectives include color rendering loss, depth rendering loss, and expert-gated color regularization. Among them, color rendering loss is used to constrain the consistency between the rendered color image and the real RGB image, depth rendering loss is used to constrain the consistency between the rendered depth image and the real depth image, and expert-gated color regularization is used to suppress the unstable changes of Gaussian color parameters during incremental mapping.

[0015] Beneficial effects This invention innovatively incorporates expert-gated color regularization, error-aware Gaussian densification strategy, and Gaussian map refinement process into the Gaussian splash dense RGB-D SLAM framework, effectively improving the appearance stability, local map integrity, and global map quality during incremental mapping. Figure 1 To the point of being responsive.

[0016] First, this invention effectively solves the problem of fixed color regularization weights in existing Gaussian splash SLAM methods by employing an expert-gated color regularization mechanism. In traditional Gaussian map updates, different Gaussian primitives typically use a uniform color regularization intensity, which is difficult to adapt to differences in observation count, spatial location, scale, color sensitivity, and local reconstruction state. For Gaussian primitives that have already converged and been observed multiple times, overly weak color constraints can easily lead to appearance drift; while for newly inserted or not yet fully optimized Gaussian primitives, overly strong color constraints will limit their appearance adaptability. This invention utilizes a lightweight hybrid expert gating module to adaptively generate color regularization gating weights based on the local state of each Gaussian primitive, thereby achieving differentiated color consistency constraints and maintaining necessary local update capabilities while improving appearance stability.

[0017] Secondly, the error-aware Gaussian compaction strategy proposed in this invention can more accurately identify and supplement under-reconstructed local areas. Existing Gaussian compaction methods often rely on depth error or visibility statistics, which can easily lead to inaccurate identification of insufficiently represented regions in scenarios with sparse observations, depth noise, complex geometry, and local occlusion. This invention comprehensively utilizes rendering contour coverage, color residuals, and depth residuals to construct a compaction mask, simultaneously determining whether the Gaussian map is underrepresented from three perspectives: map coverage, appearance error, and geometric error, and selectively inserting new Gaussian primitives in the corresponding effective regions. This approach reduces local rendering errors and improves the map representation capability and 3D reconstruction integrity of complex areas.

[0018] Furthermore, this invention alleviates the sequential dependency problem in incremental mapping through a Gaussian map refinement stage. Since SLAM systems typically optimize maps frame-by-frame in chronological order, early observation areas or areas that appear only briefly may lack sufficient supervision in subsequent optimizations, while frequently revisited areas may overfit recent observations, resulting in uneven global map convergence. This invention, after completing online tracking and mapping on all input frames, revisits the accumulated keyframe set and performs additional Gaussian map optimization based on keyframe RGB-D observations, giving different regions a more balanced optimization opportunity, thereby improving global map convergence. Figure 1 Consistency and high-fidelity reconstruction quality.

[0019] Furthermore, this invention employs explicit 3D Gaussian primitives as scene representation, combined with a differentiable Gaussian splash rendering mechanism, enabling simultaneous output of camera trajectory, 3D Gaussian map, color-rendered image, depth-rendered image, and contour image. Compared to traditional sparse or semi-dense SLAM maps, the 3D Gaussian map generated by this invention possesses superior appearance representation and rendering capabilities; compared to SLAM methods based on implicit neural representations, this invention offers a more intuitive map structure and higher rendering efficiency, making it more suitable for practical applications such as robotics, drones, augmented reality, and intelligent transportation.

[0020] Finally, this invention has strong application scalability. The method can be deployed in mobile robots, drones, vehicle platforms, or handheld RGB-D devices for tasks such as autonomous navigation, digital twin construction, augmented reality interaction, and intelligent traffic environment modeling. By constructing a stable, complete, and high-fidelity Gaussian map, this invention can provide a more reliable 3D environment representation for intelligent systems. Attached Figure Description

[0021] Figure 1 is a flowchart of the overall process framework of the present invention.

[0022] Figure 2 shows the visualization results of the UAV dataset in this invention. Detailed Implementation

[0023] Dense Color Depth-SLAM (RGB-D SLAM) requires the system to construct a dense 3D scene map with geometric consistency and high-fidelity appearance representation while estimating camera pose. Addressing the problems of unstable color optimization, insufficient reconstruction of local regions, and uneven global map convergence in the incremental mapping process of existing Gaussian splash SLAM, this invention proposes a Gaussian splash dense RGB-D SLAM framework that integrates expert-gated color regularization, error-aware Gaussian compaction, and Gaussian map refinement. Through expert-gated color regularization Gaussian map update, error-aware Gaussian compaction, and keyframe revisit-based Gaussian map refinement, a more stable, complete, and high-fidelity dense RGB-D SLAM is achieved. This invention constructs a high-fidelity Gaussian splash SLAM framework for dense RGB-D scene reconstruction by introducing an expert-gated color regularization mechanism, an error-aware Gaussian compaction strategy, and a Gaussian map refinement stage, achieving a more stable, complete, and high-fidelity dense RGB-D SLAM.

[0024] The core of this invention lies in constructing a 3D Gaussian map using RGB-D image sequences, stabilizing the Gaussian appearance optimization through expert-gated color regularization, supplementing under-reconstructed local areas through error-aware Gaussian densification during incremental SLAM, and improving the global map quality through Gaussian map refinement. Figure 1Consistency. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and specific steps. For example... Figure 1 As shown.

[0025] 1. Acquisition and Preprocessing of RGB-D Image Sequences First, a continuous RGB-D image sequence of the scene to be processed is acquired using a robot, drone, vehicle-mounted platform, or handheld RGB-D sensor. The RGB-D image sequence includes synchronously acquired RGB images and depth images, where the RGB images provide scene appearance information and the depth images provide scene geometric structure information.

[0026] Before system operation, camera intrinsic parameters, RGB images, depth images, and corresponding frame index information are read. For different data sources, depth scale normalization can be performed according to their depth encoding methods, such as converting depth maps stored in millimeters to depth values ​​in meters. At the same time, invalid depths, excessively close depths, or excessively far depths in the depth map are filtered to avoid abnormal depth values ​​affecting subsequent 3D backprojection, camera tracking, and Gaussian map construction.

[0027] After preprocessing, each frame of RGB-D data contains RGB image, depth image and camera intrinsic parameter information, providing input for subsequent differentiable rendering, camera pose estimation and 3D Gaussian map optimization.

[0028] 2. Initialization of the 3D Gaussian map After acquiring the initial RGB-D image frame, the system performs 3D backprojection using the effective depth pixels in the first frame and camera intrinsic parameters, converting 2D pixels into 3D spatial points. For each effective depth pixel, the system calculates its corresponding 3D position based on its pixel coordinates, depth value, and camera intrinsic parameters, and uses this 3D position as the center position of the initial Gaussian unit.

[0029] Each Gaussian primitive includes at least a 3D center position, a scale parameter, a color parameter, and an opacity parameter. The 3D center position represents the Gaussian primitive's location in scene space; the scale parameter describes the spatial influence range of the Gaussian primitive; the color parameter is initialized with the corresponding RGB pixel values; and the opacity parameter describes the contribution strength of the Gaussian primitive to the rendering result.

[0030] Using the above method, the system generates an initial set of 3D Gaussian primitives from the initial RGB-D image frame, thus forming an initial 3D Gaussian map. This Gaussian map serves as the basis for subsequent camera tracking, error-aware Gaussian compaction, and Gaussian map updates.

[0031] 3. Perform Gaussian map updates frame-by-frame guided by camera tracking, error-aware Gaussian compaction, and expert-gated color regularization. After completing the initial 3D Gaussian map construction, the system reads each frame sequentially, starting from the second frame of the RGB-D image sequence. It then iteratively performs camera tracking, error-aware Gaussian compaction, and expert-gated color regularization-guided Gaussian map updates for each frame until all input frames have been processed. This frame-by-frame iterative process constitutes the main flow of the online SLAM stage of this invention.

[0032] 3.1 Camera Tracking Based on Differentiable Rendering For the current input RGB-D frame, the system first predicts the initial pose of the current frame based on the historical camera pose and using a constant velocity motion model. Under this initial pose, the current 3D Gaussian map is rendered into a color image, a depth image, and a contour image using differentiable Gaussian splashing.

[0033] Subsequently, the rendered color image is compared with the real RGB image of the current frame to calculate the color residual; the rendered depth image is compared with the real depth image of the current frame to calculate the depth residual. By jointly minimizing the color residual and the depth residual, the camera pose of the current frame is iteratively optimized.

[0034] To improve the robustness of the tracking process, the system uses rendered contour images or cumulative opacity to filter reliable pixel regions. Only when the cumulative opacity of a pixel location is greater than a preset threshold is the pixel considered to belong to a region that can be reliably interpreted by the current Gaussian map and participate in the camera tracking loss calculation; regions with low cumulative opacity are regarded as new observation regions, occluded regions, or unstable regions and are not included in the current frame pose optimization.

[0035] 3.2 Error-aware Gaussian compaction After completing the camera pose estimation for the current frame, the system determines whether new Gaussian primitives need to be inserted based on the rendering feedback of the current Gaussian map at the viewpoint of that frame. This invention constructs an error-aware Gaussian compactification mask that comprehensively considers three types of information: contour coverage, color residual, and depth residual.

[0036] Specifically, when the cumulative opacity of a certain pixel position is lower than a preset threshold, it indicates that the region is not fully covered by existing Gaussian pixels and can be identified as a region with insufficient Gaussian coverage; when the color residual between the rendered color image and the real RGB image is greater than a preset color error threshold, it indicates that the appearance modeling of the region is insufficient and can be identified as a region with insufficient appearance modeling; when the depth residual between the rendered depth and the real depth map is greater than a preset depth error threshold, the corresponding region can be identified as a region with potential geometric missing features.

[0037] The system combines the three types of regions mentioned above to form a compacted mask, and inserts new Gaussian elements into the selected effective depth regions of the compacted mask. The newly inserted Gaussian elements inherit the RGB color information and depth geometry information of the corresponding pixels and are added to the current 3D Gaussian map to supplement local under-reconstructed areas, reduce local rendering errors, and enhance the integrity of map representation.

[0038] 3.3 Expert-gated color regularization-guided Gaussian map update After completing camera tracking and error-aware Gaussian compaction for the current frame, the system enters the Gaussian map update phase. The system maintains a keyframe database, which selects keyframes based on viewpoint overlap. The system selects the five keyframes with the highest viewpoint overlap with the current frame from the keyframe database. If there are fewer than five keyframes, the actual number is used, and these are combined with the current frame to form a local mapping window.

[0039] In each iteration of the mapping optimization, the system randomly samples a frame from the local mapping window as a supervision frame and renders the current 3D Gaussian map as a color image and a depth image. Then, the rendered color image is compared with the RGB image of the supervision frame to calculate the color rendering loss; the rendered depth image is compared with the depth image of the supervision frame to calculate the depth rendering loss. Through these color and depth rendering losses, the system optimizes the 3D center, scale, color parameters, and opacity parameters of the Gaussian units.

[0040] Simultaneously, this invention introduces an expert-gated color regularization mechanism during Gaussian map updates. The system constructs a state description feature for each Gaussian element. This state description feature includes at least the Gaussian element's opacity, cumulative observation count, spatial location, scale information, and color importance weight. The system inputs this state description feature into a lightweight hybrid expert gating module. This hybrid expert gating module includes a routing network and multiple expert networks. The routing network generates assigned weights for different experts based on the input state description feature, and the multiple expert networks output candidate gating responses. These responses are then weighted and fused to obtain the corresponding color regularization gating weights for the Gaussian elements. For example, the lightweight hybrid expert gating module can employ a structure consisting of four lightweight experts and a learnable routing network. Each expert network has different parameters, enabling it to output different gating weights. The routing network is implemented as a two-layer MLP with dimensions 5→16→4, and each expert network is implemented as a two-layer MLP with dimensions 5→16→1. Both use ReLU activation.

[0041] Finally, the system utilizes expert-gated weights to modulate the difference between the current Gaussian color parameters and the cached color parameters from the previous optimized state, thereby adaptively adjusting the color regularization intensity for different Gaussian primitives. For stable and frequently observed Gaussian primitives, the system imposes strong color consistency constraints to suppress color drift; for newly inserted or not yet fully optimized Gaussian primitives, the system retains a larger color update space to improve local adaptability. Through this mechanism, the system can improve the appearance stability and rendering consistency during incremental mapping.

[0042] 4. Gaussian map refinement After all frames in the input RGB-D image sequence have undergone the aforementioned frame-by-frame iterative processing, the system enters the Gaussian map refinement stage. The purpose of this stage is to alleviate the problems of local overfitting and uneven global convergence caused by sequential dependencies during online incremental mapping.

[0043] Specifically, the system randomly samples keyframes from the accumulated keyframe set and performs additional optimization on the current 3D Gaussian map based on the RGB-D observations of these keyframes. This stage reuses the mapping optimization function from the Gaussian map update stage, that is, it combines color rendering loss, depth rendering loss, and expert-gated color regularization term to further adjust the Gaussian meta-parameters.

[0044] By revisiting accumulated keyframes, early and sparse observation areas receive additional supervision, thus mitigating optimization biases introduced by relying solely on local time windows. This process improves the global consistency, appearance stability, and reconstruction quality of the 3D Gaussian map. In practical applications, the Gaussian map refinement stage can optimize only the Gaussian map parameters, or it can jointly optimize the Gaussian map parameters and camera pose parameters according to task requirements.

[0045] Camera pose and 3D Gaussian map output After all processing steps are completed, the system outputs an optimized camera trajectory and a 3D Gaussian map. The camera trajectory includes the camera pose estimation results for each frame in the input RGB-D image sequence; the 3D Gaussian map consists of an optimized set of Gaussian elements, capable of representing the geometric structure and appearance information of the scene.

[0046] Based on the output 3D Gaussian map, the system can generate color, depth, and contour images from any viewpoint using a differentiable Gaussian splash rendering method. The output results can be further used for tasks such as robot autonomous navigation, UAV intelligent perception, augmented reality, and intelligent traffic environment perception.

[0047] Experimental verification and effect evaluation To verify the effectiveness and practical value of this invention, the method was quantitatively and qualitatively analyzed using three categories of indicators: rendering quality, geometric reconstruction quality, and camera tracking accuracy. Rendering quality was evaluated using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Perceptual Similarity Error (LPIPS); geometric reconstruction quality was evaluated using Depth L1 and Reconstruction Integrity Index (F1); and camera tracking accuracy was evaluated using Absolute Trajectory Error (ATE). Among these indicators, higher values ​​for PSNR, SSIM, and F1 indicate better performance, while lower values ​​for LPIPS, Depth L1, and ATE indicate better performance.

[0048] This invention was experimentally validated on commonly used RGB-D SLAM datasets and UAV-view datasets. The Replica dataset was used to evaluate the system's rendering quality, geometric reconstruction accuracy, and camera tracking performance in indoor scenes; the UAV-view dataset was used to further verify the invention's scene reconstruction capabilities under complex viewpoint changes and flight motion conditions. To verify the performance advantages of this invention, its method was compared with several typical dense RGB-D SLAM methods, including Vox-Fusion, NICE-SLAM, Point-SLAM, GS-SLAM, SplaTAM, and HF-SLAM. Vox-Fusion and NICE-SLAM are methods based on neural implicit representations, Point-SLAM is based on neural point cloud representations, and GS-SLAM, SplaTAM, HF-SLAM, and this invention are all methods based on Gaussian primitive representations.

[0049] Experimental results show that the present invention achieves superior overall performance on the Replica dataset. In terms of rendering quality, the present invention achieves the highest PSNR and SSIM, indicating that it can generate clearer and more structurally consistent rendered images. Regarding perceptual similarity, the present invention achieves lower LPIPS, indicating that the rendered results are closer to the real images in terms of visual perception. In geometric reconstruction, the present invention achieves the lowest Depth L1, indicating that it can obtain more accurate depth geometric representations. In camera tracking, the present invention achieves competitive ATE results, demonstrating that it maintains stable pose estimation capabilities while improving map quality.

[0050] The experimental results of the method of this invention on the Replica dataset are shown in Table 1 below. The best results are indicated in bold, and the second-best results are indicated in underlined italics.

[0051] Table 1. Experimental results of the method of the present invention on the Replica dataset.

[0052] Furthermore, this invention also conducted visualization experiments on drone-perspective data, such as... Figure 2 As shown in the figure. Experimental results show that in scenes with large changes in viewing angle and motion trajectories closer to those of a flight platform, existing methods are prone to problems such as image blurring, missing local structures, color artifacts, and incomplete reconstruction, while the present invention can generate rendering results that are closer to realistic images. In the living room sequence, the PSNR of the rendering result of the present invention reaches 35.63 dB, which is higher than the comparison method; in the office sequence, the PSNR of the rendering result of the present invention reaches 31.47 dB, also showing better visual quality. These results indicate that the present invention is not only applicable to conventional indoor RGB-D SLAM scenes, but can also adapt to more complex observation conditions and motion patterns from the perspective of drones.

Claims

1. A Gaussian splatter dense color depth-SLAM method based on expert-gated color regularization, characterized by: Includes the following steps: (1) RGB-D image sequence acquisition and data preprocessing: acquire the RGB-D image sequence of the scene to be processed, the RGB-D image sequence includes synchronously acquired RGB images and depth images; read the RGB images, depth images and camera intrinsic parameters; (2) Initialization of three-dimensional Gaussian map: Initialize the three-dimensional Gaussian map using the first frame RGB-D image; the three-dimensional Gaussian map is composed of several Gaussian elements, each Gaussian element includes the three-dimensional center position, scale, color parameters and opacity; and set the camera pose of the first frame to the preset initial pose; (3) Starting from the second frame, read each frame in the RGB-D image sequence in sequence, and perform the following steps in a loop on the current frame: camera tracking based on differentiable rendering, error-aware Gaussian compaction, and expert-gated color regularization-guided Gaussian map update until all input frames have been processed. (4) Gaussian map refinement: After all frames in the input RGB-D image sequence have completed the processing of steps (2) and (3) above, the Gaussian map refinement stage is entered; key frames are randomly selected from the accumulated key frame database, and the Gaussian map is optimized based on the RGB-D image corresponding to the selected key frame; the optimization is to use the optimization function to further adjust the Gaussian meta-parameters; the optimization is repeated multiple times. (5) Camera pose and 3D map output: In practical applications, the input is the RGB-D image sequence to be processed, and the above steps of 3D Gaussian map initialization, camera tracking based on differentiable rendering, error-aware Gaussian compaction, expert-gated color regularization-guided Gaussian map update and Gaussian map refinement are executed; finally, the camera trajectory and 3D Gaussian map are output.

2. The method according to claim 1, characterized in that: It also includes step (6) three-dimensional Gaussian map rendering, in which the three-dimensional Gaussian map generates color images, depth images and contour images from any viewpoint through differentiable Gaussian splash rendering.

3. The method according to claim 1, characterized in that: In step (2), the initialization of the three-dimensional Gaussian map includes: back-projecting two-dimensional pixels into three-dimensional space based on the effective depth pixels and camera intrinsic parameters in the initial RGB-D image frame to generate the three-dimensional center position of the initial Gaussian primitive; using the corresponding RGB pixel values ​​as the initial color parameters of the Gaussian primitive; determining the initial scale of the Gaussian primitive based on the depth value and the camera projection relationship; and setting the initial opacity for each Gaussian primitive to form the initial three-dimensional Gaussian map.

4. The method according to claim 1, characterized in that: In step (3), the camera tracking based on differentiable rendering is as follows: predict the initial pose of the current frame based on the historical camera pose and using a constant velocity motion model; render the current 3D Gaussian map into a color image, a depth image, and a contour image according to the initial pose, and compare the rendering results with the RGB-D image shown in the current frame; iteratively optimize the camera pose of the current frame by minimizing the color residual and the depth residual. Calculate the cumulative opacity of the current 3D Gaussian map, and calculate the tracking loss only in reliable pixel areas where the cumulative opacity is greater than or equal to a preset threshold.

5. The method according to claim 1, characterized in that: In step (3), the error-aware Gaussian densification is specifically as follows: After completing the camera tracking of the current frame, it is determined whether new Gaussian primitives need to be inserted based on the densification mask of the current 3D Gaussian map in the current frame; the densification mask is constructed by integrating the error-aware Gaussian densification process of contour coverage, color residual and depth residual; wherein, pixel areas with cumulative opacity lower than a preset threshold are determined as insufficient Gaussian coverage areas, pixel areas with color residual greater than a preset threshold are determined as insufficient appearance modeling areas, and pixel areas with depth residual greater than a preset threshold are determined as geometrically missing areas; the insufficient Gaussian coverage areas, insufficient appearance modeling areas or geometrically missing areas in the 3D Gaussian map are marked as selected areas of the densification mask, and new Gaussian primitives are inserted in these areas.

6. The method according to claim 1, characterized in that: In step (3), the expert-gated color regularization-guided Gaussian map update is as follows: After completing the camera tracking and Gaussian compaction of the current frame, the 3D Gaussian map is updated based on the current frame and the key frame database in the SLAM architecture. The key frame database selects key frames according to the view overlap. The 5 key frames with the highest view overlap with the current frame are selected from the key frame database. If there are less than 5, the actual number is taken. The key frames are combined with the current frame to form a local mapping window. The mapping optimization function is used to perform mapping optimization iteration to update each parameter of each Gaussian element. In each iteration, a frame is randomly sampled from the local mapping window as a supervision frame to optimize the 3D Gaussian map. During the optimization process, for each Gaussian element, a state description feature containing opacity, number of observations, spatial location, scale information, and color importance weight is constructed. The state description features are input into the lightweight hybrid expert gating module. Different expert weights are generated for each feature through the routing network, and the color regularization gating weights of the corresponding Gaussian elements are jointly output by multiple expert networks. The residual between the current Gaussian color parameter and the color parameter cached in the previous optimized state or initialization state is minimized by using the expert gating weights, and the color regularization intensity of different Gaussian elements is adaptively adjusted.

7. The method according to claim 5, characterized in that: For pixel regions where the depth residual is greater than a preset threshold, a new Gaussian element is generated; the new Gaussian element inherits the RGB color information and depth geometric information of the corresponding pixel and is inserted into the current 3D Gaussian map.

8. The method according to claim 6, characterized in that: The state description features include at least the opacity of Gaussian elements, the cumulative number of observations, spatial location, scale information, and color importance weights. The lightweight hybrid expert gating module includes a routing network and multiple expert networks. The routing network generates the assigned weights of each expert network based on the state description features. The multiple expert networks output candidate gating responses respectively, and the color regularization gating weights of the current Gaussian are obtained through weighted fusion.

9. The method according to claim 6, characterized in that: In optimizing the 3D Gaussian map, the optimization objectives include color rendering loss, depth rendering loss, and expert-gated color regularization. Among them, color rendering loss is used to constrain the consistency between the rendered color image and the real RGB image, depth rendering loss is used to constrain the consistency between the rendered depth image and the real depth image, and expert-gated color regularization is used to suppress unstable changes in Gaussian color parameters during incremental mapping.

10. The application of the method according to claim 2, characterized in that... This method can be applied to robot autonomous navigation, UAV intelligent perception, augmented reality, digital twins, or intelligent transportation environment perception.

Images