Slam acceleration method based on three-dimensional gaussian rendering

Abstract

The application relates to the field of computer vision and navigation, and provides a SLAM acceleration method based on three-dimensional Gaussian rendering, which comprises the following steps: initializing a three-dimensional Gaussian point cloud and camera pose parameters based on input data, performing camera tracking on a target frame in the input data, and outputting optimized camera pose parameters based on a staged switching strategy and an iteration mechanism, wherein the camera tracking utilizes a reduced-precision floating-point format; performing a hierarchical screening mechanism in the optimized camera pose parameters to determine key frames from color images and depth images, using the key frames and the optimized camera pose parameters, and performing mapping through a renderer supporting multi-channel data to update the three-dimensional Gaussian point cloud. The method can quickly converge the pose optimization through staged switching and iteration, uses the reduced-precision floating-point format to reduce the numerical processing burden, and the hierarchical screening mechanism can reduce the calculation cost of frame-by-frame fine comparison, thereby solving the problem of low efficiency of the SLAM process.

Description

Technical Field

[0001] This application relates to the fields of computer vision and navigation technology, and in particular to a SLAM acceleration method based on 3D Gaussian rendering. Background Technology

[0002] 3D point cloud processing technology, combined with high-performance GPU computing, drives the development of SLAM (Simultaneous Localization and Mapping) algorithms. SLAM is used for real-time localization and map building, and is applied in dense scene reconstruction, dynamic environment perception, and robot autonomous navigation. The 3D Gaussian distribution representation method can accurately model the point cloud distribution and provide continuous scene representation. User requirements include high-precision map building, reliable pose estimation, and high-quality scene reconstruction, while requiring efficient computing and low-latency response. It is suitable for fields such as robotics, augmented reality, and virtual reality.

[0003] Some technologies use 3D Gaussian point clouds as SLAM carriers to replace traditional geometric methods or neural radiation fields. 3D Gaussian SLAM generates point clouds by backprojecting RGB-D data and uses differentiable rendering to optimize scene parameters and camera trajectories. This method can densely represent the scene and improve map accuracy and pose estimation reliability. It includes camera pose optimization and 3D rendering pipeline. It backpropagates the difference between the synthesized image and the real observation. The optimization process relies on full-precision floating-point calculation and fixed iteration strategy.

[0004] However, the above methods result in huge computational loads for rendering and point cloud updates, low efficiency of matrix operations, insufficient utilization of GPU parallelism, frequent repetitive calculations, high memory consumption and bandwidth pressure due to full-precision floating-point calculations, redundant keyframe selection, fixed loss iterations, and separate CUDA core calls, which limit the real-time application of the system. In other words, the above methods result in low efficiency of the SLAM process while ensuring that the accuracy of map reconstruction and pose estimation remains basically unchanged. Summary of the Invention

[0005] This application provides a SLAM acceleration method based on 3D Gaussian rendering to solve the problem of low efficiency in the SLAM process while ensuring that the accuracy of map reconstruction and pose estimation remain basically unchanged.

[0006] This application provides a SLAM acceleration method based on 3D Gaussian rendering, including:

[0007] Acquire input data, which includes color images and depth images;

[0008] Based on the first frame of the input data, initialize the 3D Gaussian point cloud and camera pose parameters;

[0009] For the target frame in the input data, camera tracking is performed, and optimized camera pose parameters are output based on a phased switching strategy and an iterative mechanism. The target frame is any frame in the input data other than the first frame. The camera tracking uses a reduced-precision floating-point format.

[0010] Based on the optimized camera pose parameters, a hierarchical filtering mechanism is executed to determine keyframes from the color image and the depth image. The hierarchical filtering mechanism includes at least two stages.

[0011] The keyframes and optimized camera pose parameters are used to perform rendering and mapping to update the 3D Gaussian point cloud.

[0012] As can be seen from the above technical solutions, this application provides a SLAM acceleration method based on 3D Gaussian rendering. The method includes: acquiring input data, the input data including a color image and a depth image; initializing a 3D Gaussian point cloud and camera pose parameters based on the first frame of the input data; performing camera tracking on a target frame in the input data; outputting optimized camera pose parameters based on a phased switching strategy and an iterative mechanism; wherein the target frame is any frame in the input data other than the first frame; and the camera tracking utilizes a reduced-precision floating-point format; performing a hierarchical filtering mechanism based on the optimized camera pose parameters to determine keyframes from the color image and the depth image; the hierarchical filtering mechanism includes at least two stages; and performing mapping using the keyframes and the optimized camera pose parameters to update the 3D Gaussian point cloud. The proposed method integrates a phased switching strategy and an iterative mechanism into the camera tracking process, enabling pose optimization to converge quickly and avoiding premature consumption of computational resources on secondary targets. The use of reduced-precision floating-point format during camera tracking alleviates the burden of numerical processing and memory subsystem bandwidth pressure. The hierarchical filtering mechanism identifies keyframes at a computational cost lower than frame-by-frame fine-grained comparison. This addresses the issue of low efficiency in the SLAM process while maintaining essentially unchanged map reconstruction and pose estimation accuracy. Attached Figure Description

[0013] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating the SLAM acceleration method based on 3D Gaussian rendering provided in this application embodiment;

[0015] Figure 2 This is a schematic diagram of the first frame initialization process provided in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of the preprocessing flow provided in the embodiments of this application;

[0017] Figure 4 This is a schematic diagram of loss function calculation provided in an embodiment of this application;

[0018] Figure 5 A simplified schematic diagram for calculating the semi-major axis provided in an embodiment of this application;

[0019] Figure 6 This is a schematic diagram of the keyframe selection process provided in an embodiment of this application. Detailed Implementation

[0020] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with this application.

[0021] With the rapid development of fields such as computer vision, autonomous robot navigation, virtual reality (VR), and augmented reality (AR), SLAM has become an important foundation for intelligent perception systems. Traditional SLAM technology is mostly based on sparse feature points or dense pixel information to achieve environmental modeling and camera pose estimation. However, its reconstruction quality and real-time performance are often limited in complex lighting, dynamic scenes, and large-scale environments.

[0022] With the development of GPU computing power and 3D sensing devices (such as depth cameras and LiDAR), methods for continuous scene modeling based on point clouds have begun to be explored. Among them, point cloud representation based on 3D Gaussian distribution has gradually become an important research direction in the field of 3D vision because it can simultaneously encode geometric structure and color distribution information and achieve a good balance between storage and rendering efficiency.

[0023] This type of method uses continuously distributed 3D Gaussian point clouds as the core representation of the scene. Each point carries attributes such as position, covariance matrix, color, and opacity, enabling continuous modeling and differentiable rendering of real-world scenes while maintaining low storage overhead. In such systems, the core workflow typically includes camera pose optimization and a 3D rendering pipeline. By calculating the difference between the rendered synthetic image and the actual observed image, scene parameters and camera trajectory are optimized in reverse.

[0024] Specifically, the workflow of the 3D Gaussian SLAM system is as follows: First, the system serially receives frames-by-frame images captured by an RGB-D camera, containing three channels of color data and one channel of depth data, and saves them as ground truth. Second, this 2D image is back-projected into the 3D camera space, generating a 3D Gaussian point for each pixel. Next, using the estimated camera pose and the existing 3D Gaussian point cloud, the 3D Gaussian points are projected onto the 2D image plane according to the camera's intrinsic and extrinsic parameters, calculating their distribution range and depth in the pixel coordinate system. Subsequently, based on the point's covariance matrix, its elliptical coverage area in screen space is determined, and all Gaussian points are weighted and rendered after depth sorting using color and opacity parameters, thereby generating a rendered image corresponding to the camera's viewpoint. This rendering process can be executed in a highly parallel manner on the GPU. Then, gradient backpropagation is performed on the Gaussian point parameters, enabling the system to achieve end-to-end joint optimization using photometric errors.

[0025] Compared with traditional reconstruction methods based on voxel meshes or triangular meshes, 3D Gaussian-based SLAM technology has higher representational flexibility and continuity, and can capture details and surface transition features without explicit topology, thereby significantly improving computational efficiency while ensuring reconstruction quality.

[0026] Some approaches, such as Splatam, utilize a single non-localized monocular RGB-D camera to achieve dense SLAM via a 3D Gaussian carrier. However, existing methods have the following drawbacks:

[0027] 1. Redundancy in Keyframe Selection Strategy. During the 3D Gaussian map update phase, considering the computational burden, SLAM needs to select some keyframes as a baseline. Based on the estimated camera poses so far, it renders and optimizes the parameters of the updated 3D Gaussian map. Splatam saves every nth frame into a candidate keyframe list and selects k frames as keyframes for optimization, including the current frame, the most recent keyframe, and k-2 preceding keyframes.

[0028] The keyframe selection module typically judges based on the degree of geometric overlap between images. Its basic process is as follows: First, a certain number of valid pixels are randomly sampled in the current frame, and these pixels are back-projected into a 3D point cloud using camera intrinsic and extrinsic parameters. Then, the point cloud is sequentially transformed into the camera coordinate system of all candidate keyframes in the candidate keyframe list, and then projected onto the frame plane again. The proportion of the projected points falling within the image range of each keyframe is calculated, which is used as the inter-frame overlap index. Finally, the top k keyframes are selected in descending order of overlap and used as the optimization input for the current frame.

[0029] As system runtime and the number of keyframes increase, the original algorithm gradually reveals several performance bottlenecks and limitations. First, the method performs complete 3D point transformation and projection operations on each keyframe individually, and its computational complexity increases linearly with the number of keyframes. This significantly slows down the system when dealing with dozens or even hundreds of keyframes. Second, because each keyframe requires independent matrix multiplication, projection, boundary judgment, and statistical operations, the GPU parallelization utilization is low, making the keyframe selection stage one of the performance bottlenecks of the real-time system.

[0030] Secondly, the algorithm lacks a hierarchical or coarse-screening mechanism, failing to distinguish between significantly overlapping frames and obviously irrelevant frames, resulting in a large amount of redundant computational overhead. For example, in situations where scene changes are minimal or keyframes are densely distributed, some keyframes have almost no overlap but are still fully included in the computation, thus wasting computational resources. Furthermore, when the number of keyframes is large but only a small number of candidate frames are actually needed, the original algorithm still performs full computation, failing to dynamically reduce the candidate range according to the target selection ratio, resulting in a mismatch between computational efficiency and task requirements.

[0031] 2. Redundancy in Data Precision. In some solutions, most matrix and tensor operations during the rendering and optimization phases are performed using full-precision (FP32) floating-point representations, including 2D and 3D covariance matrix calculations for 3D Gaussian points, homogeneous transformations of point cloud coordinates, and operations such as 2D projection and depth-weighted fusion in the rendering pipeline. This design ensures high numerical precision and reconstruction stability, but also introduces significant computational and storage overhead. Since the throughput of FP32 operation units on GPUs is relatively low, their computational latency becomes one of the main bottlenecks in the entire rendering and optimization process. Especially when millions of Gaussian points need to be processed per frame, the computational load of matrix multiplication and covariance updates is enormous, easily leading to excessive GPU computing power consumption and increased inter-frame latency. Furthermore, the large data volume of full-precision floating-point format significantly increases video memory usage and memory bandwidth consumption, making it unfriendly to power- and hardware-resource-sensitive edge devices. In high-resolution inputs or large-scale scenes, intermediate tensors such as covariance matrices, projection matrices, and gradient caches reside simultaneously in video memory, causing video memory strain or even overflow, limiting the system's scalability in more complex scenarios.

[0032] On the other hand, existing methods do not fully utilize the mixed-precision computing capabilities of GPUs. Although modern GPUs widely support half-precision (FP16) computation and have dedicated acceleration units such as Tensor Cores, all matrix and tensor operations in the original algorithm are still performed in FP32 precision. This fails to manage precision levels according to the computational needs of different stages, resulting in wasted computing resources. In the backpropagation stage, gradient calculation and parameter updates both use full-precision data, leading to long operator execution times. Furthermore, uneven load distribution occurs on GPU computing units under multi-threaded concurrency, further reducing hardware utilization.

[0033] 3. Iterative Redundancy of Loss. In existing SLAM systems based on 3D Gaussian point clouds (such as SplaTAM), camera pose estimation typically employs an iterative optimization strategy based on photometric error. This involves updating the camera pose and scene parameters by minimizing the difference between the rendered image and the actual observed image. This process generally uses adaptive optimization algorithms such as gradient descent or Adam, performing a fixed number of iterations (usually around 200 generations) in each frame of the tracking phase to obtain stable pose estimation results.

[0034] However, this fixed-step optimization design suffers from significant computational redundancy and efficiency issues in practical operation. Due to substantial differences in image overlap, lighting consistency, and depth quality between different frames, the optimization convergence speed is not constant. For most frames, the loss function tends to stabilize after the first few dozen iterations, but the system still forces all 200 rounds of backpropagation and parameter updates, resulting in a large amount of unnecessary computation. Especially during backpropagation performed on the GPU, each iteration requires recalculating the gradient, calling the rendering kernel function, and performing gradient backpropagation, resulting in extremely high time and memory overhead. This makes the optimization phase one of the main sources of overall system latency.

[0035] Furthermore, existing methods lack an adaptive control mechanism based on convergence metrics. During iteration, even when the loss function's rate of decline is extremely low or oscillations occur, the algorithm continues to perform a complete iteration without automatically terminating early, thus wasting GPU computing resources. This strategy is particularly inefficient in scenarios involving long-term, multi-frame cumulative optimization. For example, in the stable tracking phase or when scene changes are minimal, pose optimization across multiple consecutive frames may fall into repetitive computation, significantly reducing the overall real-time performance and energy efficiency of the system. On the other hand, this fixed-algebra optimization is also detrimental to hardware resource scheduling, failing to flexibly adjust the iteration depth according to dynamic changes in computational load, causing fluctuations in GPU core utilization and even affecting the efficiency of parallel task scheduling.

[0036] 4. Redundancy in CUDA kernel calls. In existing SLAM systems based on 3D Gaussian point clouds, rendering and optimization are typically based solely on three-channel color images (RGB) for photometric error constraints, while depth information is processed as a separate module, mainly used for point cloud backprojection and generating depth silhouette map constraints. While this separate structure is logically clear in its design, it leads to fragmented data flow and redundant computations in actual operation.

[0037] Because RGB rendering and depth rendering belong to different computational paths, the system needs to maintain two sets of data buffers and intermediate results, including projection point coordinates, pixel weights, occlusion masks, and other information. This not only increases memory usage and data copying overhead but also significantly increases communication latency between the rendering and optimization modules. In the GPU implementation, the rendering kernel functions for the RGB and depth channels cannot share the same scheduling flow and must be called and synchronized separately, limiting parallelization efficiency and increasing overall scheduling complexity. The original solution did not implement multi-channel fusion scheduling in its computational organization; that is, the rendering pipeline still processes a single image channel as a unit. For tasks with both color and geometric constraints, the system must first complete RGB rendering and error calculation before switching to the depth rendering stage to execute geometric consistency constraints. This serial structure leads to a significant increase in the time overhead of the rendering and backpropagation stages, making it impossible to achieve unified modeling and efficient parallel computation of RGB and depth information, thus limiting the system's performance in high-frame-rate real-time SLAM scenarios.

[0038] 5. Redundancy in loss function calculation. The system needs to calculate the loss function for the current frame during both the camera tracking and mapping phases. However, some operators (such as 2D / 3D covariance updates) involve a large number of complex calculations, such as square root operations and exponential operations. Although these calculations have limited sensitivity to subsequent loss function training, they are still executed in full using high-precision standard formulas, resulting in increased energy consumption and decreased performance overall.

[0039] On the other hand, existing solutions typically employ a fixed-structure, full-image L1 loss function, which involves a pixel-by-pixel global comparison between the rendered result and the actual observed image to minimize reconstruction error and guide parameter optimization. These methods usually perform joint backpropagation updates of camera pose parameters and the 3D Gaussian point cloud in each frame to achieve higher accuracy. Furthermore, some solutions balance depth and color errors at different stages using simple loss weight coefficients, but lack a dynamic switching mechanism, maintaining a uniform computational pattern across all stages. Simultaneously, existing methods often employ a dense computational pattern involving all pixels when calculating the loss, uniformly calculating depth and RGB errors across the entire image without distinguishing between critical regions such as edges and high-texture areas and flat regions. A few improved solutions attempt to introduce fixed thresholds or randomly sampled mask mechanisms, but these often rely on preset ratios or static mask templates, lacking adaptability and gradient awareness.

[0040] To address the aforementioned issues, some embodiments of this application provide a SLAM acceleration method based on 3D Gaussian rendering. By systematically optimizing camera tracking, mapping, loss function iteration, and data flow management processes, this method significantly accelerates the overall SLAM process while ensuring map reconstruction accuracy and pose estimation accuracy.

[0041] By quantizing the matrix calculation format, reducing the number of CUDA core accesses, optimizing the keyframe selection mechanism, using low-cost operators for approximate calculation, optimizing the loss function calculation strategy, and implementing early training stopping, this approach not only improves computational throughput but also reduces GPU memory usage. Furthermore, by considering the stage-specific characteristics of the loss function and computational resource scheduling strategies, it achieves adaptive processing capabilities for different scenarios. Moreover, the method maintains a highly modular algorithm framework, facilitating deployment on various hardware architectures and enabling efficient application to existing 3D Gaussian SLAM systems, further enhancing the algorithm's practicality and promotional value. Through these optimizations, this scheme significantly accelerates the overall SLAM process while maintaining essentially unchanged map reconstruction and pose estimation accuracy.

[0042] like Figure 1 As shown, the method includes the following steps:

[0043] S100: Obtain input data.

[0044] The input data includes color images and depth images. The color image provides visual information about the scene, while the depth image provides geometric information about the scene.

[0045] A color image is a series of two-dimensional image data frames continuously captured by an imaging device, containing scene color information. Color information typically includes three channels: red, green, and blue, used to record the appearance and texture of the scene. This sequence can originate from the color sensor in an RGB or RGB-D camera, or from image data that has undergone preprocessing such as interpolation and depigmentation. In some implementations, color images can also be represented using other color spaces.

[0046] A depth image is a two-dimensional dataset that is temporally aligned with a color image, recording the physical distance from each pixel in the scene to the camera's optical center in each frame. Depth information is typically stored as a single-channel matrix. This sequence can originate from a time-of-flight camera, a structured light depth camera, a binocular stereo vision system, or a monocular depth estimation algorithm based on a neural network.

[0047] S200: Based on the first frame of the input data, initialize the 3D Gaussian point cloud and camera pose parameters.

[0048] 3D Gaussian point cloud parameters are a set of optimizable variables used to characterize the 3D geometry and appearance of a scene. Each 3D Gaussian point corresponds to a set of parameters, which together define a point distributed in an ellipsoidal shape in 3D space. The position parameter describes the coordinates of the Gaussian point in 3D space, the scale parameter defines the extent to which the Gaussian point extends in various directions, determining the shape of the ellipsoid, the color parameter defines the color that the point appears to be when viewed from a specific viewpoint, and the transparency parameter defines the contribution weight of the point to the final rendered image.

[0049] Camera pose parameters are a set of parameters describing the position and orientation of an imaging device in three-dimensional space. Position is usually represented by a three-dimensional translation vector, while orientation can be represented by a rotation matrix, Euler angles, or quaternions. Camera pose parameters define the transformation relationship from the world coordinate system to the camera coordinate system and serve as the geometric basis for projecting a three-dimensional scene onto a two-dimensional image plane. In the SLAM process, each frame corresponds to a set of camera pose parameters to be estimated or optimized.

[0050] In some embodiments, based on the first frame of the input data, initializing the 3D Gaussian point cloud and camera pose parameters includes: performing data preprocessing on the color image of the first frame to output a preprocessed color image, the data preprocessing including color normalization; performing data preprocessing on the depth image of the first frame to output a preprocessed depth image, the data preprocessing including depth dimension rearrangement; backprojecting the preprocessed depth image based on the camera intrinsic and extrinsic matrices to generate spatial position parameters of the 3D point cloud; obtaining the color vector of the pixel corresponding to the spatial position parameters from the preprocessed color image; calculating the mean square distance between adjacent points in the 3D point cloud using a reduced-precision floating-point format, and calculating the scene radius according to a preset maximum depth value and scene scale parameters; and initializing the position parameters, color parameters, transparency parameters, and scale parameters in the 3D Gaussian point cloud based on the spatial position parameters, color vectors, mean square distance, and scene radius.

[0051] like Figure 2 As shown, specifically, data preprocessing is first performed on the color image of the first frame to output a preprocessed color image. The data preprocessing includes color normalization, which normalizes the pixel values ​​to the range of 0-1, thereby eliminating the influence of differences in imaging devices on color intensity. The color image size is (480, 640, 3), the depth image size is (480, 640, 1), and the camera parameters include intrinsics (3×3) and pose (4×4).

[0052] Simultaneously, data preprocessing is performed on the depth image of the first frame to output a preprocessed depth image. This preprocessing includes depth dimension rearrangement, reorganizing the depth image into a memory layout suitable for GPU parallel computing. This standardized input preprocessing maintains the comparability and robustness of initialization results across different data sources and camera models.

[0053] After preprocessing, the input camera parameter inverse kinematics module is used to invert the camera extrinsic matrix `pose`, obtaining the transformation matrix `w2c` (world-to-camera matrix, 4×4) from world coordinates to camera coordinates. The spatial position of the camera center in the world coordinate system is then calculated. By inverting the first three rows and three columns of `w2c`, the position vector of the camera's optical center in world coordinates is obtained, used for subsequent perspective projection matrix construction. Simultaneously, the perspective projection matrix `proj` (projection matrix, 4×4) is constructed based on the camera intrinsic matrix `K` and the extrinsic matrix `w2c`, and its product `w2c` is calculated. proj It is used to realize the coordinate projection mapping from three-dimensional points to two-dimensional image planes.

[0054] This ensures that the geometric relationship between different coordinate systems is maintained, thus providing a theoretical basis for the inverse transformation from depth values ​​to 3D coordinates. It unifies the input camera geometric information into a reversible linear mapping relationship, enabling the system to maintain spatial consistency and trackability in subsequent frames.

[0055] The camera intrinsic parameter matrix contains focal length and principal point coordinate information, while the camera extrinsic parameter matrix describes the camera's position and orientation in the world coordinate system. A back-projection transformation converts the two-dimensional pixel coordinates and corresponding depth values ​​into three-dimensional world coordinates. Then, a reduced-precision floating-point format is used to calculate the average squared distance between adjacent points in the 3D point cloud. Based on a preset maximum depth value and scene scale parameters, the scene radius is calculated. In this embodiment, the reduced-precision floating-point format uses FP16 instead of FP32, significantly reducing memory usage and computation time while maintaining computational accuracy.

[0056] In the point cloud generation stage, effective pixels are first filtered based on the depth image. A mask is generated to remove pixels with zero depth or invalid pixels, thus avoiding noise propagation from meaningless points. Subsequently, based on the camera intrinsic parameter matrix... Image plane coordinates With corresponding depth value Back projection is a 3D point in the camera coordinate system Its mathematical expression is shown in the following formula:

[0057] ;

[0058] in, This is the inverse of the camera intrinsic matrix, used to implement the transformation from pixel coordinates to camera coordinates.

[0059] Then through the Inverting the matrix yields The matrix maps a 3D point cloud from the camera coordinate system to the world coordinate system, and its calculation formula is as follows:

[0060] ;

[0061] Then, the spatial transformation of the 3D points from the camera's local coordinate system to the global world coordinate system is completed, thereby obtaining the physical position of the initial point cloud in the scene.

[0062] This approach uses FP16 instead of FP32, significantly reducing hardware resource consumption during computation. While generating the point cloud, the system calculates the average squared distance between adjacent points in the point cloud. This is used to measure the density of point clouds and the spatial scale distribution of a scene. Its calculation expression is as follows:

[0063] ;

[0064] in, The three-dimensional coordinates of adjacent points This represents the number of valid point pairs.

[0065] In the parameter initialization module, the system calculates the scene radius based on the preset maximum depth value and the set scene scale parameter. The formula is as follows:

[0066] ;

[0067] in, This is the maximum depth value. The set scene scale parameters.

[0068] The scene scale parameter is used to adaptively adjust the spatial scale of the Gaussian rendering volume, so that a reasonable Gaussian distribution density can be maintained under different depth ranges or scene sizes.

[0069] Finally, an initial 3D point cloud dataset is generated. The number of points in the point cloud is usually H×W effective pixels. While generating the point cloud, the system calculates the average squared distance between adjacent points in the point cloud to characterize the density of the point cloud and the spatial scale distribution of the scene. This transforms 2D perceptual data into 3D structural data with spatial semantics. The system also achieves automatic estimation of the scene's geometric scale through the distance measurement between points, providing a statistical basis for subsequent Gaussian kernel parameterization.

[0070] After obtaining the initial point cloud, the rendering parameters and system variables are initialized based on the point cloud data, the set total number of frames, the mean squared distance, and the selected Gaussian distribution model. The initialized rendering parameters include the Gaussian distribution center coordinates, i.e., spatial position parameters, color parameters, variance matrix, and transparency parameters; the system variables include dynamically adjustable global variables, such as scene scale parameters and densification thresholds.

[0071] S300: For the target frame in the input data, perform camera tracking and output optimized camera pose parameters based on a phased switching strategy and an iterative mechanism.

[0072] like Figure 3 As shown, when the system starts tracking the image at time t of a new frame, it first needs to read the color image (RGB) and depth image (Depth) of the current frame from the input data, and calculate the world-to-camera transformation matrix based on its true pose. This matrix is ​​obtained by inverting the true pose matrix. Subsequently, the color image and depth image are preprocessed, including channel rearrangement and normalization. This step ensures that the input data meets a uniform tensor format for efficient processing on the GPU.

[0073] The target frame is any frame in the input data other than the first frame. For each frame in the sequence after the first frame, the system performs camera tracking. The goal of camera tracking is to calculate the accurate pose of the camera in the three-dimensional space corresponding to the current frame. The tracking process adopts an iterative optimization mechanism that combines a phased switching strategy. This means that the optimization process is not static, but will intelligently adjust its focus as the iteration progresses.

[0074] The tracing process utilizes a reduced-precision floating-point format for calculations; for example, it uses half-precision floating-point numbers instead of traditional single-precision floating-point numbers. This data format conversion effectively reduces the bandwidth pressure on data transmission between processor memory and computing units, and improves the execution throughput of computing units.

[0075] When time step t > 1, a constant velocity model is introduced as a camera motion prediction mechanism to calculate the initial pose of the current frame based on the poses of the previous two frames, thereby reducing the optimization search space and improving the convergence speed.

[0076] In some embodiments, camera tracking is performed on the target frame in the input data, including: predicting an initial estimate of the camera pose of the current target frame based on the optimized camera pose parameters of the previous frame using a constant velocity model; predicting an initial estimate of the camera pose of the current target frame based on the optimized camera pose parameters of the previous frame using a constant velocity model includes: obtaining the optimized camera pose parameters of the previous two frames, and calculating the initial rotation estimate and initial translation estimate of the current target frame using a first-order extrapolation method based on the optimized camera pose parameters of the previous two frames; performing normalization processing on the initial rotation estimate to obtain an initial rotation value that meets the rotation constraints; combining the initial rotation value and the initial translation estimate to obtain an initial estimate of the camera pose of the current target frame; and performing optimization iterations starting from the initial camera pose estimate to output the optimized camera pose parameters of the current target frame.

[0077] For the initial rotation estimate, let the unnormalized rotation quaternions of the first two frames be respectively... Then the initial value of the rotation of the current frame Calculated using the following formula:

[0078] ;

[0079] That is, by using first-order extrapolation, the initial rotation value of the current frame is inferred based on the rotation trend of the previous two frames, and the quaternion is ensured to satisfy the rotation constraint through normalization operation.

[0080] For the initial translation estimate, let the camera translation vectors of the first two frames be... Then the constant-velocity extrapolation model for translation is:

[0081] ;

[0082] in, This is the initial estimate of the camera translation vector for frame t.

[0083] This method assumes that the camera's linear velocity remains constant over a short period, effectively reducing jitter or divergence caused by inaccurate initial optimization values. For the first frame of the sequence (t=0) or in the case of no historical information, the pose of the previous frame is directly copied as the initialization value to maintain temporal continuity, as shown in the following equation:

[0084] ;

[0085] The initial rotation value describes the predicted camera orientation, while the initial translation estimate describes the predicted camera position. These two pieces of information are encapsulated together into a camera pose representation, for example, combined into a 4×4 homogeneous transformation matrix. The top-left 3×3 submatrix is ​​derived from the initial rotation value, and the rightmost column vector is composed of the initial translation estimate. This combined transformation matrix is ​​the final output initial camera pose estimate, representing the system's best guess of the current camera state based on historical motion, and will serve as the initial input value for subsequent refinement optimization algorithms.

[0086] Starting with the initial camera pose estimation, optimization iterations are performed to output the optimized camera pose parameters for the current target frame. In some embodiments, this includes: using the initial camera pose estimation as the current optimization state; based on the current optimization state, synchronously generating the color rendering result and depth rendering result corresponding to the target frame through the forward rendering channel; calculating the loss function between the color rendering result and the depth rendering result and the actual observed image; updating the current optimization state based on the loss function and in conjunction with a phased switching strategy, and outputting the optimized camera pose parameters after iteration.

[0087] In the forward rendering pass, matrix calculations are performed using a reduced-precision floating-point format. The matrix calculations include covariance matrix operations and projection transformation operations. The covariance matrix operations include the calculation of the semi-major axis of a 3D Gaussian projection onto a 2D plane. The semi-major axis calculation uses an approximation based on Taylor expansion. The target frame is a 3D Gaussian point located within the view frustum.

[0088] Camera tracking is implemented by a camera tracking module. This module uses a gradient optimization mechanism to dynamically estimate the camera pose in the current frame. In each time frame, it receives the initial rotation and translation parameters provided by the initialization module from the previous stage and performs multiple rounds of optimization iterations in conjunction with the rendering loss function to obtain the optimal camera pose estimate. The core of the camera tracking module is to calculate the loss function between the rendered image and the ground truth image under the current camera pose parameters and perform iterative optimization.

[0089] The forward rendering pipeline is a unified data processing flow that combines a 3D scene model with a given camera pose to generate corresponding 2D color and depth images. It's called a pipeline because it merges what might otherwise be separate color and depth rendering paths into a single execution pipeline. In its implementation, this pipeline can be implemented using an integrated GPU kernel function that receives scene and camera parameters, calculates the color and depth values ​​for each pixel in parallel, and outputs the results synchronously.

[0090] In the forward rendering pass, matrix calculations are performed using a reduced-precision floating-point format. The matrix calculations include covariance matrix operations and projection transformation operations. The covariance matrix operations include the calculation of the semi-major axis of a 3D Gaussian projection onto a 2D plane. The semi-major axis calculation uses an approximation based on Taylor expansion. The target frame is a 3D Gaussian point located within the view frustum. Unnecessary computations are reduced by view frustum clipping.

[0091] Specifically, the color image and depth image of the current frame are first extracted from the input data in temporal index order, and their camera intrinsic and extrinsic parameter matrices are calculated. The extrinsic parameter matrix is ​​provided by the initialization module in the previous stage or by the tracking results of the previous frame. The input color image is rearranged by channel into (C,H,W) format and normalized; the depth image undergoes the same channel transformation to ensure consistent processing on the GPU in subsequent steps.

[0092] When entering a new time frame idx When the velocity is greater than 0, a constant velocity model is used to initialize an initial camera pose estimate for the current frame. This method can quickly generate the initial pose of the current frame based on the assumption of motion continuity, even without real calibration information, providing a better starting point for subsequent optimization.

[0093] The view frustum detection module is a key component of this application for filtering out invalid point clouds and reducing the computational burden of rendering and projection in 3D Gaussian point cloud rendering and tracking calculations. Implemented on the GPU as an inline device function, this module can efficiently determine whether each 3D point is within the current camera's field of view (i.e., within the camera's view frustum) even with high point cloud density and significant inter-frame differences. This avoids invalid calculations for invisible points during subsequent rasterization and Gaussian projection stages. The view frustum geometry is a truncated pyramid shape with a large base, determined by both camera intrinsic parameters and the projection matrix.

[0094] This application determines whether point P is within the view frustum by judging whether point P simultaneously satisfies the depth condition and the screen projection constraint condition. That is, the depth of the point in the camera coordinate system should be between the near clipping plane and the far clipping plane. When the distance of the point in front of the camera's field of view is less than 0.2 units, the point is regarded as an invalid point and is discarded. In addition, the point is transformed from world coordinates through the projection matrix to obtain homogeneous coordinates and the perspective division is calculated. If the projected point exceeds the normalized device coordinate (NDC) range [-1,1], it means that the point is not within the visible range of the current frame image.

[0095] A 3D Gaussian ellipsoid possesses volumetric, directional, and density distribution characteristics in space, exhibiting an ellipsoidal shape. A 3D Gaussian model can not only continuously represent surface shapes but also participate in optimization and gradient propagation in a differentiable manner, thus achieving a fine representation of both geometry and lighting. In three-dimensional space, a Gaussian ellipsoid can be represented by the following probability density function (PDF). Defined as follows:

[0096] ;

[0097] in, Spatial coordinates; It is the Gaussian center (i.e., the geometric center of the ellipsoid). for The covariance matrix is ​​used to describe the shape, scale, and orientation of an ellipsoid in three-dimensional space.

[0098] covariance matrix The characteristic decomposition form is as follows:

[0099] ;

[0100] in, Let the rotation matrix define the directions of the principal axes of the ellipsoid. The scaling matrix determines the variance of the ellipsoid along its three principal axes.

[0101] Therefore, the three-dimensional Gaussian ellipsoid can be characterized by the following six key parameters during the rendering process:

[0102] ;

[0103] If we further consider color distribution and opacity, each Gaussian point can be expanded into the following complete set of attributes: :

[0104] ;

[0105] in, Indicates color, Indicates opacity.

[0106] Under normal circumstances, The matrix is ​​non-diagonal, and the ellipsoidal shape exhibits anisotropy. In this embodiment, to reduce computational complexity, an isotropic approximation is used, i.e., let:

[0107] ;

[0108] Therefore:

[0109] ;

[0110] in, for identity matrix

[0111] This setting causes the 3D Gaussian to degenerate from a general ellipsoid into a sphere, with its isosurfaces uniformly distributed in all directions and possessing complete rotational symmetry. The Gaussian distribution in this case depends only on the Euclidean distance from the point to the center, as shown in the following equation:

[0112] ;

[0113] This greatly simplifies subsequent matrix operations, gradient backpropagation, and covariance updates in the rendering pipeline.

[0114] like Figure 5 As shown, to obtain the coverage radius when a 3D Gaussian projection is applied to a 2D pixel plane, the semi-major axis length needs to be calculated. This application uses an approximate calculation instead of the common high-complexity calculation. When a 3D point is projected onto a 2D plane using the camera projection matrix (Jacobian), the 2D covariance is calculated using the following formula:

[0115] ;

[0116] in, This is the semi-major axis length (i.e., coverage radius) of the 2D Gaussian after projection. The normalized radius of the 3D Gaussian in three-dimensional space. Let x be the x-coordinate of the 3D point on the camera normalized plane. Let y be the 3D point's y-coordinate on the camera's normalized plane. This refers to the camera's focal length.

[0117] Because square root operations are costly to implement in hardware (FPGA / GPU), this application uses Taylor expansion (an approximation over a small range) to eliminate the square root sign. Sometimes, ,in, Substituting the small quantity symbol, we get the following formula:

[0118] ;

[0119] This application simplifies the generation and data transfer of multiple matrices, the multiplication of five 3×3 matrices, two square root operations, and multiple floating-point addition, subtraction, multiplication, and division operations into simple multiplication and addition operations.

[0120] like Figure 4 As shown, in some embodiments, the specific process of the forward rendering pass includes: obtaining projected Gaussian points based on the current optimization state and the three-dimensional Gaussian points; generating key-value pairs for sorting based on the projected Gaussian points and depth information; performing global sorting according to the keys in the key-value pairs, so that the Gaussian points are arranged in order of screen blocks and depth, to determine the index range of ordered Gaussian points in each screen block, and obtaining the sorting result; and performing parallel volume rendering based on the sorting result to generate color rendering results and depth rendering results.

[0121] Parallel volume rendering includes allocating parallel rendering tasks to screen blocks; for each pixel, the color contribution and transparency contribution of the Gaussian points covering the corresponding pixel are arranged in depth order; the depth rendering result adopts the median depth approximation method, which approximates the depth value of the pixel by calculating the weighted median of the depth of the Gaussian points covering the pixel.

[0122] The forward rendering channel supports multi-channel data synchronous processing and is implemented through a multi-channel renderer unit that supports joint input of depth and RGB. In forward rendering, depth image and contour map information are output simultaneously, avoiding multiple calls to the CUDA unit, which would increase communication latency and computation time, and improving the data communication efficiency between the host and the GPU.

[0123] Multi-channel renderer units have a wide range of applications beyond depth and RGB unification. They can also be extended to unified depth-normal or depth-semantic rendering structures, generating multimodal output in a single CUDA call and reducing the number of kernel function calls. Furthermore, multi-channel renderer units can be replaced with CUDA Stream-based parallel rendering structures, achieving similar performance improvements through multi-core concurrent execution.

[0124] Specifically, firstly, the system performs a prefix summation operation on the number of Gaussian points contained in each tile using cub::DeviceScan::InclusiveSum to obtain the offset position of each tile in the global index array, and at the same time obtains the total number of Gaussian points to be rendered.

[0125] cub::DeviceScan::InclusiveSum is a parallel algorithm function in the NVIDIA CUDA tool library CUB, used to quickly calculate the number of Gaussian points accumulated in each screen tile, thereby determining the starting position of each tile's data in the global array. It is a key preparatory step for parallel data reassembly.

[0126] Subsequently, memory space is dynamically allocated based on this quantity to establish a structure for storing key-value pairs. Next, the duplicateWithKeys kernel takes the 2D projection coordinates and depth information of each Gaussian point as input and generates a corresponding key-value pair for each Gaussian point. The key is used to identify the tile to which it belongs and the depth sorting information, and the value is the index of the Gaussian point. This establishes a sortable data structure for subsequent tile rendering. The core task of the duplicateWithKeys kernel is to generate a corresponding key-value pair for each input Gaussian point data. It receives the projected Gaussian point and depth as input and calculates a "key" (for sorting) and a "value" (the index of the Gaussian point) to create a data structure for subsequent parallel sorting.

[0127] Next, by calculating the highest valid bit width of the key, the number of bits required for sorting is determined, and `cub::DeviceRadixSort::SortPairs` is called to sort the key-value pairs based on the key, achieving an ordered arrangement of all Gaussian points by tile and depth. Finally, the system uses the `identifyTileRanges` kernel to mark the start and end index range of each tile in the sorting result, assigning an independent rendering task to each screen tile, enabling subsequent parallel blending and rasterization operations on the GPU.

[0128] The `identifyTileRanges` function is executed on the sorted key-value pair array. Its core logic is to scan the sorted keys and find the positions where the parts representing the tile numbers in the keys have changed. These positions mark the start and end indices of the Gaussian point sequence corresponding to each tile in the global sorted array, thereby defining a clear data processing boundary for each tile.

[0129] Next, the volume rendering module uses the core CUDA kernel function `renderCUDA` for Gaussian volume rendering. This function is responsible for the final rasterization and color blending of the 3D Gaussian points after projection and block sorting on the screen. Its core idea is to use screen tiles as units (corresponding to thread blocks), with each thread corresponding to one pixel. It loads the Gaussian projection parameters within the current tile in batches from shared memory, including 2D position, depth, elliptic coefficient, and transparency, and calculates the coverage of each pixel with the Gaussian projection.

[0130] In the calculation, the pixel distance is attenuated using the quadratic exponential form of the Gaussian distribution, and the contribution of each Gaussian to the pixel color is obtained by combining the opacity of the Gaussian. The rendering process uses a cumulative transparency update method to achieve the blending of the foreground and background layers until the pixel is completely opaque or the tile area is processed.

[0131] In terms of implementation, the core operators, including the distance squared term and the Gaussian exponential decay term, are optimized from full precision to half precision, effectively reducing memory bandwidth and computation latency, making them suitable for hardware accelerators or low-power devices. Furthermore, the depth output uses the median depth to approximate the mean depth, simplifying the accumulation calculation.

[0132] The frustum detection, Gaussian ellipsoid radius calculation, depth sorting, and volume rendering modules in camera tracking are implemented in C++ and CUDA, and compiled and bound to the main program via Pybind11 to form a directly callable rendering module (Renderer). Traditional Renderer units only support RGB color channel data and cannot process depth channel information simultaneously. Therefore, the main program needs to call the Renderer unit twice to achieve independent rendering of RGB and depth maps, and perform multiple data transfers between the host and GPU.

[0133] To address the aforementioned efficiency issues, this application fuses and encodes depth and color images, enabling joint rendering and result return in a single GPU call. This effectively reduces the number of CUDA kernel function calls, significantly lowers data transfer overhead between the host and device, and thus improves overall rendering efficiency and system resource utilization.

[0134] like Figure 4 As shown, the L1 loss function is then calculated. This function calculates the L1 loss (i.e., absolute error loss) for the entire system. Its main function is to measure the difference between the rendering result of the current frame and the actual observation data, so as to guide the optimization and updating of Gaussian parameters and camera pose.

[0135] Specifically, the function internally employs a multi-stage control strategy to dynamically schedule the rendering mode and loss weights. In some embodiments, the stages include an early iteration stage and a late iteration stage. In the early iteration stage, the depth loss function between the depth rendering result and the real depth image is calculated, and the iteration count is less than or equal to a preset threshold. In the late iteration stage, both the depth loss function and the color loss function between the color rendering result and the real color image are calculated simultaneously, and the iteration count is greater than the preset threshold. Based on the iteration count, the early iteration stage and the late iteration stage are switched, and then optimized camera pose parameters are output based on the iteration mechanism.

[0136] The segmented strategy dynamically determines the system's computation mode at different stages by using iteration counts and stage markers: in the early iteration stage (e.g., iteration number ≤ 20), only depth domain L1 loss is enabled for fast convergence of pose estimation; in the later iteration stage (e.g., iteration number > 20), both depth domain and RGB domain L1 loss are enabled to refine color consistency and improve rendering quality; thus significantly reducing GPU computational overhead while ensuring the optimization accuracy of key parameters.

[0137] In some embodiments, in the phased switching strategy, when calculating the depth loss function and the color loss function, a gradient-aware mask strategy is applied to determine key pixel regions, including: extracting edge gradient magnitude maps of the real depth image and the real color image based on the image gradient operator; selecting high gradient regions according to the gradient magnitude distribution quantiles to form an initial mask; performing random sampling on the initial mask to form a hybrid mask; and using the hybrid mask to filter key pixel regions to calculate the depth loss function and the color loss function.

[0138] The masking strategy is used to adaptively select key pixel regions for loss calculation when calculating depth and RGB loss. Specifically, the masking strategy first uses the Sobel operator to extract edge gradient magnitude maps from the depth map and RGB grayscale map respectively. Then, it selects significant edge regions according to the gradient distribution quantile (such as the top 30% or 70% quantile) to obtain the initial mask. To prevent the mask from being too sparse and causing training instability, a portion of non-edge pixels are randomly added according to the target sampling ratio (such as 40%). Random noise is used to achieve the balance of global uniform sampling, and finally a mixed mask mixed_mask is formed.

[0139] This mask is used in both depth and RGB error calculations, allowing the model to focus on geometric changes in high texture gradient regions. This avoids interference from flat regions in the optimization during the early convergence phase, reduces computation on unimportant parts of the result, and thus improves reconstruction accuracy and convergence speed.

[0140] After the loss function is calculated, the system iteratively updates the relevant parameters using the backpropagation mechanism. First, the calculated total loss function is input into the backpropagation module, which calls the automatic differentiation interface `.backward()` to perform the gradient backpropagation operation. This operation automatically calculates the gradient values ​​of each learnable parameter (including 3D Gaussian parameters and camera pose) according to the chain rule. Subsequently, the optimization module calls the parameter update function to execute the gradient descent step `optimizer.step()`. The optimizer is an adaptive optimizer based on the Adam algorithm, which internally updates the parameters with weights according to first-order and second-order momentum estimates, thereby achieving fast and stable convergence. After the update is complete, the system automatically clears the gradient cache to prepare for the next iteration.

[0141] To reduce the waste of computational resources caused by invalid iterations, this application adaptively tracks the decreasing trend of the loss function through a dynamic iteration strategy to determine the number of iterations required for training. In some embodiments, the iteration mechanism includes dynamic early stopping control. During the iteration process, the decreasing trend of the loss function is monitored to calculate the relative decrease ratio between two adjacent iterations. When the relative decrease ratio is lower than a preset threshold multiple times consecutively, an early stopping condition is triggered. The iteration process is terminated according to the early stopping condition to output the currently optimized camera pose parameters.

[0142] Specifically, in the tracking optimization phase of each frame, this strategy dynamically adjusts the number of optimization iterations by monitoring the decrease of the loss function in real time, thereby significantly improving computational efficiency while ensuring accuracy.

[0143] Existing camera tracking iteration counts are typically set to 200, but in actual training, far more training iterations are unnecessary. The early stopping condition in this application is that the number of iterations exceeds the warm-up value, and the loss function decreases by less than a set relative threshold multiple times consecutively.

[0144] The initial few iterations (e.g., 30) serve as a warm-up phase, ensuring the loss function has sufficient time to decrease. Afterward, the system begins calculating the relative decrease ratio between adjacent iterations. When the loss decrease ratio is less than a preset threshold (e.g., 5%), the system considers it a weak decrease and accumulates the count. If the accumulated count exceeds a set patience value (e.g., 6 times), the iteration process is automatically terminated early, achieving adaptive early stopping.

[0145] In addition, this application introduces a dynamic extension mechanism, which automatically triggers an extended iteration cycle when the depth loss is detected to be higher than a set threshold, doubling the number of iterations and continuing to optimize, in order to prevent attitude estimation deviations caused by accidental stopping.

[0146] By combining dynamic early stopping with extension, the number of iterations can be automatically adjusted according to the real-time optimization convergence, avoiding the problems of overcomputation or premature convergence that exist in the traditional fixed iteration method.

[0147] Tests on specific experimental datasets show that this method can reduce iteration time by more than 50% without reducing tracking accuracy, significantly improving overall real-time performance and energy efficiency.

[0148] S400: Based on optimized camera pose parameters, a hierarchical filtering mechanism is executed to determine keyframes from color and depth images. The hierarchical filtering mechanism includes at least two stages.

[0149] Keyframes are one or more representative frames selected from the entire image sequence for scene model optimization and expansion. Keyframes typically meet conditions such as sufficient visual overlap with other frames and high image quality. Using keyframes instead of global optimization for every frame can reduce computation and prevent the model from overfitting to noisy frames.

[0150] Since SLAM systems typically require hundreds or even thousands of images as input, even if only one keyframe is selected every 3 or 5 frames, the system still needs to maintain a very large list of keyframes in the later stages. At this point, each new keyframe must be compared with every frame in this huge list for overlap, resulting in a significant computational burden. Therefore, based on the existing keyframe selection algorithm based on depth overlap rate, a hierarchical filtering mechanism is implemented. This mechanism is a method for selecting representative keyframes from a large number of image frames. It includes multiple filtering stages; in some embodiments, the hierarchical filtering mechanism includes at least a coarse filtering stage and a fine filtering stage.

[0151] like Figure 6 As shown, in this embodiment, the number of existing keyframes is obtained; if the number of existing keyframes is greater than the number threshold and the target selection ratio is lower than the ratio threshold, a hierarchical screening mechanism is executed. The hierarchical screening mechanism includes: in the coarse screening stage, sparse sampling is performed on the depth image of the current frame and back-projected into a three-dimensional point cloud to calculate the overlap with the candidate keyframes; highly overlapping candidate frames are selected based on the overlap; in the fine screening stage, pixel sampling is performed on the highly overlapping candidate frames to calculate the overlap; and keyframes are selected based on the overlap.

[0152] In the coarse screening stage, the system uses only one-quarter (400 pixels) of the original algorithm's sampling points for rapid estimation. It sparsely samples the depth map of the current frame and back-projects it into a 3D point cloud, projecting this cloud onto the camera coordinate system of the candidate keyframes. The overlap between the keyframe and the current frame is calculated based on the proportion of projected points within the field of view. After sorting by overlap, the algorithm retains only highly overlapping candidate frames (three times the original number of target frames) as the candidate keyframe set. This process quickly eliminates most irrelevant frames with low computational cost, achieving approximately 75% computational pruning.

[0153] During the fine screening stage, the system performs complete overlap calculation and sorting on candidate frames in the coarse screening results, and uses complete pixel sampling to restore high-precision evaluation. If the number of candidate frames has met the target number k, the fine screening stage is skipped and the coarse screening results are returned directly, thereby significantly reducing computational latency in large-scale datasets or low selection ratio scenarios.

[0154] It is understood that in this embodiment, when the number of existing keyframes is greater than the number threshold and the target selection ratio is lower than the ratio threshold, the coarse screening stage is performed first, followed by the fine screening stage. When the number of existing keyframes is less than or equal to the number threshold and the target selection ratio is greater than or equal to the ratio threshold, only the fine screening stage is performed.

[0155] The computational logic of the entire process includes operations such as depth validity screening, 3D backprojection, coordinate transformation, projection onto a 2D plane, and boundary filtering. The layered screening strategy in this application is only activated when the total number of keyframes is large (e.g., more than 50 frames) and the target selection ratio is low (e.g., less than 30%). It can achieve a 2–4 ​​times speedup and a significant reduction in memory usage while maintaining the same screening accuracy as the original algorithm. This mechanism is compatible with FP16 or fixed-point mixed-precision matrix operation modules, possesses good hardware acceleration adaptability, and provides an efficient keyframe input set for subsequent Gaussian rendering and incremental modeling.

[0156] S500: Utilizes keyframes and optimized camera pose parameters to perform rendering and mapping to update the 3D Gaussian point cloud.

[0157] The core of the mapping process is updating and maintaining the parameters of the 3D Gaussian point cloud to more accurately reflect the real scene. During this process, the system fixes the camera pose of the keyframes and focuses on optimizing the representation of the scene itself, that is, adjusting the position, shape, color and other attributes of the 3D Gaussian points so that the virtual image rendered from the keyframe perspective is as consistent as possible with the real keyframe image.

[0158] In some embodiments, a frame is sampled from the selected keyframes, and the optimized camera pose parameters corresponding to the keyframes are obtained; based on the optimized camera pose parameters, a reconstruction loss function is calculated, which includes a depth consistency term and a color reprojection error term; according to the gradient magnitude, a pruning operation and a densification operation are performed on the 3D Gaussian point cloud to update the 3D Gaussian point cloud, which also includes a 3D Gaussian point cloud obtained based on the difference between the depth rendering result of the current frame and the real depth image.

[0159] The system uses a 3D mapping method that combines keyframe random sampling with gradient optimization to achieve adaptive updating of Gaussian parameters for continuous frames in dynamic scenes.

[0160] In this stage, the camera pose parameters of the current frame are fixed, and a fixed number of iterations (e.g., 30 times) is used as the frame mapping optimization cycle. In each iteration, a frame or the current time image is randomly selected from the input set of current keyframes as the optimization input to enhance the global consistency and temporal robustness of the model.

[0161] In each iteration, the data structure corresponding to the current frame is first constructed, including the color image, depth map, camera intrinsic and extrinsic parameters, and historical pose sequence. Based on this, the aforementioned loss calculation function is called to obtain the reconstruction loss value of the current frame. The loss function consists of a depth consistency term, a color reprojection error term, and an optional contour constraint term. The gradients of each Gaussian parameter (including position, variance, transparency, etc.) are calculated through the aforementioned backpropagation.

[0162] In the gradient-free region, a Gaussian sieving and densification mechanism (Prune, Densify) based on gradient magnitude is further executed. Gaussians with small contributions or high gradient regions are dynamically reduced or copied. This can effectively remove inefficient Gaussians with low weights affecting the final graph construction, and copy important Gaussians with greater influence to participate in subsequent optimization. Then, an optimizer update is performed to complete the iterative optimization of parameters and gradient clearing, maintaining the compactness and local expressiveness of the model.

[0163] To ensure the continuity and numerical stability of the optimization process, the aforementioned rendering module is invoked in the current frame to obtain depth and contour information, which is then compared with the actual depth map. When the rendered contour value is lower than a set threshold, the region is determined to be non-existent. Simultaneously, when the rendered depth is significantly greater than the actual depth and the error exceeds a multiple of the median, the region is considered to contain unmodeled nearby objects. The two detection results are logically ORed to form a non-existence mask, which is used to identify regions where Gaussian parameters need to be added. Subsequently, the system extracts effective pixels from the color image and depth map based on this mask, and backprojects them into a new 3D point cloud using camera intrinsic and extrinsic parameters. Then, new Gaussian parameters (including position, variance, color, transparency, etc.) are initialized based on the spatial distribution of the point cloud, and this new Gaussian parameter is concatenated with the original Gaussian set. The model is updated in real time, and gradient accumulation and time index variables are reset simultaneously during this process to ensure the continuity and numerical stability of the optimization process.

[0164] When the entire 3D Gaussian SLAM system starts up, it first performs initialization, and then inputs color-depth image frames in sequence. For each frame, it executes the S300-S500 part, going through the preprocessing, camera tracking, adding new Gaussian, key frame selection and mapping stages, and gradually generating a scene map based on 3D Gaussian.

[0165] Through systematic optimization of the above steps, this method achieves adaptive processing capabilities for different scenarios, maintains a high degree of modularity in its algorithm framework, and facilitates deployment on various hardware architectures. This method not only improves computational throughput but also reduces GPU memory consumption, providing strong support for efficient point cloud processing in subsequent fields such as autonomous navigation, augmented reality, and robot perception.

[0166] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method for SLAM acceleration based on three-dimensional Gaussian rendering, characterized in that, The method comprises the following steps: acquiring input data, the input data comprising color images and depth images; initializing a three-dimensional Gaussian point cloud and camera pose parameters based on a first frame of the input data; performing camera tracking on target frames in the input data, outputting optimized camera pose parameters based on a staged switching strategy and an iterative mechanism, wherein the target frames are frames other than the first frame in the input data, the camera tracking uses a reduced-precision floating-point format, and the reduced-precision floating-point format adopts an FP16 format; performing a hierarchical screening mechanism based on the optimized camera pose parameters to determine key frames from the color images and the depth images, the hierarchical screening mechanism comprising at least two stages, and the hierarchical screening mechanism comprising at least a coarse screening stage and a fine screening stage; performing rendering and mapping using the key frames and the optimized camera pose parameters to update the three-dimensional Gaussian point cloud.

2. The method of claim 1, wherein, The step of initializing the three-dimensional Gaussian point cloud and the camera pose parameters based on the first frame of the input data comprises the following steps: performing data preprocessing on a color image of the first frame to output a preprocessed color image, the data preprocessing comprising a color normalization operation; performing data preprocessing on a depth image of the first frame to output a preprocessed depth image, the data preprocessing comprising a depth dimension rearrangement operation; back-projecting the preprocessed depth image based on an intrinsic parameter matrix and an extrinsic parameter matrix of a camera to generate spatial position parameters of a three-dimensional point cloud; acquiring a color vector of a pixel point corresponding to the spatial position parameters from the preprocessed color image; calculating an average squared distance of adjacent points in the three-dimensional point cloud using a reduced-precision floating-point format, and calculating a scene radius according to a preset maximum depth value and a scene scale parameter; initializing position parameters, color parameters, transparency parameters and scale parameters in the three-dimensional Gaussian point cloud according to the spatial position parameters, the color vector, the average squared distance and the scene radius. 3.The method of claim 1, wherein, The step of performing camera tracking on the target frames in the input data comprises the following steps: acquiring optimized camera pose parameters of the previous two frames; calculating a rotation initial estimate and a translation initial estimate of a current target frame by a first-order extrapolation method according to the optimized camera pose parameters of the previous two frames; performing unitization processing on the rotation initial estimate to obtain a rotation initial value conforming to a rotation constraint; combining the rotation initial value and the translation initial estimate to obtain a camera pose initial estimate of the current target frame; performing optimization iteration starting from the camera pose initial estimate to output the optimized camera pose parameters of the current target frame.

4. The method of claim 3, wherein, The step of performing optimization iteration starting from the camera pose initial estimate to output the optimized camera pose parameters of the current target frame comprises the following steps: taking the camera pose initial estimate as a current optimization state; synchronously generating a color rendering result and a depth rendering result corresponding to the target frame through a forward rendering channel based on the current optimization state, wherein the target frame is a three-dimensional Gaussian point located in a view frustum. In the forward rendering pass, matrix calculations are performed using reduced-precision floating-point formats, including covariance matrix operations and projection transformation operations, the covariance matrix operations including semi-major axis calculations for 3D Gaussian projection onto a 2D plane, the semi-major axis calculations employing Taylor series-based approximation calculations; calculating a loss function between the color rendering result and the depth rendering result and the real observation image; updating the current optimization state based on the loss function and combining a phased switching strategy, and outputting the optimized camera pose parameters after iteration.

5. The method of claim 4, wherein, The phases of the phased switching strategy include an early iteration phase and a late iteration phase. The output of the optimized camera pose parameters based on the phased switching strategy and the iteration mechanism includes: In the early iteration phase, a depth loss function between the depth rendering result and the real depth image is calculated, and in the early iteration phase, the iteration count is less than or equal to a preset threshold; In the late iteration phase, the depth loss function and a color loss function between the color rendering result and the real color image are calculated simultaneously, and in the late iteration phase, the iteration count is greater than the preset threshold; According to the iteration count, the early iteration phase and the late iteration phase are switched, and the optimized camera pose parameters are output based on the iteration mechanism.

6. The method of claim 4, wherein, The forward rendering pass supports multi-pass data synchronization processing; The color rendering result and the depth rendering result corresponding to the target frame are generated synchronously through the forward rendering pass, including: Based on the current optimization state and the 3D Gaussian point, a projected Gaussian point is obtained; Based on the projected Gaussian point and the depth information, a key-value pair for sorting is generated; According to the key in the key-value pair, a global sorting is performed to arrange the Gaussian points in a screen block and a depth order to determine the index range of the ordered Gaussian points in each screen block, and a sorting result is obtained; Based on the sorting result, a parallel volume rendering is performed to synchronously generate the color rendering result and the depth rendering result; the parallel volume rendering includes assigning parallel rendering tasks to the screen blocks; for each pixel, the color degree contribution and the transparency contribution of the Gaussian point covering the corresponding pixel are mixed and overlaid in the depth order; wherein the depth rendering result uses a median depth approximation method.

7. The method of claim 5, wherein the method further comprises: In the phased switching strategy, when calculating the depth loss function and the color loss function, a gradient-aware mask strategy is applied to determine a key pixel region, including: Based on an image gradient operator, edge gradient amplitude maps of the real depth image and the real color image are extracted; According to the gradient amplitude distribution quantile, a high gradient region is selected to form an initial mask; Random oversampling is performed on the initial mask to form a hybrid mask; The hybrid mask is used to screen the key pixel region to calculate the depth loss function and the color loss function.

8. The method of claim 1, wherein the method further comprises: The iteration mechanism includes dynamic early stopping control: The output of the optimized camera pose parameters based on the phased switching strategy and the iteration mechanism includes: During the iteration process, the relative decline ratio of adjacent two iterations is calculated by monitoring the downward trend of the loss function; trigger an early stop condition when the relative drop ratio is below a preset threshold for a plurality of times in succession; terminate the iteration process according to the early stop condition to output the current optimized camera pose parameter.

9. The method of claim 1, wherein, The keyframes are one or more, including: The hierarchical screening mechanism is executed based on the optimized camera pose parameter to determine the keyframes from the color images and the depth images, including: obtaining a number of keyframes; In the case that the number of keyframes is greater than a threshold and a target selection ratio is lower than a threshold, the hierarchical screening mechanism is executed, including: In the coarse screening stage, sparse sampling is performed on the depth image of the current frame, and back projection is performed to a three-dimensional point cloud to calculate an overlap degree with a candidate keyframe; high-overlap candidate frames are screened based on the overlap degree; In the fine screening stage, pixel sampling is performed on the high-overlap candidate frames to calculate an overlap degree; keyframes are selected based on the sorting of the overlap degrees.

10. The method of claim 1, wherein, The rendering and mapping are performed using the keyframes and the optimized camera pose parameter to update the three-dimensional Gaussian point cloud, including: a frame is sampled from the selected keyframes, and an optimized camera pose parameter corresponding to the keyframes is obtained; a reconstruction loss function is calculated based on the optimized camera pose parameter, including a depth consistency term and a color back projection error term; a pruning operation and a densification operation are performed on the three-dimensional Gaussian point cloud according to the gradient amplitude to update the three-dimensional Gaussian point cloud, and the three-dimensional Gaussian point cloud further includes a three-dimensional Gaussian point cloud obtained based on a difference between a depth rendering result of the current frame and a real depth image.

Images