SLAM acceleration method based on three-dimensional Gaussian rendering
By introducing a phased switching strategy, reducing the precision of the floating-point format, and employing a hierarchical filtering mechanism into the 3D Gaussian SLAM method, the keyframe selection and loss function calculation are optimized, solving the problem of low computational efficiency in existing technologies and achieving a more efficient SLAM process.
Patent Information
- Application Number
- CN202511881521.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-01-09
- Estimated Expiration
- 2045-12-15
AI Technical Summary
Existing 3D Gaussian SLAM methods suffer from massive computational demands for rendering and point cloud updates, low efficiency in matrix operations, insufficient GPU parallel utilization, frequent repetitive calculations, high memory consumption due to full-precision floating-point calculations, redundancy in keyframe selection, redundancy in loss iteration, redundancy in CUDA kernel calls, and redundancy in loss function calculations, thus limiting the real-time application of the system.
A phased switching strategy and iterative mechanism are adopted. Camera tracking is performed using a reduced-precision floating-point format. A hierarchical filtering mechanism is executed to determine keyframes. The loss function calculation strategy is optimized by combining the hierarchical filtering mechanism and low-cost operator approximation calculation, reducing the number of CUDA core accesses, introducing an adaptive iteration termination mechanism, and unifying the RGB and depth rendering channels.
While maintaining the accuracy of map reconstruction and pose estimation, it significantly improves the efficiency of the SLAM process, reduces memory usage and computational burden, increases GPU utilization, reduces latency, and achieves more efficient real-time performance.
Smart Images

Figure CN121304949A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision and navigation, and particularly relates to a SLAM acceleration method based on three-dimensional Gaussian rendering. BACKGROUND
[0002] Three-dimensional point cloud processing technology combined with high-performance GPU computing promotes the development of SLAM (Simultaneous Localization and Mapping, three-dimensional reconstruction and positioning technology) algorithm. SLAM is used for real-time positioning and map construction, and is applied in dense scene reconstruction, dynamic environment perception and robot autonomous navigation. The three-dimensional Gaussian distribution representation method can accurately model the point cloud distribution and provide continuous scene expression. User requirements include high-precision map construction, reliable pose estimation and high-quality scene reconstruction, and at the same time, efficient calculation and low-latency response are required, which is suitable for fields such as robots, augmented reality and virtual reality.
[0003] Part of the technology uses three-dimensional Gaussian point cloud as a SLAM carrier to replace the traditional geometric method or neural radiation field. The three-dimensional Gaussian SLAM generates point cloud by back-projection of RGB-D data, and uses differentiable rendering to optimize scene parameters and camera trajectory. This method can express the scene densely, improve the accuracy of the map and the reliability of the pose estimation, and includes camera pose optimization and three-dimensional rendering pipeline. The optimization process relies on full-precision floating-point calculation and fixed iteration strategy.
[0004] However, by the above-mentioned method, the rendering and point cloud update calculation amount is huge, the matrix operation efficiency is low, the GPU parallel utilization is insufficient, the repeated calculation is frequent, the full-precision floating-point calculation leads to high memory occupation and large bandwidth pressure, the key frame selection is redundant, the loss iteration is fixed, and the CUDA core call is separated, which limits the real-time application of the system. That is to say, by the above-mentioned method, the efficiency of the SLAM process is low under the premise that the map reconstruction accuracy and the pose estimation accuracy are basically unchanged. SUMMARY
[0005] The present application provides a SLAM acceleration method based on three-dimensional Gaussian rendering to solve the problem of low efficiency of the SLAM process under the premise that the map reconstruction accuracy and the pose estimation accuracy are basically unchanged.
[0006] The present application provides a SLAM acceleration method based on three-dimensional Gaussian rendering, comprising: obtaining input data, the input data comprising color images and depth images; initializing three-dimensional Gaussian point cloud and camera pose parameters based on the first frame of the input data; For a target frame in the input data, camera tracking is performed, and an optimized camera pose parameter is output based on a staged switching strategy and an iteration mechanism, wherein the target frame is a frame other than the first frame in the input data, and the camera tracking utilizes a reduced-precision floating-point format; Based on the optimized camera pose parameter, a hierarchical screening mechanism is performed to determine a key frame from the color image and the depth image, and the hierarchical screening mechanism includes at least two stages; Rendering and mapping are performed using the key frame and the optimized camera pose parameter to update the three-dimensional Gaussian point cloud.
[0007] According to the above technical solution, the application provides a SLAM acceleration method based on three-dimensional Gaussian rendering, which includes: obtaining input data, the input data including a color image and a depth image, and based on a first frame of the input data, initializing a three-dimensional Gaussian point cloud and a camera pose parameter; for a target frame in the input data, camera tracking is performed, and an optimized camera pose parameter is output based on a staged switching strategy and an iteration mechanism, wherein the target frame is a frame other than the first frame in the input data, and the camera tracking utilizes a reduced-precision floating-point format; based on the optimized camera pose parameter, a hierarchical screening mechanism is performed to determine a key frame from the color image and the depth image, and the hierarchical screening mechanism includes at least two stages; and mapping is performed using the key frame and the optimized camera pose parameter to update the three-dimensional Gaussian point cloud. The method integrates the staged switching strategy and the iteration mechanism into the camera tracking process, so that the pose optimization can quickly converge and avoid consuming computing resources on secondary targets too early. In the camera tracking process, the reduced-precision floating-point format is used to reduce the numerical processing burden and the memory subsystem bandwidth pressure. The hierarchical screening mechanism can identify key frames at a lower computational cost than frame-by-frame fine comparison. This solves the problem of low efficiency of the SLAM process while ensuring that the map reconstruction accuracy and the pose estimation accuracy remain basically unchanged. BRIEF DESCRIPTION OF DRAWINGS
[0008] In order to more clearly illustrate the technical solutions of the application, the following will briefly introduce the drawings needed in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.
[0009] Figure 1 A flowchart of the SLAM acceleration method based on three-dimensional Gaussian rendering provided by the embodiments of the application is shown. Figure 2 A first frame initialization process diagram provided by the embodiments of the application is shown. Figure 3 A preprocessing flowchart provided by the embodiments of the application is shown. Figure 4 A loss function calculation schematic diagram provided for an embodiment of the present application is shown in FIG. 3. Figure 5 A simplified schematic diagram for calculating the semi-major axis provided for an embodiment of the present application is shown in FIG. 4. Figure 6 A key frame selection process schematic diagram provided for an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION
[0010] The embodiments will be described in detail below with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following embodiments are not meant to represent all implementations consistent with the present application.
[0011] With the rapid development of computer vision, robot autonomous navigation, virtual reality (VR), augmented reality (AR), and other fields, SLAM has become an important foundation for intelligent perception systems. Traditional SLAM techniques are mostly based on sparse feature points or dense pixel information to achieve environment modeling and camera pose estimation, but in complex lighting, dynamic scenes, and large-scale environments, the reconstruction quality and real-time performance are often limited.
[0012] With the development of GPU computing power and three-dimensional sensing devices (such as depth cameras and laser radars), continuous scene modeling methods based on point clouds have been explored. Among them, point cloud representation based on three-dimensional Gaussian distribution (3D Gaussian Representation) gradually becomes an important research direction in the field of three-dimensional vision because it can encode both geometric structure and color distribution information while achieving a good balance between storage and rendering efficiency.
[0013] This kind of method takes a continuously distributed three-dimensional Gaussian point cloud as the core expression form of the scene, each point carries position, covariance matrix, color, and opacity attributes, and can realize continuous modeling and differentiable rendering of the real scene while maintaining low storage overhead. In this kind of system, the core process usually includes a camera pose optimization process and a three-dimensional rendering pipeline, which calculates the difference between the synthesized image obtained by rendering and the real observed image to inversely optimize the scene parameters and camera trajectory.
[0014] Specifically, the workflow of the three-dimensional Gaussian SLAM system is as follows: first, the system receives a photo containing three-channel color data and one-channel depth data taken by a color-depth camera (RGB-Depth Camera, RGB-D) in a time sequence frame by frame in series, and saves it as a true value; second, the 2D photo is back-projected to the 3D camera space to generate a 3D Gaussian point in space for each pixel; next, using the estimated camera pose and the existing 3D Gaussian point cloud, the three-dimensional Gaussian points are projected to the two-dimensional image plane according to the camera's internal and external parameters, and the distribution range and depth of the points in the pixel coordinate system are calculated; then, the elliptical coverage area of the points in the screen space is determined according to the covariance matrix of the points, and all Gaussian points are rendered after depth sorting and weighted rendering combined with color and opacity parameters, thereby generating a rendered image corresponding to the camera view. The rendering process can be performed in a highly parallel manner on a GPU. Subsequently, the gradient of the Gaussian point parameters is back-propagated, enabling the system to use photometric error to achieve end-to-end joint optimization.
[0015] Compared with the traditional reconstruction method based on voxel grid or triangular mesh, the three-dimensional Gaussian-based SLAM technology has higher representation flexibility and continuity, and can capture details and surface transition features without explicit topological structure, thereby significantly improving the calculation efficiency while ensuring the reconstruction quality.
[0016] In some schemes, such as Splatam, a single non-positioning monocular RGB-D camera is used to realize dense SLAM through a 3D Gaussian carrier. However, the existing method has the following disadvantages: 1. Redundant key frame selection strategy. In the 3D Gaussian map updating stage, considering that the calculation amount cannot be too large, the SLAM needs to select some key frames as the reference to render and optimize the parameters of the updated 3D Gaussian map according to the estimated camera pose so far. Splatam saves every nth frame into the candidate key frame list, and selects k frames as key frames for optimization, including the current frame, the most recent key frame, and k-2 previous key frames.
[0017] The key frame selection module usually judges according to the geometric overlap degree between images. The basic process is as follows: first, a certain number of valid pixel points are randomly sampled in the current frame, and these pixels are back-projected into a three-dimensional point cloud using the camera internal and external parameters; then, the point cloud is transformed into the camera coordinate system of all candidate key frames in the candidate key frame list in turn, and is projected onto the frame plane again, and the proportion of the projected points falling within the range of each key frame image is calculated as the frame overlap index; finally, the top k key frames are selected according to the descending order of the overlap degree, as the optimization input of the current frame.
[0018] As the system runs for a longer time and the number of keyframes grows, the original algorithm gradually exposes performance bottlenecks and limitations in many aspects. First, the method performs complete three-dimensional point transformation and projection operations on all keyframes one by one, and the computational complexity increases linearly with the number of keyframes, which will significantly slow down the system running speed in the scale of dozens or even hundreds of keyframes. Since each keyframe needs to perform independent matrix multiplication, projection, boundary judgment and statistical operation, the utilization rate of GPU parallelization is low, which leads to the keyframe selection stage becoming one of the performance bottlenecks of the real-time system.
[0019] Secondly, the algorithm lacks hierarchical or coarse screening mechanism and does not distinguish between significantly overlapping frames and obviously irrelevant frames, resulting in a large amount of redundant calculation overhead. For example, in the case of small scene changes or dense keyframe distribution, there is almost no overlap between some keyframes, but they are still fully involved in the calculation, wasting computing resources. In addition, in the case of a large number of keyframes and only a small number of candidate frames are needed, the original algorithm still performs full calculation, and cannot dynamically reduce the candidate range according to the target selection ratio, resulting in a mismatch between calculation efficiency and task demand.
[0020] 2. Data precision redundancy. In some schemes, most of the matrix and tensor operations in the rendering and optimization stages use full-precision (FP32) floating-point representation, including 2D and 3D covariance matrix calculation of three-dimensional Gaussian points, homogeneous transformation of point cloud coordinates, and 2D projection and depth weighted fusion in the rendering pipeline. The design guarantees high numerical precision and reconstruction stability, but at the same time brings significant calculation and storage overhead. Since the FP32 operation unit has relatively low throughput on the GPU, its operation delay becomes one of the main bottlenecks of the entire rendering and optimization process, especially in the case of millions of Gaussian points per frame, the calculation amount of matrix multiplication and covariance update is huge, which easily leads to high GPU computing occupation and increases the inter-frame delay. In addition, the data volume of full-precision floating-point format is large, which leads to a significant increase in memory occupation and memory bandwidth consumption, which is not friendly to power-sensitive and hardware resource-sensitive edge devices. In high-resolution input or large-scale scenes, intermediate tensors such as covariance matrices, projection matrices and gradient caches are simultaneously resident in the memory, which can cause memory tension and even overflow, limiting the scalability of the system in more complex scenes.
[0021] On the other hand, existing methods do not fully utilize the mixed-precision computing characteristics of GPUs. Although modern GPUs have widely supported half-precision (FP16) computing and have dedicated acceleration units such as Tensor Cores, all matrix and tensor operations in the original algorithm are still performed at FP32 precision. The precision is not managed hierarchically according to the computing requirements of different stages, resulting in waste of computing resources. In the backpropagation stage, both gradient calculation and parameter update use full-precision data, which leads to longer operator execution time and uneven load of GPU computing units in multi-thread concurrency, further reducing hardware utilization.
[0022] 3. Loss iteration redundancy. In existing 3D Gaussian point cloud-based SLAM systems (such as SplaTAM), camera pose estimation usually uses an iterative optimization strategy based on photometric error, i.e., updating camera pose and scene parameters by minimizing the difference between rendered images and real observed images. This process generally uses gradient descent or adaptive optimization algorithms such as Adam, and performs a fixed number of iterations (usually around 200 iterations) in each frame tracking stage to obtain stable pose estimation results.
[0023] However, this fixed-step optimization design has obvious computational redundancy and efficiency problems in actual operation. Due to the large differences in image overlap, lighting consistency, and depth quality between different frames, the optimization convergence speed is not constant; for most frames, the loss function has stabilized after a few dozen iterations, but the system still forces the full 200 iterations of backpropagation and parameter update, resulting in a large amount of invalid computation. Especially in the backpropagation process executed on the GPU, the gradient needs to be recalculated, the rendering kernel function needs to be called, and the gradient needs to be backpropagated, which has high time and memory overhead, making the optimization stage one of the main sources of overall system delay.
[0024] In addition, existing methods lack an adaptive control mechanism based on convergence indicators. During iteration, even if the loss function has a very low descent rate or oscillates, the algorithm still continues to perform complete iterations, which cannot automatically terminate early, thus wasting GPU computing resources. This strategy is particularly inefficient in long-term, multi-frame cumulative optimization scenarios. For example, in stable tracking or small scene change scenarios, pose optimization of consecutive multiple frames may be trapped in repeated computation, significantly reducing the overall real-time performance and energy efficiency ratio of the system. On the other hand, this fixed iteration optimization is not conducive to hardware resource scheduling, and cannot flexibly adjust the iteration depth according to the dynamic changes in computing load, causing fluctuations in GPU core utilization, and even affecting the efficiency of parallel task scheduling.
[0025] 4、CUDA kernel invocation redundancy. In existing 3D Gaussian point cloud-based SLAM systems, rendering and optimization are usually based only on three-channel color images (RGB) for photometric error constraints, while depth information (Depth) is handled separately as an independent module, mainly for point cloud back-projection and generating depth silhouette map (Silhouette Map) constraints. This separate structure is logically clear in design, but in actual operation, it causes data flow fragmentation and repeated calculations.
[0026] Because RGB rendering and depth rendering belong to different computing paths, the system needs to maintain two sets of data buffers and intermediate results, including projected point coordinates, pixel weights, and occlusion masks. This not only increases memory usage and data copy overhead, but also significantly increases communication delay between the rendering module and the optimization module. In GPU implementation, the rendering kernel functions of RGB channels and depth channels cannot share the same scheduling stream and need to be called and synchronized separately, which limits the parallelization efficiency and increases the overall scheduling complexity. The original scheme does not implement multi-channel fusion scheduling in terms of calculation organization, i.e., the rendering pipeline still takes a single image channel as the processing unit. For tasks with both color and geometric constraints, the system must first complete RGB rendering and error calculation, then switch to the depth rendering stage to perform geometric consistency constraints. This serial structure significantly increases the time overhead of the rendering and backpropagation stages, making it impossible to achieve unified modeling and efficient parallel computing of RGB and depth information, and thus limiting the system's performance in high-frame-rate real-time SLAM scenarios.
[0027] 5、Loss function calculation redundancy. The system needs to calculate the loss function of the current frame in the camera tracking stage and the mapping stage. However, some operators (such as two-dimensional / three-dimensional covariance update) have complex calculations such as square root and exponential operations. Although these calculations have limited sensitivity to subsequent loss function training, they are still executed with high-precision standard formulas, resulting in increased energy consumption and decreased performance.
[0028] On the other hand, in existing solutions, the loss function usually adopts a fixed structure of full-image L1 calculation method, that is, a global comparison is made between the rendering result and the real observed image pixel by pixel to minimize the reconstruction error and guide parameter optimization. Such methods usually simultaneously update the camera pose parameters and the three-dimensional Gaussian point cloud in each frame to obtain higher accuracy. In addition, some solutions balance the depth and color errors in different stages through simple loss weight coefficients, but lack a dynamic switching mechanism and still maintain a unified calculation mode throughout the stage. At the same time, existing methods mostly use dense calculation mode with full-pixel participation when calculating the loss, that is, the depth and RGB error are calculated uniformly in the entire image range without distinguishing between key areas such as edges and high textures and flat areas. A few improved solutions attempt to introduce a fixed threshold or random sampling mask mechanism, but often rely on a preset proportion or static mask template, lacking adaptability and gradient awareness.
[0029] To solve the above problems, some embodiments of the present application provide a three-dimensional Gaussian rendering-based SLAM acceleration method. By systematically optimizing the camera tracking, mapping, loss function iteration, and data flow management processes, the method achieves significant acceleration of the overall SLAM process while ensuring the accuracy of map reconstruction and pose estimation.
[0030] By quantizing the matrix calculation format, reducing the number of CUDA kernel accesses, optimizing the key frame selection mechanism, using low-cost operator approximation calculation, optimizing the loss function calculation strategy, and training early stopping, the method not only improves the calculation throughput but also reduces the GPU memory occupancy. At the same time, considering the phased features of the loss function and the calculation resource scheduling strategy, the method realizes adaptive processing capability for different scenes. In addition, the method maintains a high degree of modularity in the algorithm framework, facilitating deployment on various hardware architectures and efficient application in existing 3D Gaussian SLAM systems, further improving the practicality and promotional value of the algorithm. Through the above optimization, the present solution can achieve significant acceleration of the overall SLAM process while ensuring the accuracy of map reconstruction and pose estimation.
[0031] As shown in Figure 1 , the method comprises the following steps: S100: acquiring input data.
[0032] The input data includes color images and depth images. The color images provide visual information of the scene, and the depth images provide geometric structure information of the scene.
[0033] Color images are a sequence of 2D image data containing color information of a scene captured by an imaging device. Color information usually includes three channels of red, green, and blue, which are used to record the appearance texture of a scene. The sequence can be derived from a color sensor in an RGB camera, an RGB-D camera, or pre-processed image data such as interpolated, demosaiced, etc. In some implementations, color images can also be represented in other color spaces.
[0034] Depth images are a sequence of 2D data recording the physical distance from each pixel in a scene to the optical center of a camera, aligned in time with color images. Depth information is usually stored in a single channel matrix. The sequence can be derived from a time-of-flight camera, a structured light depth camera, a binocular stereo vision system, or a monocular depth estimation algorithm based on neural networks.
[0035] S200: Based on the first frame of input data, initialize the 3D Gaussian point cloud and camera pose parameters.
[0036] 3D Gaussian point cloud parameters are a set of optimizable variables used to represent the 3D geometry and appearance of a scene. Each 3D Gaussian point corresponds to a set of parameters that collectively define a point distributed in an ellipsoid shape in 3D space. Position parameters describe the coordinates of the Gaussian point in 3D space, scale parameters define the stretching of the Gaussian point in each direction, determining the shape of the ellipsoid, color parameters define the color presented by the point when viewed from a particular perspective, and transparency parameters define the contribution weight of the point to the final rendered image.
[0037] Camera pose parameters are a set of parameters describing the position and orientation of an imaging device in 3D space. Position is usually represented by a 3D translation vector, and orientation can be represented by a rotation matrix, Euler angles, or a quaternion. Camera pose parameters define the transformation from the world coordinate system to the camera coordinate system, which is the geometric basis for projecting a 3D scene onto a 2D image plane. In the SLAM process, each frame of image corresponds to a camera pose parameter to be estimated or optimized.
[0038] In some embodiments, based on the first frame of the input data, the three-dimensional Gaussian point cloud and the camera pose parameters are initialized, including: performing data preprocessing on the color image of the first frame to output a preprocessed color image, the data preprocessing including color normalization operation; performing data preprocessing on the depth image of the first frame to output a preprocessed depth image, the data preprocessing including depth dimension rearrangement operation; based on the camera intrinsic matrix and the camera extrinsic matrix, the preprocessed depth image is back projected to generate the spatial position parameters of the three-dimensional point cloud; from the preprocessed color image, the color vector of the pixel point corresponding to the spatial position parameter is obtained; the average square distance of adjacent points in the three-dimensional point cloud is calculated using a reduced precision floating point format, and the scene radius is calculated according to the preset maximum depth value and the scene scale parameter; according to the spatial position parameter, the color vector, the average square distance and the scene radius, the position parameter, the color parameter, the transparency parameter and the scale parameter in the three-dimensional Gaussian point cloud are initialized.
[0039] As shown in Figure 2 , specifically, first, data preprocessing is performed on the color image of the first frame to output a preprocessed color image, the data preprocessing including color normalization operation, the pixel value is normalized to 0-1 range, thereby eliminating the influence of imaging device difference on color intensity, wherein the color image size is (480, 640, 3), the depth image size is (480, 640, 1), and the camera parameters include intrinsic matrix (intrinsics, 3x3) and pose matrix (pose, 4x4).
[0040] At the same time, data preprocessing is performed on the depth image of the first frame to output a preprocessed depth image, the data preprocessing including depth dimension rearrangement operation, the depth image is reorganized into a memory layout suitable for GPU parallel calculation. Through this standardized input preprocessing, the comparability and robustness of the initialization results can be maintained under different data sources and camera models.
[0041] After preprocessing, the camera parameter inverse solution module is entered to inverse the input camera extrinsic matrix pose to obtain the world-to-camera matrix w2c (4x4) from the world coordinate to the camera coordinate, and further calculate the spatial position of the camera center in the world coordinate system. By taking the inverse of the first 3 rows and the first 3 columns of w2c, the position vector of the camera optical center in the world coordinate can be obtained, which is used for subsequent perspective projection matrix construction. At the same time, the perspective projection matrix proj (projection matrix, 4x4) is constructed according to the camera intrinsic matrix K and the extrinsic matrix w2c, and its continuous multiplication w2c proj is calculated, which is used to realize the coordinate projection mapping of three-dimensional points to two-dimensional image plane.
[0042] Further ensure the geometric relationship between different coordinate systems is maintained, thereby providing a theoretical basis for the inverse transformation of the depth value to the three-dimensional coordinate, and unifying the input camera geometric information into a reversible linear mapping relationship, so that the system can maintain spatial consistency and traceability in subsequent frames.
[0043] The camera intrinsic matrix contains focal length and principal point coordinate information, and the camera extrinsic matrix describes the position and orientation of the camera in the world coordinate system. The two-dimensional pixel coordinates and the corresponding depth value are converted into three-dimensional world coordinates through back projection transformation. Then the average square distance of adjacent points in the three-dimensional point cloud is calculated using the reduced precision floating point format, and the scene radius is calculated according to the preset maximum depth value and scene scale parameter. In this embodiment, the reduced precision floating point format uses FP16 format instead of FP32 format, which significantly reduces memory occupation and calculation time while ensuring calculation accuracy.
[0044] In the point cloud generation stage, first, the effective pixel screening is performed according to the depth image, and the mask is generated to remove the pixel points with zero or invalid depth, so as to avoid the noise propagation of meaningless points. Subsequently, according to the camera intrinsic matrix , the image plane coordinates and the corresponding depth value are back projected into three-dimensional points in the camera coordinate system , the mathematical expression of which is as follows: ; wherein, is the inverse matrix of the camera intrinsic matrix, which is used to realize the conversion from pixel coordinates to camera coordinates.
[0045] Then, by taking the inverse of matrix, the matrix is obtained, which maps the three-dimensional point cloud in the camera coordinate system to the world coordinate system, and the calculation formula is as follows: ; Further, the spatial transformation of the three-dimensional point from the camera local coordinate system to the global world coordinate system is completed, thereby obtaining the physical position of the initial point cloud in the scene.
[0046] This scheme uses FP16 instead of FP32, which significantly reduces the hardware resource consumption during calculation. While generating the point cloud, the system calculates the average square distance of adjacent points in the point cloud, which is used to measure the density of the point cloud and the spatial scale distribution of the scene, and the calculation expression is as follows: ; wherein, is the three-dimensional coordinates of adjacent points, is the number of effective point pairs.
[0047] In the parameter initialization module, the system calculates the scene radius based on the preset maximum depth value and the set scene scale parameter. The formula is as follows: ; in, This is the maximum depth value. The set scene scale parameters.
[0048] The scene scale parameter is used to adaptively adjust the spatial scale of the Gaussian rendering volume, so that a reasonable Gaussian distribution density can be maintained under different depth ranges or scene sizes.
[0049] Finally, an initial 3D point cloud dataset is generated. The number of points in the point cloud is usually H×W effective pixels. While generating the point cloud, the system calculates the average squared distance between adjacent points in the point cloud to characterize the density of the point cloud and the spatial scale distribution of the scene. This transforms 2D perceptual data into 3D structural data with spatial semantics. The system also achieves automatic estimation of the scene's geometric scale through the distance measurement between points, providing a statistical basis for subsequent Gaussian kernel parameterization.
[0050] After obtaining the initial point cloud, the rendering parameters and system variables are initialized based on the point cloud data, the set total number of frames, the mean squared distance, and the selected Gaussian distribution model. The initialized rendering parameters include the Gaussian distribution center coordinates, i.e., spatial position parameters, color parameters, variance matrix, and transparency parameters; the system variables include dynamically adjustable global variables, such as scene scale parameters and densification thresholds.
[0051] S300: For the target frame in the input data, perform camera tracking and output optimized camera pose parameters based on a phased switching strategy and an iterative mechanism.
[0052] like Figure 3 As shown, when the system starts tracking the image at time t of a new frame, it first needs to read the color image (RGB) and depth image (Depth) of the current frame from the input data, and calculate the world-to-camera transformation matrix based on its true pose. This matrix is obtained by inverting the true pose matrix. Subsequently, the color image and depth image are preprocessed, including channel rearrangement and normalization. This step ensures that the input data meets a uniform tensor format for efficient processing on the GPU.
[0053] The target frame is any frame in the input data other than the first frame. For each frame in the sequence after the first frame, the system performs camera tracking. The goal of camera tracking is to calculate the accurate pose of the camera in the three-dimensional space corresponding to the current frame. The tracking process adopts an iterative optimization mechanism that combines a phased switching strategy. This means that the optimization process is not static, but will intelligently adjust its focus as the iteration progresses.
[0054] The tracing process utilizes a reduced-precision floating-point format for calculations; for example, it uses half-precision floating-point numbers instead of traditional single-precision floating-point numbers. This data format conversion effectively reduces the bandwidth pressure on data transmission between processor memory and computing units, and improves the execution throughput of computing units.
[0055] When time step t > 1, a constant velocity model is introduced as a camera motion prediction mechanism to calculate the initial pose of the current frame based on the poses of the previous two frames, thereby reducing the optimization search space and improving the convergence speed.
[0056] In some embodiments, camera tracking is performed on the target frame in the input data, including: predicting an initial estimate of the camera pose of the current target frame based on the optimized camera pose parameters of the previous frame using a constant velocity model; predicting an initial estimate of the camera pose of the current target frame based on the optimized camera pose parameters of the previous frame using a constant velocity model includes: obtaining the optimized camera pose parameters of the previous two frames, and calculating the initial rotation estimate and initial translation estimate of the current target frame using a first-order extrapolation method based on the optimized camera pose parameters of the previous two frames; performing normalization processing on the initial rotation estimate to obtain an initial rotation value that meets the rotation constraints; combining the initial rotation value and the initial translation estimate to obtain an initial estimate of the camera pose of the current target frame; and performing optimization iterations starting from the initial camera pose estimate to output the optimized camera pose parameters of the current target frame.
[0057] For the initial rotation estimate, let the unnormalized rotation quaternions of the first two frames be respectively... Then the initial value of the rotation of the current frame Calculated using the following formula: ; That is, by using first-order extrapolation, the initial rotation value of the current frame is inferred based on the rotation trend of the previous two frames, and the quaternion is ensured to satisfy the rotation constraint through normalization operation.
[0058] For the initial translation estimation, let the camera translation vectors of the first two frames be... Then the constant-velocity extrapolation model for translation is: ; in, This is the initial estimate of the camera translation vector for frame t.
[0059] This method assumes that the camera's linear velocity remains constant over a short period, effectively reducing jitter or divergence caused by inaccurate initial optimization values. For the first frame of the sequence (t=0) or in the case of no historical information, the pose of the previous frame is directly copied as the initialization value to maintain temporal continuity, as shown in the following equation: ; The initial rotation value describes the predicted camera orientation, while the initial translation estimate describes the predicted camera position. These two pieces of information are encapsulated together into a camera pose representation, for example, combined into a 4×4 homogeneous transformation matrix. The top-left 3×3 submatrix is derived from the initial rotation value, and the rightmost column vector is composed of the initial translation estimate. This combined transformation matrix is the final output initial camera pose estimate, representing the system's best guess of the current camera state based on historical motion, and will serve as the initial input value for subsequent refinement optimization algorithms.
[0060] Starting with the initial camera pose estimation, optimization iterations are performed to output the optimized camera pose parameters for the current target frame. In some embodiments, this includes: using the initial camera pose estimation as the current optimization state; based on the current optimization state, synchronously generating the color rendering result and depth rendering result corresponding to the target frame through the forward rendering channel; calculating the loss function between the color rendering result and the depth rendering result and the actual observed image; updating the current optimization state based on the loss function and in conjunction with a phased switching strategy, and outputting the optimized camera pose parameters after iteration.
[0061] In the forward rendering pass, matrix calculations are performed using a reduced-precision floating-point format. The matrix calculations include covariance matrix operations and projection transformation operations. The covariance matrix operations include the calculation of the semi-major axis of a 3D Gaussian projection onto a 2D plane. The semi-major axis calculation uses an approximation based on Taylor expansion. The target frame is a 3D Gaussian point located within the view frustum.
[0062] Camera tracking is implemented by a camera tracking module. This module uses a gradient optimization mechanism to dynamically estimate the camera pose in the current frame. In each time frame, it receives the initial rotation and translation parameters provided by the initialization module from the previous stage and performs multiple rounds of optimization iterations in conjunction with the rendering loss function to obtain the optimal camera pose estimate. The core of the camera tracking module is to calculate the loss function between the rendered image and the ground truth image under the current camera pose parameters and perform iterative optimization.
[0063] The forward rendering pipeline is a unified data processing flow that combines a 3D scene model with a given camera pose to generate corresponding 2D color and depth images. It's called a pipeline because it merges what might otherwise be separate color and depth rendering paths into a single execution pipeline. In its implementation, this pipeline can be implemented using an integrated GPU kernel function that receives scene and camera parameters, calculates the color and depth values for each pixel in parallel, and outputs the results synchronously.
[0064] In the forward rendering pass, matrix calculations are performed using a reduced-precision floating-point format. The matrix calculations include covariance matrix operations and projection transformation operations. The covariance matrix operations include the calculation of the semi-major axis of a 3D Gaussian projection onto a 2D plane. The semi-major axis calculation uses an approximation based on Taylor expansion. The target frame is a 3D Gaussian point located within the view frustum. Unnecessary computations are reduced by view frustum clipping.
[0065] Specifically, the color image and depth image of the current frame are first extracted from the input data in temporal index order, and their camera intrinsic and extrinsic parameter matrices are calculated. The extrinsic parameter matrix is provided by the initialization module in the previous stage or by the tracking results of the previous frame. The input color image is rearranged by channel into (C,H,W) format and normalized; the depth image undergoes the same channel transformation to ensure consistent processing on the GPU in subsequent steps.
[0066] When entering a new time frame idx When the velocity is greater than 0, a constant velocity model is used to initialize an initial camera pose estimate for the current frame. This method can quickly generate the initial pose of the current frame based on the assumption of motion continuity, even without real calibration information, providing a better starting point for subsequent optimization.
[0067] The view frustum detection module is a key component of this application for filtering out invalid point clouds and reducing the computational burden of rendering and projection in 3D Gaussian point cloud rendering and tracking calculations. Implemented on the GPU as an inline device function, this module can efficiently determine whether each 3D point is within the current camera's field of view (i.e., within the camera's view frustum) even with high point cloud density and significant inter-frame differences. This avoids invalid calculations for invisible points during subsequent rasterization and Gaussian projection stages. The view frustum geometry is a truncated pyramid shape with a large base, determined by both camera intrinsic parameters and the projection matrix.
[0068] This application determines whether point P is within the view frustum by judging whether point P simultaneously satisfies the depth condition and the screen projection constraint condition. That is, the depth of the point in the camera coordinate system should be between the near clipping plane and the far clipping plane. When the distance of the point in front of the camera's field of view is less than 0.2 units, the point is regarded as an invalid point and is discarded. In addition, the point is transformed from world coordinates through the projection matrix to obtain homogeneous coordinates and the perspective division is calculated. If the projected point exceeds the normalized device coordinate (NDC) range [-1,1], it means that the point is not within the visible range of the current frame image.
[0069] A 3D Gaussian ellipsoid possesses volumetric, directional, and density distribution characteristics in space, exhibiting an ellipsoidal shape. A 3D Gaussian model can not only continuously represent surface shapes but also participate in optimization and gradient propagation in a differentiable manner, thus achieving a fine representation of both geometry and lighting. In three-dimensional space, a Gaussian ellipsoid can be represented by the following probability density function (PDF). Defined as follows: ; in, Spatial coordinates; It is the Gaussian center (i.e., the geometric center of the ellipsoid). for The covariance matrix is used to describe the shape, scale, and orientation of an ellipsoid in three-dimensional space.
[0070] covariance matrix The characteristic decomposition form is as follows: ; in, Let the rotation matrix define the directions of the principal axes of the ellipsoid. The scaling matrix determines the variance of the ellipsoid along its three principal axes.
[0071] Therefore, the three-dimensional Gaussian ellipsoid can be characterized by the following six key parameters during the rendering process: ; If we further consider color distribution and opacity, each Gaussian point can be expanded into the following complete set of attributes: : ; in, Indicates color, Indicates opacity.
[0072] Under normal circumstances, The matrix is non-diagonal, and the ellipsoidal shape exhibits anisotropy. In this embodiment, to reduce computational complexity, an isotropic approximation is used, i.e., let: ; Therefore: ; in, for identity matrix This setting causes the 3D Gaussian to degenerate from a general ellipsoid into a sphere, with its isosurfaces uniformly distributed in all directions and possessing complete rotational symmetry. The Gaussian distribution in this case depends only on the Euclidean distance from the point to the center, as shown in the following equation: ; This greatly simplifies subsequent matrix operations, gradient backpropagation, and covariance updates in the rendering pipeline.
[0073] like Figure 5 As shown, to obtain the coverage radius when a 3D Gaussian projection is applied to a 2D pixel plane, the semi-major axis length needs to be calculated. This application uses an approximate calculation instead of the common high-complexity calculation. When a 3D point is projected onto a 2D plane using the camera projection matrix (Jacobian), the 2D covariance is calculated using the following formula: ; in, This is the semi-major axis length (i.e., coverage radius) of the 2D Gaussian after projection. The normalized radius of the 3D Gaussian in three-dimensional space. Let x be the x-coordinate of the 3D point on the camera normalized plane. Let y be the 3D point's y-coordinate on the camera's normalized plane. This refers to the camera's focal length.
[0074] Because square root operations are costly to implement in hardware (FPGA / GPU), this application uses Taylor expansion (an approximation over a small range) to eliminate the square root sign. Sometimes, ,in, Substituting the small quantity symbol, we get the following formula: ; This application simplifies the generation and data transfer of multiple matrices, the multiplication of five 3×3 matrices, two square root operations, and multiple floating-point addition, subtraction, multiplication, and division operations into simple multiplication and addition operations.
[0075] like Figure 4As shown, in some embodiments, the specific process of the forward rendering pass includes: obtaining projected Gaussian points based on the current optimization state and the three-dimensional Gaussian points; generating key-value pairs for sorting based on the projected Gaussian points and depth information; performing global sorting according to the keys in the key-value pairs, so that the Gaussian points are arranged in order of screen blocks and depth, to determine the index range of ordered Gaussian points in each screen block, and obtaining the sorting result; and performing parallel volume rendering based on the sorting result to generate color rendering results and depth rendering results.
[0076] Parallel volume rendering includes allocating parallel rendering tasks to screen blocks; for each pixel, the color contribution and transparency contribution of the Gaussian points covering the corresponding pixel are arranged in depth order; the depth rendering result adopts the median depth approximation method, which approximates the depth value of the pixel by calculating the weighted median of the depth of the Gaussian points covering the pixel.
[0077] The forward rendering channel supports multi-channel data synchronous processing and is implemented through a multi-channel renderer unit that supports joint input of depth and RGB. In forward rendering, depth image and contour map information are output simultaneously, avoiding multiple calls to the CUDA unit, which would increase communication latency and computation time, and improving the data communication efficiency between the host and the GPU.
[0078] Multi-channel renderer units have a wide range of applications beyond depth and RGB unification. They can also be extended to unified depth-normal or depth-semantic rendering structures, generating multimodal output in a single CUDA call and reducing the number of kernel function calls. Furthermore, multi-channel renderer units can be replaced with CUDA Stream-based parallel rendering structures, achieving similar performance improvements through multi-core concurrent execution.
[0079] Specifically, firstly, the system performs a prefix summation operation on the number of Gaussian points contained in each tile using cub::DeviceScan::InclusiveSum to obtain the offset position of each tile in the global index array, and at the same time obtains the total number of Gaussian points to be rendered.
[0080] cub::DeviceScan::InclusiveSum is a parallel algorithm function in the NVIDIA CUDA tool library CUB, used to quickly calculate the number of Gaussian points accumulated in each screen tile, thereby determining the starting position of each tile's data in the global array. It is a key preparatory step for parallel data reassembly.
[0081] Subsequently, memory space is dynamically allocated based on this quantity to establish a structure for storing key-value pairs. Next, the duplicateWithKeys kernel takes the 2D projection coordinates and depth information of each Gaussian point as input and generates a corresponding key-value pair for each Gaussian point. The key is used to identify the tile to which it belongs and the depth sorting information, and the value is the index of the Gaussian point. This establishes a sortable data structure for subsequent tile rendering. The core task of the duplicateWithKeys kernel is to generate a corresponding key-value pair for each input Gaussian point data. It receives the projected Gaussian point and depth as input and calculates a "key" (for sorting) and a "value" (the index of the Gaussian point) to create a data structure for subsequent parallel sorting.
[0082] Next, by calculating the highest valid bit width of the key, the number of bits required for sorting is determined, and `cub::DeviceRadixSort::SortPairs` is called to sort the key-value pairs based on the key, achieving an ordered arrangement of all Gaussian points by tile and depth. Finally, the system uses the `identifyTileRanges` kernel to mark the start and end index range of each tile in the sorting result, assigning an independent rendering task to each screen tile, enabling subsequent parallel blending and rasterization operations on the GPU.
[0083] The `identifyTileRanges` function is executed on the sorted key-value pair array. Its core logic is to scan the sorted keys and find the positions where the parts representing the tile numbers in the keys have changed. These positions mark the start and end indices of the Gaussian point sequence corresponding to each tile in the global sorted array, thereby defining a clear data processing boundary for each tile.
[0084] Next, the volume rendering module uses the core CUDA kernel function `renderCUDA` for Gaussian volume rendering. This function is responsible for the final rasterization and color blending of the 3D Gaussian points after projection and block sorting on the screen. Its core idea is to use screen tiles as units (corresponding to thread blocks), with each thread corresponding to one pixel. It loads the Gaussian projection parameters within the current tile in batches from shared memory, including 2D position, depth, elliptic coefficient, and transparency, and calculates the coverage of each pixel with the Gaussian projection.
[0085] In the calculation, the pixel distance is attenuated using the quadratic exponential form of the Gaussian distribution, and the contribution of each Gaussian to the pixel color is obtained by combining the opacity of the Gaussian. The rendering process uses a cumulative transparency update method to achieve the blending of the foreground and background layers until the pixel is completely opaque or the tile area is processed.
[0086] In terms of implementation, the core operators, including the distance squared term and the Gaussian exponential decay term, are optimized from full precision to half precision, effectively reducing memory bandwidth and computation latency, making them suitable for hardware accelerators or low-power devices. Furthermore, the depth output uses the median depth to approximate the mean depth, simplifying the accumulation calculation.
[0087] The frustum detection, Gaussian ellipsoid radius calculation, depth sorting, and volume rendering modules in camera tracking are implemented in C++ and CUDA, and compiled and bound to the main program via Pybind11 to form a directly callable rendering module (Renderer). Traditional Renderer units only support RGB color channel data and cannot process depth channel information simultaneously. Therefore, the main program needs to call the Renderer unit twice to achieve independent rendering of RGB and depth maps, and perform multiple data transfers between the host and GPU.
[0088] To address the aforementioned efficiency issues, this application fuses and encodes depth and color images, enabling joint rendering and result return in a single GPU call. This effectively reduces the number of CUDA kernel function calls, significantly lowers data transfer overhead between the host and device, and thus improves overall rendering efficiency and system resource utilization.
[0089] like Figure 4 As shown, the L1 loss function is then calculated. This function calculates the L1 loss (i.e., absolute error loss) for the entire system. Its main function is to measure the difference between the rendering result of the current frame and the actual observation data, so as to guide the optimization and updating of Gaussian parameters and camera pose.
[0090] Specifically, the function internally employs a multi-stage control strategy to dynamically schedule the rendering mode and loss weights. In some embodiments, the stages include an early iteration stage and a late iteration stage. In the early iteration stage, the depth loss function between the depth rendering result and the real depth image is calculated, and the iteration count is less than or equal to a preset threshold. In the late iteration stage, both the depth loss function and the color loss function between the color rendering result and the real color image are calculated simultaneously, and the iteration count is greater than the preset threshold. Based on the iteration count, the early iteration stage and the late iteration stage are switched, and then optimized camera pose parameters are output based on the iteration mechanism.
[0091] The segmented strategy dynamically determines the system's computation mode at different stages by using iteration counts and stage markers: in the early iteration stage (e.g., iteration number ≤ 20), only depth domain L1 loss is enabled for fast convergence of pose estimation; in the later iteration stage (e.g., iteration number > 20), both depth domain and RGB domain L1 loss are enabled to refine color consistency and improve rendering quality; thus significantly reducing GPU computational overhead while ensuring the optimization accuracy of key parameters.
[0092] In some embodiments, in the phased switching strategy, when calculating the depth loss function and the color loss function, a gradient-aware mask strategy is applied to determine key pixel regions, including: extracting edge gradient magnitude maps of the real depth image and the real color image based on the image gradient operator; selecting high gradient regions according to the gradient magnitude distribution quantiles to form an initial mask; performing random sampling on the initial mask to form a hybrid mask; and using the hybrid mask to filter key pixel regions to calculate the depth loss function and the color loss function.
[0093] The masking strategy is used to adaptively select key pixel regions for loss calculation when calculating depth and RGB loss. Specifically, the masking strategy first uses the Sobel operator to extract edge gradient magnitude maps from the depth map and RGB grayscale map respectively. Then, it selects significant edge regions according to the gradient distribution quantile (such as the top 30% or 70% quantile) to obtain the initial mask. To prevent the mask from being too sparse and causing training instability, a portion of non-edge pixels are randomly added according to the target sampling ratio (such as 40%). Random noise is used to achieve the balance of global uniform sampling, and finally a mixed mask mixed_mask is formed.
[0094] This mask is used in both depth and RGB error calculations, allowing the model to focus on geometric changes in high texture gradient regions. This avoids interference from flat regions in the optimization during the early convergence phase, reduces computation on unimportant parts of the result, and thus improves reconstruction accuracy and convergence speed.
[0095] After the loss function is calculated, the system iteratively updates the relevant parameters using the backpropagation mechanism. First, the calculated total loss function is input into the backpropagation module, which calls the automatic differentiation interface `.backward()` to perform the gradient backpropagation operation. This operation automatically calculates the gradient values of each learnable parameter (including 3D Gaussian parameters and camera pose) according to the chain rule. Subsequently, the optimization module calls the parameter update function to execute the gradient descent step `optimizer.step()`. The optimizer is an adaptive optimizer based on the Adam algorithm, which internally updates the parameters with weights according to first-order and second-order momentum estimates, thereby achieving fast and stable convergence. After the update is complete, the system automatically clears the gradient cache to prepare for the next iteration.
[0096] To reduce the waste of computational resources caused by invalid iterations, this application adaptively tracks the decreasing trend of the loss function through a dynamic iteration strategy to determine the number of iterations required for training. In some embodiments, the iteration mechanism includes dynamic early stopping control. During the iteration process, the decreasing trend of the loss function is monitored to calculate the relative decrease ratio between two adjacent iterations. When the relative decrease ratio is lower than a preset threshold multiple times consecutively, an early stopping condition is triggered. The iteration process is terminated according to the early stopping condition to output the currently optimized camera pose parameters.
[0097] Specifically, in the tracking optimization phase of each frame, this strategy dynamically adjusts the number of optimization iterations by monitoring the decrease of the loss function in real time, thereby significantly improving computational efficiency while ensuring accuracy.
[0098] Existing camera tracking iteration counts are typically set to 200, but in actual training, far more training iterations are unnecessary. The early stopping condition in this application is that the number of iterations exceeds the warm-up value, and the loss function decreases by less than a set relative threshold multiple times consecutively.
[0099] The initial few iterations (e.g., 30) serve as a warm-up phase, ensuring the loss function has sufficient time to decrease. Afterward, the system begins calculating the relative decrease ratio between adjacent iterations. When the loss decrease ratio is less than a preset threshold (e.g., 5%), the system considers it a weak decrease and accumulates the count. If the accumulated count exceeds a set patience value (e.g., 6 times), the iteration process is automatically terminated early, achieving adaptive early stopping.
[0100] In addition, this application introduces a dynamic extension mechanism, which automatically triggers an extended iteration cycle when the depth loss is detected to be higher than a set threshold, doubling the number of iterations and continuing to optimize, in order to prevent attitude estimation deviations caused by accidental stopping.
[0101] By combining dynamic early stopping with extension, the number of iterations can be automatically adjusted according to the real-time optimization convergence, avoiding the problems of overcomputation or premature convergence that exist in the traditional fixed iteration method.
[0102] Tests on specific experimental datasets show that this method can reduce iteration time by more than 50% without reducing tracking accuracy, significantly improving overall real-time performance and energy efficiency.
[0103] S400: Based on optimized camera pose parameters, a hierarchical filtering mechanism is executed to determine keyframes from color and depth images. The hierarchical filtering mechanism includes at least two stages.
[0104] Keyframes are one or more representative frames selected from the entire image sequence for scene model optimization and expansion. Keyframes typically meet conditions such as sufficient visual overlap with other frames and high image quality. Using keyframes instead of global optimization for every frame can reduce computation and prevent the model from overfitting to noisy frames.
[0105] Since SLAM systems typically require hundreds or even thousands of images as input, even if only one keyframe is selected every 3 or 5 frames, the system still needs to maintain a very large list of keyframes in the later stages. At this point, each new keyframe must be compared with every frame in this huge list for overlap, resulting in a significant computational burden. Therefore, based on the existing keyframe selection algorithm based on depth overlap rate, a hierarchical filtering mechanism is implemented. This mechanism is a method for selecting representative keyframes from a large number of image frames. It includes multiple filtering stages; in some embodiments, the hierarchical filtering mechanism includes at least a coarse filtering stage and a fine filtering stage.
[0106] like Figure 6 As shown, in this embodiment, the number of existing keyframes is obtained; if the number of existing keyframes is greater than the number threshold and the target selection ratio is lower than the ratio threshold, a hierarchical screening mechanism is executed. The hierarchical screening mechanism includes: in the coarse screening stage, sparse sampling is performed on the depth image of the current frame and back-projected into a three-dimensional point cloud to calculate the overlap with the candidate keyframes; highly overlapping candidate frames are selected based on the overlap; in the fine screening stage, pixel sampling is performed on the highly overlapping candidate frames to calculate the overlap; and keyframes are selected based on the overlap.
[0107] In the coarse screening stage, the system uses only one-quarter (400 pixels) of the original algorithm's sampling points for rapid estimation. It sparsely samples the depth map of the current frame and back-projects it into a 3D point cloud, projecting this cloud onto the camera coordinate system of the candidate keyframes. The overlap between the keyframe and the current frame is calculated based on the proportion of projected points within the field of view. After sorting by overlap, the algorithm retains only highly overlapping candidate frames (three times the original number of target frames) as the candidate keyframe set. This process quickly eliminates most irrelevant frames with low computational cost, achieving approximately 75% computational pruning.
[0108] During the fine screening stage, the system performs complete overlap calculation and sorting on candidate frames in the coarse screening results, and uses complete pixel sampling to restore high-precision evaluation. If the number of candidate frames has met the target number k, the fine screening stage is skipped and the coarse screening results are returned directly, thereby significantly reducing computational latency in large-scale datasets or low selection ratio scenarios.
[0109] It is understood that in this embodiment, when the number of existing keyframes is greater than the number threshold and the target selection ratio is lower than the ratio threshold, the coarse screening stage is performed first, followed by the fine screening stage. When the number of existing keyframes is less than or equal to the number threshold and the target selection ratio is greater than or equal to the ratio threshold, only the fine screening stage is performed.
[0110] The computational logic of the entire process includes operations such as depth validity screening, 3D backprojection, coordinate transformation, projection onto a 2D plane, and boundary filtering. The layered screening strategy in this application is only activated when the total number of keyframes is large (e.g., more than 50 frames) and the target selection ratio is low (e.g., less than 30%). It can achieve a 2–4 times speedup and a significant reduction in memory usage while maintaining the same screening accuracy as the original algorithm. This mechanism is compatible with FP16 or fixed-point mixed-precision matrix operation modules, possesses good hardware acceleration adaptability, and provides an efficient keyframe input set for subsequent Gaussian rendering and incremental modeling.
[0111] S500: Utilizes keyframes and optimized camera pose parameters to perform rendering and mapping to update the 3D Gaussian point cloud.
[0112] The core of the mapping process is updating and maintaining the parameters of the 3D Gaussian point cloud to more accurately reflect the real scene. During this process, the system fixes the camera pose of the keyframes and focuses on optimizing the representation of the scene itself, that is, adjusting the position, shape, color and other attributes of the 3D Gaussian points so that the virtual image rendered from the keyframe perspective is as consistent as possible with the real keyframe image.
[0113] In some embodiments, a frame is sampled from the selected keyframes, and the optimized camera pose parameters corresponding to the keyframes are obtained; based on the optimized camera pose parameters, a reconstruction loss function is calculated, which includes a depth consistency term and a color reprojection error term; according to the gradient magnitude, a pruning operation and a densification operation are performed on the 3D Gaussian point cloud to update the 3D Gaussian point cloud, which also includes a 3D Gaussian point cloud obtained based on the difference between the depth rendering result of the current frame and the real depth image.
[0114] The system uses a 3D mapping method that combines keyframe random sampling with gradient optimization to achieve adaptive updating of Gaussian parameters for continuous frames in dynamic scenes.
[0115] In this stage, the camera pose parameters of the current frame are fixed, and a fixed number of iterations (e.g., 30 times) is used as the frame mapping optimization cycle. In each iteration, a frame or the current time image is randomly selected from the input set of current keyframes as the optimization input to enhance the global consistency and temporal robustness of the model.
[0116] In each iteration, the data structure corresponding to the current frame is first constructed, including the color image, depth map, camera intrinsic and extrinsic parameters, and historical pose sequence. Based on this, the aforementioned loss calculation function is called to obtain the reconstruction loss value of the current frame. The loss function consists of a depth consistency term, a color reprojection error term, and an optional contour constraint term. The gradients of each Gaussian parameter (including position, variance, transparency, etc.) are calculated through the aforementioned backpropagation.
[0117] In the gradient-free region, a Gaussian sieving and densification mechanism (Prune, Densify) based on gradient magnitude is further executed. Gaussians with small contributions or high gradient regions are dynamically reduced or copied. This can effectively remove inefficient Gaussians with low weights affecting the final graph construction, and copy important Gaussians with greater influence to participate in subsequent optimization. Then, an optimizer update is performed to complete the iterative optimization of parameters and gradient clearing, maintaining the compactness and local expressiveness of the model.
[0118] To ensure the continuity and numerical stability of the optimization process, the aforementioned rendering module is invoked in the current frame to obtain depth and contour information, which is then compared with the actual depth map. When the rendered contour value is lower than a set threshold, the region is determined to be non-existent. Simultaneously, when the rendered depth is significantly greater than the actual depth and the error exceeds a multiple of the median, the region is considered to contain unmodeled nearby objects. The two detection results are logically ORed to form a non-existence mask, which is used to identify regions where Gaussian parameters need to be added. Subsequently, the system extracts effective pixels from the color image and depth map based on this mask, and backprojects them into a new 3D point cloud using camera intrinsic and extrinsic parameters. Then, new Gaussian parameters (including position, variance, color, transparency, etc.) are initialized based on the spatial distribution of the point cloud, and this new Gaussian parameter is concatenated with the original Gaussian set. The model is updated in real time, and gradient accumulation and time index variables are reset simultaneously during this process to ensure the continuity and numerical stability of the optimization process.
[0119] When the entire 3D Gaussian SLAM system starts up, it first performs initialization, and then inputs color-depth image frames in sequence. For each frame, it executes the S300-S500 part, going through the preprocessing, camera tracking, adding new Gaussian, key frame selection and mapping stages, and gradually generating a scene map based on 3D Gaussian.
[0120] Through systematic optimization of the above steps, this method achieves adaptive processing capabilities for different scenarios, maintains a high degree of modularity in its algorithm framework, and facilitates deployment on various hardware architectures. This method not only improves computational throughput but also reduces GPU memory consumption, providing strong support for efficient point cloud processing in subsequent fields such as autonomous navigation, augmented reality, and robot perception.
[0121] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.
Claims
1. A SLAM acceleration method based on 3D Gaussian rendering, characterized in that, include: Acquire input data, which includes color images and depth images; Based on the first frame of the input data, initialize the 3D Gaussian point cloud and camera pose parameters; For the target frame in the input data, camera tracking is performed, and optimized camera pose parameters are output based on a phased switching strategy and an iterative mechanism. The target frame is any frame in the input data other than the first frame. The camera tracking uses a reduced-precision floating-point format. Based on the optimized camera pose parameters, a hierarchical filtering mechanism is executed to determine keyframes from the color image and the depth image. The hierarchical filtering mechanism includes at least two stages. The keyframes and optimized camera pose parameters are used to perform rendering and mapping to update the 3D Gaussian point cloud.
2. The SLAM acceleration method based on 3D Gaussian rendering according to claim 1, characterized in that, The initialization of the 3D Gaussian point cloud and camera pose parameters based on the first frame of the input data includes: Perform data preprocessing on the color image of the first frame to output a preprocessed color image, the data preprocessing including color normalization operation; Perform data preprocessing on the depth image of the first frame to output a preprocessed depth image, the data preprocessing including depth dimension rearrangement operation; Based on the camera intrinsic and extrinsic parameters matrix, the preprocessed depth image is back-projected to generate the spatial position parameters of the 3D point cloud. From the preprocessed color image, obtain the color vector of the pixel corresponding to the spatial location parameter; Using a reduced-precision floating-point format, the average squared distance between adjacent points in the 3D point cloud is calculated, and the scene radius is calculated based on the preset maximum depth value and scene scale parameters. Based on the spatial location parameters, color vector, mean square distance, and scene radius, initialize the location parameters, color parameters, transparency parameters, and scale parameters in the 3D Gaussian point cloud.
3. The SLAM acceleration method based on 3D Gaussian rendering according to claim 1, characterized in that, Performing camera tracking on the target frame in the input data includes: Based on the constant velocity model, the initial estimate of the camera pose for the current target frame is predicted according to the optimized camera pose parameters of the previous frame. Starting from the initial estimation of the camera pose, perform optimization iterations to output the optimized camera pose parameters of the current target frame; The step of predicting the initial estimate of the camera pose for the current target frame based on the constant velocity model and the optimized camera pose parameters of the previous frame includes: Obtain the optimized camera pose parameters of the previous two frames; based on the optimized camera pose parameters of the previous two frames, calculate the initial rotation estimate and initial translation estimate of the current target frame using the first-order extrapolation method; perform normalization processing on the initial rotation estimate to obtain the initial rotation value that meets the rotation constraints; combine the initial rotation value and the initial translation estimate to obtain the initial camera pose estimate of the current target frame.
4. The SLAM acceleration method based on 3D Gaussian rendering according to claim 3, characterized in that, The step of performing optimization iterations, starting from the initial camera pose estimation, to output the optimized camera pose parameters for the current target frame includes: The initial camera pose estimate is used as the current optimization state; Based on the current optimization state, the color rendering result and depth rendering result corresponding to the target frame are generated synchronously through the forward rendering channel; the target frame is a three-dimensional Gaussian point located within the view frustum. In the forward rendering channel, matrix calculations are performed using a reduced-precision floating-point format. The matrix calculations include covariance matrix operations and projection transformation operations. The covariance matrix operations include the calculation of the semi-major axis of a three-dimensional Gaussian projection onto a two-dimensional plane. The semi-major axis calculation uses an approximate calculation based on Taylor expansion. Calculate the loss function between the color rendering result and the depth rendering result and the actual observed image; Based on the loss function and combined with a phased switching strategy, the current optimization state is updated, and the optimized camera pose parameters are output after iteration.
5. The SLAM acceleration method based on 3D Gaussian rendering according to claim 4, characterized in that, The stages include the early iteration stage and the late iteration stage; The optimized camera pose parameters output based on the phased switching strategy and iterative mechanism include: In the early stage of the iteration, the depth loss function between the depth rendering result and the real depth image is calculated. In the early stage of the iteration, the iteration count is less than or equal to a preset threshold. In the later stage of the iteration, the depth loss function and the color loss function between the color rendering result and the real color image are calculated simultaneously. In the later stage of the iteration, the iteration count is greater than the preset threshold. Based on the iteration count, the early iteration stage and the late iteration stage are switched, and then the optimized camera pose parameters are output based on the iteration mechanism.
6. The SLAM acceleration method based on 3D Gaussian rendering according to claim 4, characterized in that, The forward rendering channel supports multi-channel data synchronization processing; The process of synchronously generating the color rendering result and depth rendering result corresponding to the target frame through the forward rendering pass includes: Based on the current optimization state and the three-dimensional Gaussian points, the projected Gaussian points are obtained; Based on the projected Gaussian points and depth information, key-value pairs for sorting are generated; Global sorting is performed on the keys in the key-value pairs to arrange the Gaussian points according to screen blocks and depth order, so as to determine the index range of ordered Gaussian points in each screen block and obtain the sorting result. Based on the sorting result, a parallel volume rendering is performed to synchronously generate color rendering results and depth rendering results; the parallel volume rendering includes allocating parallel rendering tasks to the screen blocks; for each pixel, the color contribution and transparency contribution of the Gaussian points covering the corresponding pixels are arranged in the depth order; wherein, the depth rendering result adopts the median depth approximation method.
7. The SLAM acceleration method based on 3D Gaussian rendering according to claim 5, characterized in that, In the phased switching strategy, when calculating the depth loss function and color loss function, a gradient-aware mask strategy is applied to determine key pixel regions, including: Based on the image gradient operator, edge gradient magnitude maps of the true depth image and the true color image are extracted; High gradient regions are selected based on the quantiles of the gradient magnitude distribution to form the initial mask; Random sampling is performed on the initial mask to form a hybrid mask; The hybrid mask is used to filter key pixel regions to calculate depth loss and color loss functions.
8. The SLAM acceleration method based on 3D Gaussian rendering according to claim 1, characterized in that, The iterative mechanism includes dynamic early stopping control: The optimized camera pose parameters output based on the phased switching strategy and iterative mechanism include: During the iteration process, the decreasing trend of the loss function is monitored to calculate the relative decrease ratio between two adjacent iterations; When the relative decrease ratio falls below a preset threshold multiple times consecutively, an early stop condition is triggered. The iteration process is terminated according to the early stopping condition to output the current optimized camera pose parameters.
9. The SLAM acceleration method based on 3D Gaussian rendering according to claim 1, characterized in that, The tiered screening mechanism includes at least a coarse screening stage and a fine screening stage, and the keyframes are one or more, including: The step of performing a hierarchical filtering mechanism based on the optimized camera pose parameters to determine keyframes from the color image and depth image includes: Get the number of existing keyframes; If the number of existing keyframes exceeds a threshold and the target selection ratio is below a threshold, a hierarchical filtering mechanism is executed, which includes: In the coarse screening stage, sparse sampling is performed on the depth image of the current frame and back-projected into a 3D point cloud to calculate the overlap with the candidate keyframes. Based on the aforementioned overlap, highly overlapping candidate frames are selected. During the fine screening stage, pixel sampling is performed on the highly overlapping candidate frames to calculate the overlap. Keyframes are selected based on the sorting of the overlap.
10. The SLAM acceleration method based on 3D Gaussian rendering according to claim 1, characterized in that, The step of using the keyframes and optimized camera pose parameters to perform rendering and mapping to update the 3D Gaussian point cloud includes: Sample one frame from the selected keyframes and obtain the optimized camera pose parameters corresponding to the keyframe; Based on the optimized camera pose parameters, a reconstruction loss function is calculated, which includes a depth consistency term and a color reprojection error term. Based on the gradient magnitude, a pruning operation and a densification operation are performed on the 3D Gaussian point cloud to update the 3D Gaussian point cloud. The 3D Gaussian point cloud also includes a 3D Gaussian point cloud obtained based on the difference between the depth rendering result of the current frame and the real depth image.
Citation Information
Patent Citations
Dense RGB-D SLAM method based on multistage 3D Gaussian
CN118781189A
Three-dimensional reconstruction and rendering method based on non-parameter image
CN118840490A
Semantic SLAM method based on three-dimensional Gaussian sputtering
CN120431306A
Positioning mapping method and system based on three-dimensional Gaussian splash and depth camera
CN120953355A
Simultaneous positioning and dense three-dimensional reconstruction method
US20200043189A1
Cited By
Visual SLAM method and system based on 3D Gaussian spattering in dynamic environment and readable storage medium
CN121837486A
