RGB-D SLAM method for 3DGS and two-stage attitude optimization

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, and discloses an RGB-D SLAM method for 3DGS and two-stage attitude optimization. According to the method, depth information is complemented through Gaussian process regression, the pose is optimized through dual-mode tracking, the calculation efficiency is improved by combining 3DGS and two-stage optimization, the problems of depth complementation, pose stability and real-time performance of existing RGB-D SLAM in a complex environment are effectively solved, the real-time reconstruction requirement in a complex scene can be met, and the real-time reconstruction efficiency is improved. And the robustness and the precision of the RGB-D SLAM in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to an RGB-D SLAM method with 3DGS and two-stage pose optimization. Background Technology

[0002] Existing RGB-D SLAM (Simultaneous Localization and Mapping) systems face limitations in complex environments, primarily due to three factors: First, incomplete depth information. RGB-D sensors struggle to acquire depth data in low-texture or high-reflectivity areas, leading to large holes in the map and impacting the quality of 3D scene reconstruction. Second, unstable pose initialization. In low-texture or geometrically ambiguous scenes, traditional pose initialization methods are susceptible to large-scale errors, causing tracking drift and map inconsistencies. Third, a balance between computational efficiency and computational cost is difficult to achieve. While deep learning or neural network-based SLAM methods can generate more accurate 3D maps, they involve significant computational demands, making them unsuitable for real-time scenarios, especially on resource-constrained devices.

[0003] In the prior art, invention patent CN117197449A discloses a SLAM construction method and system based on RGB-D semantic segmentation. It combines RGB-D images and a depth completion algorithm, employing a Transformer-based dual-branch multi-level attention fusion RGB-D semantic segmentation model for training to ensure robust semantic information acquisition even under poor imaging conditions. Invention patent CN115984493A discloses an RGB-D SLAM method, system, and storage medium based on pseudo-depth maps. It utilizes deep learning technology to predict the depth of color images collected by a monocular camera, converts the data into pseudo-depth maps, and then inputs them into the ORB-SLAM3 system, thereby reducing conversion and deployment time.

[0004] However, existing technologies still have significant shortcomings. While the scheme in publication CN117197449A introduces a semantic segmentation model to improve robustness, it does not adequately explain how to address the issues of incomplete depth information and unstable pose initialization. Although the scheme in publication CN115984493A implements RGB-D SLAM using a pseudo-depth map, there is still room for improvement in depth estimation accuracy and real-time performance, especially when dealing with complex environments and large-scale scenes, where computational efficiency and map building quality may be affected.

[0005] In summary, while existing RGB-D SLAM technology has made some progress, it still has shortcomings in balancing depth information integrity, pose initialization stability, and computational efficiency. How to effectively complete depth information, improve pose estimation accuracy, and optimize computational efficiency has become a pressing technical challenge that requires further innovation and improvement to meet the demands of high-quality visual reconstruction and real-time performance in complex environments. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an RGB-D SLAM method based on 3DGS and two-stage pose optimization.

[0007] In a first aspect, the present invention provides an RGB-D SLAM method based on 3DGS and two-stage pose optimization, the technical solution of which is as follows: The depth map of the current frame in the RGB-D sequence is downsampled to generate the original point cloud, and the original point cloud is ellipsoidally regularized to obtain the regularized point cloud. Gaussian process regression is used to complete the missing depth regions in the depth map of the current frame, generating a completed point cloud, and then the completed point cloud is fused with the original point cloud to form the complete point cloud of the current frame; Based on the regularized point cloud, a fractionally weighted generalized iterative nearest-point algorithm is used to perform dual-mode camera pose tracking to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode. Based on the final camera pose, the overlap rate between the current frame and the trackable set is calculated. When the overlap rate is lower than a set threshold or reaches a fixed frame interval, the current frame is determined as a key frame, and the global Gaussian map or the trackable set is updated according to the type of the key frame. The global Gaussian map is rendered and loss is calculated based on 3D Gaussian splashing, and the camera pose and Gaussian parameters are optimized in a coordinated manner through two-stage pose optimization to obtain the optimized global Gaussian map.

[0008] The beneficial effects of the RGB-D SLAM method based on 3DGS and two-stage pose optimization of the present invention are as follows: The method of this invention completes depth information through Gaussian process regression, optimizes pose through dual-mode tracking, and improves computational efficiency by combining 3DGS and two-stage optimization. It effectively solves the problems of depth completion, pose stability and real-time performance of existing RGB-D SLAM in complex environments, and can meet the real-time reconstruction requirements in complex scenes, improving the robustness and accuracy of RGB-D SLAM in complex environments.

[0009] Based on the above scheme, the RGB-D SLAM method of 3DGS and two-stage pose optimization of the present invention can be further improved as follows.

[0010] In one alternative approach, the step of performing ellipsoidal regularization on the original point cloud to obtain a regularized point cloud includes: Calculate the covariance matrix of any point in the local neighborhood of any point in the original point cloud; The covariance matrix is ​​decomposed into a direction matrix and a scale matrix composed of eigenvalues, and the median of all eigenvalues ​​is calculated. The scale matrix is ​​normalized by dividing it by the median to obtain a normalized scale matrix. The regularized covariance matrix of any point is then reconstructed using the direction matrix and the normalized scale matrix. The regularized point cloud is obtained based on the regularized covariance matrix of all points in the original point cloud.

[0011] In one alternative approach, the step of using Gaussian process regression to complete the missing depth regions in the depth map of the current frame and generating a completed point cloud includes: The effective depth pixels and corresponding depth observations are extracted from the depth map to form a training dataset, and the pixels of the missing depth regions in the depth map are used as the prediction point set. Construct a spatial-luminance product kernel function that combines spatial distance information and RGB luminance information, and use the spatial-luminance product kernel function to calculate the autocovariance matrix of the training dataset and the cross-covariance matrix between the training dataset and the prediction point set; Based on the autocovariance matrix, the crosscovariance matrix, and the depth observations, the mean depth prediction of the predicted point set is calculated, and the missing depth region is filled in based on the mean depth prediction to generate the filled point cloud.

[0012] In one alternative approach, the step of performing dual-mode camera pose tracking using a fractionally weighted generalized iterative nearest-neighbor algorithm based on the regularized point cloud to obtain the final camera pose of the current frame includes: Using the frame-to-frame tracking mode, the regularized point cloud of the current frame and the regularized point cloud of the previous frame are used as inputs, and the temporary camera pose of the current frame is calculated by the fraction-weighted generalized iterative nearest point algorithm. Using the frame-to-map tracking mode, the temporary camera pose is used as the initial value for frame-to-map registration. The regularized point cloud of the current frame is registered with the trackable set. The initial value is iteratively optimized using the fractionally weighted generalized iterative nearest-point algorithm to calculate the final camera pose of the current frame.

[0013] In one alternative approach, the step of calculating the overlap rate between the current frame and the trackable set based on the final camera pose includes: Based on the final camera pose, the regularized point cloud of the current frame is registered with the trackable set, and a set of valid corresponding points that satisfy geometric consistency is obtained after registration. The number of points in the set of valid corresponding points is counted as the number of valid corresponding points, and the number of source points in the regularized point cloud of the current frame used for registration is obtained. The ratio of the number of valid corresponding points to the number of source points is determined as the overlap rate between the current frame and the traceable set.

[0014] In one alternative approach, the step of identifying the current frame as a keyframe and updating the global Gaussian map or the traceable set based on the type of the keyframe includes: When the overlap rate is lower than the set threshold, the current frame is determined as a tracking key frame; when the frame interval between the previous mapping key frame and the current frame reaches a fixed frame interval, the current frame is determined as a mapping key frame; wherein, if the overlap rate is lower than the set threshold and the frame interval between the previous mapping key frame and the current frame reaches the fixed frame interval, then the current frame is determined as the tracking key frame. When the current frame is determined to be a tracking keyframe, the regularized point cloud of the current frame is added to the trackable set, and the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map; when the current frame is determined to be a mapping keyframe, the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map.

[0015] In one alternative approach, the steps of rendering and loss calculation of the global Gaussian map based on 3D Gaussian splashing, and co-optimizing the camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map include: Using 3D Gaussian splashing technology, the global Gaussian map is rendered in a differentiable manner to generate the predicted RGB image and predicted depth image of the key frame; Using the real RGB image and real depth image of the keyframe as supervision, and combining the predicted RGB image and the predicted depth image, the color loss term and the depth loss term are calculated and a joint loss function is constructed; The joint loss function is minimized through a two-stage pose optimization process. In the first stage, a joint rigid body transformation is performed on the camera pose of the key frame and all Gaussian point sets in the view of the key frame. In the second stage, the Gaussian parameters are first fixed and the camera pose is optimized, and then the Gaussian parameters are optimized while the camera pose is fixed. Based on the results of the two-stage pose optimization process, the Gaussian parameters in the global Gaussian map and the camera pose of the keyframes are updated to obtain the optimized global Gaussian map.

[0016] Secondly, the present invention provides an RGB-D SLAM system with 3DGS and two-stage pose optimization, the technical solution of which is as follows: It includes: a processing module, a fusion module, a tracking module, an update module, and an optimization module; The processing module is used to: downsample the depth map of the current frame in the RGB-D sequence to generate the original point cloud, and perform ellipsoidal regularization on the original point cloud to obtain a regularized point cloud; The fusion module is used to: use Gaussian process regression to complete the missing depth regions in the depth map of the current frame, generate a completed point cloud, and fuse the completed point cloud with the original point cloud to form the complete point cloud of the current frame; The tracking module is used to: perform dual-mode camera pose tracking based on the regularized point cloud using a fractionally weighted generalized iterative nearest-point algorithm to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode; The update module is used to: calculate the overlap rate between the current frame and the trackable set based on the final camera pose; when the overlap rate is lower than a set threshold or reaches a fixed frame interval, convert the complete point cloud into a Gaussian point set and add it to the global Gaussian map, and set the current frame as a key frame to update the trackable set. The optimization module is used to: render and calculate the loss of the global Gaussian map based on 3D Gaussian splashing, and to perform collaborative optimization of camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map.

[0017] The beneficial effects of the RGB-D SLAM system with 3DGS and two-stage pose optimization of the present invention are as follows: The system of this invention completes depth information through Gaussian process regression, optimizes pose through dual-mode tracking, and improves computational efficiency by combining 3DGS and two-stage optimization. It effectively solves the problems of depth completion, pose stability and real-time performance of existing RGB-D SLAM in complex environments, and can meet the real-time reconstruction needs in complex scenes, improving the robustness and accuracy of RGB-D SLAM in complex environments.

[0018] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the RGB-D SLAM method with 3DGS and two-stage pose optimization as described in this invention.

[0019] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the RGB-D SLAM method with 3DGS and two-stage pose optimization as described in this invention.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of the RGB-D SLAM method for 3DGS and two-stage pose optimization according to the present invention. Figure 2 This is a SLAM framework diagram; Figure 3 The image shows the completion result of Gaussian process regression; Figure 4 Comparison of reconstruction results for the TUM RGB-D dataset; Figure 5 Comparison chart of reconstruction results for the Replica dataset; Figure 6 This is a schematic diagram of an embodiment of the RGB-D SLAM system with 3DGS and two-stage pose optimization according to the present invention. Figure 7 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0022] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0023] Figure 1This diagram illustrates a flowchart of an embodiment of an RGB-D SLAM method with 3DGS and two-stage pose optimization provided by the present invention. This RGB-D SLAM method with 3DGS and two-stage pose optimization can be executed by electronic devices such as terminal devices or servers. Figure 1 As shown, it includes the following steps: S1. Downsample the depth map of the current frame in the RGB-D sequence to generate the original point cloud, and perform ellipsoidal regularization on the original point cloud to obtain a regularized point cloud.

[0024] In this context, the RGB-D sequence refers to a data stream consisting of a series of temporally continuous and synchronized RGB images and depth maps, used as input to a SLAM system. For example, the fr1 / desk sequence in the TUM RGB-D dataset contains multiple frames of RGB images and corresponding depth maps, with each frame capturing color and depth information of a desktop scene. The current frame refers to the latest frame of the RGB-D sequence being processed; for example, when processing the fr1 / desk sequence, the current frame is the latest image and depth map being processed. The operation involves processing the RGB image and depth map of the frame. A depth map is a two-dimensional image where each pixel value represents the distance information to a corresponding point in the scene; for example, the depth map of the current frame. In this context, pixel values ​​represent the distance from the camera to the surface of the desktop object. The raw point cloud refers to a set of points generated by back-projecting the effective pixels of the depth map into 3D space, without any completion processing; for example, the current frame depth map... After downsampling, a set of 3D points containing the effective depth pixels is obtained. Ellipsoidal regularization refers to the process of scaling the covariance matrix of each point in a point cloud to improve numerical stability; for example, when calculating the original point cloud... The local neighborhood covariance of each point is decomposed, and the eigenvalues ​​are normalized to the median. Regularized point clouds refer to point clouds that have undergone ellipsoidal regularization, where the covariance matrix has a uniform scale; for example, for the original point cloud... After ellipsoidal regularization, a scale-normalized point cloud is obtained for tracking.

[0025] S2. Gaussian process regression is used to complete the missing depth regions in the depth map of the current frame, generating a completed point cloud, and the completed point cloud is merged with the original point cloud to form the complete point cloud of the current frame.

[0026] Gaussian process regression refers to a nonparametric regression method based on a probability model used to predict missing values; for example, using the current frame depth map. The effective pixels in the depth map are used to predict missing depth regions using a spatial-luminance kernel function. Missing depth regions refer to pixel areas in the depth map where effective depth values ​​cannot be obtained due to sensor limitations; for example, in the depth map... Holes in the image are caused by desktop glare or low texture. Point cloud completion refers to the generation of a 3D point set after filling in missing depth regions using Gaussian process regression; for example, using the predicted mean. Complete the missing regions to obtain point clouds. A complete point cloud refers to the complete 3D point set obtained by fusing the original point cloud and the completed point cloud; for example, when... and Merge to form the complete point cloud of the current frame. .

[0027] S3. Based on the regularized point cloud, a fractionally weighted generalized iterative nearest-point algorithm is used to perform dual-mode camera pose tracking to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode.

[0028] The fractional weighted generalized iterative nearest-point algorithm refers to an improved ICP algorithm that uses a fractional residual cost function to enhance registration robustness. The final camera pose refers to the position and orientation of the camera in the current frame within the global coordinate system, obtained after dual-mode tracking; for example, the pose obtained after frame-to-frame and frame-to-map tracking. Frame-to-frame tracking mode refers to the rapid estimation of camera pose by registering point clouds between adjacent frames; for example, registering the regularized point cloud of the current frame with the point cloud of the previous frame to obtain a temporary pose. Frame-to-map tracking mode refers to registering the current frame's point cloud with the trackable set in the global map to correct drift; for example, using... As initial values, the current frame point cloud is compared with the traceable set. Registration obtained .

[0029] S4. Based on the final camera pose, calculate the overlap rate between the current frame and the trackable set. When the overlap rate is lower than a set threshold or reaches a fixed frame interval, determine the current frame as a key frame, and update the global Gaussian map or the trackable set according to the type of the key frame.

[0030] Here, the traceable set refers to a local map composed of point clouds of historical keyframes, used for frame-to-map tracking; for example, a set This includes a downsampled point cloud containing recently tracked keyframes. The overlap rate refers to the ratio of the current frame's point cloud to the number of valid corresponding points in the trackable set; for example, the overlap rate can be calculated using a formula. Setting a threshold refers to a preset lower limit for the overlap rate, used to trigger keyframe selection; for example, a threshold... Set to 0.3, when The tracking keyframes are triggered periodically. The fixed frame interval refers to the preset number of frames between keyframe generation intervals to ensure map update frequency; for example, every... The frame forces the generation of a mapping keyframe. A keyframe is a frame selected based on overlap ratio or a fixed interval for updating the map; for example, when... Or the frame interval reaches At that time, the current frame is selected as the keyframe. The type of keyframe refers to its classification, including tracking keyframes and mapping keyframes; for example, tracking keyframes are used to update the trackable set, while mapping keyframes only update the global map. A global Gaussian map refers to a global 3D map composed of the Gaussian point set of all keyframes; for example, a map... It includes parameters for the position, covariance, color, and opacity of each Gaussian point.

[0031] S5. Render and calculate the loss of the global Gaussian map based on 3D Gaussian splashing, and perform collaborative optimization of camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map.

[0032] Two-stage pose optimization refers to a pose refinement method that first performs joint rigid body transformation during the mapping process, and then performs decoupled optimization. For example, in stage one, SE(3) joint correction is performed, and in stage two, pose and Gaussian parameters are alternately optimized. Camera pose refers to the rotation and translation parameters of the camera in the global coordinate system. For example, pose This represents the camera's 6 degrees of freedom pose. Gaussian parameters refer to the properties of each Gaussian point in a 3D Gaussian point set, including position, covariance, color, and opacity; for example, Gaussian points include the mean. Covariance ,color and transparency The optimized global Gaussian map refers to a global map updated after two-stage pose optimization, exhibiting higher geometric consistency; for example, a map updated by minimizing the loss function to adjust the Gaussian parameters and pose. .

[0033] The technical solution of this embodiment completes the depth information through Gaussian process regression, optimizes the pose through dual-mode tracking, and improves the computational efficiency by combining 3DGS and two-stage optimization. It effectively solves the problems of depth completion, pose stability and real-time performance of existing RGB-D SLAM in complex environments, and can meet the real-time reconstruction needs in complex scenes, and improve the robustness and accuracy of RGB-D SLAM in complex environments.

[0034] In one alternative approach, the step of performing ellipsoidal regularization on the original point cloud to obtain a regularized point cloud includes: Calculate the covariance matrix of any point in the local neighborhood of any point in the original point cloud.

[0035] The local neighborhood refers to the set of neighboring points around a given point in a point cloud, used to calculate local features; for example, for a point... The covariance matrix is ​​calculated by selecting its k nearest neighbors. The covariance matrix is ​​a matrix that describes the local neighborhood distribution characteristics of a point cloud; for example, the point... covariance matrix It represents the spatial distribution of its neighboring points.

[0036] The covariance matrix is ​​decomposed into a direction matrix and a scale matrix consisting of eigenvalues, and the median of all eigenvalues ​​is calculated.

[0037] Here, the direction matrix refers to the rotation matrix obtained after eigenvalue decomposition of the covariance matrix, representing the local principal direction; for example, the matrix after covariance matrix decomposition. The eigenvalues ​​represent the orientation of the point cloud neighborhood. Eigenvalues ​​are the eigenvalues ​​obtained from the eigenvalue decomposition of the covariance matrix, representing the scale of the neighborhood along each principal direction; for example, eigenvalues... The variance of the corresponding neighborhood along the three principal axes. The scale matrix is ​​a diagonal matrix composed of eigenvalues, representing the scale information of the neighborhood; for example, the scale matrix... The median is the middle value in a set of values ​​arranged in ascending order; for example, when calculating a set of eigenvalues. the median of .

[0038] The scale matrix is ​​normalized by dividing it by the median to obtain a normalized scale matrix. The regularized covariance matrix of any point is then reconstructed using the direction matrix and the normalized scale matrix.

[0039] The normalized scale matrix refers to the matrix obtained by dividing the scale matrix by the median of its eigenvalues; it is used for scale unification. For example... The regularized covariance matrix refers to the covariance matrix reconstructed using the direction matrix and the normalized scaling matrix; for example, .

[0040] The regularized point cloud is obtained based on the regularized covariance matrix of all points in the original point cloud.

[0041] Among the above-mentioned optional methods, the processing accuracy and robustness of the original point cloud are further improved. Ellipsoidal regularization effectively reduces the impact of noise, making the point cloud data more uniform and consistent with the structure of the real scene, providing high-quality input for subsequent depth completion and pose tracking.

[0042] In one alternative approach, the step of using Gaussian process regression to complete the missing depth regions in the depth map of the current frame and generating a completed point cloud includes: The effective depth pixels and corresponding depth observations are extracted from the depth map to form a training dataset, and the pixels in the missing depth regions of the depth map are used as the prediction point set.

[0043] Here, effective depth pixels refer to pixels in the depth map that have effective depth values; for example, depth map. Pixels that are neither zero nor outliers. A depth observation refers to the actual depth value corresponding to a valid depth pixel; for example, the depth value of a valid pixel. As training data. The training dataset refers to: the set of valid depth pixels and their depth values ​​used for Gaussian process regression; for example, from... 10% of the effective pixels are selected to form the training set. The prediction point set refers to the set of pixels in the depth map representing the missing depth regions, with the depth values ​​to be predicted; for example, the set of pixels representing the missing regions. Missing depth region pixels refer to the pixel locations in the depth map where the depth value is missing; for example, holes caused by noise or occlusion.

[0044] Construct a spatial-luminance product kernel function that combines spatial distance information and RGB luminance information, and use the spatial-luminance product kernel function to calculate the autocovariance matrix of the training dataset and the cross-covariance matrix between the training dataset and the prediction point set.

[0045] Spatial distance information refers to the Euclidean distance between pixels in the image space; for example, the distance in the kernel function. Distance calculation based on pixel coordinates. RGB brightness information refers to the brightness or color value of pixels in an RGB image; for example, the brightness or color value of pixels in a kernel function. Calculations are based on pixel brightness differences. The spatial-luminance product kernel function refers to a kernel function that combines spatial distance and RGB brightness, used for Gaussian process regression; for example, , , The self-covariance matrix refers to the covariance matrix among pixels within the training dataset; for example, a matrix... The cross-covariance matrix refers to the covariance matrix between the training dataset and the prediction point set; for example, the covariance matrix... .

[0046] Based on the autocovariance matrix, the crosscovariance matrix, and the depth observations, the mean depth prediction of the predicted point set is calculated, and the missing depth region is filled in based on the mean depth prediction to generate the filled point cloud.

[0047] Here, the depth prediction mean refers to the average value of the missing depth predicted by Gaussian process regression; for example, according to , calculate As the depth of completion.

[0048] Among the above-mentioned optional methods, the integrity and accuracy of depth information are further enhanced. By using Gaussian process regression combined with spatial and brightness information, missing depth regions are accurately predicted, generating complete and reliable point clouds, laying the foundation for building a continuous and consistent 3D scene.

[0049] In one alternative approach, the step of performing dual-mode camera pose tracking using a fractionally weighted generalized iterative nearest-neighbor algorithm based on the regularized point cloud to obtain the final camera pose of the current frame includes: Using the frame-to-frame tracking mode, the regularized point cloud of the current frame and the regularized point cloud of the previous frame are used as inputs, and the temporary camera pose of the current frame is calculated by the fractional weighted generalized iterative nearest point algorithm.

[0050] Temporary camera pose refers to the initial camera pose estimated in frame-to-frame tracking mode; for example, the pose is obtained by registering the point clouds of the current frame with those of the previous frame. .

[0051] Using the frame-to-map tracking mode, the temporary camera pose is used as the initial value for frame-to-map registration. The regularized point cloud of the current frame is registered with the trackable set. The initial value is iteratively optimized using the fractionally weighted generalized iterative nearest-point algorithm to calculate the final camera pose of the current frame.

[0052] Here, the initial value for frame-to-map registration refers to the initial pose value used during frame-to-map tracking; for example, the temporary pose... Used as the initial value for frame-to-map optimization.

[0053] Among the above-mentioned optional methods, the stability and reliability of camera pose tracking are further improved. The dual-mode tracking combines the advantages of frame-to-frame and frame-to-map, quickly responds to scene changes and accurately matches, effectively suppresses accumulated errors, and ensures the accuracy of real-time positioning.

[0054] In one alternative approach, the step of calculating the overlap rate between the current frame and the trackable set based on the final camera pose includes: Based on the final camera pose, the regularized point cloud of the current frame is registered with the trackable set, and a set of valid corresponding points that satisfy geometric consistency is obtained after registration.

[0055] Geometric consistency refers to the degree to which corresponding points in the point cloud satisfy geometric constraints after registration; for example, point pairs with a distance less than a threshold after registration are considered valid correspondences. The set of valid corresponding points refers to the set of point pairs that satisfy geometric consistency after registration; for example, a set... It contains the current frame point cloud and the points that match in the trackable set.

[0056] The number of points in the set of valid corresponding points is counted as the number of valid corresponding points, and the number of source points in the regularized point cloud of the current frame used for registration is obtained.

[0057] Here, the number of valid correspondences refers to the number of points in the set of valid correspondences; for example... This indicates the number of valid matching points. The source point count refers to the number of points in the current frame's point cloud used for registration; for example, This indicates the number of points in the current frame that are involved in registration.

[0058] The ratio of the number of valid corresponding points to the number of source points is determined as the overlap rate between the current frame and the traceable set.

[0059] Among the above-mentioned optional methods, the key frame selection strategy and map update mechanism have been further optimized. Based on the overlap rate, the tracking or mapping needs are dynamically judged, key frames are accurately selected and the map is updated efficiently to ensure the real-time performance and global consistency of map construction.

[0060] In one alternative approach, the step of identifying the current frame as a keyframe and updating the global Gaussian map or the traceable set based on the type of the keyframe includes: When the overlap rate is lower than the set threshold, the current frame is determined as a tracking key frame; when the frame interval between the previous mapping key frame and the current frame reaches a fixed frame interval, the current frame is determined as a mapping key frame; wherein, if the overlap rate is lower than the set threshold and the frame interval between the previous mapping key frame and the current frame reaches the fixed frame interval, the current frame is determined as the tracking key frame.

[0061] Here, tracking keyframes refer to keyframes selected due to insufficient overlap, used to update the trackable set; for example, when At this time, the current frame is set as the tracking keyframe. The previous mapping keyframe refers to the frame that was most recently selected as the mapping keyframe; for example, the frame in the sequence... The frame is the previous mapping keyframe. The frame interval refers to the difference in the number of frames between the current frame and the previous mapping keyframe; for example, if the current frame is the [number]th keyframe... Frame, the previous mapping keyframe was frame number 1. Frame, then the interval is Mapping keyframes refer to keyframes selected at fixed frame intervals, used solely for updating the global map; for example, when the frame interval reaches a certain value... At that time, the current frame is set as the keyframe for mapping.

[0062] When the current frame is determined to be a tracking keyframe, the regularized point cloud of the current frame is added to the trackable set, and the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map; when the current frame is determined to be a mapping keyframe, the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map.

[0063] Among the above-mentioned optional methods, the keyframe processing flow is further refined. For different types of tracking and mapping keyframes, they are reasonably allocated to the trackable set or the global Gaussian map to improve map maintenance efficiency and enhance adaptability and resource utilization.

[0064] In one alternative approach, the steps of rendering and loss calculation of the global Gaussian map based on 3D Gaussian splashing, and co-optimizing the camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map include: Using 3D Gaussian splashing technology, the global Gaussian map is rendered in a differentiable manner to generate the predicted RGB image and predicted depth image of the key frame.

[0065] Differentiable rendering refers to generating images using differentiable graphics methods, which facilitates gradient optimization; for example, using 3D Gaussian splashing technology to render predicted images.

[0066] Using the real RGB image and real depth image of the keyframe as supervision, and combining the predicted RGB image and the predicted depth image, the color loss term and the depth loss term are calculated, and a joint loss function is constructed.

[0067] Here, the predicted RGB image refers to an RGB image generated through differentiable rendering; for example, the predicted RGB image of a keyframe obtained from rendering a global Gaussian map. A predicted depth image refers to a depth image generated through differentiable rendering; for example, the predicted depth image of a keyframe rendered from the given depth. A true RGB image refers to the actual RGB image captured for a keyframe; for example, the real image corresponding to a keyframe. True depth images refer to the actual depth images captured for keyframes; for example, the true depth map corresponding to a keyframe. The color loss term refers to a measure of the difference between the predicted RGB image and the true RGB image; for example, It includes L1 loss and SSIM loss. The depth loss term refers to a measure of the difference between the predicted depth image and the true depth image; for example, This is the L1 loss. The joint loss function is a weighted combination of the color loss and the depth loss; for example, .

[0068] The joint loss function is minimized through a two-stage pose optimization process. In the first stage, a joint rigid body transformation is performed on the camera pose of the keyframe and all Gaussian point sets in the view of the keyframe. In the second stage, the Gaussian parameters are first fixed and the camera pose is optimized, and then the Gaussian parameters are optimized while the camera pose is fixed.

[0069] Here, joint rigid body transformation refers to the rigid body transformation simultaneously applied to the camera pose and Gaussian point set in the first stage of the two-stage optimization; for example, through the formula... The pose and Gaussian center are updated consistently.

[0070] Based on the results of the two-stage pose optimization process, the Gaussian parameters in the global Gaussian map and the camera pose of the keyframes are updated to obtain the optimized global Gaussian map.

[0071] The result of the two-stage pose optimization process refers to the updated camera pose and Gaussian parameters obtained after optimization; for example, after optimization in stages one and two, a more accurate pose and Gaussian map are obtained.

[0072] Among the above-mentioned optional methods, the accuracy and efficiency of the global Gaussian map are further improved. By combining three-dimensional Gaussian splashing differentiable rendering with two-stage optimization, the camera pose and Gaussian parameters are jointly adjusted to achieve efficient collaborative optimization and ensure the visual quality and geometric accuracy of the reconstructed scene.

[0073] It should be noted that the method in this embodiment achieves efficient and stable 3D scene reconstruction and camera tracking by integrating modules such as Gaussian process regression depth completion, keyframe management, global Gaussian map construction, and camera pose optimization. Compared with existing technologies, this invention can maintain continuous and complete geometric information even in the presence of depth loss, weak texture, or occlusion, and significantly improves the accuracy and robustness of trajectory estimation through staged pose optimization, thereby enhancing the overall reconstruction quality. While ensuring real-time performance at 30 FPS, it also balances high accuracy in pose estimation and high fidelity in map construction, making it widely applicable to scenarios such as robot autonomous navigation, augmented reality, and virtual reality. Figure 2 As shown, the main processing flow includes: (1) Tracking process: Input RGB image and depth map, downsample the depth map to generate original point cloud, and perform ellipsoidal regularization on the original point cloud to obtain regularized point cloud; for depth missing regions, Gaussian process regression is used to perform depth completion under space-luminance kernel constraints to generate complete point cloud, and the complete point cloud is fused with the original point cloud to form complete point cloud. Subsequently, based on the regularized point cloud, fractional weighted generalized iterative nearest point algorithm is used for dual-mode camera pose tracking: first, temporary camera pose is quickly obtained in frame-to-frame tracking mode, and then frame-to-map tracking is performed on the trackable set with the result as the initial value to obtain the globally consistent final camera pose, which is used as the initial value for mapping optimization.

[0074] (2) Keyframe Management: Keyframes are dynamically selected in consecutive frames based on criteria such as overlap rate and fixed frame interval. When the overlap rate is lower than a set threshold, the current frame is determined as a tracking keyframe; when the fixed frame interval is reached, the current frame is determined as a mapping keyframe. Tracking keyframes are used to update the trackable set to stabilize subsequent pose estimation, and also to convert its complete point cloud into a Gaussian point set and add it to the global Gaussian map; mapping keyframes are only used to convert the complete point cloud into a Gaussian point set and add it to the global Gaussian map, thereby achieving a balance between computational overhead and map accuracy.

[0075] (3) Construction of 3D Gaussian map: The complete point cloud of the key frame is converted into a Gaussian point set and added to a global Gaussian map; each Gaussian contains parameters such as position, covariance (after ellipsoidal regularization), color and opacity. This explicit representation is more compact and more conducive to differentiable rendering and parallel optimization, which can improve appearance and geometric reproducibility while ensuring storage efficiency.

[0076] (4) Camera pose optimization: A two-stage pose optimization strategy is adopted. In the first stage, a joint rigid body transformation is performed on the camera pose of the key frame and all Gaussian point sets in its field of view to quickly absorb the tracking residual. In the second stage, decoupling optimization is adopted. First, the Gaussian parameters are fixed to optimize the camera pose, and then the Gaussian parameters are fixed to optimize the camera pose to obtain higher trajectory accuracy and global consistency.

[0077] To better illustrate the technical solution of this embodiment, the following examples are used for explanation: S10. Downsample the depth map of the current frame in the RGB-D sequence to generate the original point cloud, and then perform ellipsoidal regularization on the original point cloud to obtain the regularized point cloud. Specifically, the depth map of the i-th frame... Uniform downsampling yields the original point cloud used for tracking. ,in This represents the depth map of the current frame, where each pixel value represents the distance information of the corresponding scene point; This represents the set of 3D points generated by backprojection of effective depth pixels. The covariance matrix of the source and target points is calculated in the local neighborhood. and ,in Represents the local covariance matrix of the source point cloud. Let represent the local covariance matrix of the target point cloud. To obtain a stable scale representation, the covariance matrix is ​​decomposed into the following form: in, This is the direction matrix, representing the main direction of the local point cloud distribution; The scaling matrix is ​​composed of eigenvalues. The structure represents the scale information of the neighborhood along the three principal axes. Ellipsoidal regularization is applied to the scale to unify the dimensions without altering the principal axis directions and relative shape. in, Represents the set of eigenvalues the median; This is the normalized scaling matrix. This process improves the numerical stability and convergence of subsequent 3D Gaussian splash optimization.

[0078] S20. Gaussian process regression is used to complete the missing depth regions in the current frame's depth map, generating a completed point cloud. From the depth map... The training dataset is constructed by extracting valid pixels and their depth observations. ,in The set of coordinates of effective depth pixels. The corresponding depth observations are used; pixels in the missing depth regions of the depth map are used as the prediction point set. Deep completion and confidence assessment are performed using the predicted mean and variance: The kernel function takes the form of a spatial product of luminance: in, In the formula, For kernel function components based on spatial distance information, The kernel function components are based on RGB brightness information; and These are the kernel function parameters. Derived from the predicted mean. Generate a complete point set The completed point cloud is then merged with the original point cloud to form the complete point cloud for the current frame. It should be noted that the tracking process only uses the original point cloud. The point set is used to complete the mapping optimization process.

[0079] S30. Based on regularized point clouds, a fractionally weighted generalized iterative nearest-point algorithm is used for dual-mode camera pose tracking. Rigid body transformation. The registration residual applied to the source point is defined as: in, and These are the source point and the target point, respectively. The ordinary generalized iterative nearest-point algorithm solves for the pose by minimizing the Mahalanobis distance: The formula for calculating the joint covariance matrix is: in, For transformation The rotating part, This represents a small noise term. To improve the robustness of the algorithm in real-world scenarios, a fractionally weighted generalized iterative nearest-point algorithm is adopted. First, the inverse of the joint covariance matrix is ​​normalized using the norm to obtain the accuracy matrix: in, Here are the diagonal regularization parameters. Then, the quadratic cost is replaced with a fractional residual cost function: in, Control the residual saturation rate. In frame-to-frame tracking mode, use the regularized point cloud of the current frame. Regularized point cloud compared to the previous frame As input, the temporary camera pose is obtained by minimizing the above cost function: S40. In frame-to-map tracking mode, using temporary camera pose. As the initial value for frame-to-map registration, the regularized point cloud of the current frame is used. With traceable sets Perform registration to obtain the final camera pose for the current frame: in, It is a traceable set, containing point clouds of historical keyframes; This is a global Gaussian map.

[0080] Based on the final camera pose, calculate the overlap rate between the current frame and the trackable set: in, To satisfy the set of valid corresponding points that are geometrically consistent, The number of source points used for registration. When the overlap rate... Below the set threshold Or reach a fixed frame interval When this happens, the current frame is designated as a keyframe. Keyframes are divided into two categories: tracking keyframes and mapping keyframes. If the frame interval between the current frame and the previous mapping keyframe reaches a certain threshold, then the current frame is designated as the tracking keyframe; If a frame meets both conditions, it is designated as a mapping keyframe; if both conditions are met, it is prioritized as a tracking keyframe. When the current frame is designated as a tracking keyframe, its regularized point cloud is added to the trackable set. The complete point cloud is then converted into a Gaussian point set and added to the global Gaussian map. When a point cloud is identified as a mapping keyframe, only the complete point cloud is converted into a Gaussian point set and added. .

[0081] The S50 renders and calculates the loss of a global Gaussian map based on 3D Gaussian splashing, and performs collaborative optimization of camera pose and Gaussian parameters through two-stage pose optimization. For each pixel, a set of visible Gaussian parameters is combined according to depth from near to far. The system generates predicted RGB and predicted depth images through differentiable rendering. in, For color values, Center depth, Transmission weights are used. The keyframes are represented by their actual RGB images. and true depth images As a supervisor, a joint loss function is constructed. The color loss term is: The depth loss term is: The total loss function is: in, , , These are the weighting coefficients.

[0082] S60. Camera Pose Resolution Jacobian Calculation: To avoid the runtime overhead of automatic differentiation, an explicit analytical method is used in CUDA to calculate the Jacobian matrix of the loss function with respect to camera pose. In Lie algebra space, it is represented as Its derivative on the manifold is defined as: in, and These are the exponential and logarithmic mappings over Lie algebras, respectively. This is a composition operator on Lie groups. Its definition provides the foundation for accurate gradient computation on manifolds.

[0083] The chain rule can be used to derive the relationship between Gaussian parameters in image space and camera pose. The derivative of the transformation from world coordinates to camera coordinates. The projection point of the Gaussian center onto the image plane. The derivative with respect to pose is: in, The three-dimensional position of the Gaussian center in the camera coordinate system. The projection of the Gaussian covariance onto the image plane. The derivative with respect to pose is more complex, involving the projection Jacobian. and affine transformation matrix : in, and These are the intermediate variables involved in projecting a 3D Gaussian ellipse onto the image plane.

[0084] The key geometric Jacobian matrix in the above formula can be explicitly given. Gaussian center. relative pose The derivative is: in, It is a 3x3 identity matrix. For vectors The antisymmetric matrix. Affine transformation matrix. The derivative with respect to pose involves the antisymmetric matrix of its column vectors: in, Representation matrix The List, Similarly, it represents the antisymmetric matrix of vectors.

[0085] Based on equations (21) to (24), the gradient of each pixel can be efficiently constructed in the CUDA kernel, and a 6x6 approximate Hessian matrix with respect to the camera pose can be accumulated, thereby using optimization algorithms such as the Gauss-Newton method to quickly solve the pose increment.

[0086] S70. Minimize the above loss function through a two-stage pose optimization process. Stage determination is based on the cumulative number of observations in keyframes. and unit pixel loss :when or Then, proceed to Phase 1, performing a joint rigid body transformation on the camera pose and all Gaussian point sets within the viewport: in, For the Lie algebra increment. When and Then, proceed to stage two: first, fix the Gaussian parameters and optimize the camera pose; then, fix the camera pose and optimize the Gaussian parameters (including the center). Covariance ,color Opacity ): An adaptive learning rate mechanism is introduced for the Gaussian points generated by the completion, and the actual learning rate is: The confidence weights are determined by the variance of the Gaussian process regression predictions. in, This represents the cumulative number of times the Gaussian element is visible. Through the above process, the Gaussian parameters and camera poses of keyframes in the global Gaussian map are updated, ultimately resulting in an optimized global Gaussian map.

[0087] It should be noted that the present invention verifies the effectiveness of the technical solution in the RGB-DSLAM task through the following experimental setup and result analysis. The experiment is divided into three parts: dataset and sequence selection, hardware and implementation configuration, and comparison methods and evaluation metrics, comprehensively evaluating the tracking accuracy, reconstruction quality, and real-time performance of the method. Specifically: ① Experimental setup: Experiments were conducted using the widely used TUM RGB-D dataset and the Replica dataset for validation. The TUM RGB-D dataset contains standard sequences of real-world indoor scenes (such as fr1 / desk, fr2 / xyz, fr3 / office, etc.) to evaluate the robustness and generalization ability of tracking and reconstruction. The Replica dataset provides high-quality synthetic indoor scenes (such as the Room and Office series) to evaluate reconstruction accuracy and appearance realism. All sequences were input at full resolution, using calibrated camera intrinsics, without preheating frames, and processed online strictly in chronological order.

[0088] The experimental hardware platform was configured with an NVIDIA RTX 4080 graphics card (16GB VRAM), an Intel Core i9-14900KF processor, and 64GB of RAM. The tracking and mapping processes were executed in parallel using dual threads, sharing a global Gaussian map. The mapping thread employed CUDA rasterization and analytical Jacobian calculations to avoid the overhead of automatic differentiation. In the point cloud processing stage, the original point cloud was obtained by uniformly downsampling the depth map for tracking, while the completed point cloud generated through Gaussian process regression was used only for mapping optimization. The keyframe triggering strategy was based on an overlap rate threshold and a fixed frame interval. The mapping thread randomly sampled keyframes for optimization, and the switching between the two-stage pose optimization was determined based on an observation count threshold and a unit pixel residual threshold.

[0089] Gaussian process regression employs a spatial-luminosity product kernel function. Training pixels are randomly sampled from the effective depth region (approximately 10%), and when the missing region is too large, it backtracks to fill at the global mean depth. In the fractional-weighted generalized iterative nearest neighbor algorithm, a diagonal regularization term is added to the joint covariance and Frobenius norm normalization is performed. The fractional residual parameter is used to control the saturation rate. Normal consistency and bidirectional checks are introduced during the nearest neighbor establishment process to improve matching reliability. Regarding the learning rate setting, the base learning rate is [value missing], and an adaptive learning rate mask is used for Gaussian points completed by Gaussian process regression, with interval pruning to maintain numerical stability.

[0090] The comparison method selected representative SLAM schemes based on 3D Gaussian splashing, including Photo-SLAM, GS-ICP-SLAM, SplaTAM, MonoGS, Splat-SLAM, and RTG-SLAM. Evaluation metrics covered three aspects: pose accuracy was measured by the root mean square value of the absolute trajectory error (ATE RMSE); rendering quality was evaluated using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual patch similarity (LPIPS); and real-time performance was measured by end-to-end frame rate (FPS).

[0091] ② Gaussian process regression completion effect: like Figure 3 As shown, in scenarios with large-area depth loss and boundary noise, the Gaussian process regression completion method used in this embodiment achieves continuous and geometrically consistent reconstruction results on a single frame. Specifically, it uses RGB images to provide appearance and spatial priors ( Figure 3 a) The original depth map has obvious holes and missing measurements. Figure 3 b), the initial point cloud obtained by back projection only contains effective depth pixels ( Figure 3 c). Based on this, the depth mean predicted by Gaussian process regression is used to complete the missing regions, generating a complete point cloud. Figure 3 d), which is then used for the construction and optimization of global Gaussian maps.

[0092] The technical effects of this completion operation are reflected in four aspects: First, holes in the depth map are effectively filled, significantly enhancing the connectivity of the scene surface; second, the completion result maintains consistency with RGB textures at object edges and in high-contrast areas, reducing cracks and artifacts; third, due to more complete geometric information during the mapping stage, the joint loss of color and depth decreases faster, thereby improving rendering quality; finally, an adaptive learning rate mechanism based on uncertainty and cumulative observation count is introduced for the Gaussian points generated by the completion. The learning rate is amplified in the initial stage when observations are insufficient and uncertainty is low, and then decays exponentially to the baseline value as the number of observations increases, effectively avoiding oscillations in the later optimization process and ensuring overall stability.

[0093] ③ Tracking performance analysis: The tracking results on the TUM RGB-D dataset are shown in Table 1. In this embodiment, the sequences were continuously processed at a frame rate of 30 FPS in the real-time running group (FPS≥20). The absolute trajectory errors on the fr1 / desk, fr2 / xyz, and fr3 / office sequences were 1.83, 0.83, and 1.70, respectively, with an average of 1.45, which is significantly lower than GS-ICP-SLAM's 2.40 and close to Photo-SLAM's 1.33 (frame rate 24 FPS). Although offline methods such as RTG-SLAM have an average error of 0.96, their frame rate is only 5.10 FPS, which cannot meet the real-time requirements.

[0094] Table 1: Tracking Accuracy of TUM RGB-D Dataset The tracking accuracy on the Replica dataset is shown in Table 2. This embodiment achieves the lowest or near-lowest error values ​​in multiple room and office scenes (R0–R2 and OF0–OF4), with a mean absolute trajectory error of 0.12, while maintaining a real-time processing capability of 30 FPS, which is superior to GS-ICP-SLAM (0.18, 30 FPS) and RTG-SLAM (0.19, 5.3 FPS). The results show that this embodiment achieves real-time high-precision tracking on both datasets, significantly reducing trajectory error while maintaining a 30 FPS frame rate, demonstrating excellent tracking stability.

[0095] Table 2: Tracking Accuracy of the Replica Dataset ④ Evaluation of mapping results: The mapping results on the TUM RGB-D dataset are shown in Table 3. This embodiment maintains high values ​​for both PSNR and SSIM, while exhibiting a low LPIPS value, indicating that the generated images are closer to real-world observations in terms of detail fidelity and perceptual quality. Specifically, on the fr1 / desk, fr2 / xyz, and fr3 / office sequences, the average PSNR of this method is 24.67 dB, the average SSIM is 0.830, and the average LPIPS is 0.156, all superior to the comparison schemes. Figure 4 As shown, in the fr1 / desk and fr3 / office scenes, this method can effectively restore texture continuity, avoid edge blurring and local artifacts, and make the overall rendering effect closer to the real scene.

[0096] Table 3: Rendering results of the TUM RGB-D dataset The mapping results on the Replica dataset are shown in Table 4. Our method achieves an average PSNR of 39.72 dB, an average SSIM of 0.979, and an average LPIPS of 0.032 across multiple scenarios, all of which are optimal results. Figure 5 As shown, even in scenes with complex lighting and rich geometric details, this method can still maintain clear structure and accurate texture, avoiding the distortion and damage common in contrast methods at boundaries and in reflection areas. In summary, this method significantly improves the global consistency and detail restoration capability of the reconstructed map while maintaining real-time performance.

[0097] Table 4: Rendering effect of the Replica dataset The experimental results above demonstrate that the method proposed in this embodiment outperforms existing mainstream solutions in terms of tracking accuracy, reconstruction quality, and real-time performance, and can effectively meet the requirements of SLAM tasks in complex environments.

[0098] Figure 6 A schematic diagram of an embodiment of an RGB-D SLAM system 200 with 3DGS and two-stage pose optimization provided by the present invention is shown. Figure 6 As shown, the system 200 includes: a processing module 210, a fusion module 220, a tracking module 230, an update module 240, and an optimization module 250; The processing module 210 is used to: perform downsampling processing on the depth map of the current frame in the RGB-D sequence to generate an original point cloud, and perform ellipsoidal regularization processing on the original point cloud to obtain a regularized point cloud; The fusion module 220 is used to: use Gaussian process regression to complete the missing depth regions in the depth map of the current frame, generate a completed point cloud, and fuse the completed point cloud with the original point cloud to form the complete point cloud of the current frame; The tracking module 230 is used to: perform dual-mode camera pose tracking based on the regularized point cloud using a fractionally weighted generalized iterative nearest-point algorithm to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode; The update module 240 is used to: calculate the overlap rate between the current frame and the trackable set based on the final camera pose; when the overlap rate is lower than a set threshold or reaches a fixed frame interval, determine the current frame as a key frame; and update the global Gaussian map or the trackable set according to the type of the key frame. The optimization module 250 is used to: render and calculate the loss of the global Gaussian map based on three-dimensional Gaussian splashing, and perform collaborative optimization of camera pose and Gaussian parameters through two-stage pose optimization to obtain an optimized global Gaussian map.

[0099] In an alternative embodiment, the processing module 210 is specifically used for: Calculate the covariance matrix of any point in the local neighborhood of any point in the original point cloud; The covariance matrix is ​​decomposed into a direction matrix and a scale matrix composed of eigenvalues, and the median of all eigenvalues ​​is calculated. The scale matrix is ​​normalized by dividing it by the median to obtain a normalized scale matrix. The regularized covariance matrix of any point is then reconstructed using the direction matrix and the normalized scale matrix. The regularized point cloud is obtained based on the regularized covariance matrix of all points in the original point cloud.

[0100] In an alternative embodiment, the fusion module 220 is specifically used for: The effective depth pixels and corresponding depth observations are extracted from the depth map to form a training dataset, and the pixels of the missing depth regions in the depth map are used as the prediction point set. Construct a spatial-luminance product kernel function that combines spatial distance information and RGB luminance information, and use the spatial-luminance product kernel function to calculate the autocovariance matrix of the training dataset and the cross-covariance matrix between the training dataset and the prediction point set; Based on the autocovariance matrix, the crosscovariance matrix, and the depth observations, the mean depth prediction of the predicted point set is calculated, and the missing depth region is filled in based on the mean depth prediction to generate the filled point cloud.

[0101] In an alternative embodiment, the tracking module 230 is specifically used for: Using the frame-to-frame tracking mode, the regularized point cloud of the current frame and the regularized point cloud of the previous frame are used as inputs, and the temporary camera pose of the current frame is calculated by the fraction-weighted generalized iterative nearest point algorithm. Using the frame-to-map tracking mode, the temporary camera pose is used as the initial value for frame-to-map registration. The regularized point cloud of the current frame is registered with the trackable set. The initial value is iteratively optimized using the fractionally weighted generalized iterative nearest-point algorithm to calculate the final camera pose of the current frame.

[0102] In an alternative embodiment, the update module 240 is specifically used for: Based on the final camera pose, the regularized point cloud of the current frame is registered with the trackable set, and a set of valid corresponding points that satisfy geometric consistency is obtained after registration. The number of points in the set of valid corresponding points is counted as the number of valid corresponding points, and the number of source points in the regularized point cloud of the current frame used for registration is obtained. The ratio of the number of valid corresponding points to the number of source points is determined as the overlap rate between the current frame and the traceable set.

[0103] In an alternative embodiment, the update module 240 is specifically used for: When the overlap rate is lower than the set threshold, the current frame is determined as a tracking key frame; when the frame interval between the previous mapping key frame and the current frame reaches a fixed frame interval, the current frame is determined as a mapping key frame; wherein, if the overlap rate is lower than the set threshold and the frame interval between the previous mapping key frame and the current frame reaches the fixed frame interval, then the current frame is determined as the tracking key frame. When the current frame is determined to be a tracking keyframe, the regularized point cloud of the current frame is added to the trackable set, and the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map; when the current frame is determined to be a mapping keyframe, the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map.

[0104] In an alternative embodiment, the optimization module 250 is specifically used for: Using 3D Gaussian splashing technology, the global Gaussian map is rendered in a differentiable manner to generate the predicted RGB image and predicted depth image of the key frame; Using the real RGB image and real depth image of the keyframe as supervision, and combining the predicted RGB image and the predicted depth image, the color loss term and the depth loss term are calculated and a joint loss function is constructed; The joint loss function is minimized through a two-stage pose optimization process. In the first stage, a joint rigid body transformation is performed on the camera pose of the key frame and all Gaussian point sets in the view of the key frame. In the second stage, the Gaussian parameters are first fixed and the camera pose is optimized, and then the Gaussian parameters are optimized while the camera pose is fixed. Based on the results of the two-stage pose optimization process, the Gaussian parameters in the global Gaussian map and the camera pose of the keyframes are updated to obtain the optimized global Gaussian map.

[0105] It should be noted that the beneficial effects of the 3DGS and two-stage pose optimization RGB-D SLAM system 200 provided in the above embodiments are the same as those of the 3DGS and two-stage pose optimization RGB-D SLAM method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0106] The 3DGS and two-stage pose optimization RGB-D SLAM system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the 3DGS and two-stage pose optimization RGB-D SLAM system 200 of the present invention is an application software that can be used to execute the corresponding steps in the 3DGS and two-stage pose optimization RGB-D SLAM method of the present invention.

[0107] In some embodiments, the 3DGS and two-stage attitude optimization RGB-D SLAM system 200 of the present invention can be implemented in a combination of hardware and software. As an example, the 3DGS and two-stage attitude optimization RGB-D SLAM system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the 3DGS and two-stage attitude optimization RGB-D SLAM method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0108] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0109] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned 3DGS and two-stage pose optimization RGB-D SLAM methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the 3DGS and two-stage pose optimization RGB-D SLAM method shown in any embodiment of the present invention by calling the computer program.

[0110] In one alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0111] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0112] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0114] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0115] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0116] It should be noted that, Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0117] An embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-mentioned RGB-D SLAM methods for 3DGS and two-stage pose optimization.

[0118] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0119] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0120] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0121] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A 3DGS and two-stage pose optimization RGB-D SLAM method, characterized in that, include: The depth map of the current frame in the RGB-D sequence is downsampled to generate the original point cloud, and the original point cloud is ellipsoidally regularized to obtain the regularized point cloud. Gaussian process regression is used to complete the missing depth regions in the depth map of the current frame, generating a completed point cloud, and then the completed point cloud is fused with the original point cloud to form the complete point cloud of the current frame; Based on the regularized point cloud, a fractionally weighted generalized iterative nearest-point algorithm is used to perform dual-mode camera pose tracking to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode. Based on the final camera pose, the overlap rate between the current frame and the trackable set is calculated. When the overlap rate is lower than a set threshold or reaches a fixed frame interval, the current frame is determined as a key frame, and the global Gaussian map or the trackable set is updated according to the type of the key frame. The global Gaussian map is rendered and loss is calculated based on 3D Gaussian splashing, and the camera pose and Gaussian parameters are optimized in a coordinated manner through two-stage pose optimization to obtain the optimized global Gaussian map.

2. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 1, characterized in that, The step of performing ellipsoidal regularization on the original point cloud to obtain a regularized point cloud includes: Calculate the covariance matrix of any point in the local neighborhood of any point in the original point cloud; The covariance matrix is ​​decomposed into a direction matrix and a scale matrix consisting of eigenvalues, and the median of all eigenvalues ​​is calculated. The scale matrix is ​​normalized by dividing it by the median to obtain a normalized scale matrix. The regularized covariance matrix of any point is then reconstructed using the direction matrix and the normalized scale matrix. The regularized point cloud is obtained based on the regularized covariance matrix of all points in the original point cloud.

3. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 2, characterized in that, The steps of using Gaussian process regression to complete the missing depth regions in the depth map of the current frame and generating a completed point cloud include: The effective depth pixels and corresponding depth observations are extracted from the depth map to form a training dataset, and the pixels in the missing depth regions of the depth map are used as the prediction point set. Construct a spatial-luminance product kernel function that combines spatial distance information and RGB luminance information, and use the spatial-luminance product kernel function to calculate the autocovariance matrix of the training dataset and the cross-covariance matrix between the training dataset and the prediction point set; Based on the autocovariance matrix, the crosscovariance matrix, and the depth observations, the mean depth prediction of the predicted point set is calculated, and the missing depth region is filled in based on the mean depth prediction to generate the filled point cloud.

4. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 1, characterized in that, Based on the regularized point cloud, the steps of using a fractionally weighted generalized iterative nearest-neighbor algorithm to perform dual-mode camera pose tracking and obtain the final camera pose of the current frame include: Using the frame-to-frame tracking mode, the regularized point cloud of the current frame and the regularized point cloud of the previous frame are used as inputs, and the temporary camera pose of the current frame is calculated by the fraction-weighted generalized iterative nearest point algorithm. Using the frame-to-map tracking mode, the temporary camera pose is used as the initial value for frame-to-map registration. The regularized point cloud of the current frame is registered with the trackable set. The initial value is iteratively optimized using the fractionally weighted generalized iterative nearest-point algorithm to calculate the final camera pose of the current frame.

5. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 1, characterized in that, The step of calculating the overlap rate between the current frame and the trackable set based on the final camera pose includes: Based on the final camera pose, the regularized point cloud of the current frame is registered with the trackable set, and a set of valid corresponding points that satisfy geometric consistency is obtained after registration. The number of points in the set of valid corresponding points is counted as the number of valid corresponding points, and the number of source points in the regularized point cloud of the current frame used for registration is obtained. The ratio of the number of valid corresponding points to the number of source points is determined as the overlap rate between the current frame and the traceable set.

6. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 5, characterized in that, The steps of identifying the current frame as a keyframe and updating the global Gaussian map or the traceable set according to the type of the keyframe include: When the overlap rate is lower than the set threshold, the current frame is determined as a tracking key frame; when the frame interval between the previous mapping key frame and the current frame reaches a fixed frame interval, the current frame is determined as a mapping key frame; wherein, if the overlap rate is lower than the set threshold and the frame interval between the previous mapping key frame and the current frame reaches the fixed frame interval, then the current frame is determined as the tracking key frame. When the current frame is determined to be a tracking keyframe, the regularized point cloud of the current frame is added to the trackable set, and the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map; when the current frame is determined to be a mapping keyframe, the complete point cloud of the current frame is converted into a Gaussian point set and added to the global Gaussian map.

7. The RGB-D SLAM method with 3DGS and two-stage pose optimization according to claim 1, characterized in that, The steps include rendering and loss calculation of the global Gaussian map based on 3D Gaussian splashing, and co-optimizing the camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map, including: Using 3D Gaussian splashing technology, the global Gaussian map is rendered in a differentiable manner to generate the predicted RGB image and predicted depth image of the key frame; Using the real RGB image and real depth image of the keyframe as supervision, and combining the predicted RGB image and the predicted depth image, the color loss term and the depth loss term are calculated and a joint loss function is constructed; The joint loss function is minimized through a two-stage pose optimization process. In the first stage, a joint rigid body transformation is performed on the camera pose of the key frame and all Gaussian point sets in the view of the key frame. In the second stage, the Gaussian parameters are first fixed and the camera pose is optimized, and then the Gaussian parameters are optimized while the camera pose is fixed. Based on the results of the two-stage pose optimization process, the Gaussian parameters in the global Gaussian map and the camera pose of the keyframes are updated to obtain the optimized global Gaussian map.

8. A 3DGS and two-stage pose optimization RGB-D SLAM system, characterized in that, include: Processing module, fusion module, tracking module, update module, and optimization module; The processing module is used to: downsample the depth map of the current frame in the RGB-D sequence to generate the original point cloud, and perform ellipsoidal regularization on the original point cloud to obtain a regularized point cloud; The fusion module is used to: use Gaussian process regression to complete the missing depth regions in the depth map of the current frame, generate a completed point cloud, and fuse the completed point cloud with the original point cloud to form the complete point cloud of the current frame; The tracking module is used to: perform dual-mode camera pose tracking based on the regularized point cloud using a fractionally weighted generalized iterative nearest-point algorithm to obtain the final camera pose of the current frame; wherein, the dual modes include: frame-to-frame tracking mode and frame-to-map tracking mode; The update module is used to: calculate the overlap rate between the current frame and the trackable set based on the final camera pose; when the overlap rate is lower than a set threshold or reaches a fixed frame interval, determine the current frame as a key frame; and update the global Gaussian map or the trackable set according to the type of the key frame. The optimization module is used to: render and calculate the loss of the global Gaussian map based on 3D Gaussian splashing, and to perform collaborative optimization of camera pose and Gaussian parameters through two-stage pose optimization to obtain the optimized global Gaussian map.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the RGB-D SLAM method with 3DGS and two-stage pose optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the RGB-D SLAM method for 3DGS and two-stage pose optimization as described in any one of claims 1 to 7.

Images