Positioning mapping method and system based on three-dimensional Gaussian splash and depth camera

Abstract

The invention relates to the technical field of synchronous positioning and mapping, in particular to a positioning mapping method and system based on three-dimensional Gaussian splashing and a depth camera, and the method comprises the steps: collecting the image and depth information of a target scene in real time, and obtaining the data of a current frame and the corresponding Gaussian point cloud data; based on the current frame data and the corresponding Gaussian point cloud data, adopting an Adam optimization algorithm and analyzing a Jacobian matrix to estimate the camera pose of the current frame; selecting a key frame from a plurality of current frames based on the camera pose, and constructing a dynamic key frame window based on the key frame; and according to the key frame data and the corresponding Gaussian point cloud data, adding a Gaussian point cloud in an area with insufficient coverage, and carrying out iterative rendering on the key frame in the dynamic key frame window to complete the construction of the three-dimensional map. According to the invention, rapid and high-quality positioning mapping can be realized.

Description

Technical Field

[0001] This invention relates to the field of synchronous positioning and mapping technology, and in particular to a positioning and mapping method and system based on 3D Gaussian splashing and depth camera, which is applicable to application scenarios such as drones, robots, and autonomous driving that require autonomous positioning and environmental modeling in dynamic scenes. Background Technology

[0002] 3DGS (3D Gaussian Splatting) models a scene using a set of Gaussian ellipsoids and rasterizes these ellipsoids for efficient rendering. A scene is essentially a set of points in space, each described by a Gaussian distribution, including attributes such as position, color, size, and orientation. Each point in the scene is represented by a Gaussian function, and these functions collectively describe the entire 3D scene. As an explicit representation, 3DGS utilizes Gaussian primitives, offering interpretability and editability. Through Gaussian sputtering, it achieves faster, differentiable rasterized rendering, generating higher-fidelity views and excelling in scene editing and real-time rendering.

[0003] Simultaneous Localization and Mapping (SLAM) is crucial for applications such as drones, robots, and autonomous driving. As the demands of applications like drones become increasingly widespread, localization and mapping become exceptionally critical for autonomous movement in complex environments. Depth camera-based SLAM methods have become the preferred choice for drone and robotic applications due to their portability and accuracy. The need to combine 3DGS with SLAM stems from the limitations of traditional SLAM map representation in terms of real-time performance, storage efficiency, and dynamic adaptability. 3DGS, as an emerging method for representing 3D scenes, describes the environment through a parameterized Gaussian distribution set, bringing revolutionary optimizations to SLAM systems.

[0004] Existing solutions include the open-source framework SplaTAM, which relies solely on automatic differentiation for camera pose estimation, resulting in excessive time consumption and failing to meet the demands of high-real-time mapping. Secondly, selecting keyframes based solely on frame sequence numbers is redundant in scenes with limited texture and slow motion, and lacks robustness in dynamic scenes with rapid movement. Furthermore, region detection based solely on depth and contour information, with localized Gaussian point additions, achieves macroscopic Gaussian point coverage but lacks refinement for complex textured regions. Therefore, this invention proposes a localization and mapping method and system based on 3D Gaussian splashing and a depth camera. Summary of the Invention

[0005] The purpose of this invention is to provide a localization and mapping method and system based on 3D Gaussian splash and depth camera, which can achieve fast and high-quality localization and mapping in indoor dynamic scenes, so as to solve the problems of low efficiency of camera pose estimation, unreasonable key frame selection and insufficient optimization of Gaussian point distribution in the prior art.

[0006] To achieve the above objectives, in one aspect, the present invention provides a localization and mapping method based on three-dimensional Gaussian splashing and a depth camera, comprising:

[0007] Real-time acquisition of image and depth information of the target scene, obtaining current frame data and corresponding Gaussian point cloud data;

[0008] Based on the current frame data and the corresponding Gaussian point cloud data, the camera pose of the current frame is estimated using the Adam optimization algorithm and the analytical Jacobian matrix.

[0009] Based on the camera pose, select keyframes from several current frames, and construct a dynamic keyframe window based on the keyframes;

[0010] Based on the keyframe data and the corresponding Gaussian point cloud data, Gaussian point clouds are added in the insufficient coverage areas, and the keyframes in the dynamic keyframe window are iteratively rendered to complete the construction of the 3D map.

[0011] Optionally, the current frame data includes sequence number, pose, camera intrinsic parameters, observation color map, and observation depth map; the Gaussian point cloud data includes position, color, scale, rotation, and opacity.

[0012] Optionally, based on the current frame data and the corresponding Gaussian point cloud data, the camera pose of the current frame is estimated using the Adam optimization algorithm and the analytical Jacobian matrix, including:

[0013] The camera coordinate system of the first frame is used as the world coordinate system, and the camera pose of the first frame is determined as the identity matrix.

[0014] Initialize the pose tracking Adam optimizer and initialize the camera pose of the current frame based on the camera pose of the previous frame;

[0015] The current frame data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the tracking loss is calculated.

[0016] The tracking loss is backpropagated, and the gradient direction is provided by analyzing the Jacobian matrix to speed up the pose convergence time and reduce the pose estimation error. The gradient of the pose perturbation is obtained from the pose tracking Adam optimizer, the pose perturbation of the current frame is obtained and the pose of the current frame is updated. The process is iteratively converged to obtain the final estimated pose of the current frame.

[0017] Optionally, calculating the tracking loss includes:

[0018] L Tracking =∑ p M1(w1L1(C(p))+w2L1(D(p)));

[0019] Where p represents a pixel; w1 and w2 are the weight coefficients of the two losses in the tracking loss; L1(C(p)) and L1(D(p)) are the L1 loss for color rendering and the L1 loss for depth rendering, respectively; M1 = (S(p) > δ s1 S(p) is the contour visibility mask during the pose tracking stage, S(p) is the rendered contour, and δ is the depth mask. s1 The threshold for visibility of the outline.

[0020] Optionally, selecting keyframes from several current frames based on the camera pose and constructing a dynamic keyframe window based on the keyframes includes:

[0021] Based on the camera pose, a key frame is selected from several current frames according to a preset key frame selection strategy.

[0022] The keyframe is added to a dynamic keyframe window of fixed size k, and the updated dynamic keyframe window is obtained. If the size of the dynamic keyframe window already meets k before the keyframe is added to the dynamic keyframe window, the overlap coefficient between the keyframe and each keyframe in the dynamic keyframe window is calculated, the keyframe with the smallest overlap coefficient is removed, and then the keyframe is added to the dynamic keyframe window.

[0023] Optionally, the keyframe selection strategy includes:

[0024] If the sequence number of the current frame is a multiple of 10, then the current frame is a keyframe;

[0025] If the relative displacement between the current frame and the previous keyframe is greater than the product of a preset displacement coefficient and the median of the observation depth of the current frame, then the current frame is a keyframe.

[0026] If the intersection-union ratio (CUI) of the current frame and the previous keyframe is less than a preset CUI threshold, then the current frame is a keyframe.

[0027] Optionally, adding Gaussian point clouds in the insufficiently covered areas based on keyframe data and corresponding Gaussian point cloud data includes:

[0028] Rendering is performed using a differentiable rasterization channel based on keyframe data and corresponding Gaussian point cloud data to obtain the mask of contour anomaly regions.

[0029] A new Gaussian point cloud is obtained based on the current frame data. Then, it is filtered by the contour anomaly region mask. Gaussian point clouds are added in the insufficient coverage areas, and the added Gaussian point clouds are merged into the original Gaussian point cloud.

[0030] Optionally, iterative rendering of keyframes in the dynamic keyframe window includes:

[0031] Initialize the Adam optimizer for graph building;

[0032] The dynamic keyframe window is traversed, and each keyframe is iterated repeatedly. In each iteration, the keyframe data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the mapping loss is calculated.

[0033] Backpropagation is performed based on the graphing loss to obtain the gradient of the Gaussian point cloud data from the graphing Adam optimizer and update the Gaussian point cloud. The optimized Gaussian point cloud data is obtained through continuous iteration. When the total number of iterations is a multiple of 200, Gaussian pruning and densification operations are performed.

[0034] Optionally, calculating the mapping loss includes:

[0035] L Mapping =∑ p w3(M3L1(D(p)))+w4L1(C(p))+w5(1-SSIM(C(p)));

[0036] Where p represents a pixel, w3, w4, and w5 are the weight coefficients of the three loss terms in the mapping loss; L1(C(p)) and L1(D(p)) are the L1 loss for color rendering and the L1 loss for depth rendering, respectively; SSIM(C(p)) is the structural similarity loss for the color map; M3 = (S(p) > δ s3 S(p) is the outline visibility mask during the mapping stage, S(p) is the rendered outline, and δ is the depth mask. s3 This is the contour threshold.

[0037] On the other hand, the present invention also provides a localization and mapping system based on three-dimensional Gaussian splashing and a depth camera, comprising:

[0038] The data acquisition and processing module is used to acquire image and depth information of the target scene in real time, and obtain the current frame data and the corresponding Gaussian point cloud data;

[0039] The camera pose tracking module is used to estimate the camera pose of the current frame based on the current frame data and the corresponding Gaussian point cloud data, using the Adam optimization algorithm and analytical Jacobian matrix.

[0040] The keyframe selection module is used to select keyframes from several current frames based on the camera pose, and to construct a dynamic keyframe window based on the keyframes.

[0041] The 3D map construction module is used to add Gaussian point clouds in areas with insufficient coverage based on keyframe data and corresponding Gaussian point cloud data, and to iteratively render the keyframes in the dynamic keyframe window to complete the construction of the 3D map.

[0042] The beneficial effects of this invention are as follows:

[0043] (1) This invention uses a differentiable rasterized channel combined with an analytical Jacobian matrix to explicitly derive the derivative of the camera pose with respect to the 3D Gaussian parameters, so that the error can be backpropagated to the camera pose parameters to guide the optimization direction. This avoids the computational overhead of automatic differentiation, significantly improves the optimization efficiency, and enhances the efficiency, accuracy and robustness of pose estimation.

[0044] (2) The present invention selects key frames based on the current frame number, the intersection-over-interference ratio of the current frame and the previous key frame and the relative displacement, and obtains a dynamic key frame window to adapt to scene changes, effectively avoiding key frame redundancy and missing cases, and only performs mapping operations on key frames, thereby improving mapping efficiency and robustness.

[0045] (3) Based on the depth and contour information of key frames, this invention adds Gaussian points to the under-covered areas and combines gradient micro-refinement operations to achieve rapid response to new areas. At the same time, it improves details in complex texture areas through gradient methods. By dynamically managing the number, scale and distribution of Gaussian points, it achieves detail enhancement, coverage expansion and redundancy filtering, balances reconstruction accuracy and computational efficiency, and can adapt to different scene requirements. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a localization and mapping system based on three-dimensional Gaussian splashing and a depth camera, according to an embodiment of the present invention.

[0049] Figure 3 This is a flowchart of the Adam optimization algorithm according to an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] On the one hand, this embodiment provides a localization and mapping method based on 3D Gaussian splashing and a depth camera, including:

[0053] Real-time acquisition of image and depth information of the target scene, obtaining current frame data and corresponding Gaussian point cloud data;

[0054] Based on the current frame data and the corresponding Gaussian point cloud data, the camera pose of the current frame is estimated using the Adam optimization algorithm and the analytical Jacobian matrix.

[0055] Based on the camera pose, select keyframes from several current frames, and construct a dynamic keyframe window based on the keyframes;

[0056] Based on the keyframe data and the corresponding Gaussian point cloud data, Gaussian point clouds are added in the insufficient coverage areas, and the keyframes in the dynamic keyframe window are iteratively rendered to complete the construction of the 3D map.

[0057] Specifically, this embodiment employs a differentiable rasterization channel combined with an analytical Jacobian matrix to explicitly derive the derivative of the camera pose with respect to the 3D Gaussian parameters. This allows errors to propagate back to the camera pose parameters, guiding the optimization direction and avoiding the computational overhead of automatic differentiation. This significantly improves optimization efficiency, pose estimation efficiency, accuracy, and robustness. Keyframes are selected based on the current frame number, the intersection-over-union ratio (IoU) between the current frame and the previous keyframe, and the relative displacement. A dynamic keyframe window is obtained to adapt to scene changes, effectively avoiding keyframe redundancy and missing frames. Mapping operations are performed only on keyframes, improving mapping efficiency and robustness. Gaussian points are added to under-covered areas based on the depth and contour information of the keyframes. Gradient micro-refinement operations are combined to achieve rapid response to new areas. In complex texture areas, gradient methods are used to enhance details. By dynamically managing the number, scale, and distribution of Gaussian points, detail enhancement, coverage expansion, and redundancy filtering are achieved, balancing reconstruction accuracy and computational efficiency, and adapting to different scene requirements.

[0058] like Figure 1As shown, the localization and mapping method based on 3D Gaussian splash and depth camera provided in this embodiment specifically includes the following steps:

[0059] The first step is to acquire image and depth information of the target scene in real time, obtain the current frame data and the corresponding Gaussian point cloud data. The current frame data includes the sequence number, pose, camera intrinsic parameters, observation color map, and observation depth map; the Gaussian point cloud data includes position, color, scale, rotation, and opacity.

[0060] Specifically, in this embodiment, image and depth information are acquired to initialize the current frame data and Gaussian point cloud data. PyTorch is used to set the pose perturbation of the current frame and the Gaussian point cloud data to automatic differentiation so as to calculate their gradients during backpropagation. The current frame data includes sequence number, pose, camera intrinsic parameters, observation color map, and observation depth map; the Gaussian point cloud data includes position, color, scale, rotation, and opacity.

[0061] The second step, based on the current frame data and the corresponding Gaussian point cloud data, uses the Adam optimization algorithm and analytical Jacobian matrix to estimate the camera pose of the current frame, including:

[0062] The camera coordinate system of the first frame is used as the world coordinate system, and the camera pose of the first frame is determined as the identity matrix.

[0063] Initialize the pose tracking Adam optimizer and initialize the camera pose of the current frame based on the camera pose of the previous frame;

[0064] The current frame data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the tracking loss is calculated.

[0065] The tracking loss is backpropagated, and the gradient direction is provided by analyzing the Jacobian matrix to speed up the pose convergence time and reduce the pose estimation error. The gradient of the pose perturbation is obtained from the pose tracking Adam optimizer, the pose perturbation of the current frame is obtained and the pose of the current frame is updated. The process is iteratively converged to obtain the final estimated pose of the current frame.

[0066] The second step specifically includes the following steps:

[0067] 1. Initialize the camera pose T1 of the first frame as the identity matrix, and use the camera coordinate system of the first frame as the world coordinate system; for t=2, initialize T2=T1, and the camera pose of each subsequent frame is first processed by the formula: T t+1 =T t +(T t -T t-1 Initialize the pose tracking Adam optimizer, and set the pose perturbation ΔT of the current frame to the initialization step. t(Including rotational change Δrot and displacement change Δtrans) are used as parameters to be optimized, and their learning rate α and momentum decay rates β1 and β2 are set respectively, where T t This represents the camera pose in frame t.

[0068] 2. Based on the current frame data and Gaussian point cloud data, the color C(p), depth D(p), and contour S(p) of pixel p are obtained through differentiable rasterization channels, and used to calculate the tracking loss L. Tracking The tracking loss function L Tracking As the objective function in the Adam optimization method;

[0069] 3. Then, backpropagation is performed using the tracking loss. The gradient direction is directly provided by the analytical Jacobian matrix, which speeds up the pose convergence time and reduces the pose estimation error. The updated pose perturbation is obtained from the pose tracking Adam optimizer. And using the formula: Obtain the updated current frame camera pose like The sum of all elements is less than 10. -4 If the pose of the current frame has converged, the iteration is terminated; otherwise, the pose of the current frame is perturbed by ΔT. t Reset the values ​​to zero and proceed to the next iteration. Continue this iterative process until the final estimated pose of the current frame is obtained.

[0070] The tracking loss function is expressed as follows:

[0071] L Tracking =∑ p M1(w1L1(C(p))+w2L1(D(p)));

[0072] Where p represents a pixel; w1 and w2 are constants used as loss weights; L1(C(p)) and L1(D(p)) are the L1 losses for color and depth rendering, respectively; M1 = (S(p) > δ s1 ) is the outline visibility mask, where S(p) is the rendered outline, and δ s1 The constant is used as the visibility threshold for the outline, with a value of 0.99.

[0073] The Jacobian matrix is ​​analyzed as follows:

[0074]

[0075] Where, μ C T represents the coordinates of the Gaussian point in the camera coordinate system. CW W is the transformation matrix from the world to the camera; T is the transformation matrix from the world to the camera. CW The rotation matrix part in W; :，iLet be the i-th column vector of W; the × symbol represents the antisymmetric matrix of the vectors; I is the identity matrix.

[0076] More specifically, such as Figure 3 As shown, the Adam optimization algorithm calculates the exponentially decaying average m of the first moment of each parameter. t and the exponentially decaying average v of the second moment t And obtain their correction values. and This allows for adjustments to the model parameters, such as the learning rate α and the gradient g. t Then adjust the model parameter θ t To minimize the objective function, thereby improving the model's predictive accuracy. The exponentially decaying average m of the first and second moments. t v t and their correction values and Model parameters θ t The update expression is as follows:

[0077] m t =β1m t-1 +(1-β1)g t ;

[0078]

[0079] Where β1 and β2 are the decay rates of the first-order moment estimate and the second-order moment estimate, respectively, which are preset in the model parameters; α is the learning rate, which is preset in the model parameters; and g... t It is the objective function with respect to parameter θ at the t-th iteration. t The gradient of , where ∈ is a local minimum, such as 10. -8 To avoid division by zero errors.

[0080] More specifically, the expressions for the color C(p), depth D(p), and contour S(p) of a pixel p rendered by a differentiable rasterization channel are as follows:

[0081]

[0082]

[0083] Where N is the number of Gaussian points, c i Let α be the color of the i-th Gaussian point. i Let d be the opacity of the i-th Gaussian point. i Let be the depth of the i-th Gaussian point.

[0084] More specifically, the pixel coordinates μ of a 3D Gaussian point I With the two-dimensional covariance matrix Σ I The expression is as follows:

[0085]

[0086] Σ I =JWΣ W W T J T ;

[0087] Where K is the camera intrinsic parameter matrix; μ W Σ represents the coordinates of the 3D Gaussian point; d represents the depth of the Gaussian point in the camera coordinate system; W T is the covariance matrix of the three-dimensional Gaussian points; J is the projected Jacobian matrix; T CW W is the transformation matrix from the world to the camera; T is the transformation matrix from the world to the camera. CW The rotation matrix part in μ; C The coordinates of the 3D Gaussian point in the camera coordinate system.

[0088] Analyzing the Jacobian matrix and pixel coordinates μ I and the two-dimensional covariance matrix Σ I The relationship according to the chain rule is as follows:

[0089]

[0090] The third step involves selecting keyframes from several current frames based on the camera pose, and constructing a dynamic keyframe window based on these keyframes, including:

[0091] Based on the camera pose, a key frame is selected from several current frames according to a preset key frame selection strategy.

[0092] The keyframe is added to a dynamic keyframe window of fixed size k, and the updated dynamic keyframe window is obtained. If the size of the dynamic keyframe window already meets k before the keyframe is added to the dynamic keyframe window, the overlap coefficient between the keyframe and each keyframe in the dynamic keyframe window is calculated, the keyframe with the smallest overlap coefficient is removed, and then the keyframe is added to the dynamic keyframe window.

[0093] The keyframe selection strategy includes:

[0094] If the sequence number of the current frame is a multiple of 10, then the current frame is a keyframe;

[0095] If the relative displacement between the current frame and the previous keyframe is greater than the product of a preset displacement coefficient and the median of the observation depth of the current frame, then the current frame is a keyframe.

[0096] If the intersection-union ratio (CUI) of the current frame and the previous keyframe is less than a preset CUI threshold, then the current frame is a keyframe.

[0097] The third step specifically includes the following steps:

[0098] 1. Determine whether the current frame is a key frame based on the key frame selection strategy. If the current frame number is a multiple of 10, then the current frame is a key frame; or if the relative displacement between the current frame and the previous key frame is greater than the product of the preset displacement coefficient and the median of the observation depth of the current frame, then the current frame is a key frame; or if the intersection-union ratio (IOU) between the current frame and the previous key frame is less than the IOU threshold, then the current frame is a key frame.

[0099] 2. If the current frame is determined to be a keyframe, add the current frame to a dynamic keyframe window with a fixed maximum size of k. If the dynamic keyframe window already has a size of k before adding the current frame, calculate the overlap coefficient (OC) between the current frame and each keyframe in the dynamic keyframe window, exclude the keyframe with the smallest overlap coefficient, and then add the current frame to the dynamic keyframe window to obtain the current dynamic keyframe window.

[0100] The preset displacement coefficient is 0.08, the crossover ratio threshold is 0.9, and k is 10.

[0101] The formula for calculating the intersection-union ratio is as follows:

[0102]

[0103] in, Represents the visible Gaussian points of the current frame, where i is the sequence number of the current frame; is the visible Gaussian point of the previous keyframe, and j is the keyframe number.

[0104] The formula for calculating the overlap coefficient is as follows:

[0105]

[0106] in, Represents the visible Gaussian points of the current frame, where i is the sequence number of the current frame; is the visible Gaussian point of the keyframe, and j is the keyframe number.

[0107] Keyframes are dynamically managed using three conditions: time interval, geometric displacement, and intersection-over-union (IoU). A dynamic keyframe window is constructed based on the overlap coefficient. Time interval serves as a safety net to ensure a basic keyframe density, preventing map update lag. Trigger thresholds are adaptively adjusted through displacement and depth linkage, dynamically increasing the displacement trigger threshold for distant scenes with the median depth, reducing redundant insertions and suppressing interference from dynamic objects. IoU thresholds are used to accurately capture new scene regions, suppressing redundant keyframes in scenes with repetitive textures. Highly correlated historical keyframes are selected based on the overlap coefficient (OC), ensuring that keyframes within the window are always highly relevant to the current scene, focusing on effective constraints while avoiding local optima. This adaptive approach to scene changes effectively improves the efficiency and robustness of 3D mapping.

[0108] The fourth step is to add Gaussian point clouds in the insufficient coverage area based on the keyframe data and the corresponding Gaussian point cloud data, and to iteratively render the keyframes in the dynamic keyframe window to complete the construction of the 3D map.

[0109] Among them, adding Gaussian point clouds in areas with insufficient coverage based on keyframe data and corresponding Gaussian point cloud data includes:

[0110] Rendering is performed using a differentiable rasterization channel based on keyframe data and corresponding Gaussian point cloud data to obtain the mask of contour anomaly regions.

[0111] A new Gaussian point cloud is obtained based on the current frame data. Then, it is filtered by the contour anomaly region mask. Gaussian point clouds are added in the insufficient coverage areas, and the added Gaussian point clouds are merged into the original Gaussian point cloud.

[0112] Iterative rendering of keyframes in the dynamic keyframe window includes:

[0113] Initialize the Adam optimizer for graph building;

[0114] The dynamic keyframe window is traversed, and each keyframe is iterated repeatedly. In each iteration, the keyframe data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the mapping loss is calculated.

[0115] Backpropagation is performed based on the graphing loss to obtain the gradient of the Gaussian point cloud data from the graphing Adam optimizer and update the Gaussian point cloud. The optimized Gaussian point cloud data is obtained through continuous iteration. When the total number of iterations is a multiple of 200, Gaussian pruning and densification operations are performed.

[0116] The fourth step specifically includes the following steps:

[0117] 1. Rendering is performed using differentiable rasterization channels based on the current keyframe data and Gaussian point cloud data to obtain the rendered depth map D(p) and contour map S(p), thus obtaining the contour anomaly region mask M2. A new Gaussian point cloud is obtained based on the current frame data, and then filtered through the contour anomaly region mask. Gaussian point clouds are added to areas with insufficient coverage and merged into the existing Gaussian point cloud. By using depth and contour information to determine areas with insufficient Gaussian coverage, the initialization of the new Gaussian point cloud locations is guided. New Gaussian point clouds only need to be initialized in uncovered areas, improving map building efficiency. When dynamically expanding the map, Gaussian point clouds are added only to insufficiently represented areas, avoiding redundant parameters and enabling rapid response to new areas.

[0118] The formula for masking out-of-contour regions is as follows:

[0119] M2=(S(p)<δ s2 )+(D gt (p)<D(p))(L1(D(p))> λMDE);

[0120] Where, δ s2 The threshold value for contour anomalies is 0.5; λ is used as the depth error weight and is set to 50; D gt (p) represents the observed depth; MDE represents the median depth error.

[0121] 2. First, initialize the mapping Adam optimizer, taking each Gaussian point cloud attribute (including: position, color, scale, rotation, and opacity) as the parameter to be optimized, and set their learning rate α and momentum decay rates β1 and β2 respectively, which will be used to optimize the Gaussian point cloud parameters below;

[0122] 3. Traverse the dynamic keyframe window, iterating 10 times for each keyframe. In each iteration, based on the keyframe data and the current Gaussian point cloud data, render the color map C(p), depth map D(p), and contour map S(p) through a differentiable rasterization channel. These are used to calculate the mapping loss L. Mapping The mapping loss function L Mapping The objective function in the Adam optimization method is used. Then, backpropagation is performed through the mapping loss. Based on the total number of mapping iterations, it is determined whether to perform Gaussian point cloud pruning and densification. The gradient of the Gaussian point cloud data is obtained from the mapping Adam optimizer and the current Gaussian point cloud is updated. The process is repeated to obtain the optimized Gaussian point cloud data.

[0123] The mapping loss function is as follows:

[0124] L Mapping =∑ p w3(M3L1(D(p)))+w4L1(C(p))+w5(1-SSIM(C(p)));

[0125] Where w3, w4, and w5 are constants, serving as weighting coefficients; SSIM(C(p)) is the structural similarity loss for rendering the color map; M3 = (S(p) > δ s3 ) represents the outline visibility mask, δ s3 is a constant, used as the contour threshold, with a value of 0.5; w3, w4, and w5 are set to 1, 0.4, and 0.1, respectively.

[0126] 4. If the total number of mapping iterations is a multiple of 200, perform Gaussian pruning and densification, including: cloning Gaussian points with position gradients greater than the position gradient threshold and scales less than the scale threshold to fill in details; splitting Gaussian points with position gradients greater than the position gradient threshold and scales greater than the scale threshold to expand coverage; and deleting Gaussian points with 2D projection radii greater than the 2D projection radius threshold or opacity less than the opacity threshold to remove redundant Gaussian points.

[0127] The location gradient threshold, scale threshold, 2D projection radius threshold, and opacity threshold are set to 0.0002, 0.01, 20, and 0.7, respectively.

[0128] By combining macroscopic contour coverage and microscopic gradient refinement operations, a rapid response to new regions is achieved. At the same time, in complex texture regions, details are improved through gradient methods. By dynamically managing the number, scale, and distribution of Gaussian points, detail enhancement, coverage expansion, and redundancy filtering are achieved, balancing reconstruction accuracy and computational efficiency, ensuring the integrity of geometric details, and improving the accuracy of 3D mapping.

[0129] On the other hand, this embodiment also provides a positioning and mapping system based on three-dimensional Gaussian splashing and a depth camera, such as Figure 2 As shown, it includes:

[0130] The data acquisition and processing module is used to acquire image and depth information of the target scene in real time, and obtain the current frame data and the corresponding Gaussian point cloud data;

[0131] The camera pose tracking module is used to estimate the camera pose of the current frame based on the current frame data and the corresponding Gaussian point cloud data, using the Adam optimization algorithm and analytical Jacobian matrix.

[0132] The keyframe selection module is used to select keyframes from several current frames based on the camera pose, and to construct a dynamic keyframe window based on the keyframes.

[0133] The 3D map construction module is used to add Gaussian point clouds in areas with insufficient coverage based on keyframe data and corresponding Gaussian point cloud data, and to iteratively render the keyframes in the dynamic keyframe window to complete the construction of the 3D map.

[0134] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A localization and mapping method based on 3D Gaussian splashing and a depth camera, characterized in that, include: Real-time acquisition of image and depth information of the target scene, obtaining current frame data and corresponding Gaussian point cloud data; Based on the current frame data and the corresponding Gaussian point cloud data, the camera pose of the current frame is estimated using the Adam optimization algorithm and the analytical Jacobian matrix. Based on the camera pose, select keyframes from several current frames, and construct a dynamic keyframe window based on the keyframes; Based on the keyframe data and the corresponding Gaussian point cloud data, Gaussian point clouds are added in the insufficient coverage areas, and the keyframes in the dynamic keyframe window are iteratively rendered to complete the construction of the 3D map.

2. The localization and mapping method based on 3D Gaussian splashing and depth camera according to claim 1, characterized in that, The current frame data includes sequence number, pose, camera intrinsic parameters, observed color map, and observed depth map; the Gaussian point cloud data includes position, color, scale, rotation, and opacity.

3. The localization and mapping method based on 3D Gaussian splashing and a depth camera according to claim 1, characterized in that, Based on the current frame data and the corresponding Gaussian point cloud data, the camera pose of the current frame is estimated using the Adam optimization algorithm and the analytical Jacobian matrix, including: The camera coordinate system of the first frame is used as the world coordinate system, and the camera pose of the first frame is determined as the identity matrix. Initialize the pose tracking Adam optimizer and initialize the camera pose of the current frame based on the camera pose of the previous frame; The current frame data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the tracking loss is calculated. The tracking loss is backpropagated, and the gradient direction is provided by analyzing the Jacobian matrix to speed up the pose convergence time and reduce the pose estimation error. The gradient of the pose perturbation is obtained from the pose tracking Adam optimizer, the pose perturbation of the current frame is obtained and the pose of the current frame is updated. The process is iteratively converged to obtain the final estimated pose of the current frame.

4. The localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to claim 3, characterized in that, Calculating the tracking loss includes: L Tracking =∑ p M1(w1L1(C(p))+w2L1(D(p))); Where p represents a pixel; w1 and w2 are the weight coefficients of the two losses in the tracking loss; L1(C(p)) and L1(D(p)) are the L1 loss for color rendering and the L1 loss for depth rendering, respectively; M1 = (S(p) > δ s1 S(p) is the contour visibility mask during the pose tracking stage, S(p) is the rendered contour, and δ is the depth mask. s1 The threshold for visibility of the outline.

5. The localization and mapping method based on 3D Gaussian splashing and a depth camera according to claim 1, characterized in that, Selecting keyframes from several current frames based on the camera pose, and constructing a dynamic keyframe window based on the keyframes, includes: Based on the camera pose, a key frame is selected from several current frames according to a preset key frame selection strategy. The keyframe is added to a dynamic keyframe window of fixed size k, and the updated dynamic keyframe window is obtained. If the size of the dynamic keyframe window already meets k before the keyframe is added to the dynamic keyframe window, the overlap coefficient between the keyframe and each keyframe in the dynamic keyframe window is calculated, the keyframe with the smallest overlap coefficient is removed, and then the keyframe is added to the dynamic keyframe window.

6. The localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to claim 5, characterized in that, The keyframe selection strategy includes: If the sequence number of the current frame is a multiple of 10, then the current frame is a keyframe; If the relative displacement between the current frame and the previous keyframe is greater than the product of a preset displacement coefficient and the median of the observation depth of the current frame, then the current frame is a keyframe. If the intersection-union ratio (CUI) of the current frame and the previous keyframe is less than a preset CUI threshold, then the current frame is a keyframe.

7. The localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to claim 1, characterized in that, Adding Gaussian point clouds to areas with insufficient coverage based on keyframe data and corresponding Gaussian point cloud data includes: Rendering is performed using a differentiable rasterization channel based on keyframe data and corresponding Gaussian point cloud data to obtain the mask of contour anomaly regions. A new Gaussian point cloud is obtained based on the current frame data. Then, it is filtered by the contour anomaly region mask. Gaussian point clouds are added in the insufficient coverage areas, and the added Gaussian point clouds are merged into the original Gaussian point cloud.

8. The localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to claim 1, characterized in that, Iterative rendering of keyframes in the dynamic keyframe window includes: Initialize the Adam optimizer for graph building; The dynamic keyframe window is traversed, and each keyframe is iterated repeatedly. In each iteration, the keyframe data and the corresponding Gaussian point cloud data are rendered through a differentiable rasterization channel, and the mapping loss is calculated. Backpropagation is performed based on the graphing loss to obtain the gradient of the Gaussian point cloud data from the graphing Adam optimizer and update the Gaussian point cloud. The optimized Gaussian point cloud data is obtained through continuous iteration. When the total number of iterations is a multiple of 200, Gaussian pruning and densification operations are performed.

9. The localization and mapping method based on three-dimensional Gaussian splashing and a depth camera according to claim 8, characterized in that, The calculation of the mapping loss includes: L Mapping =∑ p w3(M3L1(D(p)))+w4L1(C(p))+w5(1-SSIM(C(p))); Where p represents a pixel, w3, w4, and w5 are the weight coefficients of the three loss terms in the mapping loss; L1(C(p)) and L1(D(p)) are the L1 loss for color rendering and the L1 loss for depth rendering, respectively; SSIM(C(p)) is the structural similarity loss for the color map; M3 = (S(p) > δ s3 S(p) is the outline visibility mask during the mapping stage, S(p) is the rendered outline, and δ is the depth mask. s3 This is the contour threshold.

10. A localization and mapping system based on 3D Gaussian splashing and a depth camera, characterized in that, include: The data acquisition and processing module is used to acquire image and depth information of the target scene in real time, and obtain the current frame data and the corresponding Gaussian point cloud data; The camera pose tracking module is used to estimate the camera pose of the current frame based on the current frame data and the corresponding Gaussian point cloud data, using the Adam optimization algorithm and analytical Jacobian matrix. The keyframe selection module is used to select keyframes from several current frames based on the camera pose, and to construct a dynamic keyframe window based on the keyframes. The 3D map construction module is used to add Gaussian point clouds in areas with insufficient coverage based on keyframe data and corresponding Gaussian point cloud data, and to iteratively render the keyframes in the dynamic keyframe window to complete the construction of the 3D map.

Images