Gaussian slam method and device based on self-crossing feature network and hybrid map representation

Abstract

Embodiments of the present specification provide a Gaussian SLAM method and device based on a self-cross feature network and a hybrid map representation, wherein the Gaussian SLAM method based on the self-cross feature network and the hybrid map representation extracts sparse features and solves an initial pose through a self-cross attention mechanism; generates a sparse skeleton by using local beam adjustment; constructs a hybrid map of a sparse point cloud and a dense Gaussian cell rigid binding, and performs Gaussian cell initialization and progressive rendering optimization in parallel by using front and back threads; multiplexes a feature network initial value to construct an adaptive closed-loop controller in combination with a differentiable rendering residual, and triggers multi-view optimization on demand; and solves a correction matrix through global pose graph optimization, and synchronously updates the sparse point cloud and the dense Gaussian cell. The present application realizes low-delay, high-precision pose tracking and globally consistent three-dimensional reconstruction.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of 3D scene reconstruction technology, and in particular to a Gaussian SLAM method based on self-crossing feature networks and hybrid map representation. Background Technology

[0002] With the increasing demands for accuracy and fidelity in environmental perception from applications such as intelligent robots, augmented reality, and virtual reality, map representation in visual simultaneous localization and mapping (SLAM) systems has become a research hotspot. Dense visual SLAM, which can provide rich scene semantics and texture details, is considered one of the core perception technologies and is widely used in fields such as indoor robot inspection, digital twins, and automated spatial measurement.

[0003] Currently, most existing technologies for dense scene reconstruction employ implicit neural radiation field models, but these methods suffer from rendering time consumption issues when handling large-scale complex spaces. As an alternative, explicit 3D Gaussian primitives and differentiable rasterizable 3D Gaussian sputtering models have been proposed in recent years, effectively capturing transient and steady-state high-frequency texture details and achieving ultra-real-time rendering. However, SLAM methods based on 3D Gaussian models still have the following technical drawbacks: First, existing methods cannot meet the robust tracking requirements of robots under complex motion. They rely heavily on pure dense pixel-level photometric alignment, have a narrow convergence domain for the objective function, and lack prior geometric constraints for non-ideal motions such as rapid rotation, severe vibration, or high-dynamic field-of-view switching, making tracking prone to failure.

[0004] Second, the system suffers from high latency and potential spatial inconsistencies. Existing technologies mostly employ a serial structure that combines tracking and mapping optimization. Pose tracking must wait for the time-consuming optimization of dense photometric gradients to complete, leading to computational bottlenecks and runtime delays. Some attempts have tried decoupling tracking and mapping, but due to the lack of strict synchronization constraints, optimization objectives can easily conflict between sparse trajectories and dense maps, inducing spatial inconsistencies and rendering ghosting.

[0005] Third, it overlooks the excessive computational resource consumption of loop closure detection in large-scale scenes. Existing loop closure mechanisms mostly rely on heavy Gaussian subgraph dense rendering and alignment, lacking a mechanism to control computational overhead. This leads to long delays when loop closure is triggered, making it impossible to eliminate accumulated drift online, which seriously affects the real-time stable operation of the system.

[0006] Therefore, how to design an effective decoupled hybrid map representation and adaptive closed-loop mechanism to significantly reduce the computational load and achieve the dual goals of saving computing power and high-precision mapping, while ensuring high-fidelity reconstruction performance and low computational latency, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] In view of this, embodiments of this specification provide a Gaussian SLAM method based on self-crossing feature networks and hybrid map representations. One or more embodiments of this specification also relate to a Gaussian SLAM apparatus based on self-crossing feature networks and hybrid map representations, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0008] According to a first aspect of the embodiments of this specification, a Gaussian SLAM method based on a self-crossing feature network and a hybrid map representation is provided, comprising: A feature extraction and matching network is constructed using self-attention and cross-attention mechanisms. The network is then used to extract the feature correspondence between image frames and calculate the initial pose of the camera. A local beam adjustment module based on co-view relationship is established between the feature tracker and the back-end map. The initial pose is used as the initial value for optimization. A sliding window model is used to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. A hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units is constructed. The sparse geometric point clouds are composed of 3D landmark positions and feature descriptors in the sparse skeleton, and the sparse geometric point clouds and dense Gaussian units are rigidly bound together. Configure separate foreground and background threads for hybrid map representation to perform synchronous initialization and asynchronous progressive rendering optimization of Gaussian primitives, respectively; The geometric initial values ​​output by the feature extraction and matching network are reused, and the residuals generated by differentiable rasterization are combined to construct a residual adaptive closed-loop state feedback controller. The multi-view optimization operation is activated based on the energy value of the joint rendering residuals, and the relative pose constraints after closed-loop correction are output. A rigid body correction and update model is constructed, which includes a global pose graph and a hybrid map representation. Relative pose constraints are introduced into the global pose graph. The correction transformation matrix is ​​solved and the explicit properties of sparse keyframe poses and dense Gaussian elements in all subgraphs are updated synchronously.

[0009] In one possible implementation, the initial pose of the camera is calculated through a feature extraction and matching network, which includes: inputting the input frame and the previous frame into the network to obtain a set of matching points; performing random sampling consistency screening on the set of matching points to obtain a set of inliers; using depth information to backproject the two-dimensional matching points in the set of inliers into three-dimensional landmarks; reprojecting the three-dimensional landmarks onto the image plane through perspective projection; and using the Huber robust kernel function to minimize the nonlinear geometric reprojection error, thereby obtaining the optimal relative pose estimate as the initial pose.

[0010] In one possible implementation, the local beam adjustment module for generating the sparse skeleton includes: maintaining a sliding window containing a predetermined number of recent keyframes within the activated subgraph; using the initial pose as the initial estimate of the keyframes within the sliding window; constructing an objective function that includes a reprojection error term and a normalization fixation term; introducing a penalty factor in the normalization fixation term to constrain the pose drift of the oldest keyframe in the sliding window; performing local beam adjustment by minimizing the objective function; and outputting the jointly optimized local trajectory and 3D spatial landmarks as the sparse skeleton.

[0011] In one possible implementation, constructing a hybrid map representation involves: dividing the scene into multiple sub-maps, defining a sparse geometric point cloud set and a dense Gaussian tuple set in each sub-map. Each element in the sparse geometric point cloud set consists of the 3D landmark positions in the sparse skeleton and a high-dimensional feature descriptor output by the feature extraction and matching network. Each element in the dense Gaussian tuple set consists of a 3D mean position vector, a covariance matrix, opacity, and view-dependent RGB color. Spatial rigid binding constraints are applied to the dense Gaussian tuples through the sparse geometric point cloud.

[0012] In one possible implementation, configuring a background thread to perform asynchronous progressive rendering optimization includes: constructing a rendering loss function that integrates photometric loss, direct depth constraints, and an isotropic regularization function. The photometric loss is calculated jointly using a norm and a structural similarity index. The direct depth constraints are used to incorporate observation information from a depth sensor. The isotropic regularization function is calculated based on the ratio of the average scale of Gaussian elements within a subgraph to the scale of a single Gaussian element to suppress long strip artifacts in texture-deficient regions. The dense Gaussian element set is progressively updated by minimizing the rendering loss function.

[0013] In one possible implementation, constructing a residual adaptive closed-loop state feedback controller includes: extracting full-image features and calculating cross-similarity through a global descriptor network, and triggering closed-loop candidate detection based on the cross-similarity; responding to the triggering of closed-loop candidates, reusing a feature extraction and matching network to obtain the initial relative pose between the target sub-image and the source sub-image, and calculating the sub-image transformation; calculating the joint error under the current closed-loop view using a differentiable rendering process, and fusing photometric error and depth error in the joint error; when the joint error is lower than a preset threshold, directly outputting the initial relative pose as a relative pose constraint; when the joint error is not lower than the preset threshold, triggering multi-view pose refinement, constructing a weighted rotation average objective function with the joint error as the weight, and minimizing the Frobenius norm of the objective function to estimate the global rotation matrix, and outputting the estimated global rotation matrix as a relative pose constraint.

[0014] In one possible implementation, constructing the rigid body correction update model includes: constructing a global pose graph optimization problem, adding relative pose constraints as edges to the global pose graph, and solving for the optimal sequence of correction deviation matrices; using the sequence of correction deviation matrices as Lie algebraic rotation and translation constraints, and simultaneously updating the 3D landmark positions of the sparse geometric point cloud and the 3D mean position vector and covariance matrix of the dense Gaussian elements in each subgraph to eliminate the system's cumulative drift and satisfy the global spatial consistency condition.

[0015] According to a second aspect of the embodiments of this specification, a Gaussian SLAM device based on a self-crossing feature network and a hybrid map representation is provided, comprising: The pose calculation module is configured to construct a feature extraction and matching network through self-attention and cross-attention mechanisms, and use the network to extract the feature correspondence between image frames and calculate the initial pose of the camera. The skeleton generation module is configured to establish a local beam adjustment module based on co-view relationship between the feature tracker and the backend map. It uses the initial pose as the initial value for optimization and adopts a sliding window model to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. The map representation module is configured to construct a hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units. The sparse geometric point clouds are composed of 3D landmark positions and feature descriptors in a sparse skeleton, and the sparse geometric point clouds and dense Gaussian units are rigidly bound together. The thread configuration module configures independent foreground and background threads for hybrid map representations, so as to perform synchronous initialization and asynchronous progressive rendering optimization of Gaussian primitives respectively. The pose constraint module is configured to reuse the geometric initial values ​​output by the feature extraction and matching network, and construct a residual adaptive closed-loop state feedback controller by combining the joint rendering residual generated by differentiable rasterization. The controller determines whether to activate the multi-view optimization operation based on the energy value of the joint rendering residual, and outputs the relative pose constraint after closed-loop correction. The model update module is configured to construct a rigid body correction update model that includes a global pose graph and a hybrid map representation. It introduces relative pose constraints into the global pose graph, solves the correction transformation matrix, and synchronously updates the sparse keyframe poses and the explicit properties of dense Gaussian elements in all subgraphs.

[0016] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is instructed to perform the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0019] This specification provides a Gaussian SLAM method and apparatus based on self-crossing feature networks and hybrid map representation. The Gaussian SLAM method extracts sparse features and calculates the initial pose through a self-crossing attention mechanism; generates a sparse skeleton using local beam adjustment; constructs a hybrid map rigidly bound to sparse point clouds and dense Gaussian primitives; and performs Gaussian primitive initialization and progressive rendering optimization in parallel using foreground and background threads. It reuses the initial values ​​of the feature network and combines them with differentiable rendering residuals to construct an adaptive closed-loop controller, triggering multi-view optimization as needed. Finally, it solves for the correction matrix through global pose graph optimization, synchronously updating the sparse point cloud and dense Gaussian primitives. This application achieves low-latency, high-precision pose tracking and globally consistent 3D reconstruction. Attached Figure Description

[0020] Figure 1 This is a flowchart of a Gaussian SLAM method based on self-crossing feature networks and hybrid map representations provided in one embodiment of this specification; Figure 2 This is a comparison chart of experimental results of a Gaussian SLAM method based on self-crossing feature networks and hybrid map representation provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the reconstruction result of a Gaussian SLAM method based on a self-crossing feature network and hybrid map representation provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a Gaussian SLAM device based on a self-crossing feature network and hybrid map representation provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0021] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0022] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0024] This specification provides a Gaussian SLAM method based on self-crossing feature networks and hybrid map representations. This specification also relates to a Gaussian SLAM device based on self-crossing feature networks and hybrid map representations, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0025] See Figure 1 , Figure 1 A flowchart of a Gaussian SLAM method based on a self-crossing feature network and hybrid map representation according to an embodiment of this specification is shown, specifically including the following steps.

[0026] Step 101: Construct a feature extraction and matching network through self-attention and cross-attention mechanisms, and use the network to extract the feature correspondence between image frames and solve the initial pose of the camera; Step 102: Establish a local beam adjustment module based on co-view relationship between the feature tracker and the back-end map, use the initial pose as the initial value for optimization, and use the sliding window model to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. Step 103: Construct a hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units. The sparse geometric point clouds are composed of 3D landmark positions and feature descriptors in the sparse skeleton, and the sparse geometric point clouds and dense Gaussian units are rigidly bound together. Step 104: Configure separate foreground and background threads for the hybrid map representation to perform synchronous initialization of Gaussian meta-databases and asynchronous progressive rendering optimization, respectively; Step 105: Reuse the geometric initial values ​​output by the feature extraction and matching network, and construct a residual adaptive closed-loop state feedback controller by combining the joint rendering residual generated by differentiable rasterization. Determine whether to activate the multi-view optimization operation based on the energy value of the joint rendering residual, and output the relative pose constraint after closed-loop correction. Step 106: Construct a rigid body correction update model that includes a global pose graph and a hybrid map representation. Introduce relative pose constraints into the global pose graph, solve for the correction transformation matrix, and synchronously update the sparse keyframe poses and dense Gaussian explicit properties in all subgraphs. The method of this scheme also includes verifying the pose estimation accuracy and rendering quality of the system under conditions of rapid motion and high-resolution reconstruction.

[0027] Among them, self-attention and cross-attention mechanisms refer to the computational units in neural networks used to capture long-distance dependencies between pixels within a single frame and pixel correspondences across frames. For example, the self-attention layer calculates the correlation weights of different positions within the same feature map, and the cross-attention layer calculates the correlation weights between feature maps of two consecutive frames, so that robust sparse feature matching point pairs can still be obtained when the viewpoint changes drastically.

[0028] Feature extraction and matching network: refers to a neural network architecture based on deep learning. Its input is two or more consecutive RGB images, and its output is a set of two-dimensional feature points matched between the images and the corresponding confidence scores. This network enhances the robustness of matching in regions with poor texture and motion blur through an attention mechanism.

[0029] Initial pose: refers to the relative rotation matrix and translation vector estimated by the camera during continuous movement based on the feature matching relationship between the current frame and the previous frame, which serve as the initial values ​​for subsequent local optimization and map building.

[0030] Co-view relationship: refers to the correlation between multiple keyframes that jointly observe the same batch of 3D spatial landmarks. By analyzing the degree of co-view, it is possible to determine which keyframes have strong geometric constraints with the current frame, thereby constructing a local optimization window.

[0031] Local beam adjustment module: refers to an algorithm unit that performs joint nonlinear optimization of keyframe pose and 3D landmark position within a sliding window, reducing local cumulative drift by minimizing reprojection error.

[0032] Sparse skeleton: refers to a set of point clouds composed of landmarks in three-dimensional space and their corresponding feature descriptors. These landmarks are sparsely distributed but geometrically precise, serving as rigid anchors to constrain the deformation of dense Gaussian elements.

[0033] Hybrid map representation: refers to expressing a scene simultaneously as both sparse geometric point clouds and dense Gaussian units, where sparse point clouds provide structural constraints and dense Gaussian units are responsible for high-fidelity texture rendering.

[0034] Rigid binding: refers to the fixed spatial positional relationship between each landmark in a sparse point cloud and multiple neighboring Gaussian elements. When the position of a landmark is updated, the spatial position and shape parameters of the Gaussian elements bound to it are updated synchronously according to the same rigid body transformation.

[0035] Foreground threads and background threads: These refer to two parallel task flows created at the operating system level. The foreground thread is responsible for camera tracking and Gaussian primitive instantiation, which have high real-time requirements, while the background thread is responsible for differentiable rendering optimization and map refinement, which have higher computational costs.

[0036] Synchronous initialization and asynchronous progressive rendering optimization: Synchronous initialization refers to the foreground thread immediately generating new Gaussian primitives based on depth information to cover the newly added field of view when the current frame arrives; asynchronous progressive rendering optimization refers to the background thread performing multiple iterations to optimize the color, opacity, covariance, and other properties of historical Gaussian primitives without blocking foreground tracking.

[0037] Differentiable rasterization refers to the process of projecting three-dimensional Gaussian primitives onto a two-dimensional image plane and generating a rendered image. This process supports gradient backpropagation, thereby allowing the optimization of Gaussian primitive properties through a rendering loss function.

[0038] Joint rendering residual: refers to the difference vector between the rendered image and the real image in two dimensions: luminance (RGB color) and depth. For example, it is a scalar value obtained by weighted summation of the absolute values ​​of pixel color differences and depth differences.

[0039] Residual adaptive closed-loop state feedback controller: refers to a control unit that dynamically decides whether to perform closed-loop refinement based on the size of the current rendering residual. When the residual is small, it directly accepts the initial closed-loop constraint to save computing power, and when the residual is large, it triggers more refined multi-view optimization.

[0040] Relative pose constraint: refers to the transformation relationship between the target subgraph output by the closed-loop detection process and the source subgraph. It is usually represented as a combination of quaternions or rotation matrices and translation vectors, and is used as an edge in the global pose graph optimization.

[0041] Global pose graph: refers to a graph structure in which nodes represent the poses of each keyframe or subgraph, and edges represent the relative transformation constraints between nodes (including sequence constraints and loop closure constraints). Globally consistent optimal pose estimates can be obtained through graph optimization algorithms.

[0042] Corrected transformation matrix: refers to the similarity transformation or Euclidean transformation matrix obtained by optimization through the global pose graph, used to correct the cumulative drift of the system, including rotation, translation and optional scaling components.

[0043] Verification steps: This involves running the system on a simulation dataset or a real robot platform, comparing the estimated camera trajectory with the ground truth trajectory to calculate the absolute trajectory error, and comparing the rendered image with the real image to calculate metrics such as peak signal-to-noise ratio and structural similarity, in order to prove the effectiveness of the method.

[0044] The aforementioned method steps of this application will be further described below through a detailed embodiment.

[0045] This embodiment applies to a mobile robot performing inspection tasks in a large indoor warehouse environment. The robot is equipped with an RGB-D depth camera, and the scene range is approximately 20 meters × 30 meters, including complex structures such as shelves, boxes, and forklifts. The robot needs to move rapidly while simultaneously building a high-fidelity 3D map in real time, and also handle sudden rotations and accelerations.

[0046] First, the system executes step one: constructing a feature extraction and matching network using self-attention and cross-attention mechanisms. This network employs an encoder-decoder structure based on the Transformer architecture. When the robot camera acquires images at 30 frames per second, the system inputs the current frame and the previous frame into the network. The self-attention layer in the network performs contextual aggregation on the feature maps within each frame, ensuring that pixels on the same object receive similar feature representations; the cross-attention layer calculates the correlation matrix between the feature maps of two frames and decodes matching point pairs from it. The network outputs hundreds of matching points, each including the pixel coordinates in the previous frame and the pixel coordinates in the subsequent frame, as well as the matching confidence. Subsequently, the system uses these matching points to calculate the camera's initial pose—that is, the rotation matrix and translation vector from the previous frame to the current frame—by minimizing the reprojection error. Although this initial pose has some drift, it is sufficient as a valid initial value for subsequent optimization.

[0047] Next, in step two, a local beamforming module based on co-view relationships is established between the feature tracker and the backend map. The system maintains an active subgraph containing the 10 most recent keyframes. When a new keyframe is inserted, the module uses the initial pose obtained in step one as the initial estimate for that keyframe, while simultaneously collecting the 3D landmarks jointly observed by that keyframe and other keyframes in the subgraph. The local beamforming module constructs a nonlinear least squares problem, aiming to minimize the sum of reprojection errors of all observed landmarks on each keyframe, and adds an additional constraint to suppress pose drift of the oldest keyframe in the sliding window. Through iterative optimization using the Gauss-Newton method or the Levenberg-Marquardt method, the locally corrected keyframe poses and 3D landmark positions are finally obtained. These 3D landmarks are geographically sparse but precisely located, collectively forming a sparse skeleton describing the scene's geometry.

[0048] Step 3: Construct a hybrid map representation consisting of sparse geometric point clouds and dense Gaussian primitives. The system divides the entire warehouse scene into multiple sub-maps, each covering an area of ​​approximately 5 meters × 5 meters. In each sub-map, the sparse geometric point cloud directly uses the 3D landmarks and their feature descriptors from the sparse skeleton obtained after optimization in Step 2. The dense Gaussian primitives consist of a large number of ellipsoids, each primitive being described by its spatial center position, covariance matrix (determining shape and orientation), opacity, and spherical harmonic coefficients (determining view-related color). The system establishes a rigid binding relationship between each Gaussian primitive and its nearest few sparse landmarks through explicit rules: that is, the update of the spatial position and covariance matrix of the Gaussian primitive must follow the rigid transformation of its bound landmarks.

[0049] Step 4: Configure independent foreground and background threads for the hybrid map representation. The foreground thread runs in a high-priority task of the real-time operating system. Upon receiving a new RGB-D image frame, it immediately generates new Gaussian primitives within the current field of view based on depth information. It then projects existing Gaussian primitives onto the image plane using the currently estimated camera pose and performs a rapid comparison with the real image, without performing time-consuming gradient iteration. The background thread runs at a lower frequency (e.g., 5 times per second), independently acquiring the Gaussian primitive set from the historical sub-image queue. It constructs a joint loss function that fuses photometric loss, depth loss, and isotropic regularization loss. It calculates the gradient of the loss relative to the Gaussian primitive properties using a differentiable rasterizer renderer, and then progressively updates the 3D center, covariance, opacity, and color parameters of the Gaussian primitives using the Adam optimizer. Data is exchanged between the foreground and background threads via a double buffer to ensure that the tracking process is not blocked by rendering optimization.

[0050] Step 5: Reuse the initial geometric values ​​output by the aforementioned feature extraction and matching network, and construct a residual adaptive closed-loop state feedback controller by combining the joint rendering residuals generated by differentiable rasterization. When the robot returns to a previously visited area after patrolling the warehouse, the system first extracts the full-image features of the current frame through the global descriptor network NetVLAD, calculates the similarity with the feature vectors of historical keyframes, and finds candidate sub-images that may form a closed loop. Then, the controller directly reuses the feature extraction and matching network from Step 1 to calculate the initial relative pose between the current frame and the matching frame in the candidate sub-image. Next, the controller uses the differentiable rasterization renderer from Step 4 to generate a rendered image and depth map from the current viewpoint, and calculates the joint residual pixel by pixel with the real image and depth sensor data—the weighted sum of the absolute values ​​of color difference and depth difference. If the joint residual is lower than a pre-set threshold (e.g., color difference less than 5 gray levels and depth difference less than 3 cm), the controller directly outputs the initial relative pose as a valid closed-loop constraint without further optimization, thus saving a significant amount of computation time. If the joint residual is higher than the threshold, the controller triggers multi-view refinement: it calls an optimization algorithm based on weighted rotation average to perform joint pose optimization on multiple views in the closed-loop candidate frame and its neighborhood to obtain a higher-precision global rotation matrix, which is then used as the final closed-loop constraint output.

[0051] Step Six: Construct a rigid body correction and update model incorporating a global pose graph and a hybrid map representation. The system maintains a global pose graph, where nodes represent the reference pose of each subgraph, and edges include relative transformations between sequentially adjacent subgraphs and the closed-loop constraints output in Step Five. After the closed-loop constraints are added, the system calls a graph optimization solver (such as g2o or Ceres Solver) for global optimization, calculating the optimal correction bias matrix sequence for each subgraph pose. Subsequently, the system uses the correction bias matrix sequence as Lie algebraic rotation and translation constraints to perform a rigid body update on the hybrid map representation within each subgraph: for each landmark in the sparse geometric point cloud, its 3D position is left-multiplied by the correction transformation matrix to obtain the new position; for dense Gaussian elements, their 3D center mean vector and covariance matrix are simultaneously updated according to the same rotation and translation transformation. This synchronous update ensures the global consistency of the map, avoiding tearing or ghosting.

[0052] Step 7: Verify the pose estimation accuracy and rendering quality of the system under conditions of rapid motion and high-resolution reconstruction. After running all the aforementioned steps on simulation datasets (such as TUM-RGBD, ScanNet) and real physical robots, the system records the root mean square error of the absolute trajectory error between the camera-estimated trajectory and the ground truth trajectory, and calculates the peak signal-to-noise ratio, structural similarity, and learned perceptual image patch similarity between the rendered image and the ground truth image. Experimental data show that this method can maintain centimeter-level positioning accuracy even when the robot moves at a linear velocity of 1 meter per second and an angular velocity of 90 degrees per second, while outputting high-fidelity rendered views at a rate of more than 30 frames per second, with a peak GPU memory usage of less than 4GB.

[0053] The beneficial effects of this embodiment include: by constructing a Gaussian SLAM method that fuses a self-crossing feature network with a hybrid map representation, sparse geometric skeletons are rigidly bound to dense Gaussian primitives, achieving parallel decoupling of tracking and mapping, significantly reducing system latency and improving real-time mapping capabilities for large-scale scenes. Simultaneously, by using a residual adaptive closed-loop controller to trigger multi-view optimization on demand, the computational resource waste caused by frequent dense alignment in traditional methods is avoided. The global rigid body correction update model ensures the consistency of the map space after cumulative drift elimination. Overall, this method achieves robust localization and mapping for fast-moving and complex large-scale scenes while maintaining high-fidelity rendering quality.

[0054] In one possible implementation, the initial pose of the camera is calculated through a feature extraction and matching network, which includes: inputting the input frame and the previous frame into the network to obtain a set of matching points; performing random sampling consistency screening on the set of matching points to obtain a set of inliers; using depth information to backproject the two-dimensional matching points in the set of inliers into three-dimensional landmarks; reprojecting the three-dimensional landmarks onto the image plane through perspective projection; and using the Huber robust kernel function to minimize the nonlinear geometric reprojection error, thereby obtaining the optimal relative pose estimate as the initial pose.

[0055] Among them, random sampling consistency screening refers to an iterative algorithm used to randomly select the smallest subset from the set of matching points containing outliers (false matches) to estimate the model parameters (here, the essential matrix or homography matrix), count the number of interior points that conform to the model, and finally select the model with the most interior points as the effective model, thereby eliminating false matching point pairs.

[0056] Interior point set: refers to the set of matching point pairs whose reprojection error is less than a preset threshold after being screened by the random sampling consensus algorithm. These point pairs are identified as correct geometric correspondence points and can be used for subsequent pose calculation.

[0057] Back projection: refers to the process of calculating the spatial coordinates of a point in the camera's three-dimensional coordinate system based on the two-dimensional pixel coordinates and the corresponding depth value of that pixel through the inverse operation of the camera's intrinsic parameter matrix.

[0058] Perspective projection operation: refers to the process of mapping points in three-dimensional space to a two-dimensional image plane through the camera intrinsic parameter matrix. It usually includes homogeneous coordinate transformation, dividing by its depth value to obtain normalized coordinates, and then multiplying by the focal length and principal point offset to obtain pixel coordinates.

[0059] Huber robust kernel function: refers to a loss function that behaves as a quadratic function when the error is small and as a linear function when the error is large. It is used to suppress the impact of large errors (such as mismatches) on the optimization results and improve the robustness of pose estimation.

[0060] Nonlinear geometric reprojection error: refers to the sum of squared Euclidean distances between the coordinates of the projected points of 3D landmarks on the current frame and the coordinates of the actual observed feature points. This error is nonlinear with respect to camera pose and is usually solved iteratively using a perturbation model on Lie algebra.

[0061] Optimal relative pose estimation: refers to the rotation matrix and translation vector that minimizes the sum of reprojection errors, usually represented as an element on a special Euclidean group.

[0062] In conjunction with the foregoing embodiments, this section further describes in detail the specific implementation method of solving the initial pose of the camera through a feature extraction and matching network.

[0063] During robot inspection, let's assume the previous frame is denoted as frame A and the current frame as frame B. First, frames A and B are input into a pre-trained self-crossing feature network. The network outputs two sets of feature maps and uses a feature matching layer to calculate the corresponding matching point in frame B for each feature point in frame A, resulting in an initial set of matching points. This set may contain hundreds of matching points, some of which may be mismatched.

[0064] To eliminate false matches, the system executes a random sampling consensus algorithm: 8 pairs of matching points are randomly selected from the initial set of matching points, the fundamental matrix or essential matrix is ​​calculated, and then the projection error of the remaining matching points under the model is calculated. Matching points with an error of less than 3 pixels are marked as inliers. This random sampling process is repeated 500 times, and the model with the most inliers is selected. The set of inliers corresponding to this model is then used as the valid set of matching points.

[0065] Next, the system uses the depth information provided by the depth camera to backproject each 2D matching point in the inlier set. Specifically, for a pixel coordinate in frame A, the system reads the depth value of that pixel from the depth map, and then calculates the 3D coordinates of that point in the camera coordinate system of frame A based on the focal length and principal point coordinates of the camera's intrinsic parameters. Similarly, matching points in frame B can also be backprojected into 3D points, but typically only the depth of one frame is used to construct the 3D landmark.

[0066] The system transforms the 3D landmark obtained by backprojection (located in the frame A coordinate system) to the frame B coordinate system using the current relative pose to be estimated (rotation matrix and translation vector), and then calculates its projected coordinates on the image plane of frame B through perspective projection. The Euclidean distance between these projected coordinates and the actual matching point coordinates in frame B is the reprojection error.

[0067] Finally, the system uses the sum of squared reprojection errors of all inliers as the optimization objective and introduces the Huber robust kernel function to replace the ordinary squared loss. The optimization variable is the Lie algebra representation of the relative pose, and the Levenberg-Marquardt algorithm is used for iterative solution. In each iteration, the error Jacobian matrix is ​​calculated, the linear increment equation is solved, and the pose parameters are updated until the error change is less than a threshold or the maximum number of iterations is reached. The final output optimal relative pose estimate is the initial pose described in the above embodiment.

[0068] The beneficial effects of this embodiment include: by combining the matching point set output by the self-cross feature network with random sampling consistency screening, mismatches caused by motion blur or texture repetition are effectively eliminated, improving the robustness of pose estimation. The Huber robust kernel function is used to further suppress the influence of abnormal errors, ensuring that the initial pose estimation maintains convergence under complex motion, providing reliable initial values ​​for subsequent local beam adjustment and loop closure detection.

[0069] In one possible implementation, the local beam adjustment module for generating the sparse skeleton includes: maintaining a sliding window containing a predetermined number of recent keyframes within the activated subgraph; using the initial pose as the initial estimate of the keyframes within the sliding window; constructing an objective function that includes a reprojection error term and a normalization fixation term; introducing a penalty factor in the normalization fixation term to constrain the pose drift of the oldest keyframe in the sliding window; performing local beam adjustment by minimizing the objective function; and outputting the jointly optimized local trajectory and 3D spatial landmarks as the sparse skeleton.

[0070] Among them, the activated sub-map refers to a set of keyframes and their associated 3D landmarks that are currently being actively updated and optimized by the system and covered by the current camera's field of view. The entire map is usually divided into multiple sub-maps according to spatial proximity, and each time only the sub-map closest to the current location is activated for local optimization.

[0071] Sliding window: refers to a fixed-length first-in-first-out queue that stores the pose information and observed landmark indices of the most recent keyframes. Older keyframes outside the window are fixed or marginalized and no longer participate in the current local optimization.

[0072] Predetermined number of nearest keyframes: refers to the sliding window capacity set by system parameters. For example, if it is set to 10 keyframes, the oldest keyframe will be removed from the window when the 11th keyframe is inserted.

[0073] Normalization fixed term: refers to an additional cost term added to the objective function to constrain the pose of the oldest keyframe in the sliding window from changing significantly. It is usually in the form of the Mahalanobis distance between the current estimated pose of the oldest frame and its prior value multiplied by a penalty factor.

[0074] Penalty factor: refers to a positive scalar coefficient used to control the weight of the normalization fixed term in the entire optimization objective. The larger the value, the tighter the pose of the oldest frame is constrained, which helps to prevent drift from propagating outside the window.

[0075] Jointly optimized local trajectories and 3D spatial landmarks: refers to the pose sequence of all keyframes within the sliding window obtained after local beam adjustment optimization, as well as the coordinates of all 3D landmarks jointly observed by these keyframes. These results have local consistency.

[0076] In conjunction with the foregoing embodiments, this section describes in detail the specific process by which the local beam adjustment module generates a sparse skeleton.

[0077] During continuous robot movement, the system maintains an activation subgraph. Assume the sliding window capacity is set to the 10 most recent keyframes, and the area within the window is denoted as a keyframe group. When a new keyframe (frame 11) is determined by the system to need insertion, the sliding window mechanism removes the oldest keyframe (frame 1) from the window, but retains its observed landmarks as fixed references.

[0078] First, the system uses the initial pose calculated in the above embodiment as the initial estimate of the new key frame, and the poses of the remaining 9 key frames in the window use the result of the previous round of local optimization as the initial value.

[0079] Then, a target function for local beam adjustment is constructed. This function consists of two parts: the first part is the reprojection error term, which calculates the sum of squared Huber norms between the projected coordinates and the actual observed coordinates of each 3D landmark observed in all keyframes within the window; the second part is the normalization fixation term, which multiplies the Mahalanobis distance between the Lie algebra difference between the current estimated pose of the oldest keyframe in the window (currently frame 2, since frame 1 has been removed) and the previous prior pose after optimization by a penalty factor.

[0080] The value of the penalty factor is adaptively adjusted according to the system runtime. For example, the penalty factor is smaller at the beginning of system startup to allow for larger adjustments, and the penalty factor is increased after the system stabilizes to suppress drift.

[0081] The objective function is solved iteratively using a graph optimization tool. Each iteration calculates the Jacobian matrix of all error terms relative to the keyframe pose and 3D landmark position, then solves the linear equations to obtain the incremental step size, and updates the state variables. The iteration stops when the number of iterations reaches 20 or the pose increment is less than 0.001 radians / cm.

[0082] After optimization, the poses of all keyframes within the sliding window are locally and consistently adjusted, and the spatial coordinates of the 3D landmarks observed by these keyframes are also optimized synchronously. These 3D landmarks constitute a sparse point cloud, and each landmark is associated with the feature descriptor described in the above embodiments. The system uses these optimized 3D landmark sets as a sparse skeleton and outputs it to the subsequent hybrid map construction module.

[0083] The beneficial effects of this embodiment include: by using a sliding window local beam adjustment mechanism, optimization is performed only on neighboring keyframes and landmarks, avoiding the high computational overhead of global optimization. Normalization of fixed terms effectively constrains the drift of the oldest frame, preventing errors from accumulating too quickly. The output sparse skeleton has high local geometric accuracy, providing reliable rigid anchor points for dense Gaussian elements, thereby improving the overall mapping accuracy.

[0084] In one possible implementation, constructing a hybrid map representation involves: dividing the scene into multiple sub-maps, defining a sparse geometric point cloud set and a dense Gaussian tuple set in each sub-map. Each element in the sparse geometric point cloud set consists of the 3D landmark positions in the sparse skeleton and a high-dimensional feature descriptor output by the feature extraction and matching network. Each element in the dense Gaussian tuple set consists of a 3D mean position vector, a covariance matrix, opacity, and view-dependent RGB color. Spatial rigid binding constraints are applied to the dense Gaussian tuples through the sparse geometric point cloud.

[0085] Among them, a subgraph refers to a large-scale scene divided into several overlapping regions according to spatial location. Each subgraph independently manages its internal keyframes, sparse point clouds, and Gaussian units, which facilitates parallel processing and closed-loop correction.

[0086] Sparse geometric point cloud set: refers to a point cloud composed of multiple sparse landmarks, each landmark having three-dimensional coordinates and a high-dimensional feature descriptor, the number of which is much smaller than the number of dense Gaussian units.

[0087] High-dimensional feature descriptor: refers to a floating-point vector extracted from the image by the feature extraction and matching network in the above embodiments, used to characterize the local image texture and geometric information. It usually has a dimension of 128, 256 or higher, and is used for subsequent feature matching and loop closure detection.

[0088] Dense Gaussian primitive set: refers to a set consisting of a large number of Gaussian ellipsoids, each primitive independently storing its spatial properties (3D mean vector, covariance matrix) and optical properties (opacity, spherical harmonic coefficients), for use in differentiable rasterization rendering.

[0089] Three-dimensional mean position vector: refers to the coordinates of the center point of the Gaussian element in three-dimensional space, usually represented by three components.

[0090] Covariance matrix: refers to a three-dimensional symmetric positive definite matrix used to control the scale of Gaussian elements in various directions. Its eigenvectors determine the orientation of the elements, and its eigenvalues ​​determine the length of each axis.

[0091] Opacity: A scalar value between 0 and 1 used to control the transparency of Gaussian elements during rendering.

[0092] View-dependent RGB color: refers to the color presented by Gaussian elements at a specific viewpoint, usually obtained by interpolation calculation of different viewpoints using spherical harmonic function coefficients.

[0093] Spatial rigid binding constraint: refers to mathematically establishing a transformation synchronization relationship between sparse landmarks and surrounding dense Gaussian elements. When the position of a landmark moves due to global optimization, the mean vector and covariance matrix of the Gaussian elements bound to it are updated according to the same rigid body transformation.

[0094] In conjunction with the foregoing embodiments, this section describes in detail the process of constructing a hybrid map representation.

[0095] The system first divides the warehouse scene into multiple sub-maps, each covering a cube region with a side length of 5 meters. Adjacent sub-maps overlap by 1 meter to ensure a smooth transition. For each sub-map, the system maintains its map content independently.

[0096] When constructing the sparse geometric point cloud set, the system directly obtains the 3D landmark locations from the sparse skeleton output in the above embodiments. Simultaneously, for each landmark, the system backtracks to observe the keyframes of that landmark and extracts a high-dimensional feature descriptor (256-dimensional floating-point vector) of the image location of that landmark from the feature extraction and matching network in the above embodiments. This descriptor is stored together with the landmark location.

[0097] When constructing a dense Gaussian primitive set, the system employs a densification strategy. For each new keyframe, a Gaussian primitive is generated at each pixel location based on the depth image. The 3D mean position vector of this primitive is set to the spatial coordinates obtained by backprojecting the pixel; the covariance matrix is ​​initialized as an isotropic diagonal matrix, with diagonal elements set to a scale value proportional to the depth value (e.g., one percent of the depth value); the opacity is initialized to 0.5; and the color is initialized to the zero-order term of the spherical harmonic coefficients corresponding to the pixel's RGB value. As the system runs, a background thread continuously optimizes the properties of these primitives.

[0098] To establish a rigid binding constraint between sparse geometric point clouds and dense Gaussian primitives (GRPs), the system employs a nearest neighbor association strategy. For each GRP, the Euclidean distance between its 3D mean position and all sparse landmarks within the subgraph is calculated. The three nearest landmarks are selected as anchor points for this primitive, and the anchor point indices and the local offset vector of the GRP relative to each anchor point are stored. When subsequent sparse landmarks update their positions due to global loop closure optimization, the system calculates a correction transformation for each GRP based on the difference between the old and new anchor positions using a weighted average. This transformation is then applied to the mean vector and covariance matrix of the GRP. Specifically, the mean vector is directly left-multiplied by a rotation-translation matrix, and the covariance matrix is ​​updated through a similarity transformation. That is, the new covariance is equal to the rotation matrix multiplied by the original covariance and then multiplied by the transpose of the rotation matrix. Simultaneously, the scaling factor is adjusted proportionally according to the translation portion.

[0099] Through the above method, the hybrid map representation within each subgraph is constructed, and the sparse skeleton and dense primitives form a rigid binding relationship, laying the foundation for subsequent real-time tracking and high-fidelity rendering.

[0100] The beneficial effects of this embodiment include: by dividing the scene into subgraphs and independently constructing a sparse-dense hybrid representation for each subgraph, efficient management of large-scale scenes is achieved. Sparse geometric point clouds provide accurate geometric anchor points, while dense Gaussian primitives provide high-fidelity textures. The rigid binding constraint between the two ensures the spatial consistency of the map during global correction, avoiding map tearing caused by pose updates.

[0101] In one possible implementation, configuring a background thread to perform asynchronous progressive rendering optimization includes: constructing a rendering loss function that integrates photometric loss, direct depth constraints, and an isotropic regularization function. The photometric loss is calculated jointly using a norm and a structural similarity index. The direct depth constraints are used to incorporate observation information from a depth sensor. The isotropic regularization function is calculated based on the ratio of the average scale of Gaussian elements within a subgraph to the scale of a single Gaussian element to suppress long strip artifacts in texture-deficient regions. The dense Gaussian element set is progressively updated by minimizing the rendering loss function.

[0102] Background threads refer to low-priority computational tasks that run in parallel with the main tracing thread. They are used to iteratively optimize the constructed Gaussian meta-levels without blocking the real-time front-end.

[0103] Asynchronous progressive rendering optimization: refers to a background thread independently executing optimization iterations at a low frequency (e.g., several times per second), updating only a portion of the high-order primitives or a portion of the subgraph each time, gradually improving the rendering quality.

[0104] Photometric loss: refers to the difference between the rendered image and the real RGB image. It usually includes two parts: L1 norm (absolute difference sum) and structural similarity index. The former measures the pixel-level brightness difference, and the latter measures the similarity of local texture structure.

[0105] Norm: Here it refers to the L1 norm, which is the sum of the absolute differences between the three color channels of each pixel.

[0106] Structural similarity index: refers to an index that measures the similarity between two images in three dimensions: brightness, contrast and structure. The value ranges from 0 to 1, and the closer it is to 1, the more similar the images are.

[0107] Depth direct constraint: refers to using the difference between the real depth map collected by the depth sensor and the rendered depth map as the optimization target, usually calculated using L1 or L2 norm.

[0108] Isotropic regularization function: refers to a penalty term used to encourage Gaussian elements to have scales as close as possible in all directions, that is, the eigenvalues ​​of the covariance matrix tend to be equal, thereby avoiding the appearance of elongated striped artifacts in texture-poor or weakly observed regions.

[0109] Average scale: refers to the arithmetic mean of the three scale components (square roots of the eigenvalues ​​of the covariance matrix) of all Gaussian elements within the subgraph.

[0110] Long stripe artifacts: These refer to unnatural stripes or elongated spots that appear in a rendered image, usually caused by the Gaussian unit being too large in a certain direction.

[0111] Incremental update: This refers to performing only one step of gradient descent on the loss function in each optimization iteration, updating a small portion of the Gaussian meta-attributes, and gradually converging after multiple iterations.

[0112] In conjunction with the foregoing embodiments, this section describes in detail the process of background threads performing asynchronous progressive rendering optimization.

[0113] During robot operation, the system creates a separate background thread. This thread is set to run 5 times per second, or one round of optimization every 200 milliseconds. The background thread does not directly participate in camera tracking or keyframe insertion, but focuses on improving the rendering quality of the already constructed map.

[0114] The background thread first retrieves the dense Gaussian set of the current active subgraph and its two neighboring subgraphs, along with the corresponding keyframe data, from shared memory. Then, it constructs a rendering loss function consisting of three components: The first component is the photometric loss. A background thread randomly selects four keyframes from the keyframes within the sub-image. For each keyframe, a differentiable rasterizer is called to project Gaussian primitives onto the viewpoint of that keyframe, generating a rendered RGB image. Then, the rendered image is compared with the actual RGB image of that keyframe. Specifically, the photometric loss equals the L1 norm plus the complement of a structural similarity metric with a weight of 0.2. The L1 norm calculates the sum of the absolute differences of the three channels of all pixels, and the structural similarity metric calculates the structural similarity between the two images; taking its reciprocal minimizes the loss while maximizing the similarity.

[0115] The second component is the direct depth constraint. For the same batch of selected keyframes, the renderer simultaneously outputs a depth map (i.e., the center depth of the Gaussian pixel with the highest opacity on each pixel). A background thread retrieves the true depth map corresponding to the keyframe, removes invalid depths (such as zero values ​​or depths exceeding a threshold), and calculates the average of the absolute differences between the rendered depth and the true depth. This average is multiplied by a weighting factor of 0.1 and then added to the total loss.

[0116] The third component is the isotropic regularization function. A background thread iterates through all Gaussian primitives within the current subgraph, calculating the square roots of the three eigenvalues ​​of the covariance matrix for each primitive (i.e., the scales along the three axes). Then, the average scale of all primitives within the subgraph is calculated. For each primitive, the isotropic regularization term equals the sum of the variances of the ratios of the primitive's three scales to the average scale, multiplied by a small regularization coefficient (e.g., 0.01). This term encourages the three scales of each primitive to approach equality, thus avoiding the generation of elongated stripe artifacts in regions lacking depth constraints (such as white walls).

[0117] The total loss function is a weighted sum of the above three terms. A background thread calculates the gradient of the total loss with respect to the mean vector, covariance matrix, opacity, and spherical harmonic coefficients of each Gaussian element, and then performs a gradient descent update on the Gaussian attributes using the Adam optimizer at a learning rate of 0.01. After the update, the background thread writes the updated Gaussian elements back to shared memory and waits for the next scheduling cycle.

[0118] Because the background thread and the foreground tracing thread run in parallel and their shared data is protected by double buffering or read-write locks, the real-time performance of the foreground is unaffected. After hundreds of asynchronous iterations, the rendering quality of Gaussian primitives gradually improves to its optimal level.

[0119] The beneficial effects of this embodiment include: by asynchronously executing progressive rendering optimization in a background thread, time-consuming gradient calculations are decoupled from real-time tracking, ensuring a high frame rate for the front end. The loss function, which integrates photometric, depth, and isotropic regularization, allows Gaussian primitives to maintain high-fidelity textures while suppressing artifact generation in texture-poor regions. The progressive update mechanism avoids excessive computational load in a single optimization, enabling the system to continuously improve map quality with limited computing power.

[0120] In one possible implementation, constructing a residual adaptive closed-loop state feedback controller includes: extracting full-image features and calculating cross-similarity through a global descriptor network, and triggering closed-loop candidate detection based on the cross-similarity; responding to the triggering of closed-loop candidates, reusing a feature extraction and matching network to obtain the initial relative pose between the target sub-image and the source sub-image, and calculating the sub-image transformation; calculating the joint error under the current closed-loop view using a differentiable rendering process, and fusing photometric error and depth error in the joint error; when the joint error is lower than a preset threshold, directly outputting the initial relative pose as a relative pose constraint; when the joint error is not lower than the preset threshold, triggering multi-view pose refinement, constructing a weighted rotation average objective function with the joint error as the weight, and minimizing the Frobenius norm of the objective function to estimate the global rotation matrix, and outputting the estimated global rotation matrix as a relative pose constraint.

[0121] Among them, the global descriptor network refers to a deep learning-based image global feature extraction network, such as NetVLAD. It takes the entire image as input and outputs a fixed-length feature vector, which encodes the layout and appearance of the entire scene for fast scene retrieval.

[0122] Cross-similarity: refers to the inner product or cosine distance between the global descriptor of the current frame and the global descriptor of historical keyframes. The higher the similarity, the closer the scene appearance is, and the more likely it is to form a closed loop.

[0123] Loop closure candidate detection: refers to filtering out several historical keyframes or sub-graphs that may form a loop with the current frame based on the cross-similarity exceeding a certain threshold.

[0124] Reusing the feature extraction and matching network: This refers to directly calling the same feature extraction and matching network used for front-end tracking in the above embodiments, without retraining or loading a new network, thereby saving computing resources.

[0125] Initial relative pose: refers to the coarse transformation relationship between the source subgraph and the target subgraph obtained through feature matching and essential matrix decomposition.

[0126] Joint error: refers to the weighted sum of photometric and depth errors between the rendered image obtained by differentiable rendering and the real image, and is used as an adaptive metric to measure the closed-loop quality.

[0127] Multi-view pose refinement: When the coarse transformation is not accurate enough, the accuracy of pose estimation is further improved by using the observation data of multiple candidate matching frames and through weighted averaging or nonlinear optimization.

[0128] Weighted rotation average: refers to a method that integrates multiple relative rotation estimates to obtain the global optimal rotation. The weight of each relative rotation is determined by its corresponding joint error, and the smaller the error, the greater the weight.

[0129] Frobenius norm: refers to the square root of the sum of squares of matrix elements, used to measure the deviation between the estimated value and the true value of a rotated matrix.

[0130] In conjunction with the foregoing embodiments, this section describes in detail the construction and operation process of the residual adaptive closed-loop state feedback controller.

[0131] When the robot returns to a previously visited area after a large-scale patrol, the system initiates a loop closure detection process. First, the global descriptor network NetVLAD performs forward inference on the RGB image of the current frame, outputting a 4096-dimensional global feature vector. The system calculates the inner product of this vector with the global feature vectors of all historical keyframes to obtain a similarity score. When the highest similarity score exceeds a threshold of 0.7 and the corresponding historical keyframe is more than 50 frames apart from the current frame, loop closure candidate detection is triggered. The subgraph to which the historical keyframe belongs is marked as the target subgraph, and the subgraph to which the current frame belongs is marked as the source subgraph.

[0132] In response to the triggering of closed-loop candidates, the system reuses the same self-crossing feature extraction and matching network used in the above embodiments, directly takes the matching frame with the highest similarity between the current frame and the target subgraph as input, outputs the matching point set, and then obtains the initial relative pose, i.e. the rotation matrix and translation vector from the source subgraph to the target subgraph, through the random sampling consensus algorithm and essential matrix decomposition.

[0133] Next, the system uses the differentiable rendering process described in the above embodiments to generate a rendered image and a rendered depth map from the perspective of the target sub-image, and compares them with the real image and real depth map of the current frame. The joint error is calculated as follows: the average of the absolute photometric errors of all pixels is multiplied by a weight of 0.7, and the average of the absolute depth errors of all effective depth pixels is multiplied by a weight of 0.3 to obtain a single scalar value.

[0134] The system presets a joint error threshold, for example, the average grayscale difference of 5 and the average depth difference of 0.03 meters, which is normalized to 0.1. If the calculated joint error is less than 0.1, it means that the initial relative pose is accurate enough. The controller directly outputs the initial relative pose as a closed-loop constraint to the global pose graph optimization module without performing any additional optimization, thereby avoiding unnecessary computational overhead.

[0135] If the joint error is not less than 0.1, it indicates a significant deviation in the initial relative pose, requiring refinement. The controller triggers multi-view pose refinement: First, select the five keyframes with the highest co-view degree from both the target and source sub-images to form a multi-view set. Then, repeat the aforementioned feature matching and initial pose estimation process for each pair (source keyframe, target keyframe) in the set to obtain multiple relative rotation estimates, each with a joint error as a weight. The controller constructs a weighted rotation average objective function to find a global rotation matrix that minimizes the weighted sum of squared errors between this matrix and each relative rotation estimate, using the Frobenius norm as the error metric. This optimization problem can be solved using iterative weighted least squares or gradient descent on a Riemannian manifold. The optimal global rotation matrix obtained after optimization convergence, along with the translation vector recalculated based on the rotation matrix, constitutes the final relative pose constraint and is output to the global pose graph optimization module.

[0136] The beneficial effects of this embodiment include: By employing a residual adaptive mechanism, multi-view refinement is dynamically determined based on the joint rendering error, directly using the initial pose in most loop closure scenarios, significantly reducing the computational load of loop closure detection. Refinement is only activated when the initial pose accuracy is insufficient, ensuring the reliability of loop closure correction. The weighted rotational averaging method utilizes the joint error as weights, effectively suppressing the impact of abnormal matching. Overall, on-demand computation for loop closure detection is achieved, significantly reducing system lag and wasted computing power.

[0137] In one possible implementation, constructing the rigid body correction update model includes: constructing a global pose graph optimization problem, adding relative pose constraints as edges to the global pose graph, and solving for the optimal sequence of correction deviation matrices; using the sequence of correction deviation matrices as Lie algebraic rotation and translation constraints, and simultaneously updating the 3D landmark positions of the sparse geometric point cloud and the 3D mean position vector and covariance matrix of the dense Gaussian elements in each subgraph to eliminate the system's cumulative drift and satisfy the global spatial consistency condition.

[0138] The global pose graph optimization problem refers to a nonlinear least squares problem organized in a graph structure. Nodes in the graph represent the poses of each subgraph or keyframe, and edges represent the relative transformation constraints between nodes (including order constraints and loop constraints). The optimization objective is to minimize the sum of squared errors of all edges in order to obtain the globally consistent optimal pose.

[0139] Edge: Refers to a directed or bidirectional connection between two nodes in the graph. Each edge carries a measurement value (relative pose) and an information matrix (uncertainty). In this method, closed-loop constraints are added as new edges to the global pose graph.

[0140] Correction bias matrix sequence: refers to the adjustment amount of the pose of each node obtained by graph optimization, usually expressed as a tangent vector on the Lie algebra or directly as a transformation matrix on the Lie group.

[0141] Lie algebraic rotation and translation constraints refer to treating the modified transformation matrix as an element on a special Euclidean group, updating it by left-multiplying the coordinates of the sparse point cloud and Gaussian elements, while preserving the rigid body properties of the transformation.

[0142] Synchronous algebraic update: refers to applying the same transformation matrix to all map elements (sparse landmarks and dense Gaussian elements) at the same time, ensuring that the entire subgraph moves as a rigid body and avoiding internal relative deformation.

[0143] Global spatial consistency condition: After correction, the poses and map elements of all sub-graphs match each other in the world coordinate system, and the stitching error of the closed loop region is less than a preset threshold.

[0144] In conjunction with the foregoing embodiments, this section describes in detail the construction and execution process of the rigid body correction update model.

[0145] After the controller outputs one or more closed-loop constraints, the system initiates global pose graph optimization. The global pose graph already contains all subgraph nodes and their sequentially adjacent edges (i.e., relative transformations of the subgraphs in adjacent time or spatial locations). Now, closed-loop constraints are added as new edges to the pose graph. Each closed-loop edge connects the source subgraph node and the target subgraph node, and the edge measurement value is the output relative pose constraint. The information matrix is ​​set according to the inverse of the joint error.

[0146] The constructed global pose graph contains, for example, 50 subgraph nodes and several edges. The system calls a graph optimization solver (such as Ceres Solver) for optimization. The optimization variable is the six-DOF pose (3D translation plus 3D rotation, represented using quaternions or Lie algebras) of each subgraph node. The optimization objective is the sum of squared Mahalanobis distances of the errors on all edges. The solver uses the Levenberg-Marquardt algorithm iteratively, converging when the number of iterations reaches 100 or the pose change is less than 1e-6. The optimization result outputs a sequence of correction bias matrices for each subgraph node. Each correction bias matrix is ​​a 4x4 rigid body transformation matrix, representing the translation and rotation to be applied to that subgraph.

[0147] After obtaining the correction bias matrix sequence, the system performs synchronous algebraic updates on the hybrid map representation within each subgraph. For each 3D landmark location in the sparse geometric point cloud within the subgraph, the system left-multiplies its homogeneous coordinates by the correction bias matrix to obtain new 3D coordinates. For each primitive in the dense Gaussian element set, the system simultaneously updates its 3D mean position vector (also left-multiplied by the correction bias matrix) and covariance matrix. The covariance matrix is ​​updated as follows: the new covariance equals the top-left 3x3 rotated portion of the correction bias matrix multiplied by the original covariance, and then multiplied by the transpose of that rotated portion, while ensuring that the translation portion does not affect the covariance. This update ensures that the shape and orientation of the Gaussian elements rotate and scale along with the rigid body of the subgraph without shear deformation.

[0148] After all sub-maps are updated, the system recalculates the map stitching error of the closed-loop region. Experiments show that after using this rigid body correction update model, the point cloud stitching error of the closed-loop region can be reduced from tens of centimeters before the loop closure to within centimeters. The rendered image no longer shows ghosting or breaks at the closed-loop boundary, satisfying the global spatial consistency condition.

[0149] The beneficial effects of this embodiment include: updating the model through global pose graph optimization and rigid body correction efficiently propagates closed-loop constraints to all subgraphs and map elements. The synchronous algebraic update method avoids the high computational cost of per-primary-level re-optimization while ensuring spatial consistency between sparse point clouds and dense Gaussian properties. Lie algebraic rotation and translation constraints preserve the rigidity of the map, effectively eliminating cumulative drift generated over long-term operation, resulting in good global consistency in large-scale scene reconstruction.

[0150] In one overall embodiment, the technical solution of this application is specifically described as follows.

[0151] In practical robot perception processes, a single dense representation model often struggles to fully capture the combined demands of complex tracking constraints and high-fidelity rendering. Therefore, to overcome the limitations of traditional implicit or explicit deterministic models, this application adopts the following hybrid map representation architecture based on sparse point clouds and dense Gaussian clouds: We divide the scene into multiple sub-graphs For efficient management, the blended map contains a sparse geometric point cloud for each submap. and dense Gausky set The characterization equation for a three-dimensional Gaussian set is defined as follows: (1.1) in This represents the three-dimensional mean position vector of the Gaussian element in space. This represents the covariance matrix that controls its shape and orientation. Indicates opacity. This represents the RGB color related to the viewpoint. To ensure a rigid binding between the geometric skeleton and the dense texture, we define the sparse geometry set as: (1.2) in, For the location of three-dimensional road signs, This is a high-dimensional feature descriptor extracted from a self-crossing feature network. After local optimization, the subgraph will rigorously encapsulate these two representations, forming a rigid body. Since the tracking process of a SLAM system must be robust to rapid camera movement, the matching results from a self-crossing feature network are incorporated into the optimization framework. Given an input frame... Compared to the previous frame The neural network outputs a reliable set of matching points. The optimal relative pose estimation is obtained by minimizing the nonlinear geometric reprojection error. : (1.3) in, This represents the camera intrinsic parameter matrix. It is the set of interior points that have undergone rigorous screening using the Random Sampling Consensus (RANSAC) method. This represents the feature pixels extracted from the current frame. This represents a 3D landmark obtained by backprojection using depth information. This indicates a perspective projection operation. This represents the Huber robust kernel function for suppressing nonlinear perturbation gain.

[0152] To overcome local drift and complete local graph optimization within a specified time, the system maintains a region containing [a specific element] within the activated subgraph. A sliding window of the most recent keyframe And perform local beam adjustment (BA). The optimized objective function is designed as follows: (1.4) in Indicates in keyframe The set of landmark indices observed in the data, with the latter term introducing a penalty factor. The normalized fixed term is used to enforce constraints on the oldest frame in the sliding window. The pose drift.

[0153] The system performs a decoupled and parallel progressive Gaussian mapping in the background. An optimized rendering loss function is also used. It integrates multiple indicators: (1.5) Photometric loss through Norm and Structural Similarity-Indices (SSIM) jointly constrain: (1.6) At the same time, direct constraints for depth sensors were introduced. And an isotropic regularization function To suppress the generation of long strip artifacts in areas with missing textures: (1.7) in Indicates the first The scale of a Gaussian unit, The average scale within the subgraph.

[0154] To address the unavoidable cumulative errors in large-scale scenes and achieve on-demand resource allocation, this application designs an adaptive residual closed-loop registration triggering mechanism: firstly, full-image features are extracted through the global descriptor network NetVLAD, and then cross-similarity is calculated. This triggers loop closure candidates. Subsequently, the front-end self-crossing feature network is directly reused to obtain the target sub-graph. With source graph Initial relative pose between Calculate subgraph transformation The joint error of the current closed-loop viewpoint is calculated using differentiable rendering. : (1.8) in and For depth and opacity masks, through - The mixture weights the photometric and depth losses. When the calculated residual error... Below the strict threshold In this case, it is directly transmitted to the global pose graph as the final closed-loop constraint, thus avoiding iterative redundancy. If This triggers multi-view pose refinement, constructing a weighted rotation average objective function to estimate a robust global rotation matrix. : (1.9) in This represents the Frobenius norm. This nonlinear mechanism, which uses residual values ​​as weights, forces the system to suppress outliers of hyperluminosity errors during closed-loop optimization. Finally, the system constructs a global pose graph optimization to solve for the optimal sequence of correction bias matrices. To completely eliminate system cumulative drift, perform the following algebraic update on the rigid bodies in the hybrid map: (1.10) (1.11) This satisfies the stability condition of global spatial consistency, which shows that the hybrid representation SLAM system of this application can achieve a high degree of matching between large-scale mapping and positioning accuracy with millisecond-level latency.

[0155] This application targets publicly available indoor datasets (such as TUM-RGBD, ScanNet, and Replica) with highly dynamic and complex motion and large-scale features, as well as real-world physical mobile robot applications. Figure 2 Tables 1 through 4 illustrate the superior performance of this application. Table 1 shows the camera pose estimation results of different SLAM methods on the TUM dataset; Table 2 shows the camera pose estimation results of different SLAM methods on the Scannet dataset; Table 3 shows the rendering quality of different SLAM methods based on novel perspective synthesis on the TUM, Scannet, and Replica datasets; and Table 4 shows the spatiotemporal performance metrics of different SLAM methods based on novel perspective synthesis on the Replica dataset. Figure 2Table 3, together with the high-fidelity rendering qualitative images and quantitative evaluation metrics (PSNR, SSIM, LPIPS) of this application on different datasets, is presented. The results show that this method effectively overcomes the defects of structural distortion and over-smoothing in existing technologies, successfully reconstructing a dense model with extreme details (such as bicycle wheel spokes and desk edges in complex scenes), achieving extremely high spatial consistency and rendering fidelity. Tables 1 and 2 detail the camera pose estimation results (absolute trajectory error ATE-RMSE) of the system on the TUM and ScanNet datasets, respectively. Data validation shows that after introducing the feature extraction network and closed-loop correction mechanism of this application, the system effectively eliminates trajectory drift under large-scale and complex viewpoint changes, achieving highly robust centimeter-level high-precision tracking. Table 4 further presents a quantitative comparison of the spatiotemporal performance metrics of different SLAM methods. Thanks to the decoupled asynchronous dual-path parallel mapping architecture, this system not only maintains a super real-time frame rate (FPS) while ensuring high-quality dense reconstruction, but also has a peak GPU memory usage that is much lower than existing 3D high-fidelity mapping methods, which greatly reduces the hardware resource threshold and computational congestion risk for high-fidelity mapping. Figure 3 The paper demonstrates the trajectory of a real physical mobile robot in a real-world testing environment and the results of online scene reconstruction. This intuitively and further proves that the algorithm proposed in this application is fully capable of meeting the requirements of online deployment in complex real-world environments, and possesses strong engineering application potential and operational robustness.

[0156] Table 1

[0157] Table 2

[0158] Table 3

[0159] Table 4

[0160] This application constructs a hybrid map representation system with strict spatial structure constraints, relying on sparse point clouds and dense 3D Gaussian primitives. Under this representation architecture, the system utilizes highly robust geometric point clouds extracted by a self-crossing feature network as underlying physical anchors and innovatively proposes a parallel progressive mapping method oriented towards the temporal evolution state of subgraphs. Specifically, this mechanism divides the rendering update process of the Gaussian map into active and historical states in the spatiotemporal dimension: on the one hand, the tracking thread, while maintaining pose calculation, collaboratively drives the synchronous instantiation and initial state mapping of Gaussian primitives in the currently active subgraph; on the other hand, an independent rendering refinement thread, based on an independent subgraph memory management strategy, fully takes over the historical subgraph queue and performs asynchronous continuous differentiable rendering and regularization correction. This key technology, through the isolation of the subgraph lifecycle and efficient scheduling of computing resources, fundamentally avoids the memory race and computing power congestion problems induced by global parameter coupling in traditional 3D Gaussian systems during continuous scene expansion, achieving high-frequency running frame rates while ensuring high-dimensional light fidelity.

[0161] This application achieves a deep cross-domain integration of deep learning feature matching capabilities and explicit 3D Gaussian differentiable rendering for large-scale scenes. Unlike the current conventional closed-loop approach that blindly performs high-frequency dense alignment across the entire image, this key technology first reuses a self-crossing feature network to efficiently calculate the initial values ​​of rigid transformation constraints between subgraphs. It then innovatively uses "rasterization rendering residuals (combining photometric and depth metrics)" as an internal dynamic judgment variable, achieving a lightweight closed-loop registration threshold for "on-demand refinement." This mechanism can adaptively evaluate whether multi-view refinement optimization needs to be initiated based on rendering quality, ensuring global accuracy. Figure 1 Under the premise of consistency, redundant gradient iteration calculations are minimized, significantly reducing the computational load and latency of the system.

[0162] This application addresses the unavoidable long-term pose drift problem in practical applications by proposing a direct rigid body update method based on hybrid map representation of structural characteristics. After completing pose map optimization and obtaining the correction deviation matrix, the system no longer iteratively calculates the spherical harmonics or color attributes of the 3D Gaussian elements. Instead, it directly uses the correction matrix as a Lie algebraic rotation and translation constraint between the Gaussian element spatial coordinates (mean vector) and shape attributes (covariance matrix). This technique effectively avoids map tearing and ghosting during closed-loop adjustments, providing an efficient, high-precision, and robust global map correction scheme for visual SLAM in complex, large-area real-world production environments.

[0163] Corresponding to the above method embodiments, this specification also provides an embodiment of a Gaussian SLAM device based on self-crossing feature networks and hybrid map representations. Figure 4This specification illustrates a schematic diagram of a Gaussian SLAM device based on a self-crossing feature network and hybrid map representation, according to one embodiment of this specification. Figure 4 As shown, the device includes: The pose calculation module 401 is configured to construct a feature extraction and matching network through self-attention and cross-attention mechanisms, and use the network to extract the feature correspondence between image frames and calculate the initial pose of the camera. The skeleton generation module 402 is configured to establish a local beam adjustment module based on co-view relationship between the feature tracker and the back-end map, using the initial pose as the initial value for optimization, and using a sliding window model to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. The map representation module 403 is configured to construct a hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units. The sparse geometric point clouds consist of 3D landmark positions and feature descriptors in a sparse skeleton, and the sparse geometric point clouds and dense Gaussian units are rigidly bound together. The thread configuration module 404 configures independent foreground and background threads for the hybrid map representation, so as to perform synchronous initialization and asynchronous progressive rendering optimization of Gaussian primitives respectively. The pose constraint module 405 is configured to reuse the geometric initial values ​​output by the feature extraction and matching network, and construct a residual adaptive closed-loop state feedback controller by combining the joint rendering residual generated by differentiable rasterization. The controller determines whether to activate the multi-view optimization operation based on the energy value of the joint rendering residual, and outputs the relative pose constraint after closed-loop correction. The model update module 406 is configured to construct a rigid body correction update model that includes a global pose graph and a hybrid map representation, introduce relative pose constraints into the global pose graph, solve the correction transformation matrix, and synchronously update the sparse keyframe poses and the explicit properties of dense Gaussian elements in all subgraphs.

[0164] In one possible implementation, the initial pose of the camera is calculated through a feature extraction and matching network, which includes: inputting the input frame and the previous frame into the network to obtain a set of matching points; performing random sampling consistency screening on the set of matching points to obtain a set of inliers; using depth information to backproject the two-dimensional matching points in the set of inliers into three-dimensional landmarks; reprojecting the three-dimensional landmarks onto the image plane through perspective projection; and using the Huber robust kernel function to minimize the nonlinear geometric reprojection error, thereby obtaining the optimal relative pose estimate as the initial pose.

[0165] In one possible implementation, the local beam adjustment module for generating the sparse skeleton includes: maintaining a sliding window containing a predetermined number of recent keyframes within the activated subgraph; using the initial pose as the initial estimate of the keyframes within the sliding window; constructing an objective function that includes a reprojection error term and a normalization fixation term; introducing a penalty factor in the normalization fixation term to constrain the pose drift of the oldest keyframe in the sliding window; performing local beam adjustment by minimizing the objective function; and outputting the jointly optimized local trajectory and 3D spatial landmarks as the sparse skeleton.

[0166] In one possible implementation, constructing a hybrid map representation involves: dividing the scene into multiple sub-maps, defining a sparse geometric point cloud set and a dense Gaussian tuple set in each sub-map. Each element in the sparse geometric point cloud set consists of the 3D landmark positions in the sparse skeleton and a high-dimensional feature descriptor output by the feature extraction and matching network. Each element in the dense Gaussian tuple set consists of a 3D mean position vector, a covariance matrix, opacity, and view-dependent RGB color. Spatial rigid binding constraints are applied to the dense Gaussian tuples through the sparse geometric point cloud.

[0167] In one possible implementation, configuring a background thread to perform asynchronous progressive rendering optimization includes: constructing a rendering loss function that integrates photometric loss, direct depth constraints, and an isotropic regularization function. The photometric loss is calculated jointly using a norm and a structural similarity index. The direct depth constraints are used to incorporate observation information from a depth sensor. The isotropic regularization function is calculated based on the ratio of the average scale of Gaussian elements within a subgraph to the scale of a single Gaussian element to suppress long strip artifacts in texture-deficient regions. The dense Gaussian element set is progressively updated by minimizing the rendering loss function.

[0168] In one possible implementation, constructing a residual adaptive closed-loop state feedback controller includes: extracting full-image features and calculating cross-similarity through a global descriptor network, and triggering closed-loop candidate detection based on the cross-similarity; responding to the triggering of closed-loop candidates, reusing a feature extraction and matching network to obtain the initial relative pose between the target sub-image and the source sub-image, and calculating the sub-image transformation; calculating the joint error under the current closed-loop view using a differentiable rendering process, and fusing photometric error and depth error in the joint error; when the joint error is lower than a preset threshold, directly outputting the initial relative pose as a relative pose constraint; when the joint error is not lower than the preset threshold, triggering multi-view pose refinement, constructing a weighted rotation average objective function with the joint error as the weight, and minimizing the Frobenius norm of the objective function to estimate the global rotation matrix, and outputting the estimated global rotation matrix as a relative pose constraint.

[0169] In one possible implementation, constructing the rigid body correction update model includes: constructing a global pose graph optimization problem, adding relative pose constraints as edges to the global pose graph, and solving for the optimal sequence of correction deviation matrices; using the sequence of correction deviation matrices as Lie algebraic rotation and translation constraints, and simultaneously updating the 3D landmark positions of the sparse geometric point cloud and the 3D mean position vector and covariance matrix of the dense Gaussian elements in each subgraph to eliminate the system's cumulative drift and satisfy the global spatial consistency condition.

[0170] The above is an illustrative scheme of a Gaussian SLAM device based on self-crossing feature networks and hybrid map representations according to this embodiment. It should be noted that the technical solution of this Gaussian SLAM device based on self-crossing feature networks and hybrid map representations belongs to the same concept as the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above. Details not described in detail in the technical solution of the Gaussian SLAM device based on self-crossing feature networks and hybrid map representations can be found in the description of the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0171] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0172] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0173] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0174] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0175] The processor 520 executes computer-executable instructions that, when executed by the processor, implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above. The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above belong to the same concept. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0176] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0177] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representation described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representation described above.

[0178] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, the computer performs the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations described above.

[0179] The above is an illustrative example of a computer program in this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the aforementioned Gaussian SLAM method based on self-crossing feature networks and hybrid map representations. Details not described in detail in the computer program's technical solution can be found in the description of the aforementioned Gaussian SLAM method based on self-crossing feature networks and hybrid map representations.

[0180] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0181] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0182] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A Gaussian SLAM method based on self-crossing feature networks and hybrid map representations, characterized in that, include: A feature extraction and matching network is constructed using self-attention and cross-attention mechanisms. The network is then used to extract the feature correspondence between image frames and calculate the initial pose of the camera. A local beam adjustment module based on co-view relationship is established between the feature tracker and the back-end map. The initial pose is used as the initial value for optimization. A sliding window model is used to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. A hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units is constructed. The sparse geometric point clouds are composed of the three-dimensional landmark positions and feature descriptors in the sparse skeleton, and the sparse geometric point clouds and the dense Gaussian units are rigidly bound together. Configure independent foreground and background threads for the hybrid map representation to perform synchronous initialization and asynchronous progressive rendering optimization of Gaussian primitives, respectively; The geometric initial values ​​output by the feature extraction and matching network are reused, and a residual adaptive closed-loop state feedback controller is constructed by combining the joint rendering residual generated by differentiable rasterization. The controller determines whether to activate the multi-view optimization operation based on the energy value of the joint rendering residual and outputs the relative pose constraint after closed-loop correction. A rigid body correction update model is constructed, which includes a global pose graph and the hybrid map representation. The relative pose constraints are introduced into the global pose graph. The correction transformation matrix is ​​solved and the explicit properties of sparse keyframe poses and dense Gaussian elements in all subgraphs are updated synchronously.

2. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation as described in claim 1, characterized in that, The initial pose of the camera is calculated through the feature extraction and matching network as follows: inputting the input frame and the previous frame into the network to obtain a set of matching points; performing random sampling consistency screening on the set of matching points to obtain a set of inliers; using depth information to backproject the two-dimensional matching points in the set of inliers into three-dimensional landmarks; reprojecting the three-dimensional landmarks onto the image plane through perspective projection; and using the Huber robust kernel function to minimize the nonlinear geometric reprojection error, thereby obtaining the optimal relative pose estimate as the initial pose.

3. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation as described in claim 1, characterized in that, The method for generating a sparse skeleton by establishing a local beam adjustment module includes: maintaining a sliding window containing a predetermined number of recent keyframes within an activated subgraph; using the initial pose as the initial estimate of the keyframes within the sliding window; constructing an objective function that includes a reprojection error term and a normalization fixation term; introducing a penalty factor in the normalization fixation term to constrain the pose drift of the oldest keyframe in the sliding window; performing local beam adjustment by minimizing the objective function; and outputting the jointly optimized local trajectory and 3D spatial landmarks as the sparse skeleton.

4. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation according to claim 1, characterized in that, Constructing a hybrid map representation includes: dividing the scene into multiple sub-maps, defining a sparse geometric point cloud set and a dense Gaussian primitive set in each sub-map. Each element in the sparse geometric point cloud set consists of the 3D landmark position in the sparse skeleton and the high-dimensional feature descriptor output by the feature extraction and matching network. Each element in the dense Gaussian primitive set consists of a 3D mean position vector, covariance matrix, opacity, and view-dependent RGB color. Spatial rigid binding constraints are applied to the dense Gaussian primitives through the sparse geometric point cloud.

5. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation according to claim 1, characterized in that, Configuring a background thread to perform asynchronous progressive rendering optimization includes: constructing a rendering loss function that integrates photometric loss, direct depth constraints, and an isotropic regularization function. The photometric loss is calculated jointly using a norm and a structural similarity index. The direct depth constraints are used to incorporate observation information from a depth sensor. The isotropic regularization function is calculated based on the ratio of the average scale of Gaussian elements within a subgraph to the scale of a single Gaussian element to suppress long strip artifacts in texture-deficient regions. The dense Gaussian element set is progressively updated by minimizing the rendering loss function.

6. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation according to claim 1, characterized in that, The residual adaptive closed-loop state feedback controller includes: extracting full-image features and calculating cross-similarity through a global descriptor network, and triggering closed-loop candidate detection based on the cross-similarity; responding to the triggering of the closed-loop candidate, reusing the feature extraction and matching network to obtain the initial relative pose between the target sub-image and the source sub-image, and calculating the sub-image transformation; calculating the joint error under the current closed-loop view using a differentiable rendering process, wherein the joint error integrates photometric error and depth error; when the joint error is lower than a preset threshold, directly using the initial relative pose as the relative pose constraint output; when the joint error is not lower than the preset threshold, triggering multi-view pose refinement, constructing a weighted rotation average objective function with the joint error as the weight, and minimizing the Frobenius norm of the objective function to estimate the global rotation matrix, and using the estimated global rotation matrix as the relative pose constraint output.

7. The Gaussian SLAM method based on self-crossing feature networks and hybrid map representation according to claim 1, characterized in that, The rigid body correction update model includes: constructing a global pose graph optimization problem, adding the relative pose constraints as edges to the global pose graph, and solving for the optimal correction deviation matrix sequence; using the correction deviation matrix sequence as Lie algebraic rotation and translation constraints, and performing synchronous algebraic updates on the 3D landmark positions of the sparse geometric point cloud and the 3D mean position vector and covariance matrix of the dense Gaussian elements in each subgraph to eliminate system cumulative drift and satisfy the global spatial consistency condition.

8. A Gaussian SLAM device based on self-crossing feature networks and hybrid map representations, characterized in that, include: The pose calculation module is configured to construct a feature extraction and matching network through self-attention and cross-attention mechanisms, and use the network to extract the feature correspondence between image frames and calculate the initial pose of the camera. The skeleton generation module is configured to establish a local beam adjustment module based on co-view relationship between the feature tracker and the backend map, using the initial pose as the initial value for optimization, and using a sliding window model to jointly optimize the local trajectory and 3D spatial landmarks to generate a sparse skeleton. The map representation module is configured to construct a hybrid map representation consisting of sparse geometric point clouds and dense Gaussian units, wherein the sparse geometric point clouds are composed of three-dimensional landmark positions and feature descriptors in the sparse skeleton, and the sparse geometric point clouds and the dense Gaussian units are rigidly bound together. The thread configuration module configures independent foreground and background threads for the hybrid map representation, so as to perform synchronous initialization and asynchronous progressive rendering optimization of Gaussian primitives respectively. The pose constraint module is configured to reuse the geometric initial values ​​output by the feature extraction and matching network, and construct a residual adaptive closed-loop state feedback controller by combining the joint rendering residual generated by differentiable rasterization. The controller determines whether to activate the multi-view optimization operation based on the energy value of the joint rendering residual, and outputs the relative pose constraint after closed-loop correction. The model update module is configured to construct a rigid body correction update model that includes a global pose graph and the hybrid map representation, introduce the relative pose constraints into the global pose graph, solve the correction transformation matrix, and synchronously update the sparse keyframe poses and the explicit properties of dense Gaussian elements in all subgraphs.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representation as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the Gaussian SLAM method based on self-crossing feature networks and hybrid map representations as described in any one of claims 1 to 7.

Images