A dense visual scene reconstruction method and system

Abstract

The present application discloses a dense visual scene reconstruction method and system, which relates to the field of scene reconstruction. The method comprises constructing a dense visual scene reconstruction architecture including a 3D Gaussian volume representation module, a camera tracking module, a key frame management module and a map update module; and reconstructing a dense visual scene using the dense visual scene reconstruction architecture according to the image data to be reconstructed. The present application can achieve high-quality scene reconstruction and fast scene rendering under complex lighting conditions.

Description

Technical Field

[0001] The present application relates to the field of scene reconstruction, and in particular to a method and system for reconstructing a dense visual scene. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology is a technology that solves the task of main body positioning and orientation and environmental map reconstruction of mechanical equipment in unknown environments. It is considered to be the key to realizing truly fully autonomous mobile robots. With the continued hot research in the fields of computer vision, artificial intelligence, and autonomous driving, the research on SLAM has continued to deepen, and higher requirements have been put forward for positioning and navigation accuracy and scene reconstruction quality. Visual SLAM (Visual SLAM) relies on camera sensors to capture visual information, and completes the estimation of device position and modeling of the environment by processing continuous image sequences. It has the advantages of low cost, low power consumption, and small size. A lot of work has been carried out on visual SLAM based on different forms such as camera sensor types, front-end processing methods, and back-end processing methods, but further research is still needed in terms of weak textures, illumination changes, and real-time performance.

[0003] Implicit Neural Representation (INR) is a coordinate-based representation method that can represent signal parameters as a continuous function and use neural networks to approximate the signal function. INR avoids the disadvantages of discrete data storage in traditional methods, and can perform high-precision interpolation to calculate the occupancy of any position in the scene and render three-dimensional scenes of any perspective, which makes INR perform well in SLAM in handling complex environmental scenes, maintaining high-resolution details, and supporting continuous time mapping. Relatively outstanding works include RDSP-SLAM, Occupancy SLAM, and NeRF-based SLAM. However, NeRF-based SLAM is widely criticized for its slow training / rendering speed, which is not conducive to the application and deployment of real-time SLAM systems. Some literatures have made a detailed review of the subsequent improvement methods of NeRF, such as using hashing or sparse 3D grids to improve NeRF-based SLAM, but there are still problems such as over-smoothing, susceptibility to influence, and low computational efficiency. As an explicit radiation field technology, 3D GS models 3D scenes through differentiable 3D Gaussian shape primitives. Once launched, it has attracted widespread attention from researchers around the world. Unlike NeRF, which relies on volume rays for dense sampling and results in slow rendering, 3D GS uniquely combines the advantages of differentiable pipelines and point-based rendering technology, allowing 3DGS-based SLAM to retain the ideal characteristics of continuous volume radiation fields while avoiding the computational overhead associated with rendering empty spaces.

[0004] However, in terms of dense visual SLAM, 3DGS-based SLAM has shown great improvements in performance such as rendering speed and rendering quality. Although 3DGS-based SLAM has shown amazing performance in various aspects, in the process of 3D GS application, researchers often construct loss functions based on inertial experience, such as directly using function to prevent interference from extreme values ​​and noise. This choice is simple and effective and can reduce computational complexity, but its performance is unsatisfactory in complex lighting environments (such as dim light or drastic changes). The loss function may not be able to effectively capture the true distribution of local light variation data when dealing with highly nonlinear and uncertain global situations. As an emerging alternative, the SSIM loss function shows higher sensitivity when dealing with image quality and structural fidelity. Compared with the 3DGS-based SLAM loss, SSIM takes into account the brightness, contrast and structural information of the image, and can better reflect the similarity between image contents. This feature enables the application of the SSIM loss function in SLAM to effectively improve the reconstruction and rendering accuracy, especially when dealing with scenes with complex textures and structural features. In the 3DGS-based SLAM system, the use of the SSIM loss function can help the algorithm better align image data from different perspectives and reduce errors introduced by environmental noise and dynamic objects. However, there is currently little research on how to optimize the SSIM loss function and the combination strategy for optimizing the loss function. It is necessary to further explore the mathematical mechanism and reasonably optimize the loss function to improve the overall SLAM performance.

[0005] Therefore, based on the above problems, there is an urgent need to provide a new dense visual scene reconstruction method that can achieve high-quality scene reconstruction and fast scene rendering under complex lighting conditions. Summary of the invention

[0006] The purpose of this application is to provide a dense visual scene reconstruction method and system, which can achieve high-quality scene reconstruction and fast scene rendering under complex lighting conditions.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a dense visual scene reconstruction method, the dense visual scene reconstruction method comprising:

[0009] A dense visual scene reconstruction architecture is constructed; the dense visual scene reconstruction architecture includes: a 3D Gaussian volume representation module, a camera tracking module, a keyframe management module and a map update module; the 3D Gaussian volume representation module is used to initialize the input image data, and convert the data into a set of anisotropic 3D Gaussian volumes through reparameterization processing; and then a double-layer densification is performed using an RGB mask layer and an opacity screening layer; the camera tracking module is used to analyze and control the relationship between the camera pose and the rasterization process based on the input image data using the Jacobian matrix, and adopt The loss function of the camera iteration update is constructed by combining the function and the L-SSIM function to estimate the pose information of the online camera of the input image data; the key frame management module is used to retrieve, select and delete key frames according to the pose information and the 3D Gaussian volume visibility principle; the map update module is used to optimize the insertion of the 3D Gaussian volume according to the pose information, and use the isotropic regularization term to remove artifacts, and use The loss function in the mapping process is constructed by combining the function and the L-SSIM function, and finally the Gaussian parameters of the three-dimensional scene are updated and the continuous reconstruction scene is maintained;

[0010] Obtain image data to be reconstructed;

[0011] According to the image data to be reconstructed, a dense visual scene reconstruction architecture is adopted to reconstruct the dense visual scene.

[0012] Optionally, the 3D Gaussian volume representation module traverses the initialized Gaussian volume by rasterization; and splashes multiple Gaussian volumes from the front end to the back end along a channel ch for mixed rendering.

[0013] Optionally, the mixing and rendering of splashing multiple Gaussian bodies from the front end to the back end along a channel ch specifically includes:

[0014] Using the formula Determine the color of pixel i after mixed rendering ;

[0015] in, is the opacity, is the density calculated by multiplying the 2D covariance by the opacity, is the offset between the 2D Gaussian mean and the pixel coordinates, is the covariance matrix of the 2D Gaussian, is the set of real numbers, is the color of pixel j, the superscript T is the transpose, N is the number of Gaussians, is the density corresponding to the k-th Gaussian body, and k is the k-th Gaussian body corresponding to the j-th Gaussian body.

[0016] Optionally, the RGB mask layer uses the formula Determine the mask ;

[0017] The opacity filter layer uses the formula Conduct screening;

[0018] in, , and The functions represent the stopping gradient, sigmoid function and threshold index respectively. is the index number of the Gaussian body, is the threshold of the mask, is the opacity of the Gaussian, Parameters corresponding to the RGB mask.

[0019] Optionally, when the input image data is an RGB image, the formula Determining photometric color residuals ;

[0020] When the input image data has image depth, use Determining the depth residual ;

[0021] Using the formula Determine pose information ;

[0022] in, is the rendered image, is the original image, is the corresponding opacity, is the RGB mask, represents element-by-element multiplication, is the L-SSIM function, is the Gaussian filter term, is the depth of the rendered image, is the original image depth, is the depth mask, is a hyperparameter, , is the hyperparameter corresponding to the photometric loss, Mathematically defined function.

[0023] Optionally, the keyframe management module uses the formula Confirm the selected keyframe ;

[0024] Using the formula Confirm to delete the keyframe ;

[0025] in, , are Gaussian functions of the mth and nth key frames respectively, is the intersection-and-union ratio, is the overlap coefficient; , are the threshold functions when depth information is unavailable and available, respectively. is the relative displacement; when depth information is available, is the median depth information of the mth key frame, A threshold function defined for deleting keyframes.

[0026] Optionally, the map update module uses the formula Regularization term to determine the homogeneity of each term;

[0027] in, is the scale of the Gaussian body, is the corresponding mean, is the set of real numbers, for.

[0028] In a second aspect, the present application provides a computer system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dense visual scene reconstruction method.

[0029] According to the specific embodiments provided in this application, this application has the following technical effects:

[0030] The present application provides a dense visual scene reconstruction method and system. By constructing a dense visual scene reconstruction architecture (LLGS-SLAM), the 3D Gaussian body representation module in LLGS-SLAM is used for the initialization, representation and densification of the three-dimensional Gaussian body; the camera tracking module is mainly used to estimate the camera posture of the current incoming image; the key frame management module is mainly responsible for the retrieval, selection and deletion of key frames; the map update module is mainly used to update the Gaussian parameters of the three-dimensional scene and maintain a continuous reconstruction scene. The loss function constructed by the map update module and the camera tracking module can enhance the ability to capture details of brightness while maintaining computational convenience, and realize accurate reconstruction and rendering in complex lighting environments; by adding RGB mask layers and opacity screening layers for double-layer densification, the densification operation of 3D GS in the scene reconstruction process can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 This is a flowchart of a dense visual scene reconstruction method in one embodiment of the present application;

[0033] Figure 2 A three-dimensional Gaussian volume of a specific embodiment of the present application and a rendering showing a synthesis of new perspective images;

[0034] Figure 3 It is a schematic diagram of the structure of the LLGS-SLAM method under complex lighting conditions;

[0035] Figure 4 A schematic diagram showing the comparison of visual rendering performance of different loss functions in the Replica dataset Room0 and Room2 scenes;

[0036] Figure 5 Schematic diagram of the visualization effect of LLGS-SLAM in different scenes of the Replica dataset. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] In an exemplary embodiment, Figure 1 As shown, a dense visual scene reconstruction method is provided, which includes the following S101 to S103. Wherein:

[0040] S101, constructing a dense visual scene reconstruction architecture; the dense visual scene reconstruction architecture includes: a 3D Gaussian volume representation module, a camera tracking module, a key frame management module and a map update module; the 3D Gaussian volume representation module is used to initialize the input image data, and convert the data into a set of anisotropic 3D Gaussian volumes through reparameterization processing; and then use the RGB mask layer and the opacity screening layer for double-layer densification; the camera tracking module is used to use the Jacobian matrix to analyze and control the relationship between the camera pose and the rasterization process according to the input image data, and use The loss function of the camera iteration update is constructed by combining the function and the L-SSIM function to estimate the pose information of the online camera of the input image data; the key frame management module is used to retrieve, select and delete key frames according to the pose information and the 3D Gaussian volume visibility principle; the map update module is used to optimize the insertion of the 3D Gaussian volume according to the pose information, and use the isotropic regularization term to remove artifacts, and use The loss function in the mapping process is constructed by combining the function and the L-SSIM function, and finally the Gaussian parameters of the three-dimensional scene are updated and the continuous reconstruction scene is maintained;

[0041] like Figure 2 As shown, by inputting depth data RGBD or RGB data (left), LLGS-SLAM is innovative The combination of the 3D Gaussian volume representation module, the camera tracking module, the keyframe management module, and the map update module is used to reconstruct high-fidelity 3D scenes in real time. Figure 2As shown in the figure, LLGS-SLAM can accurately capture the lighting details of the scene and accurately depict light sources similar to table lamps and the halo shadows produced under multi-level illumination conditions. This application can effectively improve the performance of various indicators of 3D GS-based SLAM in RGB and RGBD datasets, and can perform real-time scene reconstruction at a forward rendering speed of 1195.35 FPS.

[0042] The determination process of L-SSIM function is as follows: , the resulting single pixel p The L-SSIM function is as follows:

[0043] represents the contrast function of brightness, The contrast function represents the contrast, represents the contrast function of the structure, , Represents the proportion of each feature in the SSIM measurement, Free setting; function ,function ; and is the mean of the coordinate points, and The covariance of the coordinate points, is the covariance between the coordinate points, is a constant term, , , and are two constants much smaller than 1, L is the dynamic range of the image pixel value; for ease of calculation, it is often taken based on experience , , , Therefore, the common expression of SSIM is:

[0044] make , , then we can get:

[0045] .

[0046] in, ;

[0047] ;

[0048] ;

[0049] For the back-propagation derivatives, each region P exists , , so we can get the following formula:

[0050] ;

[0051] ;

[0052] ;

[0053] From the above formula, we can see that , , The part can be expressed as the Gaussian filter term The relevant multiplication function. Then use the chain rule to solve:

[0054] ;

[0055] ;

[0056] Now, by substituting each term into the L-SSIM formula, we can obtain:

[0057] ;

[0058] , They are two regions centered on pixel p, “*” represents convolution, and “·” represents point-by-point multiplication;

[0059] The L-SSIM function not only retains the convenient nature of the SSIM function in derivation and calculation, but also considers the brightness-related part separately. In the SLAM system, low-light conditions or environments with drastic light changes are very challenging scenarios. For example, in a low-light environment, the brightness information in the image may become blurred, the noise increases, and it will have a great impact on the recognition and feature extraction of the scene. When the light changes drastically, the rapid change in brightness will interfere with the system's judgment of the object's position, shape and other information. By considering the brightness part separately, the L-SSIM function can deal with the problems caused by brightness changes in a more targeted manner. According to the characteristics of brightness changes, the SLAM system can use the L-SSIM function to improve the accuracy of image similarity evaluation in complex lighting environments, thereby better completing tasks such as positioning and map construction.

[0060] It is further expressed as the L-SSIM loss function in region P It is expressed as:

[0061] ; where p is the pixel index, is the center pixel, is the total number of pixels in area P.

[0062] Taking the derivative of all pixels q in region P, we get:

[0063] ;

[0064] in, and Represent the processed pixel value and original value of area P respectively, is a Gaussian filter associated with pixel q. , , , , and Parameters such as will be fixed as the search area is determined, and and The parameters will greatly affect the quality of the results trained by the SSIM loss function. It not only measures the degree of dispersion of pixel values ​​in the area relative to the mean, but also plays an overall smoothing and noise reduction effect. For example, when a white noise point appears in a darker area, due to By weighted summing of the surrounding dark pixels, the influence of this noise point on the overall result will be reduced when calculating the mean and standard deviation, thus achieving the effect of smoothing and noise reduction.

[0065] From the L-SSIM derivation, we know that there is the following relationship between the two:

[0066] ;

[0067] Therefore, L-SSIM also follows these three basic principles. .because and As a result, L-SSIM improves its ability to capture details of complex lighting environments in scene reconstruction. The range of change cannot be too large, otherwise the dynamic range of the image will be compressed and the details of different brightness levels cannot be accurately represented. For example, in the highlight area or shadow part, the contrast of the image is excessively enhanced or weakened, so that some details are submerged in the overly bright or dark area.

[0068] This application adopts a combination of multiple loss functions to effectively overcome the shortcomings of a single function. In order to avoid L-SSIM using only a Gaussian filter during back propagation, it is necessary to Function point multiplication is used to make the algorithm run smoothly. The following combination is used to construct the overall loss function of the literature:

[0069] ;

[0070] in, The function can ignore the influence of local structure and treat all error details in the region uniformly, preserving the color brightness of the image; L-SSIM inherits the contrast and structure display capabilities of SSIM, and further improves the performance of the overall function in terms of brightness. is not sensitive to the value of; for Gaussian filters, it is defined as a constant for ease of calculation.

[0071] 3D Gaussian Splatting (3D GS) uses a set of Gaussian ellipsoids to model the scene, thereby achieving effective rendering by rasterizing the Gaussian ellipsoids into images. In addition to the fast rendering speed, the explicit representation of 3D GS facilitates downstream tasks such as dynamic reconstruction, geometry editing, and physical simulation. 3D GS can achieve efficient, fast, and high-quality scene reconstruction and rendering, and has great research potential.

[0072] 3D GS-based SLAM adds a Gaussian ellipsoid to the SLAM system to flexibly represent the map, fully utilizes the dense photometric loss of each pixel to improve the mapping effect, and uses direct parameter gradient flow to accelerate the optimization process. In the process of optimization and mapping, different 3DGS-based SLAM methods often use different loss function strategies. SplaTAM uses the same strategy as the original 3D GS. Combine the SSIM term to construct the loss function of depth and color rendering; GS-SLAM optimizes scene representation by minimizing photometric color loss and geometric depth loss. When depth information is available, Gaussian Splatting SLAM uses The combination of photometric loss and geometric error, plus the weighted isotropic regularization, together constitutes the mapping loss function. Gaussian SLAM uses a weighted function of photometric color, geometric depth, and isotropic regularization for optimization, in which Combined with SSIM. The method of constructing the minimum loss function of RGBD GS-ICP SLAM is similar to that of Gaussian SLAM, but D-SSIM is used to replace SSIM. The selection and construction of different loss functions will affect the final evaluation index of the algorithm.

[0073] like Figure 3 As shown in the figure, good new perspective synthesis is obtained while reconstructing dense scenes, and finally high-quality scene reconstruction and fast scene rendering are achieved under complex lighting conditions. LLGS-SLAM supports image data input in both RGB-D or RGB-only formats.

[0074] In order to build a dense 3D GS-based SLAM with rich details, fast rendering and high quality, the 3D scene is represented as a set of coupled representations G of anisotropic 3D Gaussian bodies. The specific representation is:

[0075] ;

[0076] A single Gaussian Contains three attributes: position coordinates in the world coordinate system , the three-dimensional covariance matrix , Opacity , which ensures that 3D GS can be used for continuous three-dimensional representation. Compared with the standard Gaussian representation function, 3D GS removes the scale factor in front of the exponential part (but this does not affect the ellipsoid geometry); and the default model coordinate center is at the origin, which is convenient for rotation and scaling, and translation is added when it is placed in the world space.

[0077] During the model training process, a series of heuristic methods are used to edit the Gaussian to better render the image. The three-dimensional covariance matrix of each Gaussian can be expressed as:

[0078] ;

[0079] Among them, the rotation matrix ,scale .

[0080] In the construction of the Gaussian body, the position of the Gaussian body and Parameters such as PyTorch need to use the automatic differentiation framework of PyTorch to perform stochastic gradient descent. However, due to the optimization of PyTorch's automatic differentiation framework When you need to display Gradients ensure the normal operation of the algorithm, so 3D GS Compared with NeRF, which moves along the camera ray, 3D GS traverses the initialized Gaussian volume by rasterization, which can skip the blank area and greatly improve the running speed. The camera pose transformed to the camera coordinate system When known, the 3D Gaussian body in the 3D world coordinate system 2D Gaussian is obtained by projective transformation :

[0081] ;

[0082] in is the projection operation; , whose i-th element is 1 and the rest are 0. The Jacobian matrix representing the affine transformation in the projective transformation; yes The rotation component of .

[0083] After projecting the 3D Gaussian onto the image plane, all the N constructed Gaussian volumes are sorted in depth order. Similar to Gaussian Splatting SLAM, in order to further streamline the algorithm architecture and speed up rendering, the spherical harmonic function is discarded.

[0084] The 3D Gaussian volume representation module traverses the initialized Gaussian volume through rasterization; multiple Gaussian volumes are splashed and mixed and rendered from the front end to the back end along the direction of a channel ch.

[0085] Using the formula Determine the color of pixel i after mixed rendering ;

[0086] in, is the opacity, is the density calculated by multiplying the 2D covariance by the opacity, is the offset between the 2D Gaussian mean and the pixel coordinates, is the covariance matrix of the 2D Gaussian, is the set of real numbers, is the color of pixel j, the superscript T is the transpose, N is the number of Gaussians, is the density corresponding to the k-th Gaussian body, and k is the k-th Gaussian body corresponding to the j-th Gaussian body.

[0087] In order to ensure that LLGS SLAM can compactly represent the scene while maintaining excellent performance, a double-layer densification operation is used in the 3D Gaussian volume representation module.

[0088] For non-negative scale attributes , an additional RGB mask parameter is introduced , based on which a binary mask is generated Then the direct estimation method is used and the gradient mask is combined to calculate the new RGB mask , as follows:

[0089] ;

[0090] in , and The functions represent the stopping gradient, sigmoid function and threshold index respectively. Represents the index number of the Gaussian body, Represents the threshold of the mask. When getting a new RGB mask After that, the image can be processed pixel by pixel to filter out redundant Gaussian points. When depth information is available, the depth mask can be obtained in the same way. .

[0091] In order to solve the problem that low-transparency Gaussian bodies have a negative impact on scene construction, an opacity screening layer is further introduced. The function of this screening layer is to ensure that only Gaussian bodies with high enough transparency are used in the system, thereby improving the quality and efficiency of scene construction. In order to avoid redundant calculations, transparency is not combined with mask calculations, but a separate judgment standard is set to achieve fast screening. By comparing the actual transparency of a Gaussian body with the preset threshold, it is determined whether it can participate in scene construction, as follows:

[0092] ;

[0093] in, , and The functions represent the stopping gradient, sigmoid function and threshold index respectively. is the index number of the Gaussian body, is the threshold of the mask, is the opacity of the Gaussian, Parameters corresponding to the RGB mask.

[0094] In the camera tracking module, the purpose of camera tracking is to estimate the camera pose of the current incoming image. During the tracking process, LLGS SLAM focuses on the motion estimation of the camera without involving the enhancement or update of the environment model. In the new camera pose, the pose parameters are initialized by uniform forward projection in the camera center + quaternion space. In order to avoid the overhead of automatic differentiation and achieve accurate tracking, CUDA is used to rasterize the derivatives of the solution parameters, and at least 50 gradient descents are required per frame to ensure the optimization speed of the unified rendering pipeline. Using the camera pose The analytical Jacobian matrix of the camera pose is used to more accurately analyze and control the relationship between the camera pose and the rasterization process, thereby improving the performance of the rasterization. To differentiate the expression (), we get the following chain expansion:

[0095] ;

[0096] The partial derivative on the manifold is defined as:

[0097] ;

[0098] in , , " " represents the group operator, and They represent the exponential function and logarithmic function mapping from Lie algebra to Lie group respectively.

[0099] From this we can get the following formula:

[0100] ;

[0101] in," " represents a symmetric matrix of three-dimensional vectors, Represents the i-th column of the matrix. Substituting equation () back into equation () yields the optimized equation for the camera posture during rasterization.

[0102] When the input image data is an RGB image, that is, only monocular data, use the formula Determining photometric color residuals ;

[0103] When the input image data has image depth, the camera pose is further optimized by considering both color and depth information. Determining the depth residual ;

[0104] Differentiable rendering of color and depth information via gradient-based optimization and updating of camera parameters using the Adam optimizer;

[0105] Using the formula Determine pose information ; To minimize subsequent losses, the Gaussian parameters are kept frozen during tracking and the map is not updated.

[0106] in, is the rendered image, is the original image, is the corresponding opacity, is the RGB mask, represents element-by-element multiplication, is the L-SSIM function, is the Gaussian filter term, is the depth of the rendered image, is the original image depth, is the depth mask, is a hyperparameter, , is the hyperparameter corresponding to the photometric loss, Mathematically defined function.

[0107] In the process of map construction, the reasonable selection and management of keyframes can improve computational efficiency and map quality. The management and optimization of Gaussian bodies usually requires the processing of a large amount of data, and the computational complexity increases with the increase in the number of frames. Therefore, it is introduced as a prerequisite for keyframe management. Gaussian body visibility judgment refers to the process of determining which Gaussian bodies should be included in the optimization by analyzing which Gaussian bodies can effectively provide constraints. In 3D GS technology, Gaussian bodies interact with light and scene geometry based on the distance principle, avoiding the heavy computational burden caused by NeRF's light sampling and color accumulation in the volume. In the actual physical sense, explicit 3D GS sorts Gaussian bodies based on the principle of distance, which can be understood as representing the order of geometric bodies appearing in the field of view according to the visibility principle in the actual observation process, and naturally representing the front and back, occlusion and other relationships of the scene geometry. In LLGS SLAM, the following formula is used to perform the visibility of Gaussian bodies in the rasterization process. express:

[0108]

[0109] in, The cumulative number of rays.

[0110] Once the visibility of the Gaussian body is determined, the keyframes can be selected based on the co-visibility of the Gaussian distribution. Based on the goal of SLAM real-time rendering, the research usually uses keyframes selected from specific areas to provide non-redundant information and achieve the best multi-view constraints with the least computing power. A sliding window is used to retrieve and select keyframes, and each window selects 8 to 10 keyframes depending on the data set. At the same time, in order to simply and efficiently retrieve keyframes, the target tracking frame m is compared with the last keyframe n detected. When qualified, the m frame will be inserted as a new keyframe; and when the relationship between the old keyframe and the new keyframe does not meet the threshold condition, the old keyframe will be deleted to achieve a clear and accurate scene. In the actual operation process, the formula is used Confirm the selected keyframe ;

[0111] Using the formula Confirm to delete the keyframe ;

[0112] in, , are Gaussian functions of the mth and nth key frames respectively, is the intersection-and-union ratio, is the overlap coefficient; , are the threshold functions when depth information is unavailable and available, respectively. is the relative displacement; when depth information is available, is the median depth information of the mth key frame, A threshold function defined for deleting keyframes.

[0113] In the absence of depth information, the threshold determination method relies on a fixed threshold standard to determine the importance of the frame without considering the complexity and dynamics of the scene. This will lead to the misrecognition of some key frames and chain-effect the determination of nearby visible Gaussian bodies, resulting in reduced accuracy.

[0114] The purpose of map updating is to optimize the insertion of Gaussian bodies based on a set of currently estimated online camera poses, so as to update the Gaussian parameters of the 3D scene and maintain a continuous reconstruction scene. In the SLAM view, the under-observed areas do not have good field of view coverage of the original 3D GS, so such areas are prone to lose constraints and produce artifacts, causing the Gaussian body to grow uncontrollably in unobserved areas. In order to avoid excessive stretching of the Gaussian body in sparsely covered or almost unobserved areas, LLGS SLAM encourages isotropy in the optimization process and introduces an isotropic regularization term to achieve the goal:

[0115] ;

[0116] in, is the scale of the Gaussian body, is the corresponding mean, is the set of real numbers, Mathematically defined function.

[0117] The isotropic regularization term can help control the expansion of the Gaussian body, so that it maintains a moderate size in the under-observed area and reduces the appearance of artifacts. However, the spherical Gaussian body may expand too much in the under-observed area, causing unnecessary misleading to the scene reconstruction.

[0118] Use the following formula to update:

[0119] ;

[0120] ;

[0121] In the process of mapping, a similar method is used to build the and ; and They are and The corresponding hyperparameter weights.

[0122] S102, obtaining image data to be reconstructed;

[0123] S103, reconstructing a dense visual scene using a dense visual scene reconstruction architecture according to the image data to be reconstructed.

[0124] Test and evaluate on two commonly used and widely recognized datasets: Replica and TUM. TUM includes TUM monocular data and TUM-RGBD dataset. Replica contains synthetic scenes, small pose changes, and accurate and complete synthetic depth maps. It is a relatively mature and effective benchmark dataset that can be used to evaluate RGB-D SLAM. The image data collected in the TUM dataset is of low quality, so there is more motion blur in its color images and its depth maps are relatively sparse, which puts high demands on the algorithm's ability to capture details.

[0125] The same camera pose estimation and rendering performance evaluation indicators as MonoGS are used. When measuring RGB rendering performance, three common indicators, PSNR, SSIM and LPIPS, are used for evaluation. For camera pose estimation and tracking, the mean absolute trajectory error (ATE RMSE) is used.

[0126] The main benchmark method is MonoGS from CVPR 2024. The model is compared with NeRF-based SLAM (such as Vox-Fusion, NICE-SLAM, Point-SLAM) and 3D GS-based SLAM (SplaTAM, GS-SLAM, MonoGS) in appropriate datasets. At the same time, in order to compare the experimental results when only RGB data is input, it is also compared with ORB-SLAM2, MonoGS, and MotionGS models in the TUM RGB dataset. The basis for the comparison is the public reports of these methods on the dataset and the actual performance on the same experimental set after obtaining the open source code.

[0127] Figure 4 The visualization rendering performance comparison of different loss functions in the Replica dataset Room0 and Room2 scenes is shown. It can be seen that the combined loss function method (combined Function and L-SSIM function, function and SSIM function) and only a single loss function ( Compared with the baseline model of the function, it can capture more subtle features and elements of the scene, present the real scene more accurately, and make the output result closer to the actual situation. It further shows more details of the reconstructed mural (yellow wireframe model) in the Room0 scene, focusing on the upper left corner (red wireframe) and the bow (blue wireframe area). The function considers more global errors, so it is not sensitive in the process of reconstructing the corners of the mural, resulting in some ghosting after the corners are reconstructed. The combination of the function and the SSIM function improves the performance of the corner area, which can present the area more clearly, thereby enhancing the overall visual effect and detail expression. The combination of the function and the L-SSIM function can not only sensitively capture the corners of the mural, but also show better detail and brightness capture capabilities in complex areas such as the bow, thereby constructing a clearer scene representation. The combination of the function and the L-SSIM function has good performance in different areas of the scene, further improving the ability to capture details of different areas in complex lighting environments. The following will provide a quantitative explanation based on specific indicators.

[0128] Figure 4 Comparison of visualization rendering performance of different loss functions in the Replica dataset Room0 and Room2 scenes. The performance of this application is always better than other loss function methods, showing a strong ability to capture details in complex lighting environments. Among them, the reconstruction error of using SSIM or L-SSIM loss function alone is large and the comparison is only based on performance indicators.

[0129] In Table 1, the evaluation index results of different loss functions are compared on the Replica dataset. First of all, it is worth affirming that the baseline model MonoGS (only using The proposed L-SSIM loss function is undoubtedly an excellent work. It is significantly ahead of the other two models (using only SSIM function or L-SSIM function) among the three models that only use one loss function. This is mainly due to the adaptation of the main framework design. Compared with the model that only uses the SSIM function, the proposed L-SSIM loss function does have an optimization effect, and is slightly ahead of the SSIM function in all indicators. At the same time, it can be clearly seen that the model based on the combination of the two loss functions is generally better than the model that only uses one loss function. By The combination of function and SSIM function or L-SSIM function can effectively improve various indicators. The effect of the function and L-SSIM function is the best. Specifically, the combination Compared with the method using only the SSIM function, the PSNR, SSIM and LPIPS indicators of the method using the SSIM function and L-SSIM function were optimized by 20.63 dB, 0.200 and 0.521, respectively, and the optimization ranges were 111.15%, 25.91% and 92.38%, respectively. From the perspective of complementary mechanism, there are three main aspects to explain this. The first is from the perspective of global and local characteristics. The loss function is very effective in dealing with global errors at the pixel level, but it may ignore the structural information in the image. On the contrary, the SSIM function and the L-SSIM function enable the model to better maintain the visual consistency and details of the image by considering local structural features. The second is from the perspective of robustness. The function is relatively robust to outliers, but may cause some loss of average details. The SSIM function and L-SSIM function focus on the perceptual quality of the image and can capture more subtle structural changes. Combining the two can help the model achieve a better balance in dealing with noise and preserving details. The third is to start from the perspective of coupled optimization. The function can ensure that the pixel values ​​of the entire image are close to the true values, while the SSIM function and L-SSIM function will guide the model to optimize the structure and visual perception. The combined error function can help the model maintain a low error while maintaining the natural structure and contrast of the image, thus optimizing the test indicators.

[0130] The following uses the office0 scene in Replica as an example. Both office0 and office1 are scenes with insufficient light, which can highlight the sensitivity of the L-SSIM function to low-light environments. Compared with using only the SSIM function, The model with the combined function of L-SSIM increased the PSNR index from 20.48 dB to 43.08 dB, an increase of 22.60 dB, an increase of 110.35%; increased the SSIM index from 0.794 to 0.982, an increase of 0.188, an increase of 23.68%; and reduced the LPIPS index from 0.625 to 0.031, a decrease of 0.594, a decrease of 95.04%. It can be found that The loss function reduces the global noise in the input image by minimizing the absolute error at the pixel level, thereby reducing the mean square error and directly improving the PSNR index. This can also be seen in the comparison of a single loss function. After combining the L-SSIM function, the model enhances the local features and visual consistency of the image by optimizing the structural similarity, indirectly improving the PSNR index slightly. In terms of the SSIM index, The function reduces the global error, and the L-SSIM function optimizes the structure and details of the image. The combination of the two can reduce detail loss while maintaining structural consistency and improve the SSIM index. In terms of the LPIPS index, The function reduces the loss of basic image quality, and the similarity of structure and local features emphasized by the L-SSIM function is also the focus of the LPIPS indicator. Therefore, the combination of the two improves the perceived quality of the image while ensuring pixel accuracy, significantly reducing the LPIPS indicator by an order of magnitude (which is also the most obvious improvement among the three indicators), reflecting better visual similarity.

[0131] Table 1

[0132]

[0133] Table 1 compares the rendering performance of different loss functions on the Replica dataset. The performance of this application is always better than other loss function methods. It should be noted that MonoGS originally used Loss function. The data on Replica is taken from MonoGS and the test data under the same configuration environment.

[0134] In Tables 2-4, LLGS-SLAM is compared with some advanced baseline models on three datasets (Replica, TUM RGBD, and TUM RGB).

[0135] On the relatively simple Replica dataset, after providing high-quality images and depth information, it can be clearly seen that 3D GS-based methods are generally superior to Nerf-based methods. At the same time, compared with the previous dense SLAM model SOTAPoint-SLAM, this application has basically achieved accuracy surpassing the average trajectory error in most scenes; compared with the new 3D GS-based SLAM, this application has also greatly reduced the trajectory error in each scene representation based on the baseline model MonoGS, and the difference with the best performance SplaTAM model is only 0.12 cm. On the TUM-RGBD dataset, the algorithm model not only needs to face extremely high motion blur, but also needs to estimate the camera pose based on sparse depth information. In this regard, LLGS-SLAM has gone a step further on the basis of the baseline model MonoGS and achieved the best average trajectory error of 1.52 cm. LLGS-SLAM is slightly better than other 3D GS-based SLAMs, and is significantly ahead of Nerf-based SLAM methods. The main reason for the analysis is that the L1 function used in MonoGS is based on the Laplace prior, which can be used as a high-frequency filtering operator to enhance the ability to tolerate outliers, so the average error trajectory in extreme environments is better; while LLGS-SLAM inherits the strong robustness of MonoGS while introducing the L-SSIM function, considering local structural features and maintaining image details. When there is no depth information input, LLGS-SLAM verifies the camera pose tracking of the algorithm on the TUM RGB dataset. On the TUM-RGB dataset with only image color input, this application can improve the accuracy of MonoGS and lead other 3D GS-basedSLAM. At the same time, the average trajectory error accuracy of this application is 3.08 cm, which is 1.48 cm different from the feature-based method ORB-SLAM2, and this effect is achieved on the basis of LLGS-SLAM without loop detection. In general, the LLGS-SLAM method inherits the excellent properties of MonoGS, and on this basis, it has better improved the average trajectory error of the 3D GD SLAM method on different datasets.

[0136] Table 2 Online camera pose estimation results for the Replica dataset

[0137]

[0138] Table 3 Online camera pose estimation results for the TUM RGBD dataset

[0139]

[0140] Table 4 Online camera pose estimation results for TUM RGB dataset

[0141]

[0142] Tables 2-4 show the online camera pose estimation results (ATE RMSE↓[cm]) of different SLAM methods on three datasets. The method of this application consistently outperforms all SOTA dense baseline models on Replica and TUM RGBD, and is also competitive on TUM RGB. The baseline model data on Replica, TUM RGBD and TUM RGB are taken from SplaTAM, MonoGS and MotionGS.

[0143] In Table 5, LLGS-SLAM is compared with some advanced benchmark models on the Replica dataset. As can be seen from Table 5, LLGS-SLAM has achieved good results in all aspects, among which the average PSNR index is 39.19 dB, 1.69 dB ahead of the second place; the average LPIPS index is 0.043, 0.027 ahead of the second place; the average SSIM index is 0.972, only 0.003 behind the first place. At the same time, the rendering FPS of LLGS-SLAM can reach 1195.35 FPS, which is hundreds of times faster than Nerf-based SLAM, and is also much better than 3D GS-based SLAM. In terms of PSNR, the results of this application have been fully close to or even partially exceeded the 40 dB boundary, and above 40dB indicates that the image quality is excellent (that is, very close to the original image), and 30~40dB only indicates that the image quality is good (that is, the distortion is perceptible but acceptable). It is worth noting that Point-SLAM uses the ground truth depth of the image as input and uses the sensor depth (ground truth depth in Replica) to guide the sampling along the ray. LLGS-SLAM can still achieve better rendering effects when only the RGBD image is input. In terms of SSIM, Point-SLAM and GS-SLAM are slightly ahead of the LLGS-SLAM model in some scenes, but LLGS-SLAM ranks second with its stable and good effects in various scenes. Analysis of the reasons shows that Point-SLAM, as a NeRF-based SLAM, uses neural 3D point clouds and related neural features to model the radiation field, which improves the SSIM index but the effect is unstable, and it also performs poorly in other indicators and rendering speed. GS-SLAM, as a 3D GS-based SLAM, introduces a coarse-to-fine optimization method in its design, which first uses the coarse pose / filtered 3D Gaussian representation to render the full-resolution image, and then calculates the photometric loss to optimize the pose. This method can effectively improve SSIM, but it will also significantly reduce the effects of other indicators. SplaTAM, GS-SLAM and MonoGS all use different forms of weighted L1 functions. The combination of this application combines The advantages of the function and the L-SSIM function are successfully overcome by introducing the L-SSIM function. The function mainly focuses on the error calculation at the pixel level, but often ignores the shortcomings of the structure and visual perception characteristics of the image. Therefore, LPIPS, an indicator that reflects the image similarity metric and perception loss, has been greatly optimized. This application confirms that LLGS-SLAM can effectively improve the perceived quality of the image while ensuring pixel accuracy, greatly improving the visual similarity, and the explicit performance is the significant reduction of the LPIPS indicator. The rendering speed of LLGS-SLAM can reach 1195.35. Therefore, on the whole, LLGS-SLAM shows excellent performance in the three indicators and forward rendering speed, proving the advancement of the two methods of 3D GS method and combined loss function method.

[0144] Figure 5 The visualization effect of LLGS-SLAM on different scenes of the Replica dataset is shown. Compared with the baseline, LLGS-SLAM achieves higher fidelity and more scene details. Due to the addition of the L-SSIM function that can better capture local details, the rendering effect of this application shows enhanced detail expression. Details such as the screen (Office0 ~Office4), clock (Office1 and Office3), mural (Room0), pillow (Office1 and Room1) and decorations (Room2) in the scene are good examples.

[0145] Table 5 Visualization effect of LLGS-SLAM on different scenes of Replica dataset

[0146]

[0147] Table 5 shows the rendering performance comparison results of different SLAM methods on the Replica dataset. The method of this application basically outperforms all SOTA dense baseline models in terms of PSNR and LPIPS. The baseline model data is taken from SplaTAM (only two decimal places are given in SSIM and LPIPS), MonoGS and MotionGS.

[0148] In Tables 6-8, the rationality of the experimental settings is verified by ablation experiments on three datasets. LLGS-SLAM uses photometric loss and depth loss to fit the camera pose (in the camera tracking process), and uses photometric loss, depth loss and isotropic regularization to build the scene map (in the mapping process), so the three factors are ablated separately to study their role in tracking and mapping. First, the office1 scene with the worst illumination conditions in Replica is used for research in order to test the core source of LLGS-SLAM's sensitivity to light. In Table 6, when the photometric loss is ablated, LLGS-SLAM still relies on depth loss and isotropic regularization to track the camera trajectory and build the map. However, the experimental results show that all evaluation indicators are significantly affected, indicating that photometric loss plays an indispensable role in tracking and mapping. Specifically, although the system can continue to run, the accuracy and robustness are significantly reduced. Especially under the condition of missing illumination, the lack of photometric information leads to serious positioning errors (RMSE ATE is about 7 times that of the complete SLAM system after ablating photometric loss). Photometric loss mainly ensures the correct positioning of the camera in the environment by comparing the pixel brightness of the current frame with the reference frame. When this part is removed, the system loses an important visual feedback mechanism. When the depth loss is ablated, the experimental results show that LLGS-SLAM can still maintain relatively good tracking and mapping capabilities, especially indicators such as PSNR, SSIM and LPIPS are very close to the complete SLAM system. This finding shows that depth loss is not absolutely necessary to a certain extent (which also explains that LLGS-SLAM can still perform well when only RGB images are input), especially in relatively static environments. However, despite the good overall performance, the RMSE ATE increases significantly (RMSE ATE is about 5 times that of the complete SLAM system after ablating depth loss), indicating that the lack of depth information will have a certain negative impact on positioning accuracy. Depth loss provides location information by estimating the depth information of the midpoint of the scene. When this part is ablated, although the system can continue to operate by relying on other loss functions, the long-term accumulated errors are still significant (LLGS-SLAM does not have a loop detection module), resulting in inaccurate final results. When isotropic regularization is ablated, the system performs slightly worse than the complete SLAM, but the overall performance decreases to a limited extent. This shows that LLGS-SLAM has a low dependence on isotropic regularization, and the main function of isotropic regularization is to help smooth the noise and errors that may occur during map construction and avoid artifacts. At the same time, adding isotropic regularization can improve the stability and robustness of the SLAM system in complex or dynamic environments, and has a certain positive effect on improving the overall performance and various indicators.The performance of LLGS-SLAM on the TUM RGBD and TUM RGB datasets is shown in Tables 7 and 8. The ablation experimental rules are basically similar to those of LLGS-SLAM on the Replica dataset. Only the experimental results are given here for review.

[0149] Table 6 Ablation experiment results on the Replica dataset

[0150]

[0151] Table 7 Ablation experiment results on TUM RGBD dataset

[0152]

[0153] Table 8 Ablation experiment results on TUM RGB dataset

[0154]

[0155] Tables 6 to 8 show the ablation experiment results of LLGS-SLAM in the Replica office 1 scene. Since this application focuses on studying the impact of the loss function on the SLAM system, the effects of color error, geometric residual and isotropic regularization on the SLAM system are analyzed.

[0156] In Table 9, the overall performance of the system is verified by analyzing the running time of LLGS-SLAM in the TUM RGBD and RGBD dataset fr2 scenes. It can be seen that the overall running speed of the LLGS-SLAM system is 3.1 FPS without depth information and 2.48 FPS with depth information. This shows that when there is no depth information, LLGS-SLAM can rely on other features (such as color and texture) for processing, showing that the LLGS-SLAM system can still maintain a high frame rate when processing information that only relies on RGB, and has good adaptability to dynamic environments. After adding depth information, the overall running speed drops to 2.5 FPS, reflecting the significant increase in computing resources required to process depth data, further illustrating the importance of depth information in improving positioning and map construction accuracy and the sacrifice of running speed. In summary, the running time analysis not only shows the performance of LLGS-SLAM under different information inputs, but also reveals the balance between reconstruction accuracy and running time of the overall system. "FPS" refers to the end-to-end system FPS.

[0157] Table 9

[0158]

[0159] Table 9 shows the running time analysis of LLGS-SLAM in the TUM RGBD and RGBD dataset fr2 scenes, including the total execution time of the system, the FPS calculated by dividing the total number of processed frames by the total time, and the average number of mapping iterations for each newly added keyframe.

[0160] This application proposes a new dense visual SLAM method LLGS-SLAM based on 3D GS, which uses three-dimensional Gaussian to explicitly reconstruct and render scenes. When inputting RGBD and RGB image data, the complete construction, linking and interactive influence process of the four modules of the 3D GS representation module, camera tracking module, key frame management module and map update module in the system is demonstrated, realizing efficient and high-quality scene reconstruction and rendering under complex lighting conditions. At the same time, the theoretical derivation, property description and performance realization of the new loss function L-SSIM are demonstrated, highlighting that it not only inherits the excellent properties of the SSIM function, but also further emphasizes its sensitivity to complex lighting conditions. LLGS-SLAM also shows advancement in camera pose estimation, scene reconstruction and rendering, and has a very fast forward rendering speed. This application highlights the potential of 3D GS for dense visual SLAM, especially the ability to efficiently reconstruct scenes under complex lighting conditions, and provides a good idea for more complex and efficient 3D GS-based SLAM systems in the future.

[0161] In an exemplary embodiment, a computer system is provided, which may be a server or a terminal. The computer system includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer system is used to provide computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer system is used to exchange information between the processor and an external device. The communication interface of the computer system is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a dense visual scene reconstruction method is implemented.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0163] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A dense visual scene reconstruction method, characterized in that: The dense visual scene reconstruction method comprises: A dense visual scene reconstruction architecture is constructed; the dense visual scene reconstruction architecture includes: a 3D Gaussian volume representation module, a camera tracking module, a keyframe management module and a map update module; the 3D Gaussian volume representation module is used to initialize the input image data, and convert the data into a set of anisotropic 3D Gaussian volumes through reparameterization processing; and then a double-layer densification is performed using an RGB mask layer and an opacity screening layer; the camera tracking module is used to analyze and control the relationship between the camera pose and the rasterization process based on the input image data using the Jacobian matrix, and adopt The loss function of the camera iteration update is constructed by combining the function and the L-SSIM function to estimate the pose information of the online camera of the input image data; the key frame management module is used to retrieve, select and delete key frames according to the pose information and the 3D Gaussian volume visibility principle; the map update module is used to optimize the insertion of the 3D Gaussian volume according to the pose information, and use the isotropic regularization term to remove artifacts, and use The loss function in the mapping process is constructed by combining the function and the L-SSIM function, and finally the Gaussian parameters of the 3D scene are updated and the continuous reconstruction scene is maintained; Obtain image data to be reconstructed; According to the image data to be reconstructed, a dense visual scene reconstruction architecture is adopted to reconstruct the dense visual scene; When the input image data is an RGB image, use the formula Determining photometric color residuals ; When the input image data has image depth, use Determining the depth residual ; Using the formula Determine pose information ; in, is the rendered image, is the original image, is the corresponding opacity, is the RGB mask, represents element-by-element multiplication, is the L-SSIM function, is the Gaussian filter term, is the depth of the rendered image, is the original image depth, is the depth mask, is a hyperparameter, , is the hyperparameter corresponding to the photometric loss, Mathematically defined function.

2. The dense visual scene reconstruction method according to claim 1, characterized in that: The 3D Gaussian volume representation module traverses the initialized Gaussian volume through rasterization; multiple Gaussian volumes are splashed and mixed and rendered from the front end to the back end along the direction of a channel ch.

3. The dense visual scene reconstruction method according to claim 2, characterized in that: The mixing and rendering of the multiple Gaussian splashes from the front end to the back end along the direction of a channel ch specifically includes: Using the formula Determine the color of pixel i after mixed rendering ; in, is the opacity, is the density calculated by multiplying the 2D covariance by the opacity, is the offset between the 2D Gaussian mean and the pixel coordinates, is the covariance matrix of the 2D Gaussian, is the set of real numbers, is the color of pixel j, the superscript T is the transpose, N is the number of Gaussians, is the density corresponding to the k-th Gaussian body, and k is the k-th Gaussian body corresponding to the j-th Gaussian body.

4. The dense visual scene reconstruction method according to claim 1, characterized in that: Using the formula in the RGB mask layer Determine the mask ; The opacity filter layer uses the formula Conduct screening; in, , and The functions represent the stopping gradient, sigmoid function and threshold index respectively. is the index number of the Gaussian body, is the threshold of the mask, is the opacity of the Gaussian, Parameters corresponding to the RGB mask.

5. The dense visual scene reconstruction method according to claim 1, characterized in that: The formula used in the keyframe management module Confirm the selected keyframe K insert ; Using the formula Confirm to delete the keyframe K delete ; in, , are Gaussian functions of the mth and nth key frames respectively, is the intersection-and-union ratio, is the overlap coefficient; , are the threshold functions when depth information is unavailable and available, respectively. is the relative displacement; When depth information is available, is the median depth information of the mth key frame, A threshold function defined for deleting keyframes.

6. The dense visual scene reconstruction method according to claim 1, characterized in that: Using formula in map update module Regularization term to determine the homogeneity of each term; in, is the scale of the Gaussian body, is the corresponding mean, is the set of real numbers, Mathematically defined function.

7. A computer system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dense visual scene reconstruction method according to any one of claims 1 to 6.

Images