Visual dense reconstruction method based on pure geometric Gaussian splashing

Abstract

The invention discloses a visual dense reconstruction method based on pure geometric Gaussian splashing, and relates to the field of visual reconstruction, and the method comprises the steps: distributing an initial Gaussian element for each obtained initial point cloud, and forming an initial Gaussian model; generating a training set based on the acquired RGB images; determining single-view, multi-view and key point consistency loss on the basis of the training set and camera parameters in combination with a pure geometric Gaussian splash technology which only retains geometric parameters to represent a scene, and determining training loss on the basis of the loss; training the initial Gaussian model based on the training loss to obtain a pure geometric Gaussian model; and obtaining rendering depths under all views by adopting a pure geometric Gaussian model, and obtaining a visual dense reconstruction result through TSDF fusion in combination with RGB images and camera parameters corresponding to the views. According to the method, the convergence speed of the reconstruction geometry can be accelerated under the condition of reducing the computing resource and hardware performance requirements, so that the higher geometric accuracy can be achieved in a shorter time.

Description

Technical Field

[0001] This application relates to the field of visual reconstruction, and in particular to a method, apparatus, medium and product for visual density reconstruction based on pure geometric Gaussian splashing. Background Technology

[0002] Visual dense reconstruction aims to recover accurate and dense 3D models from sparse or noisy point clouds estimated from multi-view RGB images, given the camera pose (obtained via structured bundle adjustment / structure from motion (SfM)) or Simultaneous Localization and Mapping (SLAM). High-quality, realistic 3D models are crucial for downstream capabilities such as navigation, path planning, and SLAM.

[0003] Traditional dense reconstruction methods are primarily based on Multi-view Stereo (MVS) technology. MVS technology finds consistent photometric correspondences between different viewpoints, employs stereo matching and triangulation, and ultimately generates a dense 3D point cloud or mesh from the disparity map. Based on differences in geometric representation and processing flow, mainstream MVS technologies can be broadly categorized as follows: Visual Hull-based methods, such as Plane Sweeping, systematically scan a series of hypothetical planes in 3D space and evaluate their photometric consistency to sculpt the geometric contours of the target object; Depth Map-based MVS methods, which first independently estimate depth maps for each viewpoint and then fuse multiple depth maps into a consistent 3D model; and Patch-based MVS methods, which start from sparse feature points and gradually reconstruct surface details through iterative expansion and optimization of local patches.

[0004] Benefiting from strict geometric constraints and a mature theoretical framework, traditional MVS typically achieves satisfactory reconstruction results in scenes with ample texture. However, its pipeline usually involves multiple stages, resulting in relatively low overall processing time and computational efficiency. More importantly, in areas with weak texture or scenes with drastic lighting changes, the matching algorithm is prone to failure, struggling to find reliable correspondences and ultimately producing holes or geometric errors in the reconstruction results.

[0005] Neural Radiance Fields (NeRF) fundamentally reshapes the 3D scene reconstruction paradigm by introducing an implicit neural representation to achieve highly realistic new perspective synthesis. NeRF models the scene as a continuous multilayer perceptron (MLP), taking 3D coordinates and viewpoint direction as input and outputting color and volume density. The new perspective is obtained by volume rendering of camera light, and the network is optimized with the photometric error of the observed image as supervision.

[0006] Subsequent work has largely expanded upon the NeRF framework, primarily focusing on computational efficiency and rendering quality. For acceleration, hybrid explicit-implicit representations such as sparse voxel meshes (Plenoxels), tensor decomposition (TensoRF), and multi-resolution hashing (Instant-NGP) have emerged. For image quality, further improvements have been made through anti-aliasing modeling (Mip-NeRF, Zip-NeRF). However, these volume density-based representations inherently face obstacles in explicit surface extraction, limiting their use in applications requiring direct geometric manipulation or integration with traditional mesh pipelines.

[0007] To alleviate the aforementioned geometric limitations, another research approach combines volume rendering with the symbolic distance function (SDF). Early SDF methods (such as VolSDF and NeuS) first connect implicit SDF with volume rendering to achieve clean surface extraction. Building on this, NeuS provides an unbiased rendering formula, ensuring the rendered geometry is strictly aligned to the zero-level set of the SDF, significantly improving surface accuracy and establishing a new benchmark for high-fidelity reconstruction. Despite the improved geometric accuracy, early SDF methods generally suffer from the drawback of requiring lengthy optimization times for each scene, often consuming tens of hours. Subsequent methods like Neuralangelo further improved geometric quality using hybrid representations (multi-resolution hash grids), but training times could exceed 128 hours.

[0008] In summary, while implicit methods such as NeRF and SDF demonstrate superior reconstruction fidelity, they are collectively constrained by a computationally intensive paradigm: scene-by-scene optimization based on volumetric rendering. This dependency constitutes a core bottleneck in their efficiency and scalability, hindering their practical application in 3D reconstruction.

[0009] 3D Gaussian Splatting (3DGS) revolutionized scene representation by rendering a set of explicit, anisotropic Gaussian primitives using differentiable rasterization. Subsequent 3DGS-based methods applied photometric constraints to Gaussian ellipsoids, enabling rapid training and photorealistic rendering while maintaining high photometric fidelity.

[0010] Building upon this, SuGaR achieved the extraction of triangular meshes from 3DGS representations by introducing a regularization term aligned with the surface. Subsequent work expanded the focus from pure rendering quality to geometric reconstruction accuracy, primarily by improving geometric effects through modifications to primitive geometry or combinations with implicit fields.

[0011] In modifying primitive geometry, 2DGS replaces the ellipsoid with a two-dimensional Gaussian disk and proposes a view-consistent sputtering process to efficiently obtain detailed surfaces; PGSR optimizes the Gaussian into an approximately planar surface patch by constraining the primitive scale, while applying multi-view geometric consistency to enhance geometric quality; MPGS introduces a variety of Gaussian primitives to improve adaptability to different scenes.

[0012] The combination with implicit fields has also brought significant progress. GOF directly extracts surfaces using Gaussian opacity fields; Scaffold-GS organizes local 3D Gaussians using anchor points, achieving fast, high-quality rendering with fewer primitives; CarGS designed a geometry-guided densification strategy to enhance mutual learning between rendering and reconstruction, especially addressing the weight mismatch problem caused by color and depth sharing the same α blending parameter. Nevertheless, the optimization of these methods is still mainly photometric supervision, lacking direct depth supervision.

[0013] Although current technology can achieve high geometric fidelity, it still faces key limitations such as poor reconstruction efficiency and difficulty in balancing accuracy and speed, which hinders rapid application deployment.

[0014] In recent years, with the development of feedforward neural networks, a new end-to-end reconstruction paradigm based on Transformers has emerged in the field of 3D reconstruction. DUSt3R pioneered this new paradigm for dense visual reconstruction based on Transformers, directly reconstructing 3D structures from RGB images using feedforward neural networks. By transforming the pairwise reconstruction problem into a regression of point images, it relaxes the strict constraints on conventional projection camera models. However, DUSt3R only supports two image inputs; to process more than two images, it requires O(N) computation. 2 The point map is then processed, and a global optimization process is performed. This process is computationally very expensive. MASt3R improves upon the dense matching process of DUSt3R. However, it is essentially still a dense reconstruction method that uses image-pair matching.

[0015] Both FASt3R and VGGT solve this problem. Based on the Transformer architecture, they pass N images in a single forward propagation, thus bypassing the need for iterative alignment and processing hundreds of images in just a few seconds. FASt3R uses a full attention mechanism and a parallel processing architecture, enabling it to process up to thousands of images in a single forward propagation. VGGT, on the other hand, uses an innovative alternating attention mechanism. By further feeding the output to the DPT head, the network can simultaneously output camera parameters, depth maps, point maps, and 2D point tracking, achieving higher pose accuracy than FASt3R.

[0016] Transformer-based methods represent a novel direction in dense visual reconstruction, enabling dense scene reconstruction and camera pose estimation in an extremely short time. However, Transformer-based dense visual reconstruction methods still place extremely high demands on computational resources and hardware performance, making them unusable in scenarios with insufficient hardware capabilities. Summary of the Invention

[0017] The purpose of this application is to provide a visual dense reconstruction method based on pure geometric Gaussian splashing, which can accelerate the convergence speed of reconstructed geometry while reducing computational resources and hardware performance requirements, so as to achieve higher geometric accuracy in a shorter time.

[0018] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a visual density reconstruction method based on Geometry-only Gaussian Splatting (GeoGS), including: Acquire RGB images of the scene to be reconstructed, and use motion structure recovery algorithms to obtain camera parameters and initial point clouds; Assign initial Gaussian elements to each of the initial point clouds to form an initial Gaussian model; A training set is generated based on the RGB images; Based on the training set, and combined with the pure geometric Gaussian splashing technique, the single-view loss and multi-view loss are determined; wherein, the pure geometric Gaussian only retains the geometric parameters to represent the scene; Based on the camera parameters, the key point consistency loss is determined using pure geometric Gaussian splashing technology. The training loss is determined based on the single-view loss, the multi-view loss, and the keypoint consistency loss. The initial Gaussian model is trained based on the training loss until a preset number of iterations is reached, resulting in a pure geometric Gaussian model. The rendering depth of all views is obtained using the pure geometric Gaussian model, and the visually dense reconstruction result is obtained by combining the RGB image corresponding to the view and the camera parameters through TSDF fusion.

[0019] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the visual dense reconstruction method based on pure geometric Gaussian splashing provided above.

[0020] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual dense reconstruction method based on pure geometric Gaussian splashing provided above.

[0021] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the visual dense reconstruction method based on pure geometric Gaussian splashing provided above.

[0022] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a visual dense reconstruction method based on pure geometric Gaussian splashing. By employing a pure geometric Gaussian model to decouple the reconstruction geometry from the new view synthesis task, color-related parameters are removed, reducing redundancy in the model parameter space. This allows the pure geometric Gaussian model to focus on the reconstruction geometry task, significantly reducing the number of primitives required to represent scene geometry, thereby significantly reducing storage and computational overhead (i.e., computational resources). Furthermore, during training, by constructing training losses using single-view, multi-view, and keypoint consistency losses, the convergence speed of geometric accuracy can be significantly improved, achieving higher reconstruction accuracy in a shorter time and realizing more efficient visual dense reconstruction. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic flowchart of a visual density reconstruction method based on pure geometric Gaussian splashing provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the implementation process of a visual density reconstruction method based on pure geometric Gaussian splashing provided in an embodiment of this application; Figure 3The GeoGS- provided in one embodiment of this application The diagram illustrates the convergence process of visual dense reconstruction using 2DGS on the DTU dataset, with a total of 2000 iterations and an iteration interval of approximately 100 between each point. Figure 4 The GeoGS- provided in one embodiment of this application The diagram illustrates the convergence process of visual dense reconstruction using PGSR on the DTU dataset, with a total of 2000 iterations and an iteration interval of approximately 100 between each point. Figure 5 A schematic diagram showing the distribution of visualized quantitative results obtained by iterating 2000 times on the DTU dataset using different methods provided in an embodiment of this application; Figure 6 A schematic diagram showing the distribution of visualized quantitative results obtained by iterating 2000 times on the TNT dataset using different methods provided in an embodiment of this application; Figure 7 A schematic diagram showing the distribution of visualized quantitative results obtained by iterating 2000 times on the Replica dataset using different methods provided in an embodiment of this application; Figure 8 A schematic diagram showing the distribution of visualized quantitative results obtained by training different methods on the DTU dataset for 100 seconds according to an embodiment of this application; Figure 9 A schematic diagram showing the distribution of visualized quantitative results obtained by training different methods on the TNT dataset for 100 seconds according to an embodiment of this application; Figure 10 A schematic diagram showing the distribution of visualized quantitative results obtained by training different methods on the Replica dataset for 100 seconds according to an embodiment of this application; Figure 11 A schematic diagram of qualitative results arrays for DTU, TNT, and Replica datasets provided by different methods in an embodiment of this application; Figure 12 A schematic diagram showing the cumulative frequency distribution of the time ratios used by different methods when the accuracy percentage is 80%, provided for an embodiment of this application; Figure 13 A schematic diagram showing the cumulative frequency distribution of the time ratios used by different methods when the accuracy percentage is 90%, provided for an embodiment of this application; Figure 14 A schematic diagram showing the cumulative frequency distribution of the time ratios used by different methods when the accuracy percentage is 100%, provided as an embodiment of this application; Figure 15This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] In one exemplary embodiment, this application provides a visual dense reconstruction method based on pure geometric Gaussian splashing, which can be executed by a GPU under the control of computer program instructions. Figure 1 As shown, the method includes: Step 100: Obtain the RGB image of the scene to be reconstructed, and use the motion structure recovery algorithm to obtain camera parameters and initial point cloud.

[0028] Step 101: Assign initial Gaussian elements to each initial point cloud to form an initial Gaussian model.

[0029] Step 102: Generate a training set based on RGB images.

[0030] Step 103: Based on the training set, and using the pure geometric Gaussian splashing technique, determine the single-view loss and multi-view loss. The pure geometric Gaussian technique removes the color parameters required in the traditional 3DGS framework, retaining only the geometric parameters to represent the scene, thus significantly reducing the number of parameters and Gaussians, alleviating the computational burden, and accelerating convergence.

[0031] Step 104: Determine the key point consistency loss based on camera parameters and using pure geometric Gaussian splashing technique.

[0032] Step 105: Determine the training loss based on single-view loss, multi-view loss, and keypoint consistency loss.

[0033] Step 106: Train the initial Gaussian model based on the training loss until the preset number of iterations is reached, and then obtain the pure geometric Gaussian model.

[0034] Step 107: Use a pure geometric Gaussian model to obtain the rendering depth for all views, and combine the RGB images corresponding to the views and camera parameters to obtain the visually dense reconstruction result (i.e., the mesh corresponding to the Gaussian model) through TSDF fusion.

[0035] In one embodiment of this application, step 100 corresponds to data acquisition and camera pose preparation. In practical applications, multi-view RGB images of the scene to be reconstructed can be acquired, and the camera parameters, including the camera intrinsic parameter K and the extrinsic parameters (R, t) for each frame, can be obtained using the Structure from Motion (SfM) algorithm, resulting in a sparse 3D point set P_anchor (formed from the initial point cloud). These camera parameters are used to calculate the multi-view loss and keypoint consistency loss. Given the intrinsic and extrinsic parameters of two cameras (i.e., the poses of each camera), the relative pose from one camera to the other can be obtained. Based on the relative pose, the coordinates of points corresponding to the viewpoint of one camera can be projected onto the coordinate system of the other camera, thereby solving for the multi-view loss and keypoint consistency loss.

[0036] In one embodiment of this application, step 101 described above corresponds to Figure 2 The initialization process is as follows: In practical applications, an initial Gaussian model is generated from the initial point cloud to assign an initial Gaussian primitive to each point. The position of each point is initialized to the position (mean) of a Gaussian primitive. The scale of the Gaussian primitive is initialized based on the distance between the point and the nearest point. Rotation is initialized to a unit quaternion, and opacity is initialized to 0.1. Each Gaussian primitive has a position (mean) denoted as position (…). ), scale (denoted as Rotation (denoted as) ), transparency (opacity) The parameters, such as color parameters, are eliminated, eliminating the need for the SH coefficients used in traditional 3D Gaussian Splatting (3DGS). This approach reduces the number of parameters in the Gaussian model, improving training convergence speed and allowing for a focus on dense geometry reconstruction tasks.

[0037] In one embodiment of this application, in order to ensure that the geometric information of a single view is sufficiently constrained, the process of determining the single view loss based on the training set and combined with the pure geometric Gaussian splashing technique in step 103 above is replaced by the following steps 200-204.

[0038] Step 200: Determine the reference view and obtain the corresponding RGB image, rendered normal map, and depth map. Both the rendered normal map and depth map are obtained using pure geometric Gaussian splashing technology. For each view in the training set (with the same constraints as a single view), it serves as the reference view. i (reference view) 。

[0039] For example, for a reference view iThe corresponding real RGB image is C, the rendering normal map obtained by using pure geometric Gaussian splashing technology is N, and the depth map is D. Let... Indicates the current view The set of pixels in the image.

[0040] Step 201: Solve for the gradient based on the depth map to obtain the depth normal map. The depth normal map is denoted as N. D .

[0041] Step 202: Based on the depth normal map and the rendered normal map, and combined with the RGB image, obtain the single-view normal consistency loss. This application uses a color map. As edge-aware weights, used to calculate the current view Rendering normal map With depth-normal map The normals between them are consistent, and we have: .

[0042] In the formula, This represents the loss of consistency of normals in a single view. p It is a set Pixels in This represents the total number of pixels in the actual RGB image C. A first-order gradient plot representing the true RGB image. It represents the 1-norm.

[0043] Step 203: Based on the depth map and the RGB image, and using the gradient map of the RGB image as weights, obtain the depth smoothing loss. Specifically, for the depth smoothing loss, calculate the second derivative of the depth map D, and use the gradient map of the image as weights, we have: .

[0044] In the formula, This represents the depth smoothing loss.

[0045] Step 204: Obtain the single-view loss based on the single-view normal consistency loss and depth smoothing loss. The single-view loss is expressed as: .

[0046] In the formula, Indicates single-view loss. The weights representing the single-view normal consistency loss are... The weights represent the depth smoothing loss.

[0047] In one embodiment of this application, constraining only a single view is insufficient for reconstructing high-precision geometry. Constraining multiple views is also necessary. Figure 1To improve consistency and introduce absolute depth information, this application introduces enhanced multi-view photometric loss and multi-view depth consistency loss. Furthermore, for traditional multi-view... Figure 1 Consistency loss is insufficient when absolute depth information is lacking; simply constraining the consistency between views is inadequate. Therefore, this application improves upon traditional multi-view photometric loss by calculating a high proportion of SSIM metrics, using depth-based visibility masks, and calculating the cumulative error of multiple neighboring non-repeating pixels. Based on this, the process of determining the multi-view loss in step 103 above, using the training set and combining pure geometric Gaussian splashing techniques, can be replaced by steps 300-307.

[0048] Step 300: Obtain the neighbor set of the reference view based on the angle and distance of the camera that acquired the RGB image.

[0049] Step 301: Randomly sample multiple (e.g., from the neighbor set) S (Number) neighbor views.

[0050] Step 302: Project each pixel in the reference view onto randomly sampled neighboring views. Perform bilinear interpolation on pixels within a defined range in the neighboring views after projection to obtain estimated color and estimated depth, resulting in an estimated color map and an estimated depth map. For example, project each pixel in the reference view... p Projected onto the sampled adjacent view j The estimated color is calculated by bilinear interpolation of surrounding pixels in adjacent views. and estimated depth Then, the estimated color and estimated depth of all pixels are aggregated (this set of pixels is consistent with the set defined in step 200). (same), to obtain the estimated color map and estimated depth map .

[0051] Step 303: Evaluate the similarity between the true color map and the estimated color map using a weighted combination of the Structural Similarity Index (SSIM) and the L1 metric. The similarity is expressed as: .

[0052] In the formula, Represents the true color map. , This emphasizes structural similarity rather than viewpoint-dependent color values. Please note that... It is a with A matrix with the same dimensions stores the data in... The luminance loss of each pixel on the screen.

[0053] Step 304: Based on the estimated depth map and the depth map of the reference view, determine the depth-based visibility mask of the reference view relative to each neighboring view, and combine the processed visibility mask to obtain the total visibility mask. The purpose of this step is to avoid calculating pixels that are actually invisible.

[0054] For example: for each pair of reference views i and adjacent views j This application first defines the set of visible pixels as follows: .

[0055] In the formula, It is a predefined negative number that is very close to zero, which is more robust to noise than setting it directly to 0. Indicates reference view i Rendered by pure geometric Gaussian splashing technique at the pixel level p The depth at that location.

[0056] Finally, to prevent the optimization process from getting trapped in undesirable local minima, this application accumulates photometric errors from multiple adjacent views, while avoiding ambiguity caused by duplicate pixels. Directly averaging the loss over all adjacent pixels would repeatedly calculate the same pixels. Therefore, this application only accumulates non-redundant pixels between adjacent pixels. Initialization And update the set of visible pixels obtained from adjacent pixels 1 to j as follows: .

[0057] Then, for adjacent views In this context, the set of pixels involved in the loss calculation is defined as: .

[0058] in, Indicates belonging to a set But not in the collection The pixels in the depth map. Then, a binary mask with the same spatial resolution as the depth map is constructed. : .

[0059] Step 305: Obtain the multi-view photometric loss based on the total visibility mask and similarity. The multi-view photometric loss is expressed as: .

[0060] In the formula, This indicates the photometric loss in multiple views. express The number of pixels in the image.

[0061] Step 306: Determine the multi-view depth consistency loss based on the visibility mask, the projection depth from the reference view to adjacent views, and the depth of the adjacent views. Unlike the photometric loss, the multi-view depth consistency loss only calculates the loss between the reference view and the first of the sampled adjacent views, and is as follows: .

[0062] In the formula, This represents the multi-view depth consistency loss. This represents the estimated depth from the reference view to the adjacent view. Representing adjacent views j Rendered by pure geometric Gaussian splashing technique at the pixel level p The depth at that location. This represents the set of visible pixels of the first view in the sampled neighboring views.

[0063] Step 307: Obtain the multi-view loss based on the multi-view depth consistency loss and the multi-view photometric loss. The multi-view loss is as follows: .

[0064] In the formula, Indicates multi-view loss. The weights representing the photometric loss in multiple views are: The weights represent the depth consistency loss of multiple views.

[0065] Step 400: To improve keypoint localization accuracy and fully utilize reliable keypoint information from the SfM prior, this application introduces a point correspondence loss based on the SfM prior, which works in both the 3D world space and the 2D image space. Furthermore, a sparse point matching constraint is introduced to explicitly enforce consistency at keypoints. Reliable geometric priors obtained from SfM are utilized, namely the 3D world coordinates of the keypoints and their corresponding 2D observations in the visible images. To effectively utilize this information, this application also constructs a co-visibility graph that encodes the visibility relationship between points and images, mapping each 3D point to its 2D observation in all visible images.

[0066] Step 401: In the reference view There is A key effective point is to first reverse-project their two-dimensional positions into the three-dimensional camera space, thereby achieving a view... Obtain optimized 3D key points in the camera coordinate system: .

[0067] in, Represents a view The homogeneous pixel coordinates of the key points, and It is a storage A vector representing the depth of each key point. Here, " "" indicates that homogeneous coordinates are converted into three-dimensional coordinates, where the z component represents the depth value.

[0068] Step 402: Then, for Calculate the correspondence between spatial points in 3D world space and 2D image space. To maintain consistency in world space, [the following is omitted as the text is incomplete and cannot be translated]. Transform to the world coordinate system and compare with the 3D keypoints obtained based on SfM to derive the 3D position loss: .

[0069] in, From reference view The transformation matrix from the camera to the world coordinate system, and This represents the world coordinates of the keypoints obtained through the single-view matching method.

[0070] Step 403: To achieve better alignment in image space, the two-dimensional projection loss between the projected keypoints and their two-dimensional observations was further calculated from adjacent viewpoints: .

[0071] in, This represents the homogeneous pixel coordinates of key points in the first sampled neighboring view obtained from Structure from Motion (SfM), while From reference view The transformation matrix to its first sampled adjacent view. It corresponds to the z-coordinate vector and is extended into three channels.

[0072] Step 404: To handle outliers, for and All applications used truncation loss. The resulting keypoint consistency loss is defined as: .

[0073] in, and They represent and The cutoff threshold.

[0074] Based on this, the training loss obtained in step 105 of this application can be expressed as: .

[0075] In the formula, This indicates training loss.

[0076] Furthermore, based on the above description, the framework provided in this application adopts a hierarchical coarse-to-fine learning process, aiming to enhance the robustness of optimization while improving computational feasibility. Based on this, step 106 of this application optimizes the pure geometric Gaussian model by backpropagating gradients according to the training loss. In practical applications, this optimization training process can be carefully divided into three consecutive stages. The first is an initial coarse stage (iterations 0-250), with a resolution only one-quarter of the original resolution, used to establish a stable global geometric framework. This is followed by an intermediate stage (iterations 250-500), increasing the resolution to half to refine the mid-frequency structure. The final stage of the entire process is a fine-tuning stage (iterations 500-2000), performed at full input resolution, aimed at capturing the finest geometric and texture details. To maintain geometric consistency throughout the multi-scale process, the camera intrinsics are scaled proportionally with each resolution conversion, ensuring the integrity of the projection mathematics.

[0077] In one embodiment of this application, during rendering using pure geometric Gaussian splashing technology, only the rendering flow of depth and normals in 3DGS is followed. GeoGS, similar to 3DGS, utilizes a set of Gaussian primitives to optimize scene representation; each Gaussian primitive has key attributes, including position. Covariance However, SH coefficients for color fitting are not required. Each Gaussian unit... The definition is as follows: .

[0078] Among them, location This can be represented in the camera coordinate system as In the form of. It is the independent variable, representing the position of a point in space. Covariance Through the corresponding scaling matrix and rotation matrix The calculations yielded the following: .

[0079] During the rendering process, the color of a pixel Through The color of an ordered Gaussian function and combined The results were obtained from the mixed calculation method: .

[0080] in, Indicates the index of the Gaussian function. It was created by Gorski Yuan The SH coefficient represents the view-related color, while It is based on the The resulting two-dimensional Gaussian distribution and the learnable opacity of each Gaussian function are calculated.

[0081] Two important extensions of 3DGS are 2D Gaussian Splatting (2DGS) and Planar-based Gaussian Splatting (PGSR), which use the same color rendering formulas as described above.

[0082] Both 2DGS and PGSR use the same α-mix formula to draw normal vectors. : .

[0083] in, It has an index The normal vector of the Gaussian element, and It is the rotation matrix from the camera coordinate system to the world coordinate system.

[0084] The main difference lies in the introduction of more advanced depth definitions and corresponding geometric constraints in these methods. 2DGS fuses 3DGS into a set of two-dimensional oriented planar Gaussian disks and represents the desired depth by weighting the intersection depths. ,have: .

[0085] In the formula, z is the th i The z-coordinate of an ordered Gaussian function.

[0086] PGSR proposes a planar Gaussian splashing technique that generates unbiased depth values ​​by intersecting light rays with a Gaussian plane. ,have: .

[0087] In the formula, This represents the distance from the plane to the center of the camera. It is the camera intrinsic parameter matrix. It is a homogeneous coordinate representation of a two-dimensional position.

[0088] Based on the above description, this application adopts the primitives proposed by 2DGS and PGSR, but it no longer needs to render view-related colors. It only follows the rendering process of its depth and normals. When color parameters are needed, the color can be projected from the current view to adjacent views, and the color fitting estimate can be obtained through interpolation (see the above description of the calculation of multi-view loss for details). In this application, GeoGS based on 2DGS primitives is referred to as GeoGS- GeoGS based on PGSR primitives is called GeoGS- For GeoGS- and GeoGS- All of them use the geometric constraints proposed above to achieve high reconstruction efficiency.

[0089] In one embodiment of this application, the pure geometric Gaussian method provided in this application is applied based on 2DGS and PGSR, and its depth and normal rendering flow is followed. In this embodiment, for GeoGS- and GeoGS- All of them use the geometric constraints provided above to achieve high reconstruction efficiency.

[0090] To verify the reconstruction performance of the method provided in this application, experiments were conducted on various datasets, including objects, indoor and outdoor environments. Fifteen widely used scan data from the DTU dataset, six scenes from the Tanks & Temples (TNT) dataset, and eight sequences from the Replica dataset were selected. DTU and TNT are widely used datasets for evaluating multi-view reconstruction, while Replica is widely used as a benchmark for SLAM. DTU is a large object-centric dataset with 49 / 64 images per scene. TNT contains complex real-world indoor and outdoor scenes, with approximately 263 to 1106 images per scene. Replica is an indoor SLAM dataset with 2000 frames per scene. Considering the realities of dense reconstruction in SLAM, this embodiment only used keyframes generated by HI-SLAM. Each scene has approximately 80 to 110 frames. Utilizing the camera poses provided by these datasets, an initial sparse point cloud was generated for each scene using Colmap.

[0091] The performance of GeoGS is compared with state-of-the-art reconstruction methods, including 2DGS, GOF, and PGSR. First, convergence during the reconstruction process is demonstrated. Then, the reconstruction accuracy at the same time points and the training time required to achieve the same accuracy are evaluated. The method presented in this application achieves significantly state-of-the-art results on all datasets.

[0092] (1) Reconstruct the convergence curve.

[0093] To investigate the convergence of reconstruction accuracy during training, convergence curves for GeoGS and benchmark methods were first plotted on the DTU scan dataset 114. During training, the Gaussian model was saved every 10 iterations, and the corresponding training time was recorded. It's important to note that the time spent saving the model was excluded to reduce training error. Then, the chamfer distance for each model after training was calculated.

[0094] Subsequently, for a stopping criterion of 2000 iterations, data points were sampled and plotted every 100 iterations, as shown below. Figure 3 and Figure 4 As shown. GeoGS- The initial points differ from those of 2DGS because they define depth differently. For 2DGS, the median depth is used, based on its recommendation for the DTU dataset; while for GeoGS... It simply uses a common weighted depth definition to ensure its robustness.

[0095] In this embodiment, GeoGS- Unlike 2DGS, the initial point is different because different depth definitions lead to different initial Gaussian models.

[0096] These curves show that the GeoGS algorithm not only converges significantly faster than the benchmark algorithms, but also achieves higher geometric accuracy in a shorter time. For the PGSR benchmark algorithm, GeoGS shows a more significant improvement in convergence speed in the early stages. GeoGS also improves the convergence speed for the 2DGS benchmark algorithm. Although 2DGS and GeoGS-... The numerical differences between the two methods are not significant, but in reality, the accuracy difference is as high as 397%, with a chamfer distance of 0.36 (GeoGS) compared to 1.79 (2DGS) at 2000 iterations. GeoGS achieves significantly higher accuracy under the same stopping criteria.

[0097] (2) Reconstruction accuracy at the same time.

[0098] To fairly compare reconstruction efficiency, the reconstruction quality of all methods was evaluated using the same training stopping criterion (2000 iterations and 100 seconds). For a clearer comparison of results, in Figures 5-10 The average results of all methods across all datasets are presented as a scatter plot. Quantitative results are shown in... Figure 11 Presented in the middle.

[0099] For DTUs, GeoGS achieves high reconstruction accuracy under both stopping criteria while maintaining low storage requirements. Specifically, GeoGS- It achieves the highest accuracy with minimal model storage requirements. GeoGS demonstrates high reconstruction efficiency in both reconstruction quality and storage.

[0100] For TNT, GeoGS demonstrated the highest reconstruction efficiency in most indoor and outdoor scenarios. Specifically, GeoGS- It has the highest accuracy and uses only 14% to 30% of the storage space required by current methods.

[0101] For Replica, GeoGS consistently demonstrates the highest reconstruction efficiency. GeoGS- It achieves the highest precision, while GeoGS- This achieves minimal storage requirements. Note that this embodiment reconstructs the scene using only keyframes within a short time. The reconstruction efficiency is compared to current visually dense reconstruction methods, rather than SLAM methods with incremental reconstruction and offline optimization capabilities. Comparing the results of this application to SLAM is unfair because the initialization, system level, and training data size of this application are different from those of SLAM. However, the method provided in this application demonstrates the highest reconstruction efficiency among similar methods. This indicates that employing the method provided in this application will improve the performance of tasks such as SLAM. GeoGS is able to reconstruct smoother and more accurate meshes in a short time.

[0102] (3) Training time required to achieve the same reconstruction accuracy In addition to achieving the same reconstruction quality within the same timeframe, this embodiment also compares the training time required to achieve the same reconstruction accuracy on the Replica dataset. For each scenario, the highest accuracy achieved by all methods in 2000 iterations is first determined, and 80%, 90%, and 100% of the best accuracy are set as baseline accuracies. Then, the time required for each method to reach these baseline accuracies is measured. Due to the inherent differences in training time across different scenarios, the results are standardized by identifying the fastest method in each scenario and calculating the time ratio of all methods relative to the fastest method. This approach enables obtaining the time ratio of all methods across all scenarios when reaching baseline accuracy. Subsequently, a cumulative frequency plot of these time ratios is constructed for comprehensive comparative analysis.

[0103] The cumulative frequency plots of time ratios are given for accuracy levels of 80%, 90%, and 100% of the highest accuracy, as shown below. Figures 12-14As shown, the closer the cumulative frequency curve is to the time ratio 1, the more often the method exhibits faster reconstruction times in the corresponding scenarios, indicating higher overall reconstruction efficiency. Specifically, the ordinate intercept at time ratio 1 represents the proportion of scenarios where each method achieves the fastest reconstruction speed. The time ratio frequency value at the origin signifies the proportion of scenarios where this method can achieve the fastest reconstruction.

[0104] The results show that, across all accuracy levels, GeoGS- The reconstruction time is the shortest. PGSR can achieve a stable and fast reconstruction time, but its reconstruction efficiency is still lower than GeoGS-. GeoGS- It performs relatively faster under low to medium accuracy requirements, but its reconstruction speed is slower under high accuracy requirements. For primitives with 2DGS features, GeoGS performs best, and it is the most efficient in scene reconstruction.

[0105] In summary, compared with the prior art, this application also has at least the following advantages: 1) The number of primitives required to represent the same scene is greatly reduced, and the model storage and computational overhead are also significantly reduced. This advantage mainly stems from the pure geometric Gaussian model. Traditional 3DGS couples the geometry reconstruction task with the new view synthesis task. The model not only needs to express geometric details, but also requires a large number of primitives to express the rich texture details in the scene. However, this application decouples the geometry reconstruction and new view synthesis tasks, removes color-related parameters, reduces redundancy in the model parameter space, and allows the model to focus on the geometry reconstruction task, thus greatly reducing the number of primitives required to express the scene geometry.

[0106] 2) When performing visually dense reconstruction, this application significantly improves the convergence speed of geometric accuracy during training, achieving higher reconstruction accuracy in a shorter time and realizing more efficient visually dense reconstruction. This advantage stems from the aforementioned training optimization method of the pure geometric Gaussian model, as described in advantage 1), which significantly reduces the number of primitives and parameters. This greatly reduces the parameters that need to be optimized during training, accelerating the convergence process. More importantly, for the proposed pure geometric Gaussian model, this application also proposes a set of visual geometric constraints. For multi-view constraint consistency, a depth-based visibility mask is introduced to provide depth information to the training process; keypoint constraints based on the SfM sparse anchor space correspondence (i.e., keypoint consistency loss) further improve the geometric consistency of keypoints. Together, these two factors accelerate the convergence speed of the reconstructed geometry, enabling this application to achieve higher geometric accuracy in a shorter time.

[0107] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 15 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores visual dense reconstruction data based on pure geometric Gaussian splashing. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a visual dense reconstruction method based on pure geometric Gaussian splashing.

[0108] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0109] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0110] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0111] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0114] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0115] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A visual density reconstruction method based on pure geometric Gaussian splashing, characterized in that, include: Acquire RGB images of the scene to be reconstructed, and use motion structure recovery algorithms to obtain camera parameters and initial point clouds; Assign initial Gaussian elements to each of the initial point clouds to form an initial Gaussian model; A training set is generated based on the RGB images; Based on the training set, and combined with the pure geometric Gaussian splashing technique, the single-view loss and multi-view loss are determined; wherein, the pure geometric Gaussian only retains the geometric parameters to represent the scene; Based on the camera parameters, the key point consistency loss is determined using pure geometric Gaussian splashing technology. The training loss is determined based on the single-view loss, the multi-view loss, and the keypoint consistency loss. The initial Gaussian model is trained based on the training loss until a preset number of iterations is reached, resulting in a pure geometric Gaussian model. The rendering depth of all views is obtained using the pure geometric Gaussian model, and the visually dense reconstruction result is obtained by combining the RGB image corresponding to the view and the camera parameters through TSDF fusion.

2. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 1, characterized in that, Based on the training set, the process of determining the single-view loss using the pure geometric Gaussian splashing technique includes: A reference view is determined, and the corresponding RGB image, rendering normal map, and depth map are obtained; the rendering normal map and the depth map are both obtained using pure geometric Gaussian splashing technology. Gradient calculation is performed based on the depth map to obtain the depth normal map; Based on the depth normal map and the rendering normal map, and combined with the RGB image, the single-view normal consistency loss is obtained; Based on the depth map and the RGB image, and using the gradient map of the RGB image as weights, a depth smoothing loss is obtained; The single-view loss is obtained based on the single-view normal consistency loss and the depth smoothing loss.

3. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 2, characterized in that, Based on the training set, the process of determining the multi-view loss using the pure geometric Gaussian splashing technique includes: The neighbor set of the reference view is obtained based on the angle and distance of the camera that acquired the RGB image; Randomly sample multiple adjacent views from the neighbor set; Each pixel in the reference view is projected onto the randomly sampled neighboring view. Bilinear interpolation is performed on the pixels within a set range in the neighboring view after the projection to obtain the estimated color and estimated depth, and thus obtain the estimated color map and estimated depth map. The similarity between the real color map and the estimated color map is evaluated by a weighted combination of structural similarity index and L1 index; Based on the estimated depth map and the depth map of the reference view, a depth-based visibility mask of the reference view relative to each adjacent view is determined, and the total visibility mask is obtained by combining the processed visibility mask. The multi-view photometric loss is obtained based on the total visibility mask and the similarity. The multi-view depth consistency loss is determined based on the visibility mask, the projection depth from the reference view to the adjacent view, and the depth of the adjacent view. The multi-view loss is obtained based on the multi-view depth consistency loss and the multi-view photometric loss.

4. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 1, characterized in that, Based on the camera parameters, the keypoint consistency loss is determined using a pure geometric Gaussian splashing technique, including: The world coordinates are obtained by constructing reference frames and neighboring frames using a motion structure recovery algorithm and performing sparse reconstruction. Obtain the coordinates of homogeneous pixels in neighboring frames; The 3D camera coordinates are determined by inverting the intrinsic parameter matrix of the camera parameters and combining it with the depth obtained using pure geometric Gaussian splashing technology. Determine the camera transformation matrix from the reference frame to the world coordinate system; Determine the transformation matrix from the reference frame to the neighboring frame; The depth coordinates in the neighboring camera system are obtained, and the 3D position loss and 2D reprojection loss are obtained by combining the world coordinates, the homogeneous pixel coordinates, the 3D camera coordinates, the camera transformation matrix, the transformation matrix and the threshold in the truncation loss function. The keypoint consistency loss is obtained based on the 3D position loss and the 2D reprojection loss.

5. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 3, characterized in that, The similarity is expressed as: ； In the formula, Indicates similarity. Represents the structural similarity index. This refers to the L1 index. Represents the true color map. This represents the estimated color map. , .

6. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 4, characterized in that, The 3D position loss is expressed as: ； The 2D reprojection loss is expressed as: ； In the formula, Indicates 3D position loss. Indicates 3D camera coordinates. Represents the camera transformation matrix. Represents world coordinates, Represents the norm, Indicates 2D reprojection loss. Represents the transformation matrix. Represents the depth coordinates in a neighboring camera system. Indicates the coordinates of homogeneous pixels in neighboring frames. and Both represent the threshold in the truncation loss function.

7. The visual density reconstruction method based on pure geometric Gaussian splashing according to claim 1, characterized in that, During the rendering process using pure geometric Gaussian splashing technology, only the rendering workflow of depth and normals in 3DGS is followed. When color parameters are used, color rendering is achieved by projecting the current view onto adjacent views and interpolating the results.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the visual dense reconstruction method based on pure geometric Gaussian splashing as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the visual dense reconstruction method based on pure geometric Gaussian splashing as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the visual dense reconstruction method based on pure geometric Gaussian splashing as described in any one of claims 1-7.

Images