A 3D Gaussian surface reconstruction method and system based on deep supervision multi-view densification

Abstract

The present application relates to the technical field of image data processing, and especially relates to a 3D Gaussian surface reconstruction method and system based on deep supervision multi-view densification. The method comprises the following steps: obtaining an unbiased depth map by using colmap based on generated images, and performing supervised geometric optimization on rendering quality by constructing a loss function; performing multi-view adaptive densification and depth verification based on the optimization result; performing depth error guided fine pruning based on the densification result; and obtaining a three-dimensional reconstruction network. Through the deep verification multi-view adaptive densification (DVCD) and depth error guided pruning (DRP) strategies, the generation and propagation of redundant Gaussians are effectively inhibited.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method and system for 3D Gaussian surface reconstruction based on depth-supervised multi-view densification. Background Technology

[0002] Traditional 3D reconstruction methods (such as multi-view stereo (MVS) and structured light scanning) have inherent drawbacks, including high equipment costs, susceptibility to environmental interference, and limited reconstruction accuracy, posing significant challenges in complex or dynamic scenes. Typical point cloud-based methods (Poisson reconstruction, Delaunay triangulation, and moving least squares (MLS)) directly process unstructured 3D point clouds but are highly sensitive to point cloud noise; voxelization methods (moving cube, level set, and signed distance field (SDF)) are limited by voxel resolution, resulting in a trade-off between accuracy and computational cost; and multi-view geometric methods (SfM and MVS) rely on feature matching accuracy and are susceptible to incorrect matching.

[0003] In recent years, Neural Radiation Fields (NeRF) has achieved breakthroughs in novel perspective synthesis by combining deep learning with neural rendering. However, its training speed is extremely slow (ranging from hours to days) and its volumetric rendering efficiency is low (per-pixel ray tracing), severely limiting real-time interactive experiences. 3DGaussian Splatting (3DGS), as an emerging 3D reconstruction paradigm, has achieved significant breakthroughs in rendering quality and efficiency: it uses explicit 3D Gaussian primitives to represent scenes, achieves real-time (above 30fps) rendering through GPU rasterization, and compresses training time to a few minutes. However, 3DGS has the following significant shortcomings in geometric reconstruction capabilities:

[0004] (1) Geometric reconstruction relies on RGB self-supervision and lacks geometric constraints. Gaussian elements are like pebbles embedded on the surface of an object, resulting in rough and incomplete surface meshes. In complex situations such as reflective surfaces (glass, metal), transparent materials, and textureless areas in the distance (sky, clouds, distant buildings), a large number of mirrored Gaussians and floating Gaussians are easily generated, which seriously affects the accuracy of geometric reconstruction and downstream editing effects.

[0005] (2) The adaptive densification (ADC) strategy of 3DGS judges whether to split / clone Gaussian based on the cumulative mean of the gradient of the position parameter in a single view. This easily generates a large number of redundant Gaussian units that are only effective in a few views. This not only leads to a sharp increase in model size (reduced training speed and increased memory usage), but may also cause local overfitting, produce artifacts in other views, and impair the surface reconstruction quality and editing accuracy.

[0006] To address the aforementioned shortcomings, methods such as PGSR significantly improve reconstruction accuracy by flattening the ellipsoidal Gaussian shape into a planar representation (planar Gaussian) and introducing unbiased depth rendering and multi-view geometric photometric consistency constraints. However, PGSR's geometric optimization still fundamentally relies on the accuracy of RGB reconstruction—its geometric constraints are based on the Gaussian depth generation normal constraints reconstructed during the RGB supervised phase. This self-supervised approach means that the quality of geometric reconstruction is largely limited by the accuracy of RGB reconstruction. When reflective surfaces, transparent objects, or weakly textured areas exist in the scene, the inaccuracies in RGB reconstruction directly propagate to the geometric optimization process, leading to a significant decrease in geometric reconstruction quality. Furthermore, PGSR adopts the single-view densification strategy of standard 3DGS, failing to effectively suppress the generation of redundant Gaussians, resulting in poor training efficiency in large-scale scenes.

[0007] Meanwhile, some studies have attempted to improve the redundancy problem of 3DGS densification. However, these methods are all based on general 3DGS and have not designed a dedicated mechanism for the specific geometric information requirements of PGSR. They also lack simultaneous verification of depth geometry accuracy. At present, there is no method to solve the following two core problems at the same time: (1) Provide accurate real physical scale depth supervision information for 3DGS geometry optimization without relying on expensive hardware (RGB-D cameras have poor outdoor performance, and LiDAR is expensive and needs to be associated with camera pose); (2) Under the premise of ensuring reconstruction quality, accurately judge the necessity of Gaussian densification from the perspective of multi-view consistency and superimpose depth geometry verification to effectively suppress the generation of redundant Gaussians and artifact Gaussians, and improve training efficiency and geometric accuracy. Summary of the Invention

[0008] This invention aims to address the following problems in geometric surface reconstruction using existing 3DGS and its improved methods: (1) lack of real physical scale depth supervision, resulting in severe mirrored Gaussian artifacts and floating Gaussians in reflective surfaces, transparent materials, and textureless distant views, leading to insufficient geometric reconstruction accuracy; (2) single-view densification strategies cannot accurately determine the necessity of Gaussian growth, resulting in the generation of a large number of redundant Gaussians, low training efficiency, high memory overhead, and easy introduction of erroneous Gaussians, affecting the final reconstruction quality. By introducing a real-scale depth supervision mechanism driven by monocular depth estimation, and depth-verified multi-view adaptive densification (DVCD) and depth error-guided fine pruning (DRP) strategies, this invention achieves high-quality 3D surface reconstruction with higher geometric accuracy and faster training speed, while maintaining rendering quality comparable to PGSR, providing an efficient and feasible solution for high-precision geometric demand scenarios such as digital twins, cultural heritage protection, and industrial inspection.

[0009] In a first aspect, the present invention provides a 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification, which adopts the following technical solution:

[0010] A 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification includes:

[0011] Acquire SFM point cloud image data;

[0012] Preprocessing is performed based on the acquired SFM point cloud image data;

[0013] Calculate a multi-view relative depth map based on the preprocessed image data, and generate a real-scale depth map based on the relative depth map;

[0014] Based on the generated image, flat Gaussian point cloud and differentiable rendering are performed using colmap to obtain an unbiased depth map, and supervised geometric optimization of rendering quality is performed by constructing a loss function.

[0015] Multi-view adaptive densification and depth verification are performed based on the optimization results;

[0016] Fine pruning guided by depth error based on densification results;

[0017] A three-dimensional reconstruction network was obtained.

[0018] Furthermore, the preprocessing of the acquired SFM point cloud image data includes employing a pre-trained DA3 single Transformer architecture to process the input in an arbitrary view relative depth mode, performing normalization preprocessing on each image, and performing pixel value normalization by adaptive scaling to a 504×504 compatible resolution. Subsequently, for single views without known camera poses, the same shared learnable camera token is used for all views. As geometric placeholders, the camera token for each view is concatenated with the patch token obtained from image segmentation, and then fed into the Transformer backbone network. The model divides all Transformer layers into two groups in a 2:1 ratio, for a total of: ,forward The layer only performs in-view self-attention, extracting multi-scale local visual features and global semantic features for each view; afterwards... Layers alternately perform cross-view and intra-view self-attention, dynamically reorganizing tokens of all views through tensor rearrangement to achieve efficient fusion of global geometric information. Based on the multi-scale global fusion features output by the backbone network, a dual DPT decoder head with shared features is used to simultaneously perform joint prediction of depth and rays, ensuring the intrinsic geometric consistency between the two.

[0019] Furthermore, the preprocessing of the acquired SFM point cloud image data also includes using the motion estimation software COLMAP to jointly reconstruct the sparse point cloud and camera pose. First, feature extraction and matching are performed: COLMAP's feature_extractor module detects and extracts local features for each image based on the RootSIFT or SuperPoint algorithm and stores them in an SQLite database; then, cross-view feature matching is performed using feature_matcher, supplemented by multi-model geometric verification and watermark filtering, to establish the correspondence between corresponding points in the images, resolving matching ambiguities caused by viewpoint differences and texture duplication; next, incremental sparse reconstruction is performed: COLMAP's mapper module, based on the matching results, iteratively optimizes the camera's intrinsic and extrinsic parameters through bundle adjustment, using the rotation matrix R, translation vector T, and objective function... Simultaneously, the minimum set of the triangulation algorithm is used to sample the two views, forcing triangulation angle constraints. and depth positive constraints The system calculates the 3D coordinates of corresponding feature points to generate an initial sparse point cloud. Through an optimal viewpoint selection strategy and iterative retriangulation (Post-BART), it fully represents the basic geometric structure and spatial topological relationships of the scene, and performs multi-resolution pyramid scoring. Weight Number and distribution of equilibrium points.

[0020] Furthermore, the calculation of multi-view relative depth maps based on preprocessed image data, and the generation of true-scale depth maps based on the relative depth maps, includes targeted post-processing optimization under multi-view geometric constraints after obtaining the initial depth map. First, the relative depth is converted into local 3D points using the jointly predicted ray map. For a pixel (u,v) in the i-th view, its corresponding spatial ray originates from the ray origin. and unit ray direction Definition, combined with relative depth Obtain the 3D world coordinates corresponding to this pixel:

[0021] ,

[0022] Then, the reprojection error of each 3D point in other views is calculated to correct the depth value; next, edge-aware smoothing is performed, and median filtering is used to smooth noise in planar regions, combined with depth gradient priors, while strictly preserving sharp details of object edges; finally, isolated depth noise points with an area less than 100 pixels are removed, and internal depth holes of the same size are filled, outputting a multi-view relative depth map. After obtaining the relative depth maps of all views, the relative depth is calibrated to the real-world scale using a framework of RANSAC robust outlier removal + least squares fitting, combining the sparse 3D point cloud reconstructed from COLMAP with camera intrinsic and extrinsic parameters, so that the depth value of each pixel is directly in meters. By projecting the sparse 3D points of COLMAP onto the image plane of each view, the true-scale depth value of the corresponding pixel is obtained, and then the optimal scale factor s and offset t are solved.

[0023] ,

[0024] in This represents the relative depth output by the network at pixel p. The mask represents the true-scale depth projected from sparse points in a COLMAP onto pixel p. The pixel is marked as having reliable COLMAP observations, and finally the entire relative depth map is converted into a depth map with metric units through a linear transformation: .

[0025] Furthermore, the unbiased depth map is obtained by using colmap to generate a flat Gaussian point cloud and differentiable rendering based on the generated image. This includes calculating a sparse 3D point cloud using the SfM algorithm, initializing a corresponding 3D Gaussian ellipsoid, and defining the i-th Gaussian ellipsoid as: Among them, Gauss Center By directly taking the 3D coordinates of the SfM point, the covariance matrix is ​​decomposed into the product of rotation and scaling. The initial rotation is set to the identity matrix, and the initial scaling is set to a uniform small value. Simultaneously, the opacity of each Gaussian ellipsoid is initialized. A Gaussian flattening regularization loss is introduced to force each Gaussian ellipsoid to compress along its shortest axis into a plane that fits the scene surface. Specifically, for each Gaussian ellipsoid's three scale factors s1, s2, s3, derived from the scaling matrix Si = diag(s1, s2, s3), the L1 norm of its minimum value is minimized. The direction of the shortest axis is defined as the normal direction ni of the Gaussian plane. Ambiguity in the normal direction is resolved using the viewpoint direction. After completing the Gaussian planeization constraint, an α-mixing differentiable rendering pipeline consistent with 3DGS is used to render the pixel-by-pixel normal map N and the distance map D from the plane to the camera origin under the current viewpoint. For normal map rendering, all Gaussian normals covering that pixel are mixed in depth-sorted, using the following formula:

[0026] ,

[0027] Where Rc is the rotation matrix from the world coordinate system to the camera coordinate system, αi is the opacity of the i-th Gaussian plane at that pixel, and the product term is the cumulative opacity; similarly, the distance from each Gaussian plane to the camera origin is also considered. Tc represents the coordinates of the camera center in the world coordinate system, yielding the distance map:

[0028] ,

[0029] Based on the geometric relationship between the ray and the plane, the final unbiased depth map is calculated from the rendered normal map and distance map. For any pixel on the image plane... Its homogeneous coordinates are If the camera intrinsic parameter matrix is ​​K, then the unbiased depth corresponding to the pixel is: .

[0030] Furthermore, the supervised geometric optimization of rendering quality by constructing a loss function includes a total training loss composed of a weighted average of four parts: image reconstruction loss, Gaussian flattening loss, geometric regularization loss, and depth loss. This optimizes rendering quality, Gaussian flattening constraints, and global geometric consistency. The Gaussian flattening loss... It employs a weighted sum of L1 loss and SSIM loss, and introduces an exposure coefficient per image. After brightness compensation, the image is as follows: ,in The original rendered image has the following final image reconstruction loss:

[0031] The selection rules are as follows: ,in For real images, the geometric regularization loss is composed of three weighted sub-losses: single-view geometry, multi-view photometric, and multi-view geometric consistency. The overall formula is: Single-view geometric loss Based on the local plane assumption, multi-view photometric consistency loss Cross-view luminosity consistency is measured using normalized cross-correlation (NCC) of pixel patches; multi-view geometric consistency loss. The forward and backward projection errors of pixels are calculated based on planar homography; finally, the generated depth map is used as the supervised depth D. gt The unbiased depth D typically employs a robust L1 loss to avoid interference from outliers, as shown in the formula:

[0032] ,

[0033] The total loss is the weighted sum of the individual losses: .

[0034] Furthermore, the multi-view adaptive densification and depth verification based on the optimization results includes randomly selecting K views. For each densification query, the pixel-by-pixel photometric error is first calculated, given the rendered image. With real images The per-pixel photometric error is defined as the mean of the absolute errors of each channel:

[0035] ,

[0036] Where C=3 represents the number of RGB color channels, and the error is obtained by min-max normalization to the [0,1] interval. This leads to the generation of a binary photometric error mask:

[0037] ,

[0038] Empirical threshold The default value is 0.1, which strikes a balance between photometric sensitivity and noise robustness. Simultaneously, to verify the geometric consistency of the aforementioned high-photometric-error regions, unbiased depth maps are rendered in parallel. and with external supervision depth In comparison, the pixel-wise depth absolute error is defined as:

[0039] ,

[0040] Unlike using a fixed threshold to judge depth quality, this method dynamically determines the filtering criteria based on the distribution characteristics of the current view's depth error to adapt to the differences in depth scales in different scenes. The depth filtering threshold is determined by the larger of the median plus twice the standard deviation and the 75th percentile.

[0041] ,

[0042] Where σ(·) represents the standard deviation operator, and P75(·) represents the 75th percentile.

[0043] Furthermore, the multi-view adaptive densification and depth verification based on the optimization results also includes employing an adaptive strategy to ensure that the threshold is dynamically adjusted with the scene depth complexity. For regions with uniform depth distribution, the median plus the standard deviation provides a conservative estimate; for scenes with drastic depth changes, a quantile mechanism prevents over-filtering, thereby generating a depth filtering mask. The mask marks areas corresponding to reflective, transparent materials, or floating Gaussians. This avoids introducing redundant Gaussians through densification in such areas. The fusion mask performs logical operations at the pixel level, retaining only candidate areas that simultaneously satisfy high luminance error and depth error within a reasonable range. Based on the deep-verification filtering of the generation sources of mirrored Gaussians and floating Gaussians, the geometric reliability of densification is significantly improved; finally, each Gaussian... Project onto K=10 randomly sampled training views and count their two-dimensional footprints. The number of valid pixels marked by the internal blending mask:

[0044] ,

[0045] The score si+ reflects the consistency of the Gaussian's photometric contribution under multi-view depth verification. When si+>τ+ (τ+=0.3) and the gradient condition is met, the Gaussian is marked as splittable or cloneable.

[0046] Furthermore, the depth error-guided fine-tuning based on the densification results includes, after training the scene using the flat Gaussian representation, constructing a TSDF voxel mesh using the obtained convergent planarized 3D Gaussian set. First, a globally truncated signed distance function (TSDF) voxel mesh covering the entire scene is initialized, with the voxel size adaptively set according to the scene scale, and the TSDF truncation distance... The size is uniformly set to 3 times the voxel size, and the accumulated TSDF value is initialized for each voxel. and cumulative weight ,in Using the world coordinates of the voxel center, each plane Gaussian of the PGSR is treated as a local surface, and multiple 3D points are sampled near its center along the normal direction. and Gauss's normal As the surface normal at that point, for each sampling point After locating the voxel, calculate the signed distance from the voxel center to the sampling point along the normal direction. and cut it off to Within the range, the TSDF observation value of this point for this voxel was obtained. Simultaneously calculate the weight of the observation. The weights are typically related to the Gaussian opacity of the point and the distance from the voxel center to the sampling point, used to reduce the impact of noise points and distant points on the reconstruction results. A TSDF fusion operation is performed to update the cumulative value of the corresponding voxel.

[0047] ,

[0048] After all Gaussian sampling points have been fused, the final TSDF value for each voxel is...

[0049] ,

[0050] For voxels not observed by any point, their TSDF value is set to 1. Finally, the MarchingCubes algorithm is used to extract isosurfaces from the global TSDF voxel mesh. The isosurface threshold is set to 0 to obtain the triangular mesh model of the scene. Isolated triangular patches with an area smaller than the threshold can be selectively removed and light mesh smoothing can be performed to obtain the final high-precision 3D reconstruction mesh.

[0051] Secondly, a 3D Gaussian surface reconstruction system based on depth-supervised multi-view densification includes:

[0052] The data acquisition module is configured to acquire SFM point cloud image data;

[0053] The preprocessing module is configured to perform preprocessing based on the acquired SFM point cloud image data;

[0054] The depth map module is configured to calculate a multi-view relative depth map based on the preprocessed image data and generate a real-scale depth map based on the relative depth map.

[0055] The geometry optimization module is configured to use colmap to perform flat Gaussian point cloud and differentiable rendering based on the generated image to obtain an unbiased depth map, and to perform supervised geometry optimization of the rendering quality by constructing a loss function.

[0056] The depth verification module is configured to perform multi-view adaptive densification and depth verification based on the optimization results.

[0057] The pruning module is configured to perform fine pruning guided by depth error based on the densification results, resulting in a three-dimensional reconstruction network.

[0058] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification.

[0059] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the described method for 3D Gaussian surface reconstruction based on depth-supervised multi-view densification.

[0060] In summary, the present invention has the following beneficial technical effects:

[0061] (1) Effectively eliminates geometric artifacts and improves surface reconstruction accuracy. By introducing the DepthAnything3 monocular depth estimation model and combining it with the RANSAC-least-square scale matching algorithm, a depth supervision map of the real physical scale is automatically obtained from the SfM sparse point cloud and camera pose, without relying on expensive hardware such as RGB-D cameras or LiDAR. The depth supervision loss forces the Gaussian plane to accurately fit the geometric surface of the real scene, effectively eliminating the mirrored Gaussian artifacts in reflective areas and the floating Gaussian artifacts in distant areas. On 15 standard scenes of the DTU dataset, the average chamfer distance of this method is reduced to 0.50mm, which is better than the comparison methods such as 2DGS (0.80mm), NeuS (0.84mm), VolSDF (0.86mm), and SuGaR (1.33mm), and the geometric reconstruction accuracy is significantly superior.

[0062] (2) Significantly improved training speed and reduced memory overhead. Through Depth-Verified Multi-View Adaptive Descaling (DVCD) and Depth Error Guided Pruning (DRP) strategies, the generation and propagation of redundant Gaussians were effectively suppressed. In the Mip-NeRF360 dataset ablation experiments, the full training time was 39 minutes, an improvement of approximately 41% compared to the PGSR baseline (66 minutes); the number of Gaussians was approximately 0.92M, a reduction of approximately 51% compared to the baseline (1.86M); and memory usage was simultaneously reduced. Meanwhile, the rendering quality remained comparable to the baseline (PSNR: 27.11 vs 27.18, SSIM: 0.821 vs 0.831, LPIPS: 0.211 vs 0.184), within a reasonable acceptable range.

[0063] (3) It has stronger adaptability to large-scale complex scenes. On six large and complex scenes (Barn, Caterpillar, Courthouse, Ignatius, Meetingroom, Truck) of the TanksandTemples (TNT) dataset, the average F1 score of this method reaches 0.52, which is significantly better than the comparison methods such as NeuS (0.38), Geo-NeuS (0.35), SuGaR (0.19), and 2DGS (0.30). It is particularly outstanding in large scenes such as Barn (0.68), Ignatius (0.81), and Truck (0.65), showing strong generalization ability and accurate capture of complex geometric details.

[0064] (4) The training efficiency is far superior to NeRF-based methods. Only a single GPU is needed to complete the training, while NeRF-based methods usually require several days of training time and multiple GPUs. This method has a significant competitive advantage in efficiency. Attached Figure Description

[0065] Figure 1 This is the main flowchart of the method of the present invention.

[0066] Figure 2 This is the main flowchart of the multi-view adaptive densification method for depth verification of the present invention.

[0067] Figure 3 This is a visual comparison of surface reconstruction of the present invention and the PGSR method on a major dataset.

[0068] Figure 4 These are other visual representations of images involved in the process of this invention. Detailed Implementation

[0069] The present invention will be further described in detail below with reference to the accompanying drawings.

[0070] Example 1

[0071] 3DGaussianSplatting (3DGS): A 3D reconstruction method that uses explicit 3D Gaussian primitives to represent scenes and achieves real-time rendering via GPU rasterization, offering faster training and rendering speeds compared to NeRF.

[0072] PGSR (Planar-based Gaussian Splatting Reconstruction): An improved 3DGS method that flattens Gaussian elements into a planar form and introduces unbiased depth and multi-view geometric photometric consistency constraints, which can improve the quality of surface reconstruction.

[0073] SfM (StructurefromMotion): A motion recovery structure method that simultaneously recovers camera pose and 3D sparse point cloud from multiple images. The commonly used tool is COLMAP.

[0074] RANSAC (RandomSampleConsensus): Random Sample Consensus Algorithm, a robust parameter estimation method for estimating model parameters in noisy data. This invention is used for depth scale-offset robust alignment.

[0075] DVCD (Depth-Verified Multi-View-Consistent Densification): The depth-verified multi-view adaptive densification strategy proposed in this invention combines photometric error masking and adaptive depth filtering threshold to determine the splitting / cloning of Gaussian units from the perspective of multi-view consistency.

[0076] DRP (Depth-RegularizedPruning): The depth error-guided pruning strategy proposed in this invention performs fine pruning on Gaussian units with the highest combined scores of geometric loss and photometric error after densification, thereby suppressing redundant Gaussians.

[0077] TSDF (Truncated Signed Distance Function): A voxel representation method combined with the MarchingCubes algorithm for extracting fine surface meshes from 3DGS point clouds.

[0078] VCD (View-Consistent Densification): A multi-view technology proposed in FAST-GS. Figure 1 Based on the density enhancement strategy, this invention introduces a deep verification mechanism to form a DVCD.

[0079] Reference Figure 1 This embodiment of a 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification includes:

[0080] Step S1: SFM point cloud image data generation:

[0081] The system acquires a user-provided multi-view image sequence as raw input. These images are a collection of 2D RGB images captured from different spatial locations and shooting angles around the target scene, with sufficient overlap and common feature points to meet the basic conditions for multi-view geometric reconstruction. The motion estimation software COLMAP is used to jointly reconstruct sparse point clouds and camera poses. First, feature extraction and matching are performed: COLMAP's feature_extractor module detects and extracts local features from each image based on RootSIFT or SuperPoint algorithms and stores them in an SQLite database; then, feature_matcher performs cross-view feature matching, supplemented by multi-model geometric verification (including the fundamental matrix). RANSAC estimation of homography matrix H and essential matrix E, combined with threshold and Distinguishing between general / planar / panoramic scenes) and watermark filtering (similar transformation detection, thresholding) This establishes correspondences between corresponding points in the images, resolving matching ambiguities caused by differences in viewpoint and texture repetition. Next, incremental sparse reconstruction is performed: COLMAP's mapper module, based on the matching results, iteratively optimizes the camera's intrinsic parameters (focal length, distortion coefficients) and extrinsic parameters (rotation matrix R, translation vector T) through bundle adjustment. Its core optimization objective is to minimize the reprojection error of points in 3D space onto the image plane; the objective function is:

[0082] ,

[0083] In the formula: This is a robust kernel function used to suppress out-of-point interference; For camera projection functions; For camera projection matrix; Let k be the coordinates of the k-th point in three-dimensional space; Let be the pixel coordinates of the j-th two-dimensional feature point. Simultaneously, a triangulation algorithm is used to perform minimum set sampling on corresponding points in both views, forcibly satisfying the triangulation angle constraint. and depth positive constraints An initial sparse point cloud is generated by calculating the 3D coordinates of corresponding feature points through linear triangulation. To optimize the point cloud distribution and reconstruction efficiency, an optimal viewpoint selection strategy is adopted, using a multi-resolution pyramid scoring system to filter key views. The scoring formula is as follows:

[0084]

[0085] Where: weight This is used to balance the number of points in the cloud with the uniformity of their spatial distribution. Let L be the number of non-empty voxels in the l-th layer of the pyramid, and L be the total number of pyramid layers. Combined with iterative retriangulation (Post-BART), the sparse point cloud fully represents the basic geometric structure and spatial topological relationships of the scene. Finally, redundant view compression and global bundle adjustment are used to optimize and reduce accumulated errors. The formula for determining the view co-view rate is:

[0086]

[0087] In the formula: , Let a be the set of observation points for views a and b; For set intersection / union operations; The zero norm is used to count the number of elements in the point set; V is the co-view threshold. Global bundle adjustment further unifies and optimizes the poses and 3D point coordinates of all cameras, outputting a PLY format point cloud containing point coordinates and RGB colors, as well as the precise pose parameters for each image. This process can efficiently process hundreds of images under CPU / GPU hybrid acceleration (PCG iterative method is used for large-scale problems), while improving the reconstruction robustness of low-texture areas through scene graph enhancement (initialization of non-panoramic image pairs).

[0088] The system acquires user-provided multi-view image sequences as raw input data. These sequences are collections of two-dimensional RGB images taken from different spatial locations and shooting angles around the target scene or object. The images support common lossless formats such as JPG and PNG, with a resolution of no less than 640×480. The images have sufficient overlapping areas and feature correspondences to ensure the stability and integrity of subsequent multi-view geometric reconstruction.

[0089] The motion estimation software COLMAP was used to jointly reconstruct sparse point clouds and camera pose. First, feature extraction and matching were performed: COLMAP's feature_extractor module detected and extracted local features of each image based on the RootSIFT or SuperPoint algorithm and stored them in an SQLite database; then, feature_matcher was used to perform cross-view feature matching, supplemented by multi-model geometric verification and watermark filtering, to establish the correspondence between corresponding points in the images and solve the matching ambiguity problem caused by viewpoint differences and texture repetition.

[0090] Next, incremental sparse reconstruction is performed: COLMAP's mapper module optimizes the camera's intrinsic and extrinsic parameters based on the matching results through bundle adjustment iterations, using the rotation matrix R, translation vector T, and objective function as follows.

[0091] Simultaneously, a triangulation algorithm is used (minimum set sampling of two views, forcing triangulation angle constraints). and depth positive constraints The 3D coordinates of corresponding feature points are calculated to generate an initial sparse point cloud. Although this point cloud has a low density, it achieves high density through an optimal viewpoint selection strategy (multi-resolution pyramid scoring). Weight The number and distribution of equilibrium points and Post-BART iterative retriangulation have fully characterized the basic geometric structure and spatial topological relationships of the scene.

[0092] Ultimately, this is achieved through redundant view compression (camera group common view rate). The process involves global bundle adjustment and optimization to reduce accumulated errors, outputting a PLY format point cloud containing point coordinates and RGB colors, as well as precise pose parameters for each image. This workflow can efficiently process hundreds of images under CPU / GPU hybrid acceleration (PCG iterative method is used for large-scale problems), but robustness to reconstruction of low-texture regions still faces challenges, requiring scene graph enhancement (initialization of non-panoramic image pairs) to mitigate these issues.

[0093] Step S2: Point cloud image data preprocessing

[0094] The input is processed using a pre-trained DepthAnything3 (DA3) single Transformer architecture in an arbitrary view-relative depth mode. Each image undergoes normalization preprocessing: it is adaptively scaled to a compatible resolution based on 504×504, strictly preserving the original aspect ratio while supporting various common scales, and pixel value normalization is performed. Subsequently, the same shared learnable camera token c is used for all views. l As geometric placeholders, the camera token of each view is finally concatenated with the patch token obtained from image segmentation and fed into the Transformer backbone network.

[0095] Specifically,

[0096] For user-provided multi-view image sequences, the goal of this stage is to generate high-quality, detailed multi-view relative depth maps. Depth values ​​are scale- and translation-invariant relative geometric quantities, which are then calibrated to metric depth maps with real physical scale using COLMAP geometric information. A pre-trained DepthAnything3 (DA3) single Transformer architecture is used to process the input in a multi-view relative depth mode. Each image undergoes normalization preprocessing: it is adaptively scaled to a compatible resolution based on 504×504, strictly maintaining the original aspect ratio, supporting various common ratios such as 504×504, 504×378, and 896×504, while also performing pixel value normalization. Since there is no known camera pose for a single view, the same shared learnable camera token is used for all views. As geometric placeholders, the camera token for each view is concatenated with the patch token obtained from image segmentation, and then fed into the Transformer backbone network. DA3 uses the pre-trained vanillaDINOv2ViT as the sole backbone, without introducing any task-specific architectural modifications. It achieves information interaction for any number of views through an input adaptive cross-view self-attention mechanism. The model divides all Transformer layers into two groups in a 2:1 ratio, with a total of [number of layers missing]. ,

[0097] forward The layer only performs in-view self-attention, extracting multi-scale local visual features and global semantic features for each view; afterwards... Layers alternately perform cross-view and intra-view self-attention, dynamically reorganizing tokens across all views through tensor rearrangement to achieve efficient fusion of global geometric information. This design has inherent input adaptability, adapting to changes in the number of input views. The model automatically degenerates into a pure view-intra-view self-attention mode, without incurring any additional cross-view computational overhead. Based on the multi-scale global fusion features output by the backbone network, this embodiment uses a dual DPT decoder head with shared features to simultaneously predict depth and rays. This design fundamentally guarantees the inherent geometric consistency between the two. The decoder first fuses and reassembles the hierarchical features through a shared multi-scale reassembly module, and then, through two independent fusion layers and pixel-level output layers, generates a pixel-aligned single-channel relative depth map and a six-channel ray map for each view.

[0098] Step 3: Depth Map Generation

[0099] After obtaining the initial depth maps of all views, targeted post-processing optimizations are performed under multi-view geometric constraints to improve the final quality: First, the relative depth is converted into local 3D points using the jointly predicted ray map. For a pixel (u,v) in the i-th view, its corresponding spatial ray originates from the ray origin. and unit ray direction Definition, combined with relative depth The 3D world coordinates corresponding to this pixel can be obtained:

[0100] ,

[0101] The reprojection error of each 3D point in other views is calculated to correct depth values ​​with significant geometric inconsistencies. Next, edge-aware smoothing is performed, combining depth gradient priors with median filtering to smooth noise in planar regions while strictly preserving sharp details of object edges. Finally, isolated depth noise points with an area less than 100 pixels are removed, and internal depth holes of the same size are filled, outputting a multi-view relative depth map. After obtaining the relative depth maps for all views, the relative depth is calibrated to the real-world scale using a framework of RANSAC robust outlier culling and least-squares fitting, combining the sparse 3D point cloud reconstructed from COLMAP with camera intrinsic and extrinsic parameters, so that the depth value of each pixel is directly in meters. First, the sparse 3D points from COLMAP are projected onto the image plane of each view to obtain the true-scale depth value of the corresponding pixel. Then, the optimal scale factor s and offset t are solved.

[0102] ,

[0103] in This represents the relative depth output by the network at pixel p. The mask represents the true-scale depth projected from sparse points in a COLMAP onto pixel p. The pixel is marked as having a reliable COLMAP observation. Finally, the entire relative depth map is converted into a depth map with metric units through a linear transformation:

[0104] ,

[0105] DA3 uses the pre-trained vanillaDINOv2ViT as its sole backbone, without introducing any task-specific architectural modifications. It achieves information interaction across any number of views through an input-adaptive cross-view self-attention mechanism. The model divides all Transformer layers into two groups in a 2:1 ratio, with a total number of layers L = L s +L g : front L s The layer only performs in-view self-attention, extracting multi-scale local visual features and global semantic features for each view; after L... g Layers alternately perform cross-view and intra-view self-attention, dynamically reorganizing tokens across all views through tensor rearrangement to achieve efficient fusion of global geometric information. This design has inherent input adaptability, when N v When =1, the model will automatically degenerate into a pure in-view self-attention mode, without generating any additional cross-view computational overhead.

[0106] Based on the multi-scale global fusion features output by the backbone network, this embodiment uses a dual DPT decoder head with shared features to simultaneously predict depth and rays. This design fundamentally ensures the inherent geometric consistency between the two. The decoder first fuses and reassembles the hierarchical features through a shared multi-scale reassembly module, and then, through two independent fusion layers and a pixel-level output layer, generates a pixel-aligned single-channel relative depth map for each view.

[0107] After obtaining the initial depth maps of all views, targeted post-processing optimizations are performed under multi-view geometric constraints to improve the final quality: First, based on the above 3D point projection formula, the reprojection error of each 3D point in other views is calculated to correct depth values ​​with obvious geometric inconsistencies; then, edge-aware smoothing is performed, and median filtering is used to smooth noise in planar regions in combination with depth gradient priors, while strictly preserving the sharp details of object edges; finally, isolated depth noise points with an area of ​​less than 100 pixels are removed, and internal depth holes of the same size are filled to output a multi-view relative depth map.

[0108] After obtaining depth maps from all viewpoints, and combining them with the sparse point cloud and camera parameters pre-calculated by colmap, the relative depth is calibrated to a real-world scale using the same RANSAC-least-squares framework, so that the depth value of each pixel is directly expressed in meters.

[0109] ,

[0110] Given the relative depth, Dp is the true scale depth projected from the COLMAP sparse points to the same pixel. The mask mp∈{0,1} indicates whether the pixel has reliable COLMAP observations. s and t are the scale and offset obtained through RANSAC iteration, and finally calculated as follows:

[0111] Convert the entire relative depth map into a depth map with metric units.

[0112] Step S4 initializes the SFM point cloud as a flat Gaussian point cloud and renders unbiased depth using a differentiable renderer:

[0113] First, the sparse 3D point cloud is calculated using the SfM algorithm, and a corresponding 3D Gaussian ellipsoid is initialized. The i-th Gaussian ellipsoid is defined as:

[0114] ,

[0115] Among them, Gauss Center By directly taking the 3D coordinates of the SfM point, the covariance matrix is ​​decomposed into the product of rotation and scaling. The initial rotation is set to the identity matrix, the initial scaling is set to a uniform small value, and the opacity of each Gaussian ellipsoid is initialized to 0.1. The color is taken as the average RGB value of the corresponding SfM point in each view. During training, a Gaussian flattening regularization loss is introduced, forcing each Gaussian ellipsoid to be compressed along its shortest axis into a plane that fits the scene surface. Specifically, for each Gaussian ellipsoid, the L1 norm of its three scale factors s1, s2, s3 (from the scaling matrix Si = diag(s1, s2, s3)) is minimized.

[0116] ,

[0117] The direction of the shortest axis is defined as the normal direction *ni* of the Gaussian plane. Ambiguity in the normal direction is resolved using the viewpoint direction, ensuring the angle between the normal and the camera's line of sight is greater than 90 degrees, so that the normal always points towards the camera. After completing the Gaussian planeization constraint, an α-mixing differentiable rendering pipeline, consistent with 3DGS, is used to render the pixel-by-pixel normal map *N* and the distance map *D* from the plane to the camera origin under the current viewpoint. For normal map rendering, all Gaussian normals covering each pixel are mixed in depth-ordered, using the following formula:

[0118] ,

[0119] Where Rc is the rotation matrix from the world coordinate system to the camera coordinate system, αi is the opacity of the i-th Gaussian plane at that pixel, and the product term is the cumulative opacity; similarly, the distance from each Gaussian plane to the camera origin is also considered. (Tc is the coordinate of the camera center in the world coordinate system), thus obtaining the distance map.

[0120] ,

[0121] Based on the geometric relationship between the ray and the plane, the final unbiased depth map is calculated from the normal map and distance map obtained from the above rendering. For any pixel on the image plane... Its homogeneous coordinates are If the camera intrinsic parameter matrix is ​​K, then the unbiased depth corresponding to this pixel is...

[0122] ,

[0123] This method completely eliminates the depth bias problem caused by the accumulation of α blending weights in traditional 3DGS depth rendering by performing a division operation between distance and normal, and ensures that the depth value strictly falls on the Gaussian plane, consistent with the geometry of the real scene surface, and finally obtains an unbiased depth map.

[0124] Step S5: Supervised geometry optimization based on external depth priors

[0125] A loss function is constructed, with the total training loss consisting of a weighted average of four parts: image reconstruction loss, Gaussian flattening loss, geometric regularization loss, and depth loss. This optimizes rendering quality, Gaussian flattening constraints, and global geometric consistency. The Gaussian flattening loss is a key feature of S3. Image reconstruction loss with exposure compensation is used to handle lighting differences from different viewpoints and improve rendering quality. It employs a weighted sum of L1 loss and SSIM loss, and introduces a per-image exposure coefficient. After brightness compensation, the image is as follows: ,in This is the original rendered image, and the final image reconstruction loss is...

[0126] ,

[0127] here The selection rules are

[0128] ,

[0129] in For real images, weights This design ensures structural similarity while also adapting to changes in lighting.

[0130] The geometric regularization loss is composed of three weighted sub-losses: single-view geometry, multi-view photometric, and multi-view geometric consistency. The overall formula is as follows:

[0131] ,

[0132] Among them, single-view geometric loss Based on the local plane assumption and the consistency between constraint depth and normal, an image edge-aware weight is introduced to reduce the constraint intensity in edge regions. Multi-view photometric consistency loss Cross-view luminosity consistency was measured using normalized cross-correlation (NCC) of 11×11 pixel patches. The average value is used as the loss value, and the weights are... Multi-view geometric consistency loss The forward and backward projection errors of pixels are calculated based on planar homography. After filtering out high-error pixels caused by occlusion, the average is taken, with weights... These three sub-losses work together to ensure geometric consistency within a single view and between multiple views.

[0133] The depth map generated by S2 serves as the supervision depth D. gt The unbiased depth D generated by S3 typically employs a robust L1 loss to avoid interference from outliers, as shown in the formula:

[0134] ,

[0135] The total loss is the weighted sum of the individual losses:

[0136] .

[0137] Step S6: Multi-view Adaptive Densification for Depth Verification

[0138] After a certain number of training optimizations, periodic densification operations are needed to increase the number of Gaussian elements representing the scene in order to fit the scene more accurately and meticulously. This embodiment differs from conventional 3DGS methods where densification is based on whether the accumulated gradients reach a densification threshold by adding a multi-view adaptive densification threshold and depth verification.

[0139] Specifically, K views are randomly selected, and the points that need to be densified are statistically analyzed within these views. Gaussian splitting / cloning is only performed in regions where high reconstruction errors occur across multiple views, thus avoiding the introduction of redundant Gaussian that is only effective for a few views. Building upon this, depth cues are further introduced to weaken spurious RGB error signals generated in weak textures or highlight areas.

[0140] Specifically, for each densification query, the pixel-by-pixel photometric error is first calculated. Given a rendered image. With real images The per-pixel photometric error is defined as the mean of the absolute errors of each channel:

[0141] ,

[0142] Where C=3 represents the number of RGB color channels. This error is then normalized to the [0,1] interval using the min-max method. This leads to the generation of a binary photometric error mask:

[0143] ,

[0144] Empirical threshold The default value is 0.1, which strikes a balance between photometric sensitivity and noise robustness. Meanwhile, to verify the geometric consistency of the aforementioned high-photometric-error regions, the system renders unbiased depth maps in parallel. and with external supervision depth Comparison. The absolute depth error per pixel is defined as:

[0145] ,

[0146] Unlike methods that use a fixed threshold to determine depth quality, this embodiment dynamically determines the filtering criteria based on the distribution characteristics of the current view's depth error to adapt to differences in depth scale across different scenes. The depth filtering threshold is determined by the larger of the median plus twice the standard deviation and the 75th percentile.

[0147] ,

[0148] Where σ(·) represents the standard deviation operator, and P75(·) represents the 75th percentile. This adaptive strategy ensures that the threshold is dynamically adjusted according to the scene's depth complexity: for regions with uniform depth distribution, the median plus the standard deviation provides a conservative estimate; for scenes with drastic depth changes, the quantile mechanism prevents over-filtering. A depth filtering mask is generated accordingly.

[0149] ,

[0150] The mask marks regions corresponding to reflective, transparent materials, or floating Gaussians, avoiding the introduction of redundant Gaussians through densification in such areas. The fusion mask performs logical operations at the pixel level, retaining only candidate regions that simultaneously satisfy high luminance error and a reasonable depth error range.

[0151] ,

[0152] The core idea of ​​this fusion mechanism is that regions with high photometric errors but also abnormal depths are likely to have geometrical deviations, and increasing the Gaussian density in such cases will only amplify the errors. Conversely, regions with high photometric errors but consistent depths indicate that insufficient detail is the primary issue, and densification can effectively improve reconstruction quality. Through this depth verification layer, the system effectively filters out the sources of mirrored and floating Gaussians, significantly improving the geometric reliability of densification.

[0153] The subsequent steps are strictly consistent with the VCD: [The text abruptly ends here, likely due to an incomplete sentence or a Project onto K=10 randomly sampled training views and count their two-dimensional footprints. The number of valid pixels marked by the internal blending mask:

[0154] ,

[0155] The score si+ reflects the consistency of the Gaussian's photometric contribution under multi-view depth verification. When si+>τ+ (τ+=0.3) and the gradient condition is satisfied, the Gaussian is marked as splittable or cloneable. It is worth noting that the depth branch is only introduced during the densification stage and does not participate in the gradient backpropagation of the rendered image. Therefore, the additional GPU overhead is minimal, with each iteration taking less than 2ms in actual tests, which is negligible relative to the overall training time.

[0156] Step S7: Depth Error Guided Fine Pruning

[0157] In step 6, only Gaussians with depth errors were prevented from participating in densification; they were not removed to eliminate their subsequent effects. Therefore, a fine-tuning of the depth-verified multi-view adaptive densification is performed immediately after the depth-verified multi-view densification to remove Gaussians with depth errors. Render the depth map. With normal diagram Joint Truth Depth And the depth-derived normal constraints, calculate the comprehensive geometric loss view by view:

[0158] ,

[0159] Among them depth loss For deep loss, For normal loss, weight , The geometric loss and photometric error are related to the number of high-error pixels covered by a Gaussian overlay. Coupling, forming Gaussian-wise pruning fractions:

[0160] ,

[0161] in For the photometric loss of the j-th view, This represents the number of pixels within the Gaussian footprint whose photometric error exceeds a threshold. A higher score indicates a higher Gaussian accuracy. The greater the damage to the overall reconstruction quality, the better. After normalizing to [0,1], a threshold is set. Remove the top 5% Gaussian scores. This operation is performed only once after densification, without introducing additional rendering overhead. It complements the depth-verified multi-view adaptive densification mechanism, forming a "photometric-induced densification, geometric verification pruning" mechanism.

[0162] Step S8: Surface reconstruction model generation:

[0163] The training process involves transforming the initial sparse points into a Gaussian point cloud, then using operations such as densification and pruning to control the number of points in the cloud. RGB supervision is used to adjust the color of the Gaussian point cloud, and methods such as depth maps are used to supervise the position, shape, and other parameters of the Gaussian points. The current scene is still composed of Gaussian point clouds distributed in space.

[0164] After training the scene using flat Gaussians, this embodiment obtains a converged set of planarized 3D Gaussians. This embodiment can directly sample dense 3D point clouds and their corresponding normal information from these Gaussians to construct a TSDF voxel mesh. First, a globally truncated signed distance function (TSDF) voxel mesh covering the entire scene is initialized. The voxel size is adaptively set according to the scene scale, and the TSDF truncation distance... The size is uniformly set to 3 times the voxel size, and the accumulated TSDF value is initialized for each voxel. and cumulative weight ,in Let the world coordinates be the voxel centers. Next, each plane Gaussian of the PGSR is treated as a local surface, and multiple 3D points are sampled near its center along the normal direction. and Gauss's normal This serves as the surface normal at that point. For each sampling point... After locating the voxel, calculate the signed distance from the voxel center to the sampling point along the normal direction. and cut it off to Within the range, the TSDF observation value of this point for this voxel was obtained. Simultaneously calculate the weight of this observation. The weights are typically related to the Gaussian opacity of the point and the distance from the voxel center to the sampling point, used to reduce the impact of noisy points and distant points on the reconstruction results. Perform a TSDF fusion operation to update the cumulative value of the corresponding voxel:

[0165] , ,

[0166] After all Gaussian sampling points have been fused, the final TSDF value for each voxel is...

[0167] ,

[0168] For voxels not observed by any point, their TSDF value is set to 1. Finally, the MarchingCubes algorithm is used to extract isosurfaces from the global TSDF voxel mesh, with the isosurface threshold set to 0, to obtain the triangular mesh model of the scene. Subsequently, isolated triangular patches with an area smaller than the threshold can be selectively removed, and light mesh smoothing can be performed to obtain the final high-precision 3D reconstructed mesh.

[0169] Experimental verification

[0170] To verify the effectiveness of the method in this embodiment, this embodiment uses the same experimental dataset as PGSR, including various real-world datasets, including objects, and indoor and outdoor environments. For example, the widely used MiP-NeRF360 dataset is used to evaluate the synthetic performance of new views; six large and complex scenes from TNT and 15 commonly used scenes from the DTU dataset are used to evaluate the reconstruction quality.

[0171] Evaluation Criteria: This embodiment selects three widely used image evaluation metrics to validate the novel viewpoint synthesis: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Patch Similarity (LPIPS). For evaluating surface quality, this embodiment uses F1 score and chamfer distance.

[0172] Implementation Details: This embodiment builds upon the PGSR framework, setting up its experimental scheme with largely consistent training strategies and hyperparameters. Training for all scenes uses the adam optimizer with 30,000 iterations. Depth supervision in this embodiment is synchronized with PGSR's geometric supervision during the first 7,000 training iterations. This embodiment uses the TSDF fusion algorithm to generate the corresponding TSDF fields. Subsequently, this embodiment extracts the mesh from the TSDF domain. Following PGSR's settings, this embodiment only uses exposure compensation on the TANKS and TIMPLES datasets. All experiments were conducted on an NVIDIA RTX 4090 (48G) graphics processor.

[0173] This invention conducts experiments on the Mip-NeRF360 dataset to evaluate reconstruction quality, using PGSR, SSIM, and LPIPS as metrics for comparison with mainstream methods. The results are shown in Table 1.

[0174] Table 1: Comparison of 3D reconstruction and rendering quality with mainstream methods on the Mip-NeRF360 dataset.

[0175] This invention was further experimentally verified in terms of experimental efficiency with the advanced method PGSR, and the results are shown in Table 2:

[0176] Table 2 Ablation Experiment Results of Training Efficiency and Rendering Quality for Mip-NeRF360 Dataset

[0177]

[0178] To verify the accuracy of surface reconstruction, the TSDF method was used to extract surface meshes from the 3DGS point cloud, and the results were validated on the DTU and TNT datasets. State-of-the-art surface reconstruction methods and similar methods, including VolSDF, NeuS, SuGaR, and 2DGS, were compared. Experiments demonstrate that the method presented in this embodiment achieves the best results in both quantitative and qualitative comparisons.

[0179] DTU Dataset: The method in this embodiment significantly outperforms other 3DGS-based reconstruction methods. As shown in Table 3, the method in this embodiment has the lowest average chamfer distance on the DTU dataset, indicating superior geometric reconstruction accuracy. Figure 3 As shown, the method in this embodiment provides more accurate surface reconstruction in the DTU dataset scenario.

[0180] Table 3: Performance comparison with mainstream methods on the DTU dataset. (Note: Chamfer distance ↓)

[0181]

[0182] The TNT dataset is complex and large-scale, demanding higher robustness in geometric reconstruction. Table 4 shows that the method in this embodiment achieves an average F1 score higher than other surface reconstruction methods. For example... Figure 3 Visual results show that the surface reconstructed by the method in this embodiment is more accurate, and the integrity of weak texture areas (such as walls) and detailed areas (such as object edges) is significantly improved. It eliminates the surface protrusions, collapses and voids that are common in traditional methods, and verifies the effect of depth supervision on improving geometric accuracy.

[0183] Table 4 shows the performance comparison with mainstream methods on the TankandTemple dataset. (Note: F1 score increases)

[0184]

[0185] Example 2

[0186] This embodiment provides a 3D Gaussian surface reconstruction system based on depth-supervised multi-view densification, including:

[0187] The data acquisition module is configured to acquire SFM point cloud image data;

[0188] The preprocessing module is configured to perform preprocessing based on the acquired SFM point cloud image data;

[0189] The depth map module is configured to calculate a multi-view relative depth map based on the preprocessed image data and generate a real-scale depth map based on the relative depth map.

[0190] The geometry optimization module is configured to use colmap to perform flat Gaussian point cloud and differentiable rendering based on the generated image to obtain an unbiased depth map, and to perform supervised geometry optimization of the rendering quality by constructing a loss function.

[0191] The depth verification module is configured to perform multi-view adaptive densification and depth verification based on the optimization results.

[0192] The pruning module is configured to perform fine pruning guided by depth error based on the densification results, resulting in a three-dimensional reconstruction network.

[0193] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the method for reconstructing a 3D Gaussian surface based on depth-supervised multi-view densification.

[0194] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the described method for 3D Gaussian surface reconstruction based on depth-supervised multi-view densification.

[0195] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification, characterized in that, include: Acquire SFM point cloud image data; Preprocessing is performed based on the acquired SFM point cloud image data; Calculate a multi-view relative depth map based on the preprocessed image data, and generate a real-scale depth map based on the relative depth map; Based on the generated image, flat Gaussian point cloud and differentiable rendering are performed using colmap to obtain an unbiased depth map, and supervised geometric optimization of rendering quality is performed by constructing a loss function. Multi-view adaptive densification and depth verification are performed based on the optimization results; Fine pruning guided by depth error based on densification results; A three-dimensional reconstruction network was obtained; The generated image is used to perform flattened Gaussian point cloud and differentiable rendering to obtain an unbiased depth map, including calculating a sparse 3D point cloud using the SfM algorithm, and initializing a corresponding 3D Gaussian ellipsoid, defining the i-th Gaussian as: Among them, Gauss Center By directly taking the 3D coordinates of the SfM point, the covariance matrix is ​​decomposed into the product of rotation and scaling. The initial rotation is set to the identity matrix, and the initial scaling is set to a uniform small value. Simultaneously, the opacity of each Gaussian ellipsoid is initialized. A Gaussian flattening regularization loss is introduced to force each Gaussian ellipsoid to compress along its shortest axis into a plane that fits the scene surface. Specifically, for each Gaussian ellipsoid's three scale factors s1, s2, s3, derived from the scaling matrix Si = diag(s1, s2, s3), the L1 norm of its minimum value is minimized. The direction of the shortest axis is defined as the normal direction ni of the Gaussian plane. Ambiguity in the normal direction is resolved using the viewpoint direction. After completing the Gaussian planeization constraint, an α-mixing differentiable rendering pipeline consistent with 3DGS is used to render the pixel-by-pixel normal map N and the distance map D from the plane to the camera origin under the current viewpoint. For normal map rendering, all Gaussian normals covering that pixel are mixed in depth-sorted, using the following formula: ， Where Rc is the rotation matrix from the world coordinate system to the camera coordinate system, αi is the opacity of the i-th Gaussian plane at that pixel, and the product term is the cumulative opacity; similarly, the distance from each Gaussian plane to the camera origin is also considered. Tc represents the coordinates of the camera center in the world coordinate system, yielding the distance map: ， Based on the geometric relationship between the ray and the plane, the final unbiased depth map is calculated from the rendered normal map and distance map. For any pixel on the image plane... Its homogeneous coordinates are If the camera intrinsic parameter matrix is ​​K, then the unbiased depth corresponding to the pixel is: ,in, This represents the unbiased depth at pixel p. This is a distance graph; K refers to the camera's intrinsic parameters. is the second coordinate of p, and N is the pixel-weighted average plane normal vector.

2. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 1, characterized in that, The preprocessing of the acquired SFM point cloud image data includes using a pre-trained DA3 single Transformer architecture to process the input in an arbitrary view relative depth mode, performing normalization preprocessing on each image, and performing pixel value normalization by adaptively scaling to a 504×504 compatible resolution. Subsequently, for single views without known camera poses, the same shared learnable camera token is used for all views. As geometric placeholders, the camera token for each view is concatenated with the patch token obtained from image segmentation, and then fed into the Transformer backbone network. The model divides all Transformer layers into two groups in a 2:1 ratio, for a total of: ,forward The layer only performs in-view self-attention, extracting multi-scale local visual features and global semantic features for each view; afterwards... Layers alternately perform cross-view and intra-view self-attention, dynamically reorganizing tokens of all views through tensor rearrangement to achieve efficient fusion of global geometric information. Based on the multi-scale global fusion features output by the backbone network, a dual DPT decoder head with shared features is used to simultaneously perform joint prediction of depth and rays, ensuring the intrinsic geometric consistency between the two.

3. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 2, characterized in that, The preprocessing of the acquired SFM point cloud image data also includes using the motion estimation software COLMAP to jointly reconstruct the sparse point cloud and camera pose. First, feature extraction and matching are performed: COLMAP's feature_extractor module detects and extracts local features for each image based on the RootSIFT or SuperPoint algorithm and stores them in an SQLite database; then, cross-view feature matching is performed using feature_matcher, supplemented by multi-model geometric verification and watermark filtering, to establish the correspondence between corresponding points in the images, resolving matching ambiguities caused by viewpoint differences and texture duplication; next, incremental sparse reconstruction is performed: COLMAP's mapper module, based on the matching results, iteratively optimizes the camera's intrinsic and extrinsic parameters using bundle adjustment, with rotation matrix R, translation vector T, and objective function... Simultaneously, the minimum set of the triangulation algorithm is used to sample the two views, forcing triangulation angle constraints. and depth positive constraints The system calculates the 3D coordinates of corresponding feature points to generate an initial sparse point cloud. Through an optimal viewpoint selection strategy and iterative retriangulation (Post-BART), it fully represents the basic geometric structure and spatial topological relationships of the scene, and performs multi-resolution pyramid scoring. Weight Number and distribution of equilibrium points.

4. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 3, characterized in that, The process involves calculating a multi-view relative depth map based on preprocessed image data and generating a true-scale depth map based on the relative depth map. This includes targeted post-processing optimization under multi-view geometric constraints after obtaining the initial depth map. First, the relative depth is converted into local 3D points using a jointly predicted ray map. For a pixel (u,v) in the i-th view, its corresponding spatial ray originates from the ray origin. and unit ray direction Definition, combined with relative depth Obtain the 3D world coordinates corresponding to this pixel: ， Then, the reprojection error of each 3D point in other views is calculated to correct the depth value; next, edge-aware smoothing is performed, and median filtering is used to smooth noise in planar regions, combined with depth gradient priors, while strictly preserving sharp details of object edges; finally, isolated depth noise points with an area less than 100 pixels are removed, and internal depth holes of the same size are filled, outputting a multi-view relative depth map. After obtaining the relative depth maps of all views, the relative depth is calibrated to the real-world scale using a framework of RANSAC robust outlier removal + least squares fitting, combining the sparse 3D point cloud reconstructed from COLMAP with camera intrinsic and extrinsic parameters, so that the depth value of each pixel is directly in meters. By projecting the sparse 3D points of COLMAP onto the image plane of each view, the true-scale depth value of the corresponding pixel is obtained, and then the optimal scale factor s and offset t are solved. ， in This represents the relative depth output by the network at pixel p. The mask represents the true-scale depth projected from sparse points in a COLMAP onto pixel p. The pixel is marked as having reliable COLMAP observations, and finally the entire relative depth map is converted into a depth map with metric units through a linear transformation: ,in, This represents the final metric unit depth map, where s is the scale factor and t is the offset.

5. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 4, characterized in that, The method involves supervised geometric optimization of rendering quality by constructing a loss function. This includes a total training loss composed of a weighted average of four parts: image reconstruction loss, Gaussian flattening loss, geometric regularization loss, and depth loss. This optimizes rendering quality, Gaussian flattening constraints, and global geometric consistency. The Gaussian flattening loss... It employs a weighted sum of L1 loss and SSIM loss, and introduces an exposure coefficient per image. After brightness compensation, the image is as follows: ,in The original rendered image has the following final image reconstruction loss: ,in The selection rules are as follows: ,in For real images, the geometric regularization loss is composed of three weighted sub-losses: single-view geometry, multi-view photometric, and multi-view geometric consistency. The overall formula is: Single-view geometric loss Based on the local plane assumption, multi-view photometric consistency loss Cross-view luminosity consistency is measured using normalized cross-correlation (NCC) of pixel patches; multi-view geometric consistency loss. The forward and backward projection errors of pixels are calculated based on planar homography; finally, the generated depth map is used as the supervised depth D. gt The unbiased depth D typically employs a robust L1 loss to avoid interference from outliers, as shown in the formula: ， The total loss is the weighted sum of the individual losses: in, These represent color, Gaussian flattening, geometry, and depth losses, respectively. , This represents the weights of the three losses other than color.

6. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 5, characterized in that, The multi-view adaptive densification and depth verification based on the optimization results includes randomly selecting K views. For each densification query, the pixel-by-pixel photometric error is first calculated, given a rendered image. With real images The per-pixel photometric error is defined as the mean of the absolute errors of each channel: ， Where C=3 represents the number of RGB color channels, j represents the j-th view, (u,v) represents the pixel coordinates, and I(·) is the indicator function. The error is normalized to the [0,1] interval by min-max to obtain the normalized mean error. This leads to the generation of a binary photometric error mask: ， in, Indicates the j-th view Error mask at the pixel level. The indicator function (·) outputs 1 if the condition within the parentheses is met, and 0 if it is not met. An empirical threshold is also included. The default value is 0.1, which strikes a balance between photometric sensitivity and noise robustness. Simultaneously, to verify the geometric consistency of the aforementioned high-photometric-error regions, unbiased depth maps are rendered in parallel. and with external supervision depth In comparison, the pixel-wise depth absolute error is defined as: ， in, Indicates in view j The depth error at coordinates differs from using a fixed threshold to judge depth quality. Instead, a dynamic filtering standard is determined based on the distribution characteristics of the current view's depth error to adapt to differences in depth scale across different scenes. The depth filtering threshold is determined by the larger of the median plus twice the standard deviation and the 75th quantile. ， in σ(·) represents the median operator for the depth error of all pixels in the j-th view, σ(·) represents the standard deviation operator, and P75(·) represents the 75th percentile.

7. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 6, characterized in that, The multi-view adaptive densification and depth verification based on the optimization results also include using an adaptive strategy to ensure that the threshold is dynamically adjusted with the scene depth complexity. For regions with uniform depth distribution, the median plus the standard deviation provides a conservative estimate; for scenes with drastic depth changes, a quantile mechanism prevents over-filtering, thereby generating a depth filter mask. The mask marks areas corresponding to reflective, transparent materials, or floating Gaussians. This avoids introducing redundant Gaussians through densification in such areas. The fusion mask performs logical operations at the pixel level, retaining only candidate areas that simultaneously satisfy high luminance error and depth error within a reasonable range. in Indicates color error mask This represents a deep error mask, which filters the generation sources of mirrored Gaussians and floating Gaussians based on depth verification, significantly improving the geometric reliability of densification; finally, each Gaussian... Project onto K=10 randomly sampled training views and count their two-dimensional footprints. The number of valid pixels marked by the internal blending mask: ， in, represents the two-dimensional coverage area of ​​the i-th Gaussian projected onto the j-th view. The score si+ reflects the consistency of the Gaussian's photometric contribution under multi-view depth verification. K is the number of randomly selected views, which defaults to 10. j is the sampling map index. When si+>τ+ (τ+=0.3) and the gradient condition is met, the Gaussian is marked as splittable or cloneable.

8. The 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification according to claim 7, characterized in that, The depth error-guided fine-tuning based on the densification results includes: after training the scene using the flat Gaussian representation, constructing a TSDF voxel mesh using the obtained convergent planarized 3D Gaussian set; first, initializing a global truncation signed distance function (TSDF) voxel mesh covering the entire scene, with the voxel size adaptively set according to the scene scale, and the TSDF truncation distance... The size is uniformly set to 3 times the voxel size, and the accumulated TSDF value is initialized for each voxel. and cumulative weight ,in Using the world coordinates of the voxel center, each plane Gaussian of the PGSR is treated as a local surface, and multiple 3D points are sampled near its center along the normal direction. and Gauss's normal As the surface normal at that point, for each sampling point After locating the voxel, calculate the signed distance from the voxel center to the sampling point along the normal direction. and cut it off to Within the range, the TSDF observation value of this point for this voxel was obtained. Simultaneously calculate the weight of the observation. The weights are typically related to the Gaussian opacity of the point and the distance from the voxel center to the sampling point, used to reduce the impact of noise points and distant points on the reconstruction results. A TSDF fusion operation is performed to update the cumulative value of the corresponding voxel. ， ， After all Gaussian sampling points have been fused, the final TSDF value for each voxel is... ， in, This represents the cumulative TSDF value of voxel x. This represents the cumulative observation weight of voxel x. For voxels that are not observed by any point, their TSDF value is set to 1. Finally, the MarchingCubes algorithm is used to extract isosurfaces from the global TSDF voxel mesh. The isosurface threshold is set to 0 to obtain the triangular mesh model of the scene. Isolated triangular patches with an area smaller than the threshold can be selectively removed, and light mesh smoothing is performed to obtain the final high-precision 3D reconstruction mesh.

9. A 3D Gaussian surface reconstruction system based on depth-supervised multi-view densification, performing the 3D Gaussian surface reconstruction method based on depth-supervised multi-view densification as described in claim 1, characterized in that, include: The data acquisition module is configured to acquire SFM point cloud image data; The preprocessing module is configured to perform preprocessing based on the acquired SFM point cloud image data; The depth map module is configured to calculate a multi-view relative depth map based on the preprocessed image data and generate a real-scale depth map based on the relative depth map. The geometry optimization module is configured to use colmap to perform flat Gaussian point cloud and differentiable rendering based on the generated image to obtain an unbiased depth map, and to perform supervised geometry optimization of the rendering quality by constructing a loss function. The depth verification module is configured to perform multi-view adaptive densification and depth verification based on the optimization results. The pruning module is configured to perform fine pruning guided by depth error based on densification results; A three-dimensional reconstruction network was obtained.

Images