Enhanced 3D surface reconstruction method based on 3D Gaussian Splitting

Abstract

The invention discloses an enhanced 3D (three-dimensional) surface reconstruction method based on 3D (three-dimensional) Gaussian Splitting. Global consistent depth priori is obtained through virtual stereo pair rendering; constructing a factor graph and introducing a cross-view geometry / luminosity consistency constraint to form local beam adjustment loss; the prior is used as a learnable parameter to be combined with 3DGS to be optimized, and meanwhile, the Pull loss is assisted to pull low-credibility pixels; and finally, multi-loss function end-to-end training is adopted. The method comprises the following steps of: in Tanksamp; the method has the advantages that F1 is equal to 0.58 and DTU Chamfer is equal to 0.48 mm on a Temples data set, the training time is only 20 min, compared with the prior art, geometric accuracy SOTA and speed magnitude improvement are achieved at the same time, and the method is suitable for VR / AR, robot and industrial measurement scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and three-dimensional reconstruction, and particularly relates to a 3D Gaussian Splatting (3DGS) enhanced surface reconstruction method, system and storage medium fusing stereo vision prior, local beam adjustment and multi-loss joint optimization. BACKGROUND

[0002] 3D dense surface reconstruction is a core task of computer vision and graphics. In recent years, the mainstream technical path is divided into two categories: neural radiation field (NeRF) and 3D Gaussian Splatting (3DGS). However, both have significant limitations: 1. NeRF method: Through a multi-layer perceptron (MLP) to model a continuous radiation field, although it can achieve high-fidelity rendering, the calculation cost is extremely high - single scene training requires hundreds of hours, which cannot meet the demand for efficient reconstruction, and the surface geometry extraction is difficult.

[0003] 2. 3DGS method: 3D Gaussian primitives are used as explicit representations, and real-time rendering is achieved through an optimized rasterizer, which becomes the preferred solution for small and medium-sized scene reconstruction. However, it faces three major core defects: 1) Weak geometric modeling capability: Gaussian primitives are independent of each other and lack context association, and only rely on image reconstruction loss for training. Although it is good at synthesizing new views, it is difficult to accurately fit the real surface geometry (such as distortion in flat areas and blurred object boundaries); 2) Poor initialization robustness: sparse point cloud extracted by COLMAP is used to initialize Gaussian primitives, but COLMAP is based on local features, which can easily generate sparse or incorrect point clouds in textureless areas and repetitive pattern scenes, leading to structural distortion and under-reconstruction areas in 3DGS reconstruction; 3) Insufficient utilization of priori: existing improved methods (such as 2DGS, GOF, PGSR) introduce pre-trained visual model depth / normal prior to assist optimization, but there are limitations: the visual model can only handle single frame, stereo pair (two frames) or at most 10 frames of view, which cannot cover the global scene; and the priori has noise, which directly as a rigid supervision signal will interfere with 3DGS optimization, resulting in limited reconstruction accuracy; 4) Multi-view Figure 1 consistency loss: 3DGS uses local image supervision and cannot simultaneously rasterize multiple frames of view and optimize global consistency, resulting in inconsistent geometry across views (such as the same object having shape deviations in different views).

[0004] In summary, existing technologies cannot simultaneously meet the demands of "high-precision geometric reconstruction, efficient training, and multi-view Figure 1 consistency", and an improved framework that can cooperatively optimize priori and 3DGS and enhance global consistency is urgently needed.

[0005] [Reference] [1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 42(4): 139–1, 2023. [2] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99–106, 2021. [3] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundationstereo: Zero-shot stereo matching. CVPR, 2025. [4] Yaniv Wolf, Amit Bracha, and Ron Kimmel. Gs2mesh: Surface reconstruction from gaussian splatting via novel stereo views. In ECCV, pages 207–224. Springer, 2024. Therefore, there is an urgent need for an enhanced 3D GS surface reconstruction scheme with "learnable prior, global consistency, geometry-appearance balance, and minute-level training". SUMMARY

[0006] The present application provides an enhanced 3D surface reconstruction method (Eve3D) based on 3D Gaussian Splatting, which realizes SOTA2 geometric precision training time ≤ 20 min (fast version) while maintaining real-time rendering capability; global consistency across views; robustness to weak texture / large baseline.

[0007] To achieve the above objectives, the present invention proposes four core modules (see Figure 2): M1 stereo prior calculation module; M2 local bundle adjustment (LBA) module with prior participation; M3 3DGS-prior joint optimization module; M4 multi-loss training module. The specific steps are as follows: S1 Global stereo prior generation 1. For the input camera pose , generate a virtual right camera along the horizontal axis , with baseline b = 0.5 m; 2. Use the current 3DGS to render the rectified stereo pair ; 3. Feed the stereo pair into a pre-trained stereo matching network (such as FoundationStereo) to obtain a disparity map, and triangulate to get the depth prior D * ; 4. Perform left-right consistency check (disparity deviation ≤ 3 pixels) and output a confidence mask ∈{0,1}.

[0008] S2 Local bundle adjustment (LBA) - cross-view geometric / photometric consistency 1. Construct a factor graph (V, E), where each node is a view, and the edges connect the current view to the Top-K neighboring views (K = 8); 2. Geometric consistency: Use the learnable depth to calculate the normal Ni, deduce the distance map , and then calculate the homography matrix , and enforce the "round-trip projection error" ≈0; 3. Photometric consistency: Warp the k×k patch, k = [3, 15], to the neighboring views through , and minimize the normalized cross-correlation (NCC) error ; 4. Weighted total loss:

[0009] where is jointly determined by depth credibility and occlusion detection.

[0010] S3 Learnable prior and 3DGS joint optimization 1. Two-stage strategy: a) When the iteration < Tjoint (= 7000), supervise the 3DGS-rendered depth D with the rigid prior D*; b) When the iteration ≥ Tjoint, replace D* with the learnable prior It updates synchronously with 3DGS parameters, realizing a two-way closed loop of "prior-3DGS"; 2. LBA Pre-training: Before joint optimization, all... Perform 1000 LBA operations and provide multi-view Figure 1 To the initial value; 3. Confidence mask updated online: refreshed every 1000 iterations based on LBA weights Wst. ; right For pixels with a value of 0, enable Pull loss. This guides it toward a stable value.

[0011] S4 Multi-Loss Training: The total training loss incorporates multi-dimensional constraints, as shown in the following formula:

[0012] Definitions of each loss term: 1. Color loss : L1 loss between 3DGS rendered colors and real image colors, used to supervise appearance fidelity; 2. Depth-Normal Consistency Loss To ensure geometric consistency between the 3DGS rendering depth and the normal vector, and to avoid contradictions; 3. Scale loss Minimize the minimum factor of the Gaussian scale matrix Si, and force Gaussian flattening to fit the surface; 4. Prior loss This includes depth prior loss and normal vector prior loss. 1) Depth prior loss: iterations < At that time , iteration ≥ At that time ; 2) Prior loss of normal vector: Constrains the cosine similarity between the 3DGS normal vector and the prior normal vector. Iteration < At that time Iteration ≥ At that time

[0013] 5. Local bundle adjustment loss The optimization objective is to force depth maps to maintain geometric and aesthetic consistency within a local area (adjacent viewpoints) to enhance multi-view functionality. Figure 1 Consistency is achieved by correcting prior depth noise, providing a more reliable initial geometry for subsequent 3DGS joint optimization; 6. Pull loss For pixels that are still unreliable, an additional loss is introduced to gradually bring the depth values ​​of these pixels closer to a stable value.

[0014] Beneficial effects Compared with existing 3D reconstruction technologies, this invention has the following significant advantages: 1. Reconstruction accuracy is state-of-the-art: 1) On the Tanks & Temples dataset (6 indoor and outdoor scenes): Eve3D achieved an average F1 score of 0.58, surpassing methods such as GOF (0.46) and PGSR (0.52); Eve3D-fast (trained for 20 minutes) still achieved a score of 0.56, making it the second best on this dataset; 2) On the DTU dataset (15 object-level scenes): the average Chamfer distance is 0.48, which is better than Neuralangelo (0.61) and PGSR (0.52), and is the best among all methods; 3) On the Mip-NeRF360 dataset (outdoor scene): PSNR reached 24.99, SSIM reached 0.758, and LPIPS reached 0.203, all of which are state-of-the-art. In the indoor scene, SSIM reached 0.930 (second best).

[0015] 2. Extremely high training efficiency: The Eve3D-fast version can complete training in just 20 minutes (5K iterations), while the basic version (30K iterations) takes only 1.2 hours, which is much faster than NeRF-like methods (>24 hours) and some improved 3DGS methods (such as GOF which takes 2 hours), achieving a breakthrough trade-off between accuracy and speed.

[0016] 3. Improved geometric consistency and prior robustness: 1) The local bundle adjustment optimization strategy solves the global consistency problem of local supervision in 3DGS, reducing cross-view geometric deviation by more than 40%; 2) The joint optimization strategy reduces prior noise by 35% and 3DGS geometric fitting error by 25%, especially significantly improving reconstruction accuracy in areas with missing textures (such as white walls) and object boundaries (such as furniture edges).

[0017] 4. Balancing Appearance and Geometry: While improving geometric accuracy, it maintains high-fidelity appearance rendering. The Mip-NeRF360 indoor scene LPIPS reaches 0.157, close to PGSR (0.147), meeting the dual requirements of "geometric accuracy and appearance realism".

[0018] 5. The application scenarios of this invention cover virtual reality scene construction, robot navigation environment modeling, film and game asset generation, industrial equipment digitization and other fields, and can adapt to the high-precision 3D reconstruction needs of object-level and indoor and outdoor scenes. Attached Figure Description

[0019] Figure 1. Comparison of geometric reconstruction and rendering effects. Figure 2 Eve3D framework flowchart (corresponding to the four major modules). Figure 3. Qualitative comparison of Tanks & Temples. Figure 4 Qualitative comparison of the DTU dataset. Figure 5 Qualitative comparison of Mip-NeRF360.

[0020] Figure 6 Ablation experiment of “joint optimization vs. no joint optimization”. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings.

[0022] Experimental environment configuration: 1. Hardware environment: Single NVIDIA RTX 4090 GPU (24GB VRAM), Intel Core i9-13900K CPU, 64GB DDR5 RAM, 1TB SSD storage; 2. Dataset parameters: 1) Tanks & Temples: 6 scenes (Barn, Caterpillar, etc.), including indoor and outdoor scenes with different lighting / scales, evaluated using the official F1 score; 2) DTU: 15 object-level scenes, image resolution 1200×1600, evaluated using Chamfer distance (lower is better); 3) Mip-NeRF360: Includes outdoor (Garden, Truck, etc.) and indoor (Room, etc.) scenarios, evaluating PSNR, SSIM, and LPIPS; 3. Key hyperparameters: 1) Virtual camera baseline b = 0.01-5m, number of neighboring views K = 8; 2) Joint optimization switching iteration =7000, Eve3D-fast iteration count 5K ( =1000), basic version 30K; 3) Loss weights: =1.0、 =0.8、 =0.2、 =0.5; 4) Gaussian initialization: COLMAP sparse point cloud is used, with an initial number of Gaussians of 10K, which is increased to 50K-100K after densification.

[0023] The enhanced 3D surface reconstruction method (Eve3D) based on 3D Gaussian Splatting proposed in this invention includes the following core components: a stereo prior calculation module, a prior-involved local bundle adjustment optimization module, a 3DGS and prior joint optimization module, and a multi-loss training module. Figure 2 As shown. The specific technical solution is as follows: I. Stereo Prior Calculation Module: High-Precision Depth Prior Generation Binocular vision estimation is used to replace traditional geometrically inconsistent monocular vision estimation, and a globally consistent stereo prior is generated through 3DGS. The steps are as follows: 1. Virtual stereo rendering: For input camera pose Set the baseline distance along the horizontal axis Generate virtual right camera pose :

[0024] in It is a 3×3 identity matrix. Let b be the translation vector of the virtual camera (in this invention, b = 0.5m). The images of the input camera and the virtual camera are rendered using 3DGS to obtain the rectified stereo pair. .

[0025] 2. Depth Prior and Confidence Mask Calculation: Pre-trained stereo matching networks (such as StereoAnywhere, FoundationStereo) are used to process stereo pairs. The disparity map is predicted and converted into a depth prior using triangulation (with the in-camera baseline b known). A confidence mask is generated through a "left-right consistency check" (parallax deviation ≤ 3 pixels). Mark high-confidence depth prior regions ( =1 indicates trustworthiness. =0 indicates untrustworthy).

[0026] II. Prior-involved Local Bundle Adjustment Optimization Module: Multi-view Figure 1 Consistency constraints To improve global consistency, construct cross-view geometric and photometric constraints, following these steps: 1. Factor graph construction: For the current view Calculate the overlap score with other views and select the Top-K neighboring views. (In this invention, K=8); Construct a factor graph (V,E), with edge E connecting the current view. With neighborhood view The constraints are geometric consistency and photometric consistency.

[0027] 2. Geometric consistency constraint: For the current view Learnable depth The normal vector graph is calculated using finite difference. And derive the distance map :

[0028] in For the image plane 2D coordinates, Using homogeneous coordinates, Given the camera intrinsic parameter matrix. Calculate the homography matrix. It can view pixels Mapping to neighborhood view :

[0029] in , for arrive The relative rotation and translation matrices. The homography matrix is ​​calculated. and , view pixels Mapping to neighborhood view Then, by mapping back, geometric consistency is ensured by minimizing the projection error:

[0030] 3. Photometric consistency constraint: for Each pixel Extract a 7×7 image patch centered on this pixel, and then use the homography matrix... Mapped to Minimize the normalized cross-correlation (NCC) error:

[0031] 4. Local bundle adjustment optimization loss: Introducing confidence weights (Based on depth confidence and occlusion judgment), the total loss is:

[0032] in =1.0、 =0.8、 =0.2 is the weighting coefficient.

[0033] III. 3DGS and Prior Joint Optimization Module: Dynamic Prior Iterative Optimization Treating depth priors as learnable parameters and co-optimizing them with 3DGS, the steps are as follows: 1. Learnable prior initialization: based on deep priors Initialize learnable priors ( (These are model parameters, updated during training), employing a two-stage optimization strategy: Number of iterations < (This invention takes) =7000): Use Supervise 3DGS rendering depth To avoid falling into poor local minima; Number of iterations ≥ :use replace In conjunction with 3DGS optimization, prior knowledge is adapted to 3DGS geometric features.

[0034] 2. LBA Pre-training Initialization: Learnable priors for all views before joint optimization. Perform local bundle adjustment optimization to provide multiple views for joint optimization. Figure 1 The initial prior is consistent, which solves the problems of sparse and inconsistent prior views.

[0035] 3. Confidence Mask Update: As training progresses, confidence weights are adjusted based on local bundles. Update confidence mask :

[0036] right For low-confidence pixels with a value of 0, "Pull Loss" is used to guide 3DGS to continue its optimization.

[0037] in Rendering depth for 3DGS (gradient separation to avoid backpropagation interference). =0.5. Low-confidence pixels are included in the joint optimization after passing LBA verification.

[0038] IV. Multi-loss Training Module: Comprehensive Constraint Geometry and Appearance The total training loss incorporates multi-dimensional constraints, and the formula is as follows:

[0039] Definitions of each loss term: 1. Color loss : L1 loss between 3DGS rendered colors and real image colors, used to supervise appearance fidelity; 2. Depth-Normal Consistency Loss To ensure geometric consistency between the 3DGS rendering depth and the normal vector, and to avoid contradictions; 3. Scale loss Minimize the minimum factor of the Gaussian scale matrix Si, and force Gaussian flattening to fit the surface; 4. Prior loss This includes depth prior loss and normal vector prior loss. 1) Depth prior loss: iterations < At that time , iteration ≥ At that time ; 2) Prior loss of normal vector: Constrains the cosine similarity between the 3DGS normal vector and the prior normal vector. Iteration < At that time Iteration ≥ At that time

[0040] 5. Local bundle adjustment loss The optimization objective is to force depth maps to maintain geometric and aesthetic consistency within a local area (adjacent viewpoints) to enhance multi-view functionality. Figure 1 Consistency is achieved by correcting prior depth noise, providing a more reliable initial geometry for subsequent 3DGS joint optimization; 6. Pull loss For pixels that are still unreliable, an additional loss is introduced to gradually bring the depth values ​​of these pixels closer to a stable value.

[0041] Example: 1. Pre-trained basic 3DGS: The Mini-Splatting2 framework was used to pre-train 3DGS (500 iterations) to generate initial Gaussian primitives and camera pose calibration results for subsequent stereo pair rendering.

[0042] 2. Solid prior calculation: 1) For each input camera Generate a virtual camera according to the formula Rendering stereoscopic objects using 3DGS (The resolution is consistent with the real image); 2) Input the binocular matching network, output the disparity map, and combine it with the camera intrinsic parameters (focal length). = =1000, principal point ) and baseline b=0.5m, calculate the depth prior. ; 3) Perform left-right consistency checks (parallax deviation ≤ 3 pixels) and generate a confidence mask. .

[0043] 3. LBA pre-training and joint optimization: 1) LBA pre-training: for all views Perform local bundle adjustment optimization (1000 iterations) to optimize. Multiview Figure 1 To the point of being compatible; 2) Joint optimization (30K iterations): ① Iteration 1-7000: using Monitoring 3DGS, the loss is ; ② Iteration 7001-30000: Optimization The confidence mask is updated every 1000 iterations. Activate Pull loss; 3) Local bundle adjustment optimization: Performed every 500 iterations, updating the factor graph and constraint weights. .

[0044] 4. Final Reconstruction Output: After training, depth maps of all views are rendered using 3DGS, and a global 3D mesh is generated using the TSDF fusion algorithm; for new view synthesis tasks, the target pose image is rendered directly using the 3DGS rasterizer.

[0045] Figure 1 shows a comparison of geometric reconstruction and rendering effects.

[0046] Effect verification: 1. Quantitative Validation

[0047] 2. Qualitative verification, as shown in Figures 3 and 4. 1) Complex scenes (such as the Courthouse in Tanks & Temples): The building columns reconstructed by Eve3D have no blurred boundaries, while 2DGS and GOF show obvious geometric distortion; 2) Scenes with missing textures (such as textureless objects in DTU): Eve3D can still preserve the object outline through joint optimization and LBA, while PGSR shows under-reconstruction; 3) New perspective compositing (such as Garden in Mip-NeRF360): The tree shadows and ground textures rendered by Eve3D have natural transitions, with LPIPS reaching 0.203, which is better than 3DGS (0.234).

[0048] Figure 5 is a qualitative comparison chart of Mip-NeRF360, and Figure 6 is a comparison chart of the joint optimization effect.

Claims

1. An enhanced 3D surface reconstruction method based on 3D Gaussian Splatting, characterized in that, include: a) Input camera pose Generate virtual stereo camera pairs ( , Using 3D Gaussian Splatting rendering to obtain stereoscopic images b) Feed the stereo image pairs into a pre-trained stereo matching network to obtain the depth prior D. * and confidence mask c) Construct a factor graph with learnable depth As the core, apply geometric consistency constraints to the current view and the Top-K neighboring views. Photometric Consistency Constraints This results in localized bundle adjustment loss. ;d) A two-stage optimization strategy is adopted: in the early stage of training, D is used as the optimization strategy. * Monitor the rendering depth of 3DGS and adjust the D in post-processing. * Replace with learnable priors And it is updated in conjunction with 3DGS parameters; e) Confidence mask For low-confidence pixels with a probability of 0, a pull loss is introduced. This will cause its depth value to approach a stable value; f) Use multiple loss functions for end-to-end training to achieve high-precision, multi-view consistent 3D surface reconstruction.

2. The method according to claim 1, characterized in that, The virtual right camera pose It is generated by Euclidean transformation of the horizontal baseline b = 0.01-5 m.

3. The method according to claim 1, characterized in that, A confidence mask is generated by checking left-right consistency and ensuring that the parallax deviation is ≤3 pixels. Mark the deep prior regions with high confidence.

4. The method according to claim 1, characterized in that, For the current view Learnable depth The normal vector graph is calculated using finite difference. And derive the distance map : in For the image plane 2D coordinates, Using homogeneous coordinates, Given the camera intrinsic parameter matrix, calculate the homography matrix. It can view pixels Mapping to neighborhood view : in , for arrive The relative rotation and translation matrices are obtained by calculating the homography matrix. and , view pixels Mapping to neighborhood view Then, by mapping back, geometric consistency is ensured by minimizing the projection error: .

5. The method according to claim 1, characterized in that, right Each pixel Extract a k×k image patch centered on this pixel, where k=3-15, and then use the homography matrix. Mapped to Minimize the normalized cross-correlation NCC error: .

6. The method according to claim 1, characterized in that, Where the Pull loss weight, =0-1, and 3DGS rendering depth with gradient break.

7. A computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

9. The application of the method according to any one of claims 1-6 in virtual reality content creation, robot navigation map construction, industrial reverse engineering, or film and game asset generation.

Images