A 3D Gaussian splatting-based scalable reconstruction method for ultra-large scenes

Abstract

The application discloses a super-large scene scalable reconstruction method based on 3D Gaussian Splatting. The number of sub-block images is balanced through image perception adaptive blocking; low-contribution views are removed through SSIM pruning; prior depth is obtained based on virtual stereo rendering, a confidence mask is generated through multi-view verification, boundary consistency loss is constructed, and Gaussian densification is driven; and finally, seamless meshes are obtained through TSDF fusion. Experiments show that a 2.7 km2 scene is reconstructed in 6 hours, the MatrixCity-Aerial F1 reaches 0.784, which is 41% higher than CityGaussianV2, and the method balances accuracy, efficiency and boundary consistency, and is suitable for super-large scene applications such as digital twin cities and automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically involving a scalable reconstruction method, system, device and storage medium for 3D Gaussian Splatting for ultra-large scenes of 0.3km²–2.7km², which can be widely used in fields such as digital twin cities, autonomous driving maps, and digitization of large-scale infrastructure. Background Technology

[0002] Scene reconstruction is a core research direction in computer vision. Traditional technical approaches mainly include multi-view stereo matching (MVS), neural radiation field (NeRF), and 3D Gaussian Splatting (3DGS). 1. Traditional MVS methods rely on voxel representation, surface evolution, or depth map estimation. Although they have some applications in small and medium-sized scenes, they suffer from high computational complexity and poor robustness to areas with missing textures when facing ultra-large scenes, making them difficult to adapt to city-level data scales.

[0003] 2. NeRF-type methods: These methods model continuous radiation fields using multilayer perceptrons (MLPs), enabling high-fidelity rendering. However, they have extremely low training efficiency—training time for medium-scale scenes often exceeds 100 GPUs per hour, and the rendering speed is slow, making it impossible to meet the real-time requirements of ultra-large scenes.

[0004] 3.3DGS Method: Using 3D Gaussian primitives as the scene representation unit, it combines the advantages of real-time rendering and high visual fidelity, making it a preferred solution for small to medium-sized scene reconstruction. However, it faces three major bottlenecks in ultra-large scenes (such as city-level scenes): 1) Scalability defects: The number of primitives in Go is exponentially increased with the scene scale, leading to memory overflow in high-end GPUs, and the rasterization process based on depth sorting suffers from primitive redundancy, resulting in a significant decrease in rendering speed. 2) Inappropriate segmentation strategy: Existing 3DGS ultra-large scene solutions (such as CityGaussian series and VastGaussian) adopt a "divide and conquer" strategy, but rely on manually configured uniform grid segmentation, which cannot automatically ensure that the number of images in each segment is "sufficient and minimal" - insufficient images in some segments lead to incomplete reconstruction, and redundant images in some segments increase the computational burden; at the same time, occlusion factors are not considered, and artifacts are reduced only by "adding boundary images", which cannot eliminate invalid views that are occluded; 3) Insufficient accuracy in boundary and densification: Block boundary optimization relies on monocular prior depth, which cannot guarantee geometric consistency across multiple views, resulting in stitching artifacts at the boundaries; and Gaussian densification driven by image rendering loss gradient is insensitive to blurred areas and geometric defects, causing loss of scene details.

[0005] While existing 3DGS extensions such as CityGaussianV2 and VastGaussian exist, none can simultaneously meet the requirements of "high precision, high efficiency, and no border artifacts" for the reconstruction of ultra-large scenes. There is an urgent need for an improved framework for 3DGS to overcome these limitations.

[0006] [References] [1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACMTOG, 42(4):139–1, 2023. [2] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan TBarron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neuralradiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021. [3] Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songcen Xu, Youliang Yan, et al. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In CVPR, pages 5166–5175, 2024. [4] Yang Liu, Chuanchen Luo, Lue Fan, Naiyan Wang, Junran Peng, andZhaoxiang Zhang. Citygaussian: Real-time high-quality large-scale scenerendering with gaussians. In ECCV, pages 265–282. Springer, 2024. Summary of the Invention This invention provides a scalable reconstruction method for ultra-large scenes based on 3D Gaussian Splatting, achieving: automatically ensuring that the number of images in each sub-block is "minimum and sufficient"; removing low-contribution / occluded views to reduce computation by 30-50%; eliminating boundary artifacts based on stereo priors and multi-view verification; and using normal gradient-driven densification to preserve fine structures such as windows and railings. Reconstruction of 0.3-2.7 km² scenes can be completed in 6 hours on an 8×RTX 4090, with an F1 score improvement of 41% compared to CityGaussianV2.

[0007] The first technical solution of this invention is to provide a scalable reconstruction method for ultra-large scenes based on 3D Gaussian Splatting, comprising four main modules (see Figure 2): M1 Image perceptual segmentation module; M2 Coarse optimization and image pruning module; M3 Boundary consistency optimization module; M4 Global TSDF fusion module. The core process is as follows: Step S1 Image-aware adaptive block segmentation 1. Align the scene's principal axis with the global coordinate system to obtain the axis-aligned bounding box. ; 2. Use binary segmentation: Loop through the image with the most occurrences (B, C). If |C|≤Tc, keep it; otherwise, split it along the longest axis of B. and guarantee ; 3. Obtain the initial block set {(Bi,Ci)}; 4. Expand Bi in each direction by 10-50% to form a common-view block. Corresponding image set This provides redundant views for subsequent overlap optimization.

[0008] Step S2: Parallel coarse training and low-contribution image pruning 1. In each Independently train 3DGS for 30k iterations to obtain the Gaussian metaset. ; 2. For images located outside the original Bi boundary Render the full image Î and the "external Gaussian only" image I′. If SSIM(Î,I′)≥TSSIM=0.8, then discard I. 3. Gather the pruned branches This reduces redundant calculations.

[0009] Step S3 Boundary Consistency Optimization 1. Generate a virtual right camera Ri for each target pose Pi, with baseline b=0.5 m, and render the stereo pair (Ii,Ii^R) using the current 3DGS; 2. The stereo matching network outputs the prior depth. ; 3. Project onto the 8 nearest neighbor views and backproject. If the round-trip error is less than 0.1-10 pixels, they are considered consistent, and a confidence mask is generated. ; 4. with Mask depth / normal prior construction loss by Constraints To optimize the 3D Gaussian model for prior depth, the loss function is defined as:

[0010] in: Render depth for the current model. for Surface normal vector (finite difference calculation); For the prior depth, for Surface normal vector; , The weighting coefficients are λd∈[0.5,2.0] and λn∈[0.2,1.0]. The Gaussian compaction driven by the normal component is preferred to complete 30k iterations of local optimization.

[0011] Step S4 Global TSDF Fusion All blocks are rendered with depth maps (1600×1200), voxel=0.1 m, and TSDF is used to generate a seamless mesh.

[0012] The beneficial effects are as follows: 1. Reconstruction accuracy: MatrixCity-Aerial F1=0.784, +41% vs CityGaussianV2; DTU-level object Chamfer 0.48 mm, on par with state-of-the-art for small to medium-sized scenes.

[0013] 2. Training efficiency: 40k images and 2.7 km² reconstruction were completed in 6 hours on an 8×RTX 4090, a 25% improvement over VastGaussian in 8 hours.

[0014] 3. No border artifacts: Stereo priors and multi-view verification ensure that the geometric deviation at the stitching point is less than 1 cm.

[0015] 4. Expandable video memory: After being divided into blocks, each card only stores 5–8 MB of Gaussian data, avoiding video memory overflow.

[0016] 5. High versatility: compatible with aerial photography, street view, indoor and outdoor multi-source data.

[0017] Figure 1 Comparison of reconstruction effects of ultra-large scene: (a) Overview, (b) and (c) are local comparison images; Figure 2 UltraSplat framework flowchart; Figure 3 Qualitative comparison of MatrixCity; Figure 4 Qualitative comparison of GauU-Scene. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings. I. Experimental Environment Configuration 1. Hardware environment: 8 NVIDIA RTX 4090 GPUs (24GB VRAM per card), Intel Xeon Platinum 8480+ CPU, 512GB DDR5 memory, 2TB SSD storage; 2. Dataset parameters: 1) MatrixCity: Includes aerial and street scenes, with an area of ​​0.3~2.7km², 40,000+ training images, and 450+ test images; aerial images are downsampled to 1600 pixels on the long side, while street scenes are kept at 1000×1000 pixels. 2) GauU-Scene: Includes Residence, Russian Building, and Modern Building scenes, with 40,000+ training images and 450+ test images; 3. Key hyperparameters: 1) Image quantity threshold Tc: MatrixCity=500, GauU-Scene=100; 2) SSIM pruning threshold TSSIM = 0.8; 3) Number of training iterations: 30,000 coarse optimizations and 30,000 boundary optimizations; 4) The baseline distance of the stereoscopic view is b = 0.5m; 5) Loss function weights =1.0, =0.5; 6) TSDF voxel resolution = 0.1m.

[0019] II. A super-large scene reconstruction method based on 3D Gaussian Splatting (named UltraSplat), whose core includes an image perception partitioning module, a coarse optimization and image pruning module, a boundary consistency optimization module and a global TSDF fusion module. The specific technical solution is as follows: 1. Image Perception Partitioning Module: Adaptive Partitioning Based on Image Distribution Instead of the traditional uniform spatial partitioning, it takes "image view optimization" as the core to ensure that the number of images in each partition is balanced and meets the reconstruction requirements: 1. Binary Partitioning 1) First, align the principal axis of the scene with the global coordinate system to ensure that the partition is an axis-aligned bounding box; 2) Initialize a queue, and store the total bounding box of the scene and all image sets (B, C); 3) Cyclically pop (B, C) with the largest number of images from the queue: if the number of images is ≤ threshold Tc (Tc=500 for MatrixCity scene, Tc=100 for GauU-Scene scene), keep it as a final partition; otherwise, split it into (B1,C1) and (B2, C2) along the "maximum axis" of B; if the number of images in both C1 and C2 after splitting is less than Tc, enqueue them directly; otherwise, adjust the splitting mode so that the number of images in C1 and C2 is both 1 / 2 of the original C, then enqueue them; 4) Repeat the above steps until the number of images in all partitions is ≤ Tc, forming an initial partition set {(Bi, Ci)}.

[0020] 2. Covisibility Blocks Enlarging For each initial partition (B, C), expand its width and height by 10-50%, preferably 25%, to obtain an expanded partition ; update the image set covering B synchronously , forming ( , ), which provides an overlapping area and multi-view basis for subsequent boundary consistency optimization.

[0021] II. Coarse Optimization and Image Pruning Module: Parallel Training and Redundant View Culling 1. Parallel Coarse Training For each ( , ), the 3D Gaussian model is trained independently: the scene is represented by 3D Gaussian primitives ( is the center, = is the covariance matrix) to represent the scene, and a tile-based rasterizer is used to render the pixel color C= ( For the sake of transparency, For opacity, (For color), after 30,000 training iterations, the Gaussian metaset is obtained. Forming a triple ( , , ).

[0022] 2. Low-contribution image pruning For the "boundary image" in C that extends beyond the original block B, it is removed using the following method: 1) Render two images: (Use full amount ), I' (using only Gaussian elements outside the original block B); 2) Calculation The structural similarity (SSIM) with I' is used. If SSIM ≥ the threshold TSSIM (TSSIM = 0.8), the image is considered to have low contribution to reconstruction and is discarded. 3) Retain the image set after pruning. This reduces the complexity of subsequent calculations.

[0023] III. Boundary Consistency Optimization Module: Fine-grained Optimization of Multi-view Constraints 1. Synthetic Stereo Generation 1) For each target point Set the baseline distance b along the horizontal axis to generate a virtual right point. : , where t= ; 2) From The scene is rendered from a specific perspective, resulting in perfectly calibrated stereoscopic images. ; 3) Use stereo matching networks (such as StereoAnywhere, FoundationStereo) to process stereo pairs, generate disparity maps, and convert them into prior depth. .

[0024] 2. View-consistent Depth Verification 1) For each pixel, use Project it onto the 8 nearest neighbor image views, and then backproject it back to the original view using the neighbor view depth; 2) If the offset between the back-projected pixel and the original pixel is less than 0.1-10 pixels, it is determined that the adjacent view has the same depth as the original view. 3) Generate a confidence mask Only retain pixel regions with "at least two adjacent view depths that are consistent" to ensure Reliability.

[0025] 3. Geometric-guided local optimization by Constraints To optimize the 3D Gaussian model for prior depth, the loss function is defined as:

[0026] in: Render depth for the current model. for Surface normal vector (finite difference calculation); For the prior depth, for Surface normal vector; , where represents the weighting coefficients. Furthermore, since the rendering loss is insensitive to geometric defects, Gaussian compaction driven by the surface normal vector-related loss gradient is preferentially employed. Each block is optimized independently 30,000 times.

[0027] IV. Global TSDF Fusion Module: Seamless Mesh Generation For all optimized blocks, render the depth map of each image viewpoint, and use the TSDF (Truncated Signed Distance Function) fusion algorithm (such as KinectFusion) to generate a global seamless 3D mesh to complete the reconstruction of the ultra-large scene.

[0028] Example: 1. Image-aware block implementation Taking the MatrixCity-Aerial scene as an example: the initial bounding box is (X: 0~5000m, Y: 0~5000m, Z: 0~500m), and the image set C contains 8000+ images; perform binary segmentation until the number of images in each block is ≤500, and finally obtain 16 blocks; each block is expanded by 25% (e.g., the original block X: 0~312.5m → X: -39.06~351.56m), and the image set C adds 100~200 boundary images.

[0029] 2. Coarse optimization and image pruning implementation 1) Parallel training for each block: Optimizer is Adam (learning rate 1e-4), batch size=8, iterations 30,000; 2) Image pruning: For 200 boundary images in a certain block C, calculate the SSIM of I and I', remove 80 images with SSIM≥0.8, and retain 120 high contribution images.

[0030] 3. Implementation of Boundary Consistency Optimization 1) Virtual image generation: using the target image (Based on coordinates (x, y, z) and attitude angles (α, β, γ)) The coordinates are set to (x+0.5, y, z), and the attitude angle remains unchanged; 2) Stereo Matching and Depth Validation: Stereo matching networks such as StereoAnywhere and FoundationStereo are used to generate... After verifying 8 neighboring views, Retain approximately 75% of the pixel area; 3) Local optimization: with Constrain the loss function, prioritize updating Gaussian elements in the boundary region, so that the boundary... and The cosine similarity is ≥0.95.

[0031] 4. Global integration implementation Render the depth map of each image (1600×1200 resolution), and generate a mesh model (approximately 10 million triangles) by TSDF fusion, without boundary cracks or artifacts.

[0032] Effect verification 1. Quantitative validation

[0033] 2. Qualitative verification like Figure 1 As shown, compared with GSV2, the present invention has higher reconstruction accuracy, better clarity and smoothness.

[0034] like Figure 3 , Figure 4 As shown, the reconstructed building outlines are clearer (e.g., the edges of high-rise buildings in MatrixCity are not jagged), the boundary transitions are more natural (e.g., there are no geometric breaks at the junctions of blocks), and the details are more completely preserved (e.g., the textures of building windows and walls in GauU-Scene can be clearly identified).

Claims

1. A scalable reconstruction method for ultra-large scenes based on 3D Gaussian Splatting, characterized in that, include: a) Recursively perform binary segmentation by aligning the scene axis with the bounding box until the number of images in each sub-block is ≤Tc, forming an initial block set {(Bi,Ci)}; b) Expand each Bi by 10-50% to obtain the common-view block B. i and the corresponding image set C i; c) In each B i. Independently train 3D Gaussian Splatting to obtain the Gaussian metaset G. i; d) For C Images located outside the original Bi boundary in i are removed based on the SSIM threshold TSSIM to eliminate low-contribution images; For C The image in i located outside the original Bi boundary is processed using the full G. i renders the image Image I' is obtained by rendering using only the original Bi Gaussian elements, and the calculation is performed. The structural similarity SSIM with I' is used. If SSIM ≥ the threshold TSSIM, the image is considered to have low contribution to reconstruction and is discarded. e) Generate a virtual stereo camera Ri for each target pose Pi, render the stereo pair using the current 3DGS, and obtain the prior depth. ； f) will Project onto the nearest neighbor view and backproject. If the round-trip pixel offset is less than 0.1-10 pixels, it is considered consistent, and a confidence mask is generated. ; g) with The depth / normal prior of the mask is used to construct the boundary consistency loss and drive Gaussian compaction; by Constraints To optimize the 3D Gaussian model for prior depth, the loss function is defined as: in: Render depth for the current model. for Surface normal vector; For the prior depth, for Surface normal vector; , These are the weighting coefficients; h) After rendering all the block depth maps, TSDF is used to fuse them to generate a global seamless 3D mesh.

2. The method according to claim 1, characterized in that, The value of Tc ranges from 50 to 1000 images and is dynamically adjusted according to the scene scale.

3. The method according to claim 1, characterized in that, Where the SSIM threshold TSSIM∈[0.7,0.9].

4. The method according to claim 1, characterized in that, Where the baseline b∈[0.2 m,1.0 m] of the virtual stereo camera Ri is relative to Pi.

5. The method according to claim 1, characterized in that, Gaussian densification uses pixel locations with a normal component greater than 0.

3.

6. A computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1-5.

8. The application of the method according to any one of claims 1-5 in digital twin cities, high-precision maps for autonomous driving, and digitization of large-scale infrastructure.

Images