A large scene 3D reconstruction method based on 3D Gaussian sputtering

Abstract

This application discloses a large-scene 3D reconstruction method based on 3D Gaussian sputtering, which relates to computer graphics: collecting a multi-view image set of a large scene; obtaining a sparse point cloud of the scene based on the multi-view image set; using a pre-trained depth prediction network to perform monocular depth estimation on the multi-view images to obtain a monocular depth estimation prior; using the scene sparse point cloud and the monocular depth estimation prior, globally training the scene to obtain an initial 3D Gaussian model, spatially gridding the initial 3D Gaussian model to obtain multiple scene blocks with axis-aligned bounding boxes; setting image view data for each scene block; using a parallel GPU to perform depth supervision training on the Gaussian ellipsoids in the multiple scene blocks; merging the trained scene blocks to obtain a final 3D Gaussian model; in view of the low geometric structure reconstruction accuracy caused by relying only on the color information of the multi-view images in the 3D rendering of large scenes, this application improves the rendering and reconstruction accuracy of large scenes.

Description

Technical Field

[0001] The present application relates to the field of computer graphics, and more specifically, to a method for three-dimensional reconstruction of large scenes based on three-dimensional Gaussian sputtering. Background Art

[0002] With the rapid development of 3D reconstruction technology, 3D reconstruction using multi-view images has become an efficient and practical solution. This method not only provides rich spatial information but also generates high-quality 3D models in a relatively short time. Among the many 3D reconstruction techniques, 3D Gaussian Splatting (3DGS) has been widely used due to its superior rendering speed and outstanding visual effects, covering a variety of 3D scene reconstruction and new view synthesis tasks. This method uses an explicit 3D Gaussian ellipsoid to represent the scene's geometry and color information, enabling high-quality multi-view rendering and demonstrating good performance when processing small and medium-sized scenes.

[0003] However, when 3D Gaussian sputtering is extended to large-scale scenarios such as city-level, it faces several key technical challenges:

[0004] In a large scene with an area of ​​square kilometers, the scene may contain tens of millions of Gaussian ellipsoids, which places extremely large demands on memory and computing resources. As a result, standard graphics processing units (GPUs) often face problems such as memory overflow and computing delays when processing, making efficient training and real-time rendering difficult.

[0005] Traditional 3D Gaussian sputtering methods rely primarily on color information from multi-view images for optimization and lack effective geometric supervision. This makes it difficult to accurately recover geometric structure in areas with sparse or repetitive textures. Especially in large scene reconstruction, the model often prioritizes overall consistency over local fine structure, resulting in a lack of realism in the reconstruction.

[0006] Large-scale scene reconstruction involves a balance between global consistency and local detail. Training strategies directly applied to large scenes struggle to simultaneously maintain both global geometric consistency and local detail accuracy. This is especially true when dealing with distant views and complex, dense structures, which can lead to local optimality or training instability.

[0007] Therefore, there is an urgent need for a rendering method that can improve the three-dimensional reconstruction of large scenes. Summary of the Invention

[0008] In order to address the low accuracy of geometric structure reconstruction in large-scene three-dimensional rendering due to relying solely on color information from multi-view images, this application provides a large-scene three-dimensional reconstruction method based on three-dimensional Gaussian sputtering. It improves the reconstruction accuracy of geometric structures in large-scene rendering by providing geometric supervision signals through monocular depth estimation priors and a multi-level rendering strategy based on importance.

[0009] The present application provides a method for 3D reconstruction of large scenes based on 3D Gaussian sputtering, comprising: collecting a set of multi-view images of a large scene; wherein, in the present application, a large scene refers to a 3D environment with a large spatial scale and complex structure, such as an urban block, an industrial park, a campus, a large building complex, etc. Based on the multi-view image set, a sparse point cloud of the scene is obtained; a pre-trained depth prediction network is used to perform monocular depth estimation on the multi-view images to obtain a monocular depth estimation prior; the monocular depth estimation prior refers to the depth predicted from a single-view image, which is used to provide geometric prior knowledge of the scene; the scene is globally trained using the sparse point cloud of the scene and the monocular depth estimation prior to obtain an initial 3D Gaussian model, the 3D Gaussian model containing multiple Gaussian ellipsoids; the initial 3D Gaussian model is spatially gridded to obtain multiple scene blocks with axis-aligned bounding boxes; image view data for each scene block is set; the Gaussian ellipsoids within the multiple scene blocks are trained using a parallel GPU for depth supervision; and the trained scene blocks are merged to obtain a final 3D Gaussian model.

[0010] Furthermore, a sparse point cloud of the scene is obtained, including: extracting feature points of multi-view images and matching the extracted feature points; obtaining a camera pose estimate based on the matched feature points; based on the camera pose estimate, using a linear triangulation algorithm to perform multi-view geometric constraints on each pair of matching feature points to determine the position of each feature point in three-dimensional space; using a bundle adjustment method to minimize the reprojection error, jointly optimizing the camera pose estimate and the position of each feature point in three-dimensional space, and obtaining a sparse point cloud of the scene.

[0011] Among them, the linear triangulation algorithm is a computational method for determining the spatial position of a three-dimensional point. Its basic principle is to use known camera parameters and the coordinates of matching points under at least two perspectives to establish and solve a set of linear equations. In SfM, it constructs an overdetermined set of equations of the form AX=0, where each perspective provides two constraints, and then solves the least squares solution through singular value decomposition (SVD) to determine the coordinates of the feature points in three-dimensional space. This method can simultaneously utilize observation data from multiple perspectives to reduce the uncertainty of depth estimation due to a single perspective. Among them, multi-perspective geometric constraints refer to the use of geometric relationships that exist when observing the same scene from multiple perspectives to constrain three-dimensional reconstruction. These constraints mainly include: Epipolar geometric constraints: Describes the relationship between two perspectives, established through the basic matrix F or the essential matrix E, requiring matching points to satisfy ; Three-view constraint: involves the relationship between three perspectives, represented by a trilinear tensor; multi-view Figure 1 Consistency constraint: requires that the projection of a 3D point under all observation angles must be consistent with the actual observation, etc. Multi-view geometric constraints help filter out false matches.

[0012] The bundle adjustment aims to establish and minimize the reprojection error function. The reprojection error refers to the distance between the position of a 3D point reprojected onto a 2D image through the camera model and the original observed position.

[0013] Furthermore, a monocular depth estimation prior is obtained, including: obtaining the original depth prediction value of each image based on the multi-view images using a pre-trained depth prediction network; scale-correcting the original depth prediction value based on the scene sparse point cloud using a linear regression model to obtain a depth scale factor and an offset; optimizing the depth scale factor and the offset by the least squares method to align the depth prediction value with the scene sparse point cloud in spatial scale; and using the optimized depth scale factor and offset to scale the original depth prediction value to obtain a monocular depth estimation prior consistent with the actual scene scale.

[0014] The depth scale factor is the multiplication coefficient required to convert the relative depth values ​​output by the pre-trained deep network into absolute depth values ​​consistent with the actual scene scale. Since monocular depth estimation networks generally cannot obtain absolute scale information, the depth maps they output often have scale uncertainty. The depth scale factor resolves this scale ambiguity, enabling depth estimation to match the physical scale of the real world. The offset is an additive constant term in the depth correction process, used to adjust the baseline level of overall depth. Combined with the depth scale factor, a complete linear transformation relationship is formed: Corrected Depth = Depth Scale Factor × Original Depth Prediction + Offset. This linear transformation effectively maps relative depth values ​​to an absolute depth coordinate system consistent with the scene's sparse point cloud.

[0015] A monocular depth estimation prior is a scale-corrected depth map predicted from a single-view image, serving as prior knowledge of the scene's geometry. In this application, it provides additional geometric supervision, guiding the optimization of the position and shape of the 3D Gaussian ellipsoid. This helps overcome the inability to accurately recover geometric structure when relying solely on color information, particularly in areas with sparse or repetitive textures.

[0016] Furthermore, the scene sparse point cloud and monocular depth estimation prior are used to perform global training on the scene to obtain an initial three-dimensional Gaussian model, including: constructing a three-dimensional Gaussian field based on the scene sparse point cloud. The Gaussian field contains multiple Gaussian ellipsoids, and the parameters of each Gaussian ellipsoid include the center point, covariance matrix, color value and opacity; among them, the covariance matrix represents the rotation and scale of the Gaussian ellipsoid. , where R is the rotation matrix of the Gaussian ellipsoid, represented by quaternion; S is the scale matrix of the Gaussian ellipsoid, represented by a three-dimensional vector, and the value of each dimension represents the scaling ratio of the corresponding dimension; according to the color value and opacity, the color value of the rendered pixel at each image perspective is calculated to obtain the rendered image; according to the rendered image and the real image, the loss function of the global training is set ; Optimize the parameters of the Gaussian ellipsoid by minimizing the global training loss function; construct the initial three-dimensional Gaussian model based on the optimized Gaussian ellipsoid parameters.

[0017] The loss function of global training is: ,in, and is the weight coefficient, and They are the L1 loss and structural similarity SSIM loss between rendered images and real images respectively.

[0018] A 3D Gaussian field is a spatial representation system composed of multiple 3D Gaussian ellipsoids, used to describe and characterize the geometric structure and appearance of a 3D scene. It approximates the surface and volumetric properties of a scene using a collection of Gaussian ellipsoids distributed across space, forming an implicit representation of the scene. In this application, a 3D Gaussian field is constructed from a sparse point cloud of the scene.

[0019] The Gaussian ellipsoid is the basic building block of the three-dimensional Gaussian field. Each Gaussian ellipsoid is defined by the following parameters: the center point represents the position coordinates of the Gaussian ellipsoid in three-dimensional space; the covariance matrix determines the shape, size and direction of the Gaussian ellipsoid, which is expressed as , where R is the rotation matrix represented by quaternion, S is the diagonal matrix representing the scaling ratio of the three main axes; the color value represents the RGB color information of the Gaussian ellipsoid; and the opacity controls the transparency of the Gaussian ellipsoid.

[0020] Color values ​​are the appearance attributes of the Gaussian ellipsoid, typically expressed in RGB format, and determine its visual characteristics during rendering. Opacity is a scalar parameter that controls the transparency of the Gaussian ellipsoid, typically ranging from 0 to 1, where 0 represents complete transparency and 1 represents complete opacity. During the rendering process, color values ​​and opacity together determine the weight of each Gaussian ellipsoid's contribution to the final rendered pixel. In this application, volume rendering effects are achieved through alpha blending.

[0021] Furthermore, the initial three-dimensional Gaussian model is spatially gridded to obtain a plurality of scene blocks with axis-aligned bounding boxes, including: setting the spatial boundary of the Gaussian model according to the initial three-dimensional Gaussian model; gridding the area formed by the spatial boundary to generate a plurality of spatial sub-blocks; determining the spatial sub-block to which each Gaussian ellipsoid belongs according to the coordinates of the center point of the Gaussian ellipsoid; calculating the maximum and minimum values ​​of the Gaussian ellipsoid in each spatial sub-block, and using the maximum and minimum values ​​as the geometric boundaries of the corresponding spatial sub-block; setting the axis-aligned bounding box AABB of each spatial sub-block according to the geometric boundaries; and allocating the Gaussian ellipsoid parameters contained in each bounding box according to the axis-aligned bounding box AABB to obtain a plurality of scene blocks.

[0022] Furthermore, the image perspective data of each scene block is set, including: judging whether the spatial position of each camera is located within the axis-aligned bounding box of the corresponding scene block according to the acquired camera pose estimation and the axis-aligned bounding box AABB of the scene block; if so, assigning the image captured by the camera and the corresponding monocular depth estimation prior data to the scene block for training; if not, calculating the contribution weight of the camera's perspective to the corresponding scene block :

[0023] ;in, is the pose parameter of the i-th camera, LSSIM represents the calculation of the structural similarity loss between two images, Represents the image rendered using all Gaussian ellipsoids, represents the set of Gaussian ellipsoids after removing the Gaussian ellipsoid in the j-th scene block;

[0024] Determine contribution weight Is it greater than the threshold? If yes, the image captured by the corresponding camera and the corresponding monocular depth estimation prior data are assigned to the corresponding scene block for training.

[0025] Furthermore, a parallel GPU is used to perform deep supervised training on Gaussian ellipsoids in multiple scene blocks, including: assigning multiple scene blocks to multiple GPUs for parallel processing, each GPU is responsible for the training of one or more scene blocks; projecting the Gaussian ellipsoid in each scene block onto a two-dimensional imaging plane according to the assigned image perspective to obtain a two-dimensional Gaussian ellipse; dividing the projected two-dimensional imaging plane into blocks to obtain multiple image blocks, each image block containing multiple pixel points; extracting the two-dimensional Gaussian ellipses within the pixel range of each image block, and sorting the two-dimensional Gaussian ellipses according to the distance from the center point of the two-dimensional Gaussian ellipse to the two-dimensional imaging plane; according to the sorting results of the two-dimensional Gaussian ellipse, respectively calculating the color C of the rendered image and the cumulative depth D of the two-dimensional Gaussian ellipse by α-blending; setting a local loss function for the scene block. ; According to the local loss function ,The parameters of the Gaussian ellipsoid in each scene block are iteratively adjusted through back-propagation to obtain the trained scene blocks.

[0026] Furthermore, the color C of the rendered image is calculated using the following formula: , where N represents the number of two-dimensional Gaussian ellipses involved in image rendering, Represents the color value of the i-th Gaussian ellipse at the corresponding viewing angle; Indicates the opacity of the i-th Gaussian ellipse at the corresponding viewing angle; Indicates the opacity value of the j-th Gaussian ellipse (usually in the range of 0-1);

[0027] The calculation method is: ;in, is the pixel coordinate on the two-dimensional imaging plane, is the coordinate of the center point of the Gaussian ellipse on the two-dimensional imaging plane; Represents the Gaussian ellipse covariance matrix after projection onto the two-dimensional imaging plane;

[0028] Calculate the cumulative depth D of the two-dimensional Gaussian ellipse using the following formula: ,in, Represents the depth value of the center point of the i-th Gaussian ellipse along the line of sight direction, where, in the three-dimensional Gaussian sputtering of this application, along the line of sight direction refers to the direction of the ray formed by starting from the camera optical center (or observation point), passing through a specific pixel point on the image plane, and extending into the three-dimensional scene space.

[0029] Furthermore, the local loss function , the expression is: ,in, is the weight coefficient, is the L1 loss between rendered images and real images, SSIM loss is the structural similarity between the rendered image and the real image; It is the L1 loss between the depth image and the depth prior data predicted in step S3.

[0030] Further, the trained scene blocks are merged to obtain the final three-dimensional Gaussian model, including: merging the trained scene blocks to obtain the three-dimensional Gaussian model; calculating the importance score of each Gaussian ellipsoid according to the three-dimensional Gaussian model : ;in, and represent the opacity and scale of the Gaussian ellipsoid, respectively; Indicates opacity offset; Indicates the cumulative opacity of the corresponding position after removing the Gaussian ellipsoid g; divide the scene into three levels of detail, namely LOD0, LOD1 and LOD2, for image rendering at different viewing distances; set the compression ratios of levels LOD0, LOD1 and LOD2 to ѱ0, ѱ1 and ѱ2 respectively, where ѱ0 is smaller than ѱ1 and smaller than ѱ2, and the compression ratio indicates the proportion of the Gaussian ellipsoid retained; where the value range of ѱ0 is 20% to 30%; the value range of ѱ1 is 30% to 40%; the value range of ѱ2 is 40% to 60%; when rendering a new perspective, the Gaussian ellipsoid in each scene block is scored according to its importance. Sort in descending order; calculate the distance between each scene block and the camera viewpoint; select the corresponding level of detail based on the distance, and according to the level of detail and the corresponding number of Gaussian ellipsoids to participate in the rendering according to the corresponding compression rate.

[0031] Compared with the existing technology, the advantages of this application are:

[0032] (1) Traditional three-dimensional Gaussian models are difficult to apply to large-scale scenes. This application uses spatial grid division technology to decompose large scenes into multiple scene blocks with axis-aligned bounding boxes, and uses parallel GPUs to train the blocks independently, effectively breaking through the limitation of computing resources.

[0033] (2) This application adopts a global-first-local training method. First, the sparse point cloud of the scene is used for global training to obtain the initial three-dimensional Gaussian model, and then the training is refined by block division. This avoids the problem of falling into local optimality in direct refinement training and ensures the consistency of the global geometric structure.

[0034] (3) Monocular depth estimation prior is introduced as a geometric supervision signal. By adding a depth loss term to the local loss function, the position and shape optimization of the Gaussian ellipsoid are guided, which especially improves the geometric reconstruction quality of sparse or repeated texture areas and solves the problem of geometric inaccuracy when relying solely on color information for reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an exemplary flow chart of a large scene 3D reconstruction method based on 3D Gaussian sputtering of the present application;

[0036] Figure 2 This is a rendering of a farmland scene for this application;

[0037] Figure 3 This is a rendering of a river scene in this application;

[0038] Figure 4 This is a rendering of a factory scene for this application;

[0039] Figure 5 A rendering of a residential scene for this application;

[0040] Figure 6 This is a rendering of a bridge scene for this application. DETAILED DESCRIPTION

[0041] The present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] like Figure 1 As shown, a multi-view image set of a large scene is collected; a sparse point cloud of the scene is obtained based on the multi-view image set; a pre-trained depth prediction network is used to perform monocular depth estimation on the multi-view images to obtain a monocular depth estimation prior; the monocular depth estimation prior refers to the depth predicted from a single-view image, which is used to provide geometric prior knowledge of the scene; the scene sparse point cloud and the monocular depth estimation prior are used to perform global training on the scene to obtain an initial three-dimensional Gaussian model, which contains multiple Gaussian ellipsoids; the initial three-dimensional Gaussian model is spatially gridded to obtain multiple scene blocks with axis-aligned bounding boxes; image view data of each scene block is set; the Gaussian ellipsoids in multiple scene blocks are depth-supervised trained using a parallel GPU; the trained scene blocks are merged to obtain a final three-dimensional Gaussian model.

[0043] In detail, in step 1, a drone equipped with a high-resolution color camera was used to capture images of a large-scale outdoor scene at high altitude. The acquisition range was approximately 3 km², and the flight altitude was adjusted based on the scene characteristics, typically maintaining it between 80 and 120 meters. A grid-like or S-shaped flight path was designed to ensure greater than 60% image overlap. Taking into account the sun angle and weather conditions, the images were taken during periods of uniform illumination. The resulting dataset contained approximately 3,000 color images, which were used for subsequent 3D reconstruction.

[0044] S2 uses the Structure from Motion (SfM) method to estimate the camera pose of the dataset in step 1 and obtain a sparse point cloud representation of the scene. This can be divided into the following steps:

[0045] In step 2-1, local feature points and their descriptors are extracted from each image using algorithms such as Scale-Invariant Feature Transform (SIFT) or Oriented FAST and Rotated BRIEF (ORB). These features are scale- and rotation-invariant, and remain stable across different viewing angles and lighting conditions. Feature points are extracted and matched across all images, and their relative positions in space are estimated. Based on the matched feature points, the camera's rotation and translation parameters are further estimated, resulting in an accurate camera pose estimate.

[0046] In step 2-2, using the estimated camera pose, a linear triangulation algorithm is applied to each pair of matching feature points, and multi-view geometric constraints are used to improve the 3D point positioning accuracy and determine the exact position of each matching feature point in 3D space.

[0047] In steps 2-3, bundle adjustment is used to minimize the reprojection error, optimize all camera parameters and 3D point positions, and obtain accurate camera poses and sparse point clouds.

[0048] In step 3, a pre-trained depth prediction network is used to perform monocular depth estimation on the dataset images, calculating the depth prior for each image pixel by pixel. In this application, the prediction network used for monocular depth estimation can be DPTDense Prediction Transformer, MiDaS, BTS (Big to Small), DORN (Deep Ordinal Regression Network), etc.

[0049] A linear regression model is used to calculate the depth scale factor and offset of each image. The scale parameter is optimized through the least squares method to ensure that the depth estimation is consistent with the actual scene scale. The calculated scale transformation is applied to the depth map of each image for scale alignment to ensure that the estimated depth information is consistent with the actual scene scale.

[0050] In particular, the 3D Gaussian sputtering method essentially optimizes scene representation by minimizing the color difference between the rendered image and the ground-truth image. However, this purely color-based optimization suffers from a fundamental problem: the same 2D image may correspond to multiple different 3D structures (depth ambiguity). This application overcomes this limitation through monocular depth estimation.

[0051] For example, in areas lacking rich textures (such as farmland, factory rivers, etc.), it is difficult to accurately restore depth based on color information alone, resulting in the Gaussian ellipsoid being randomly distributed at any position in the line of sight. Monocular depth estimation can provide reasonable depth constraints for these areas through learned scene priors, significantly improving the accuracy of geometric reconstruction. Figure 2 and Figure 3 As shown, when the method of the present application is used to extract the three-dimensional structure of farmland and river channels, the geometric constraints provided by the monocular depth estimation effectively prevent the surface "floating" phenomenon and ensure the rationality and continuity of the surface undulations.

[0052] In addition, in areas with repeated textures (such as window grilles, brick walls, etc.), traditional multi-view matching is prone to produce false correspondences, resulting in incorrect Gaussian ellipsoid positions. The depth prior provides additional structural information, effectively suppressing these matching ambiguities and ensuring the continuity and rationality of the geometric structure. Figure 4 、 Figure 5 and Figure 6 As shown, when the method of the present application is used to perform three-dimensional reconstruction of factories, houses and bridges, the geometric details can be accurately restored even on facades with highly repetitive structures, avoiding the "ghosting" and "overlapping" artifacts commonly seen in traditional methods.

[0053] On the other hand, since monocular depth estimation usually outputs relative depth rather than absolute depth, scale correction is required to align it with the actual scene scale. This application uses a linear regression model and least squares method for scale correction to ensure that the depth estimate is consistent with the actual scene scale.

[0054] In step 4, the scene sparse point cloud and monocular depth estimation prior are used to perform global training on the scene to obtain an initial rough 3D Gaussian model, including: constructing an initial 3D Gaussian field based on the scene sparse point cloud obtained in step S2. The Gaussian field contains multiple Gaussian ellipsoids, and initializing the parameters of the Gaussian ellipsoid: the spatial coordinates (x, y, z) of the 3D point in the point cloud are used as the center point of the Gaussian ellipsoid; the covariance matrix determines the shape, size and direction of the Gaussian ellipsoid, and is expressed in a decomposed form: ; is the rotation matrix of the Gaussian ellipsoid, expressed using quaternions; It is the scale matrix of the Gaussian ellipsoid, represented by a three-dimensional vector, where the value of each dimension represents the scaling ratio of the corresponding dimension;

[0055] According to the color value and opacity, the color value of the rendered pixel of each image perspective is calculated to obtain the rendered image; in the global training of this application, the color value of the rendered image is calculated in the same way as the scene block training below.

[0056] Set the loss function for global training based on the rendered image and the real image :

[0057] ,in, and is the weight coefficient, and They are the L1 loss and structural similarity SSIM loss between rendered images and real images respectively. , where n is the total number of pixels, To render an image, ; , where SSIM is the structural similarity index, and its value range is [0, 1]. The larger the value, the more similar the structure.

[0058] Specifically, this application utilizes a sparse scene point cloud and a monocular depth estimation prior for global training, ensuring that the initial 3D Gaussian model has a globally consistent geometric structure. This global training avoids the geometric incoherence that can result from direct local optimization, providing a unified geometric foundation for subsequent block-by-block refinement training.

[0059] Furthermore, the sparse point cloud generated by SfM is organically combined with the monocular depth estimation prior, leveraging the precision of the point cloud and the density of the depth map to form more comprehensive geometric constraints. This fusion strategy specifically addresses the problem of relying solely on color information to accurately recover the geometric structure of sparsely textured or repetitive areas.

[0060] Finally, although the initial model obtained in the global training stage has limited accuracy, it captures the main structure of the scene and forms a progressive reconstruction strategy from "coarse global modeling" to "fine local optimization", which effectively solves the difficult problem of balancing global consistency and local details in large-scale scene reconstruction.

[0061] S5: For the rough 3D Gaussian field obtained in step 4, the scene is confined to a bounded cubic area through spatial grid division. The area is then evenly divided to achieve reasonable load balancing. Gaussians are generated based on global coarse training. The maximum and minimum coordinates of the Gaussian points in each sub-block are calculated, and its axis-aligned bounding box (Axis-Aligned Bounding Box) is defined to ensure that the boundary of each block completely encapsulates the local geometric details.

[0062] S6, setting the image perspective data of each scene block, including: determining whether the spatial position of the camera is within the axis-aligned bounding box of the scene block; if so, directly dividing the camera's perspective into the image perspective data of the corresponding scene block; if not, calculating the contribution weight of the camera's perspective to the corresponding scene block : ,in, is the pose parameter of the i-th camera, LSSIM represents the calculation of the structural similarity loss between two images, Represents the image rendered using all Gaussian ellipsoids, represents the image rendered after removing the Gaussian ellipsoid in the jth block;

[0063] Determine contribution weight Is it greater than the threshold? If it is greater than, it means that removing the Gaussian ellipsoid in the block will cause a significant change in the rendering result, that is, the block has an important contribution to the rendering result of this perspective, then the perspective will be assigned to this block for training;

[0064] In particular, traditional view selection methods based on frustum-AABB intersection only consider visibility while ignoring contribution. In large scenes, a distant building may be visible from multiple views, but not all views significantly contribute to its reconstruction quality. This application precisely quantifies this importance through contribution weights, allowing limited computing resources to be focused on truly important views. Furthermore, in scenes with complex occlusion relationships (such as narrow streets and densely packed buildings), view allocation based solely on distance or frustum often fails. For example, a close-up view through a partition wall may completely obscure the target area, while a distant view at a suitable angle may provide critical information. This application's contribution calculation directly assesses the visual impact and automatically identifies these unique but important views. Furthermore, this application implements on-demand loading, requiring each GPU to store only the parameters of the relevant tiles, reducing memory requirements from the total scene size to the size of a single tile. This design enables processing of extremely large scenes with limited hardware resources, breaking the scalability limits of traditional methods.

[0065] In step 7, the block-wise Gaussian field is trained using a parallel GPU. During the training process, L1 loss, SSIM loss, and L1 depth loss are used to fine-tune the internal geometric structure of each sub-block. Specifically, for each block-wise Gaussian field, the training can be divided into the following three steps.

[0066] Step 7-1: Transform the three-dimensional Gaussian point cloud projection to the two-dimensional imaging plane, and the covariance matrix of the Gaussian point cloud projected on the two-dimensional screen Expressed as: , where ∑ is the covariance matrix of the three-dimensional Gaussian point cloud, W is the transformation matrix of the viewing angle, and J is the affine approximation Jacobian matrix of the projection transformation.

[0067] The imaging plane is divided into blocks, each consisting of a number of pixels. For each image block, a two-dimensional Gaussian ellipse is extracted that is projected within the pixel range of the image block. These Gaussian ellipses are sorted according to their distance from the imaging plane, and the final pixel color is calculated by accumulating opacity (this method is called - mixed), the specific formula is as follows: ; Where N represents the number of two-dimensional Gaussian ellipses, and Represents the color value and opacity of the i-th Gaussian ellipse at the corresponding viewing angle.

[0068] The calculation method is: ;in, are coordinates in the projected space, is the coordinate of the center point of the Gaussian ellipse in the projection space. When the accumulated opacity reaches the threshold, it stops and the final color value of the pixel is output to obtain the rendered image.

[0069] Step 7-2: Similar to step 7-1, after the projection transformation, use - The blending accumulates the depth of the two-dimensional Gaussian ellipse within the range. The specific formula is as follows: ;in represents the depth value of the center point of the i-th Gaussian ellipse along the line of sight. When the accumulated opacity reaches a threshold, the rendering stops and the final depth image is output. In particular, compared to traditional depth rendering methods, α-blending depth can handle translucency and overlapping structures, which is more consistent with real-world imaging characteristics.

[0070] Step 7-3: Calculate the L1 loss and SSIM loss for the rendered image obtained in step 7-1 and the original image, and get and At the same time, the L1 loss is calculated for the depth map obtained in step 7-2 and the monocular depth estimation predicted in step 3, and we get , calculate the local loss function for: ;in, They are the preset loss weights respectively. Finally, the parameters of the Gaussian point cloud are adjusted iteratively through back propagation to improve the quality of the rendered image and the accuracy of the geometric structure.

[0071] In particular, through The depth map obtained in step 7-2 is obtained by performing depth accumulation on the three-dimensional Gaussian ellipsoid, and the monocular depth estimate predicted in step 3 is scale-corrected to be consistent with the actual scene scale. In this application, L1d is obtained by calculating the L1 loss on the depth map obtained in step 7-2 and the monocular depth estimate predicted in step 3. This loss function provides a direct geometric supervision signal for the three-dimensional Gaussian field, effectively solving the problem of geometric uncertainty in sparsely textured areas, guiding the spatial distribution of the Gaussian ellipsoid to converge to the surface of the real scene, and significantly improving the accuracy and consistency of complex geometric structures in large scene reconstruction, especially in areas that are difficult to handle with traditional methods that rely solely on color information.

[0072] Step 8: Merge the block Gaussian fields. Due to the guidance of the global geometric prior, the inter-block interference is significantly reduced, so directly connecting the Gaussian fields obtained from each training can obtain a high-quality overall model.

[0073] Step 9: Real-time rendering at different scales based on level of detail (LoD) requires compressing the Gaussian field to a corresponding degree at different levels. Specifically, it can be divided into the following two steps:

[0074] Step 9-1, for each Gaussian ellipsoid Calculate an importance score , which is calculated as follows:

[0075] ,in, and represent the opacity and scale of the Gaussian ellipsoid, respectively; Indicates opacity offset; Represents the cumulative opacity of the corresponding position after removing the Gaussian ellipsoid g. The reason for designing this formula is to consider: (1) Points with high opacity are more likely to make significant contributions to the final image, and completely transparent points can be directly filtered out, which is in line with the principle of early culling in the rendering pipeline and effectively reduces subsequent calculations. (2) Weigh the unique contribution of points, avoid retaining redundant and occluded points, and retain points that have a substantial impact on the rendering results; (3) Larger points cover more pixels and should be given higher importance;

[0076] In step 9-2, the scene is divided into three levels of detail: LoD 0 (far), LoD 1 (medium distance), and LoD 2 (near). Each LoD level corresponds to a different compression rate, preferably 25%, 34%, and 50%. When rendering a new view, the Gaussian ellipsoid in the Gaussian field of each block in the view is scored according to its importance. Sort in descending order, select the corresponding level of detail according to the distance between the block and the camera viewpoint, and select the number of Gaussian ellipsoids with the corresponding compression rate to participate in the rendering, so as to achieve rendering of different levels of detail and improve the rendering speed. Among them, the distance between the calculated block and the camera viewpoint in this application refers to the distance between the camera position and the axis-aligned bounding box of the scene block, which can be achieved by: calculating the Euclidean distance between the camera position and the center point of the AABB, or calculating the shortest distance from the camera position to the AABB surface. This application dynamically selects models with different compression rates to participate in the rendering according to the viewing distance, which greatly reduces unnecessary computational overhead while ensuring visual quality.

[0077] The invention of the present application and its implementation methods are described schematically above. This description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention of the present application, and the actual structure is not limited to this. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of the present application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. Words such as first and second are used to indicate names and do not indicate any specific order.

Claims

1. A large scene 3D reconstruction method based on 3D Gaussian sputtering, characterized in that: include: Collect multi-view image sets of large scenes; Obtain scene sparse point cloud based on multi-view image sets; Using a pre-trained depth prediction network to perform monocular depth estimation on multi-view images to obtain a monocular depth estimation prior; the monocular depth estimation prior refers to the depth predicted from a single view image, which is used to provide geometric prior knowledge of the scene; Using the scene sparse point cloud and monocular depth estimation prior, the scene is globally trained to obtain an initial three-dimensional Gaussian model, wherein the three-dimensional Gaussian model includes multiple Gaussian ellipsoids; Performing spatial grid division on the initial three-dimensional Gaussian model to obtain multiple scene blocks with axis-aligned bounding boxes; setting image view data for each scene block; Use parallel GPUs to perform deep supervised training on Gaussian ellipsoids within multiple scene blocks; Merge the trained scene blocks to obtain the final 3D Gaussian model; Deeply supervised training of Gaussian ellipsoids within multiple scene blocks using parallel GPUs, including: Assign multiple scene blocks to multiple GPUs for parallel processing, with each GPU responsible for training one or more scene blocks; The Gaussian ellipsoid in each scene block is projected onto the two-dimensional imaging plane according to the assigned image viewing angle to obtain a two-dimensional Gaussian ellipse; The projected two-dimensional imaging plane is divided into blocks to obtain multiple image blocks, each of which contains multiple pixel points; Extracting the two-dimensional Gaussian ellipses within the pixel range of each image block, and sorting the two-dimensional Gaussian ellipses according to the distance between the center point of the two-dimensional Gaussian ellipse and the two-dimensional imaging plane; According to the sorting results of the two-dimensional Gaussian ellipse, the color C of the rendered image and the cumulative depth D of the two-dimensional Gaussian ellipse are calculated respectively through α-blending method; Setting the local loss function for scene segments ; According to the local loss function ,The parameters of the Gaussian ellipsoid in each scene block are iteratively adjusted through back-propagation to obtain the trained scene blocks.

2. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 1, characterized in that: Get the scene sparse point cloud, including: Extract feature points of multi-view images and match the extracted feature points; Obtain camera pose estimation based on the matched feature points; According to the camera pose estimation, a linear triangulation algorithm is used to perform multi-view geometric constraints on each pair of matching feature points to determine the position of each feature point in three-dimensional space; The bundle adjustment method is used to minimize the reprojection error, and the camera pose estimation and the position of each feature point in 3D space are jointly optimized to obtain a sparse point cloud of the scene.

3. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 1, characterized in that: Get the monocular depth estimation prior, including: Based on multi-view images, a pre-trained depth prediction network is used to obtain the original depth prediction value of each image; According to the sparse point cloud of the scene, a linear regression model is used to scale the original depth prediction value to obtain the depth scale factor and offset; The depth scale factor and offset are optimized by the least squares method to align the depth prediction value with the scene sparse point cloud in spatial scale; The original depth prediction value is scaled using the optimized depth scale factor and offset to obtain a monocular depth estimation prior that is consistent with the actual scene scale.

4. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 3, characterized in that: Perform global training on the scene to obtain the initial 3D Gaussian model, including: Construct a three-dimensional Gaussian field based on the scene's sparse point cloud. The Gaussian field contains multiple Gaussian ellipsoids. The parameters of each Gaussian ellipsoid include the center point, covariance matrix, color value, and opacity. Calculate the color value of the rendered pixel at each image viewing angle according to the color value and opacity to obtain a rendered image; Set the loss function for global training based on the rendered image and the real image ; Optimize the parameters of the Gaussian ellipsoid by minimizing the global training loss function; According to the optimized Gaussian ellipsoid parameters, an initial three-dimensional Gaussian model is constructed.

5. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 4, characterized in that: The initial 3D Gaussian model is spatially meshed to obtain multiple scene blocks with axis-aligned bounding boxes, including: According to the initial three-dimensional Gaussian model, the spatial boundary of the Gaussian model is set; Gridding the area formed by the spatial boundary to generate multiple spatial sub-blocks; Determine the spatial sub-block to which each Gaussian ellipsoid belongs according to the coordinates of the center point of the Gaussian ellipsoid; Calculate the maximum and minimum values ​​of the Gaussian ellipsoid within each spatial sub-block, and use the maximum and minimum values ​​as the geometric boundaries of the corresponding spatial sub-block; According to the geometric boundary, set the axis-aligned bounding box AABB of each spatial sub-block; According to the axis-aligned bounding box AABB, the Gaussian ellipsoid parameters contained in each bounding box are assigned to obtain multiple scene blocks.

6. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 5, characterized in that: Set the image perspective data for each scene block, including: Based on the obtained camera pose estimate and the axis-aligned bounding box AABB of the scene block, determine whether the spatial position of each camera is located within the axis-aligned bounding box of the corresponding scene block; if so, assign the image captured by the camera and the corresponding monocular depth estimation prior data to the scene block for training; If not, calculate the contribution weight of the camera's perspective to the corresponding scene block : in, is the pose parameter of the i-th camera, LSSIM represents the calculation of the structural similarity loss between two images, Represents the image rendered using all Gaussian ellipsoids, represents the set of Gaussian ellipsoids after removing the Gaussian ellipsoid in the j-th scene block; Determine contribution weight Is it greater than the threshold? If yes, the image captured by the corresponding camera and the corresponding monocular depth estimation prior data are assigned to the corresponding scene block for training.

7. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 1, characterized in that: Calculate the color C of the rendered image using the following formula: ; Among them, N represents the number of two-dimensional Gaussian ellipses involved in image rendering, Represents the color value of the i-th Gaussian ellipse at the corresponding viewing angle; Indicates the opacity of the i-th Gaussian ellipse at the corresponding viewing angle; Represents the opacity value of the j-th Gaussian ellipse; The calculation method is: ; in, is the pixel coordinate on the two-dimensional imaging plane, is the coordinate of the center point of the Gaussian ellipse on the two-dimensional imaging plane; Represents the Gaussian ellipse covariance matrix after projection onto the two-dimensional imaging plane; Calculate the cumulative depth D of the two-dimensional Gaussian ellipse using the following formula: ; in, Represents the depth value of the center point of the i-th Gaussian ellipse along the line of sight.

8. The large-scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 1, characterized in that: Local loss function , the expression is: ; in, is the weight coefficient, is the L1 loss between rendered images and real images, SSIM loss is the structural similarity between the rendered image and the real image; is the L1 loss between the cumulative depth D and the monocular depth prior estimate.

9. The large scene 3D reconstruction method based on 3D Gaussian sputtering according to claim 1, characterized in that: Merge the trained scene blocks to obtain the final 3D Gaussian model, including: Merge the trained scene blocks to obtain a 3D Gaussian model; According to the three-dimensional Gaussian model, calculate the importance score of each Gaussian ellipsoid : ； in, and represent the opacity and scale of the Gaussian ellipsoid, respectively; Indicates opacity offset; Indicates the cumulative opacity of the corresponding position after removing the Gaussian ellipsoid g; Divide the scene into three levels of detail, LOD0, LOD1, and LOD2, for image rendering at different viewing distances; Set the compression ratios of levels LOD0, LOD1, and LOD2 to 0, 1, and 2 respectively, where 0 is less than 1 and less than 2. The compression ratio indicates the proportion of the Gaussian ellipsoid that is retained. When rendering a new perspective, for each Gaussian ellipsoid in the scene block, the importance score Sort in descending order; Calculate the distance between each scene block and the camera viewpoint; The corresponding detail level is selected according to the distance, and according to the detail level, the corresponding number of Gaussian ellipsoids are selected to participate in the rendering according to the corresponding compression rate.

Images