A three-dimensional scene reconstruction method and system based on three-dimensional Gaussian splashing

Abstract

The application relates to a three-dimensional scene reconstruction method and system based on three-dimensional Gaussian splashing, wherein the method performs density self-adaptive block on a to-be-reconstructed scene through a sparse point cloud, constructs a directed bounding box for each sub-block to form a training domain of each sub-block, assigns a corresponding training view set to the training domain of each sub-block based on the visibility relationship of the sparse point cloud and performs perspective balance sampling to perform three-dimensional reconstruction on each sub-block, divides the region of all overlapping three-dimensional Gaussian primitives in the overlapping region of adjacent sub-blocks, deletes the overlapping three-dimensional Gaussian primitives in the exclusion area, dynamically filters the overlapping three-dimensional Gaussian primitives in the competition area according to the competition score result of the overlapping three-dimensional Gaussian primitives, realizes cross-block random stitching based on the overlapping three-dimensional Gaussian primitives in the competition area after dynamic filtering and the overlapping three-dimensional Gaussian primitives in the core area, and generates a three-dimensional reconstruction model of the to-be-reconstructed scene. Thus, the application realizes seamless stitching of the three-dimensional reconstruction model and guarantees continuity.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a 3D scene reconstruction method and system based on 3D Gaussian splashing. Background Technology

[0002] In recent years, the 3D Gaussian splashing method has become an important technique for large-scale scene 3D reconstruction. The 3D Gaussian splashing method uses explicit Gaussian primitives to represent the scene, offering advantages such as fast rendering speed and high reconstruction efficiency. However, its explicit representation also leads to a rapid increase in model size as the scene scope expands, often resulting in high GPU and storage overhead in large-scale scenes. To adapt to the needs of single-card training and large-scale scene reconstruction, existing strategies typically employ spatial partitioning, dividing the scene into multiple sub-blocks for separate training, and then merging the results of each sub-block. While this strategy alleviates resource pressure during the training phase to some extent, the merged results from each sub-block exhibit cracks, inconsistencies in depth, and flickering under changing viewpoints on continuous surfaces. Furthermore, independent training of different sub-blocks introduces inconsistencies in color and lighting distribution, resulting in noticeable seams at the boundaries of sub-blocks, affecting the overall continuity and usability of large-scale scene models. Summary of the Invention

[0003] The technical problem to be solved by the present invention is: the present invention provides a three-dimensional scene reconstruction method and system based on three-dimensional Gaussian splashing, which realizes seamless stitching of three-dimensional reconstruction models and ensures the continuity and usability of three-dimensional reconstruction models.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for reconstructing a three-dimensional scene based on three-dimensional Gaussian splashing, comprising: Acquire multi-view images of the scene to be reconstructed, calculate the sparse point cloud of the scene to be reconstructed based on the multi-view images, perform density adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks, and construct a directed bounding box for each sub-block to form the training domain of each sub-block. Based on the visibility relationship of the sparse point cloud, a corresponding training view set is assigned to the training domain of each sub-block, and view equalization sampling is performed on the training view set corresponding to each sub-block to obtain the equalized view set of each sub-block. Based on the equalized view set of each sub-block, 3D reconstruction is performed on each sub-block to obtain the 3D reconstruction result of each sub-block. The 3D reconstruction result includes a 3D Gaussian set. Based on the set of 3D Gaussian elements of each sub-block, all overlapping 3D Gaussian elements in the overlapping area of ​​adjacent sub-blocks are obtained. The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated. Based on the normalized distance, the corresponding overlapping 3D Gaussian elements are divided into regions to obtain the region division results. The region division results include the core region, the competition region, and the culling region. Overlapping 3D Gaussian elements in the rejection area are deleted, and overlapping 3D Gaussian elements in the competition area are competitively scored to obtain a competition score result. Based on the competition score result, overlapping 3D Gaussian elements in the competition area are dynamically filtered to obtain overlapping 3D Gaussian elements in the competition area after dynamic filtering. Based on the overlapping 3D Gaussian elements in the competition area after dynamic filtering and overlapping 3D Gaussian elements in the core area, cross-block random stitching is performed to generate a 3D reconstruction model of the scene to be reconstructed.

[0005] The beneficial effects of this invention are as follows: Based on the sparse point cloud of the scene to be reconstructed calculated from multi-view images, the scene to be reconstructed is divided into density-adaptive blocks, breaking the limitations of traditional uniform block division. This ensures the reconstruction accuracy of scene details while reducing computational redundancy. Based on the visibility relationship of the sparse point cloud, a corresponding training view set is allocated to the training domain of each sub-block, ensuring that the allocated training view set can cover the training domain of the sub-block and avoiding interference introduced by invalid views. At the same time, view equalization sampling is performed on the training view set to solve the sub-block reconstruction distortion caused by uneven view distribution in traditional view sampling. By extracting and calculating the normalized distances of all overlapping 3D Gaussian elements within the overlapping regions of adjacent sub-blocks relative to their respective directed bounding boxes, the regions are divided. Overlapping 3D Gaussian elements in the exclusion region are deleted, reducing the amount of invalid computation in the cross-block stitching process. Overlapping 3D Gaussian elements in the competitive region are dynamically filtered through competitive scoring and randomly stitched across blocks with overlapping 3D Gaussian elements in the core region, achieving seamless stitching between adjacent sub-blocks. This eliminates discontinuities such as discontinuities, ghosting, and redundancy at the boundaries of adjacent sub-blocks during block reconstruction, ensuring the continuity and usability of the 3D reconstruction model.

[0006] Optionally, the step of performing density-adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks includes: The sparse point cloud is divided into voxel grids to obtain M voxel grids. The volume of each voxel grid and the total number of sparse points in each voxel grid are calculated. The local density of each voxel grid is calculated based on the total number of sparse points and the volume. When the local density is lower than the first lower limit density, the voxel mesh corresponding to the local density that is lower than the first lower limit density is merged with the adjacent voxel mesh to obtain a merged voxel mesh. It is determined whether the local density of the merged voxel mesh is lower than the first lower limit density. If so, the merged voxel mesh is merged with the adjacent voxel mesh a second time until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. When the local density exceeds the first upper limit density, the voxel mesh corresponding to the local density that exceeds the first upper limit density is divided into a second voxel mesh to obtain a voxel mesh after the second voxel mesh division. It is determined whether the local density of the voxel mesh after the second voxel mesh division exceeds the first upper limit density. If so, the voxel mesh after the second voxel mesh division is divided into a third voxel mesh until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. When the local density is between the first lower limit density and the first upper limit density, the voxel mesh is divided into blocks according to a preset size to obtain N sub-blocks.

[0007] As described above, by dividing the sparse point cloud into voxel grids and calculating the local density of each voxel grid, the sparsity differences in the spatial point cloud distribution in the scene to be reconstructed can be accurately identified. Voxel grids corresponding to local densities below a first lower limit are merged to reduce the number of voxel grids and lower redundant computational and storage overhead in subsequent 3D reconstruction. Voxel grids corresponding to local densities exceeding a first upper limit are further subdivided into voxel grids, i.e., further refined, which can completely preserve detailed features. This ensures that the final N sub-blocks will not suffer from detail loss due to overly coarse voxel grids, nor from invalid redundancy due to overly dense voxel grids, achieving the dual goals of detail fidelity and computational optimization.

[0008] Optionally, constructing a directed bounding box for each sub-block to form the training domain for each sub-block includes: Based on the multi-view images, calculate the extrinsic and intrinsic parameters of the observation camera corresponding to the scene to be reconstructed. The extrinsic parameters include the camera position, and the intrinsic parameters include the focal length. For each sub-block, a directed bounding box is constructed, and the average distance from the directed bounding box of each sub-block to the camera position is calculated. Based on the focal length, the spatial pixel size corresponding to the imaging resolution of the observation camera at each average distance is calculated. The spatial pixel size is then input into an expansion formula for calculation to obtain the expansion coefficient corresponding to each sub-block. The expansion formula is as follows: ; in, This represents the expansion coefficient of sub-block i. Represents the empirical coefficient. Indicates the spatial pixel size of sub-block i; The directed bounding box of each sub-block is expanded according to the corresponding expansion coefficient to form the training domain of each sub-block.

[0009] As described above, the expansion coefficients corresponding to each sub-block are dynamically calculated based on the external and internal parameters of the observation camera, thereby enabling dynamic adjustment of the training domain of each sub-block and avoiding the problems of boundary features being interrupted or blurred caused by the traditional fixed training domain.

[0010] Optionally, the step of assigning a corresponding training view set to the training domain of each sub-block based on the visibility relationship of the sparse point cloud, and performing view equalization sampling on the training view set corresponding to each sub-block to obtain an equalized view set for each sub-block, and performing 3D reconstruction on each sub-block based on the equalized view set for each sub-block, to obtain the 3D reconstruction result of each sub-block includes: Obtain the sparse point set of each sub-block within the training domain, and simultaneously obtain the view point set of all views in the multi-view image. Input the view point set and the sparse point set into the training constraints to allocate a corresponding training view set to the training domain of each sub-block. The training constraints are as follows: ; in, This represents the training view set corresponding to sub-block i. This represents the v-th view in a multi-view image. Represents all multi-view images, Let v represent the set of viewpoints observed by the v-th view in a multi-view image. This represents the sparse point set of sub-block i within the training domain. This indicates the preset view coverage threshold. This indicates the preset sub-block coverage threshold. Represents the logical OR operator; For each view in the training view set corresponding to each sub-block, key features are extracted to obtain the key parameter vector of each view relative to the corresponding sub-block. The corresponding view is then parameterized based on the key parameter vector to obtain the parameterized view. The key parameter vector is the azimuth angle and the center distance. The parameterized view is discretized into bins according to the preset azimuth angle rule and the preset center distance rule to obtain all view bins. A round-robin sampling strategy is used to randomly sample the same number of views from each view bin and merge them to obtain a balanced view set for each sub-block. The sparse point set of each sub-block in the training domain is used as the spatial constraint, and the balanced view set of each sub-block is used as the appearance constraint to perform 3D reconstruction of each sub-block, so as to obtain the 3D reconstruction result of each sub-block.

[0011] As described above, the sparse point set of each sub-block within the training domain and the view point set of all views in the multi-view image are input into the training constraints to automatically select a training view set that effectively covers the training domain for each sub-block. After parameterizing the azimuth angle and center distance for each view in the training view set corresponding to each sub-block, the view is discretized and binned. A round-robin sampling strategy is then used to obtain a balanced view set for each sub-block, achieving a balanced distribution of views in terms of azimuth angle and center distance, thus eliminating viewpoint bias in view sampling.

[0012] Optionally, the step of performing 3D reconstruction on each sub-block using the sparse point set of each sub-block in the training domain as spatial constraints and the balanced view set of each sub-block as appearance constraints, to obtain the 3D reconstruction result of each sub-block includes: The 3D Gaussian primitive parameters of each sub-block are initialized based on the balanced view set of each sub-block and the sparse point set of each sub-block in the training domain. Each sparse point in the sparse point set of each sub-block in the training domain is taken as the center position of the initial Gaussian primitive. The projection coordinates corresponding to the center position of each initial Gaussian primitive are obtained based on the balanced view set of each sub-block. The pixel color of each projection coordinate is obtained by bilinear interpolation, and the average value of all pixel colors is calculated. The average value is taken as the initial spherical harmonic DC component of the corresponding sub-block. The average distance from the center position of each initial Gaussian primitive to the k nearest neighbor sparse points is calculated by the K nearest neighbor search method. The average distance is taken as the initial scale of the initial Gaussian primitive. The center position, the initial spherical harmonic DC component, and the initial scale are combined with preset opacity and preset rotation number to construct the three-dimensional Gaussian element parameters after each sub-block is initialized. The initialized 3D Gaussian meta-parameters of each sub-block are input into a pre-trained Gaussian splash renderer and combined with the reconstruction loss function to independently optimize the initialized 3D Gaussian meta-parameters of each sub-block, thereby obtaining the 3D reconstruction result of each sub-block.

[0013] As described above, directly using each sparse point in the sparse point set of each sub-block within the training domain as the center position of the initial Gaussian primitive ensures the accuracy of the center position and the rationality of the spatial distribution. The average pixel color is obtained through bilinear interpolation and used as the initial spherical harmonic DC component of the corresponding sub-block. The average distance calculated using the K-nearest neighbor search method is used as the initial scale of the initial Gaussian primitive, thus initializing the 3D Gaussian primitive parameters and binding them to the real geometric and texture information of the scene to be reconstructed. The center position, initial spherical harmonic DC component, and initial scale are combined with preset opacity and preset rotation number to construct complete initialized 3D Gaussian primitive parameters. A pre-trained Gaussian splash renderer and reconstruction loss function are used to independently optimize the initialized 3D Gaussian primitive parameters of each sub-block, ensuring the accuracy of the 3D reconstruction results for each sub-block.

[0014] Optionally, the step of calculating the normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block, and performing region partitioning on the corresponding overlapping 3D Gaussian elements based on the normalized distance, to obtain the region partitioning results includes: The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated using a first distance formula, which is: ; in, Let p represent the normalized distance of the overlapping 3D Gaussian element at position p relative to the directed bounding box of its sub-block i. Indicates The components are a diagonal matrix with diagonal elements. Let represent the half-side length of the directed bounding box of sub-block i. Let represent the transpose of the rotation matrix of the directed bounding box of sub-block i. This represents an overlapping 3D Gaussian element at position p. Let represent the vector of the center point of the directed bounding box of sub-block i. The infinite norm of a vector; Based on the normalized distance and the region partitioning rule, the corresponding overlapping 3D Gaussian elements are partitioned into regions to obtain the region partitioning results. The region partitioning rule is as follows: .

[0015] As described above, overlapping 3D Gaussian primitives are not directly cut off. Instead, the overlapping 3D Gaussian primitives are divided into regions by the normalized distance calculated by the first distance formula and the region division rules, so as to achieve scientific and hierarchical screening of overlapping 3D Gaussian primitives.

[0016] Optionally, the competition scoring result includes its own competition scoring result and the neighborhood competition scoring result. The step of simultaneously performing competition scoring on overlapping 3D Gaussian elements within the competition region to obtain the competition scoring result, and then dynamically filtering the overlapping 3D Gaussian elements within the competition region based on the competition scoring result, includes: Obtain the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region. Input the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region into the quality formula for calculation, and obtain the self-quality score of each overlapping 3D Gaussian element within the competition region. The quality formula is: ; in, Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. The opacity of the overlapping 3D Gaussian element at position x within the competition region is represented by . Let x represent the logarithmic scaling vector of overlapping three-dimensional Gaussian elements at position x within the contention zone. This represents the actual scale vector of the overlapping 3D Gaussian elements at position x within the competition zone. This represents the average value. This represents the first positive number that is preset. The self-quality score of each overlapping 3D Gaussian element within the competition region is input into the first competition formula for competition scoring, resulting in the self-competition score of each overlapping 3D Gaussian element within the competition region. The first competition formula is as follows: ; ; in, This represents the self-competition score of the overlapping 3D Gaussian element at position x within the competition zone. Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. Indicates the distance decay weight. This represents the normalized distance of the overlapping 3D Gaussian element at position x relative to the directed bounding box of its sub-block i; Obtain the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region. Then, use the KD-Tree spatial indexing algorithm to find the k nearest neighbors of the corresponding overlapping 3D Gaussian element from all original 3D Gaussian elements in the neighboring sub-blocks. This yields a neighborhood candidate set for each overlapping 3D Gaussian element within the competition region. Calculate the neighborhood quality score for each candidate 3D Gaussian element in the neighborhood candidate set using the quality formula. Input the neighborhood quality score into a second competition formula for competition scoring to obtain the neighborhood competition score result for the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region. The second competition formula is: ; in, This represents the neighborhood competition score of the adjacent sub-block j of the sub-block to which the overlapping 3D Gaussian element at position x belongs within the competition region. Indicates the distance decay weight. This represents the average value. Let represent the neighborhood quality score of the candidate 3D Gaussian element at position y in the neighborhood candidate set. This represents the k nearest neighboring 3D Gaussian elements to the sub-block j of the overlapping 3D Gaussian element at position x within the competition zone, obtained through the KD-Tree spatial indexing algorithm. The self-competition score of each overlapping 3D Gaussian element within the competition region and its corresponding neighborhood competition score are input into a probability formula for calculation, yielding the retention probability of each overlapping 3D Gaussian element within the competition region. The probability formula is as follows: ; in, Let x represent the retention probability of an overlapping 3D Gaussian element at position x within the competition region. This represents the sigmoid function. This indicates the boundary sharpness control parameter. This represents the self-competition score of an overlapping 3D Gaussian element at position x within the competition zone. The maximum neighborhood quality score is represented in the neighborhood candidate set of the neighboring sub-block j of the overlapping 3D Gaussian element at position x within the competition zone. Bernoulli sampling is performed based on the retention probability to achieve dynamic filtering of overlapping 3D Gaussian elements within the competing region.

[0017] As described above, the overlapping 3D Gaussian elements within the competition region are evaluated not only based on their own competition scores but also by incorporating neighborhood competition scores. This multi-dimensional competition scoring accurately identifies the overlapping 3D Gaussian elements that contribute the most to boundary smoothness. By calculating the retention probability of each overlapping 3D Gaussian element using a probability formula and performing Bernoulli sampling, the overlapping 3D Gaussian elements within the competition region are no longer subject to simple, fixed-rule filtering but rather to a selective retention filtering with randomness. This ensures a more natural and smooth boundary transition in subsequent cross-block random stitching.

[0018] Optionally, it also includes: Obtain the opacity and actual scale vector of each reconstructed 3D Gaussian primitive in the 3D reconstruction model, and input the opacity and actual scale vector of each reconstructed 3D Gaussian primitive into optimization conditions in sequence to optimize the 3D reconstruction model and obtain a full-scene model. The optimization conditions are: ; in, Represents the full-scene model. Represents a 3D reconstruction model. Indicates the location in the 3D reconstruction model as Reconstructing three-dimensional Gaussian elements, Represents a logarithm. Indicates the location in the 3D reconstruction model as The opacity of the reconstructed 3D Gaussian elements. This indicates the preset opacity threshold. Indicates the location in the 3D reconstruction model as The reconstructed logarithmic scaling vector of the three-dimensional Gaussian element. Indicates the location in the 3D reconstruction model as The actual scale vector of the reconstructed 3D Gaussian element. The infinite norm of a vector. This indicates the preset scale threshold. This represents the logical AND operator; The optimization of the full-scene model is achieved by freezing the geometric parameters of all full-scene 3D Gaussian primitives in the full-scene model and introducing pseudo-quantization operators and minimum loss functions to the appearance parameters of all full-scene 3D Gaussian primitives in the full-scene model. The geometric parameters include position parameters, rotation parameters, and a logarithmic scaling vector; the appearance parameters include opacity, spherical harmonic DC component, and higher-order residual spherical harmonic coefficients; and the minimum loss function is: ; ; ; in, Let represent the minimum loss function introduced for the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents taking the expected value for all multi-view images. This represents the v-th view in a multi-view image. Represents all multi-view images, Represents the loss function. This represents a differentiable Gaussian splash rendering function. Indicates the position in the full scene model. The opacity of the full-scene 3D high-order primitives. This represents the set of opacities of all 3D Gaussian primitives in the full-scene model. Indicates the position in the full scene model. The spherical harmonic DC component of the full-scene 3D Gaussian element. This represents the set of spherical harmonic DC components of all 3D Gaussian elements in the full-scene model. Indicates the position in the full scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates that the position in the full scene model is The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements are introduced with an 8-bit pseudo-quantization operator. Indicates the position in the full scene model. The high-order residual spherical harmonic coefficients after introducing pseudo-quantization operators in the full-scene 3D Gaussian primitives This represents the set of higher-order residual spherical harmonic coefficients of all full-scene 3D Gaussian elements after introducing pseudo-quantization operators in the full-scene model. Represents the full-scene model. This represents the geometric parameters of all frozen 3D Gaussian primitives in the full-scene model. This represents the real image of the v-th view in a multi-view image.

[0019] As described above, the optimization of the 3D reconstruction model is achieved by automatically removing invalid full-scene 3D Gaussian primitives with opacity below a preset threshold and correcting full-scene 3D Gaussian primitives with abnormal actual scale vectors. By freezing the geometric parameters of all full-scene 3D Gaussian primitives in the full-scene model and only introducing pseudo-quantization operators and minimum loss functions to the appearance parameters, the optimization of the full-scene model is achieved. This ensures that the overall geometric structure of the full-scene model remains unchanged while further refining and standardizing the appearance details, thereby improving the quality of the optimized full-scene model.

[0020] Optionally, the optimized full-scene model includes: The higher-order residual spherical harmonic coefficients of each full-scene 3D Gaussian element in the optimized full-scene model are input into the spectrum formula of the third-order spherical harmonic order for calculation, to obtain the third-order spectral energy of each full-scene 3D Gaussian element in the optimized full-scene model. The spectrum formula of the third-order spherical harmonic order is as follows: ; ; in, This indicates the position in the optimized full-scene model. The full-scene 3D Gaussian element's first Spectral energy of spherical harmonic order Let o represent the spherical harmonic order, and o represent the position in the optimized full-scene model. Full-scene 3D Gaussian elements, Indicates the first The set of indices of the order of spherical harmonics. This indicates the position from the optimized full-scene model. The first spherical harmonic coefficient extracted from the high-order residuals of the full-scene 3D Gaussian elements The set of indices of the order of spherical harmonics. The norm of vector L1 is used to represent the vector L1. The third-order spectral energy of each 3D Gaussian element in the optimized full-scene model is aggregated using an aggregation formula to obtain the high-frequency energy of each 3D Gaussian element in the optimized full-scene model. The aggregation formula is as follows: ; in, This indicates the position in the optimized full-scene model. The high-frequency energy of the full-scene three-dimensional Gaussian elements, Indicates the preset first Weights of spherical harmonic orders; The high-frequency energy input gating compression formula of each full-scene 3D Gaussian element in the optimized full-scene model is subjected to spectral gating compression to obtain a spectral-gated full-scene model. The spectral-gated full-scene model is then subjected to hierarchical storage according to a preset hierarchical storage rule to obtain a lightweight full-scene model. The gating compression formula is: ; in, This indicates the position in the full-scene model after spectrum-gated compression. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates the position in the optimized full-scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, Indicates an indicator function, This indicates the preset spectrum gating threshold.

[0021] As described above, the spectrum formula and aggregation formula of the third-order spherical harmonic can accurately identify the full-scene 3D Gaussian primitives carrying rich high-frequency information in the optimized full-scene model. Combined with the gating compression formula, adaptive spectrum gating compression is achieved, preserving key details of the full-scene model while removing redundant information. By introducing hierarchical storage rules to perform hierarchical storage on the spectrum-gated compressed full-scene model, not only is storage space optimized, but subsequent loading efficiency is also improved.

[0022] In a second aspect, the present invention provides a three-dimensional scene reconstruction system based on three-dimensional Gaussian splashing, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing described in the first aspect.

[0023] The technical effects of the three-dimensional scene reconstruction system based on three-dimensional Gaussian splashing provided in the second aspect are the same as those of the three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing provided in the first aspect. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing provided in this embodiment; Figure 2 This is a schematic diagram of the overall process of a three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing provided in this embodiment; Figure 3 These are comparison images of the effects in this embodiment; Figure 4 This is a schematic diagram illustrating the dynamic filtering of overlapping three-dimensional Gaussian elements within the competition region based on the competition scoring results, as described in this embodiment. Figure 5 This is a schematic diagram of the structure of a three-dimensional scene reconstruction system based on three-dimensional Gaussian splashing provided in this embodiment.

[0025] [Explanation of Labels in the Attached Image] 1. A 3D scene reconstruction system based on 3D Gaussian splashing; 2. Processor; 3. Memory. Detailed Implementation

[0026] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0027] Example 1 Please refer to Figures 1 to 4 This invention provides a method for reconstructing a 3D scene based on 3D Gaussian splashing, comprising the following steps: S1. Obtain multi-view images of the scene to be reconstructed, calculate the sparse point cloud of the scene to be reconstructed based on the multi-view images, perform density adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks, and construct a directed bounding box for each sub-block to form the training domain of each sub-block. In this embodiment, as Figure 2 As shown, a sparse point cloud of the scene to be reconstructed is calculated based on the acquired multi-view images of the scene to be reconstructed. The sparse point cloud is calculated using the SIM (Structure from Motion) algorithm. Based on the sparse point cloud, the scene to be reconstructed is subjected to density adaptive segmentation to obtain N sub-blocks. A directed bounding box is constructed for each sub-block to form the training domain of each sub-block. The directed bounding box is constructed using an existing method.

[0028] At this point, the step S1, which involves performing density-adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks, includes: S11. Divide the sparse point cloud into voxel grids to obtain M voxel grids. Calculate the volume of each voxel grid and the total number of sparse points in each voxel grid. Calculate the local density of each voxel grid based on the total number of sparse points and the volume. S12. When the local density is lower than the first lower limit density, the voxel mesh corresponding to the local density that is lower than the first lower limit density is merged with the adjacent voxel mesh to obtain the merged voxel mesh. It is determined whether the local density of the merged voxel mesh is lower than the first lower limit density. If so, the merged voxel mesh is merged with the adjacent voxel mesh a second time until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. S13. When the local density exceeds the first upper limit density, perform a second voxel mesh division on the voxel mesh corresponding to the local density that exceeds the first upper limit density to obtain a voxel mesh after the second voxel mesh division. Determine whether the local density of the voxel mesh after the second voxel mesh division exceeds the first upper limit density. If so, perform a third voxel mesh division on the voxel mesh after the second voxel mesh division until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. S14. When the local density is between the first lower limit density and the first upper limit density, the voxel mesh is divided into blocks according to a preset size to obtain N sub-blocks.

[0029] In this embodiment, the sparse point cloud is divided into voxel grids to obtain M voxel grids. The volume of each voxel grid and the total number of sparse points within each voxel grid are calculated to determine the local density of each voxel grid. When the local density is lower than a first lower limit density, the voxel grid corresponding to the local density below the first lower limit density is merged with the adjacent voxel grids. If the local density of the merged voxel grid is still lower than the first lower limit density, the merged voxel grid is merged again with the adjacent voxel grids at this time, until the local density of the final voxel grid is between the first lower limit density and the first upper limit density. When the local density exceeds the first upper limit density, the voxel grid corresponding to the local density exceeding the first upper limit density is divided into voxel grids a second time to obtain a voxel grid after the second voxel grid division, i.e., it is subdivided again. If the local density of the voxel grid after the second voxel grid division is still higher than the first upper limit density, it is subdivided again, i.e., a third voxel grid division is performed, until the local density of the final voxel grid is between the first lower limit density and the first upper limit density. All voxel meshes with local densities between the first lower limit density and the first upper limit density are divided into N sub-blocks according to a preset size. The first lower limit density and the first upper limit density are set according to the actual situation.

[0030] At this point, the construction of a directed bounding box for each sub-block to form the training domain of each sub-block in step S1 includes: S15. Calculate the extrinsic and intrinsic parameters of the observation camera corresponding to the scene to be reconstructed based on the multi-view image. The extrinsic parameters include the camera position, and the intrinsic parameters include the focal length. S16. Construct a directed bounding box for each sub-block and calculate the average distance from the directed bounding box of each sub-block to the camera position. Based on the focal length, calculate the spatial pixel size corresponding to the imaging resolution of the observation camera at each average distance. Input the spatial pixel size into the expansion formula for calculation to obtain the expansion coefficient corresponding to each sub-block. The expansion formula is: ; in, This represents the expansion coefficient of sub-block i. Represents the empirical coefficient. Indicates the spatial pixel size of sub-block i; S17. Expand the directed bounding box of each sub-block according to the corresponding expansion coefficient to form the training domain of each sub-block.

[0031] In this embodiment, as Figure 2 As shown, to avoid insufficient boundary constraints during 3D reconstruction of sub-blocks, the directed bounding box of each sub-block is expanded according to a corresponding expansion coefficient to generate the training domain for each sub-block. The expansion coefficient is calculated based on the extrinsic and intrinsic parameters of the observation camera corresponding to the multi-view images of the scene to be reconstructed. The extrinsic layer includes the camera position, and the intrinsic parameter includes the focal length. The average distance from the directed bounding box of each sub-block to the camera position is calculated, and the spatial pixel size corresponding to the imaging resolution of the observation camera at each average distance is calculated based on the focal length. The spatial pixel size is then input into the expansion formula to obtain the expansion coefficient for each sub-block. The empirical coefficients in the expansion formula vary depending on the complexity of the scene to be reconstructed.

[0032] S2. Based on the visibility relationship of the sparse point cloud, assign a corresponding training view set to the training domain of each sub-block, and perform view equalization sampling on the training view set corresponding to each sub-block to obtain the equalized view set of each sub-block. Based on the equalized view set of each sub-block, perform 3D reconstruction on each sub-block to obtain the 3D reconstruction result of each sub-block. The 3D reconstruction result includes a 3D Gaussian set. At this point, step S2 includes: S21. Obtain the sparse point set of each sub-block in the training domain, and simultaneously obtain the view point set of all views in the multi-view image. Input the view point set and the sparse point set into the training constraints to allocate a corresponding training view set to the training domain of each sub-block. The training constraints are: ; in, This represents the training view set corresponding to sub-block i. This represents the v-th view in a multi-view image. Represents all multi-view images, Let v represent the set of viewpoints observed by the v-th view in a multi-view image. This represents the sparse point set of sub-block i within the training domain. This indicates the preset view coverage threshold. This indicates the preset sub-block coverage threshold. Represents the logical OR operator; S22. Extract key features from each view in the training view set corresponding to each sub-block to obtain the key parameter vector of each view relative to the corresponding sub-block, and parameterize the corresponding view based on the key parameter vector to obtain the parameterized view. The key parameter vector is the azimuth angle and the center distance. S23. Discretize the parameterized view into bins according to the preset azimuth angle rule and the preset center distance rule to obtain all view bins. Use a round-robin sampling strategy to randomly sample the same number of views from each view bin and merge them to obtain a balanced view set for each sub-block. S24. Using the sparse point set of each sub-block in the training domain as the spatial constraint and the balanced view set of each sub-block as the appearance constraint, perform 3D reconstruction on each sub-block to obtain the 3D reconstruction result of each sub-block.

[0033] In this embodiment, as Figure 2 As shown, the sparse point set of each sub-block in the training domain and the view point set of all views in the multi-view image are input into the training constraints. For each view in the multi-view image, the intersection of the view point set observed by each view and the sparse point set of the sub-block in the training domain is calculated. If the proportion of the intersection to the view point set observed by the corresponding view is greater than or equal to a preset view coverage threshold, or if the proportion of the intersection to the sparse point set of the corresponding sub-block in the training domain is greater than or equal to a preset sub-block coverage threshold, then the view is assigned to the sub-block, so as to allocate the corresponding training view set to the training domain of each sub-block. For each view in the training view set corresponding to each sub-block, key features are extracted to obtain the key parameter vector of each view relative to the corresponding sub-block. That is, the azimuth angle and center distance are extracted to achieve the parameterization of the view. The parameterized view is discretized and binned according to the preset azimuth angle rule and the preset center distance rule. At this time, the preset azimuth angle rule is a uniform division of the interval from 0° to 360°, and the preset center distance rule is to divide the view according to the size of the center distance from small to large. Each parameterized view falls into the corresponding view bin according to its own key parameter vector, so that the view in each view bin has similar view features, thus obtaining all view bins.

[0034] A round-robin sampling strategy is employed to randomly sample the same number of views from each viewbox and merge them to obtain a balanced view set for each sub-block. This ensures a balanced number of views across different viewpoints. The number of views is dynamically adjusted based on the total number of views in the training view set corresponding to each sub-block. Using the sparse point set of each sub-block within the training domain as a spatial constraint and the balanced view set of each sub-block as an appearance constraint, 3D reconstruction is performed on each sub-block to obtain its 3D reconstruction result.

[0035] At this point, step S24 includes: S241. The three-dimensional Gaussian primitive parameters of each sub-block are initialized based on the balanced view set of each sub-block and the sparse point set of each sub-block in the training domain. Each sparse point in the sparse point set of each sub-block in the training domain is taken as the center position of the initial Gaussian primitive. The projection coordinates corresponding to the center position of each initial Gaussian primitive are obtained based on the balanced view set of each sub-block. The pixel color of each projection coordinate is obtained by bilinear interpolation, and the average value of all pixel colors is calculated. The average value is taken as the initial spherical harmonic DC component of the corresponding sub-block. The average distance from the center position of each initial Gaussian primitive to the k nearest neighbor sparse points is calculated by the K nearest neighbor search method. The average distance is taken as the initial scale of the initial Gaussian primitive. S242. Combine the center position, the initial spherical harmonic DC component, and the initial scale with a preset opacity and a preset number of rotations to construct the three-dimensional Gaussian element parameters after each sub-block is initialized. S243. Input the initialized 3D Gaussian meta-parameters of each sub-block into the pre-trained Gaussian splash renderer and combine them with the reconstruction loss function to independently optimize the initialized 3D Gaussian meta-parameters of each sub-block, thereby obtaining the 3D reconstruction result of each sub-block.

[0036] In this embodiment, as Figure 2 As shown, each sparse point in the sparse point set of each sub-block within the training domain is taken as the center position of the initial Gaussian primitive. Based on the balanced view set of each sub-block, the projection coordinates corresponding to the center position of each initial Gaussian primitive are obtained. The projection coordinates are calculated in combination with the extrinsic and intrinsic parameters of the observation camera. The pixel color of each projection coordinate is obtained by bilinear interpolation, and this pixel color is the basic color information of the initial Gaussian primitive. At the same time, the average value of all pixel colors is calculated and used as the initial spherical harmonic DC component of the corresponding sub-block. The average distance from the center position of each initial Gaussian primitive to the k nearest neighbor sparse points is calculated by the K-nearest neighbor search method, and the average distance is used as the initial scale of the initial Gaussian primitive. The center position, initial spherical harmonic DC component, and initial scale are combined with preset opacity and preset rotation number to construct the 3D Gaussian primitive parameters after initialization of each sub-block. The preset rotation number is set using a unit rotation matrix, and the preset opacity is dynamically adjusted according to the initial spherical harmonic DC component. For example, the higher the brightness of the color corresponding to the initial spherical harmonic DC component, the greater the opacity.

[0037] The initialized 3D Gaussian meta-parameters of each sub-block are input into a pre-trained Gaussian splash renderer and combined with the reconstruction loss function to independently optimize the initialized 3D Gaussian meta-parameters of each sub-block. The pre-trained Gaussian splash renderer projects the initialized 3D Gaussian meta-parameters of each sub-block onto each view in the equalized view set of each sub-block to obtain the rendered pixel color corresponding to each view. The rendered pixel color corresponding to each view and its actual pixel color in the equalized view set are input into the reconstruction loss function to calculate the pixel loss. With the minimum pixel loss as the optimization objective, the gradient descent algorithm is used to iteratively optimize the initialized 3D Gaussian meta-parameters to obtain the 3D reconstruction result of each sub-block.

[0038] S3. Based on the set of three-dimensional Gaussian elements of each sub-block, obtain all overlapping three-dimensional Gaussian elements in the overlapping area of ​​adjacent sub-blocks. Calculate the normalized distance of each overlapping three-dimensional Gaussian element relative to the directed bounding box of its sub-block. Based on the normalized distance, perform region division on the corresponding overlapping three-dimensional Gaussian elements to obtain the region division result, which includes the core region, the competition region, and the culling region. At this point, step S3 involves calculating the normalized distance of each overlapping 3D Gaussian element relative to its directed bounding box in the sub-block, and then performing region partitioning on the corresponding overlapping 3D Gaussian elements based on the normalized distance. The resulting region partitioning includes: S31. Calculate the normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block using the first distance formula, whereby the first distance formula is: ; in, Let p represent the normalized distance of the overlapping 3D Gaussian element at position p relative to the directed bounding box of its sub-block i. Indicates The components are a diagonal matrix with diagonal elements. Let represent the half-side length of the directed bounding box of sub-block i. Let represent the transpose of the rotation matrix of the directed bounding box of sub-block i. This represents an overlapping 3D Gaussian element at position p. Let represent the vector of the center point of the directed bounding box of sub-block i. The infinite norm of a vector; S32. Based on the normalized distance and region partitioning rules, the corresponding overlapping 3D Gaussian elements are partitioned into regions to obtain the region partitioning results. The region partitioning rules are as follows: .

[0039] In this embodiment, as Figure 2 As shown, based on the set of 3D Gaussian elements of each sub-block, all overlapping 3D Gaussian elements within the overlapping region of adjacent sub-blocks are obtained. The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated using the first distance formula. According to the normalized distance and the region division rules, the corresponding overlapping 3D Gaussian elements are divided into regions. Overlapping 3D Gaussian elements with a normalized distance greater than or equal to 1 are assigned to the core region, overlapping 3D Gaussian elements with a normalized distance greater than 1.1 are assigned to the removal region, and overlapping 3D Gaussian elements with a normalized distance between (1, 1.1) are assigned to the competition region, thus obtaining the region division result.

[0040] S4. Delete overlapping 3D Homo sapiens in the rejection area, and simultaneously perform competitive scoring on overlapping 3D Homo sapiens in the competition area to obtain a competitive scoring result. Based on the competitive scoring result, dynamically filter the overlapping 3D Homo sapiens in the competition area to obtain dynamically filtered overlapping 3D Homo sapiens in the competition area. Based on the dynamically filtered overlapping 3D Homo sapiens in the competition area and the overlapping 3D Homo sapiens in the core area, perform cross-block random stitching to generate a 3D reconstruction model of the scene to be reconstructed.

[0041] In this embodiment, as Figure 2 As shown, overlapping 3D Homo sapiens in the exclusion zone are deleted, while overlapping 3D Homo sapiens in the core zone are directly retained. Overlapping 3D Homo sapiens in the competition zone are competitively scored, and dynamic filtering is performed based on the competition score results. Based on the dynamically filtered overlapping 3D Homo sapiens in the competition zone and the directly retained overlapping 3D Homo sapiens in the core zone, cross-block random stitching is achieved to generate a 3D reconstruction model of the scene to be reconstructed.

[0042] In this embodiment, as Figure 3 As shown, the renderings of the 3D reconstruction model generated by the method of this embodiment for the same scene to be reconstructed are compared with the renderings of the 3D reconstruction models generated by other existing methods. Figure 3 The magnified area in the image shows that the invention performs better in terms of boundary continuity and overall visual harmony. The magnified area refers to... Figure 4 The area represented by a rectangle.

[0043] At this point, the competition scoring result in step S4 includes the self-competition scoring result and the neighborhood competition scoring result. Simultaneously performing competition scoring on overlapping 3D Gaussian elements within the competition region to obtain the competition scoring result, and then dynamically filtering the overlapping 3D Gaussian elements within the competition region based on the competition scoring result, includes: S41. Obtain the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region. Input the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region into the quality formula for calculation to obtain the self-quality score of each overlapping 3D Gaussian element within the competition region. The quality formula is: ; in, Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. The opacity of the overlapping 3D Gaussian element at position x within the competition region is represented by . Let x represent the logarithmic scaling vector of overlapping three-dimensional Gaussian elements at position x within the contention zone. This represents the actual scale vector of the overlapping 3D Gaussian elements at position x within the competition zone. This represents the average value. This represents the first positive number that is preset. S42. Input the self-quality score of each overlapping 3D Gaussian element in the competition region into the first competition formula for competition scoring, and obtain the self-competition score result of each overlapping 3D Gaussian element in the competition region. The first competition formula is: ; ; in, This represents the self-competition score of the overlapping 3D Gaussian element at position x within the competition zone. Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. Indicates the distance decay weight. This represents the normalized distance of the overlapping 3D Gaussian element at position x relative to the directed bounding box of its sub-block i; S43. Obtain the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition zone, and use the KD-Tree spatial indexing algorithm to query the k nearest neighbors of the corresponding overlapping 3D Gaussian element from all original 3D Gaussian elements of the neighboring sub-blocks, to obtain the neighborhood candidate set of each overlapping 3D Gaussian element within the competition zone. Calculate the neighborhood quality score of each candidate 3D Gaussian element in the neighborhood candidate set of each overlapping 3D Gaussian element using the quality formula, and input the neighborhood quality score into the second competition formula for competition scoring, to obtain the neighborhood competition score result of the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition zone. The second competition formula is: ; in, This represents the neighborhood competition score of the adjacent sub-block j of the sub-block to which the overlapping 3D Gaussian element at position x belongs within the competition region. Indicates the distance decay weight. This represents the average value. Let represent the neighborhood quality score of the candidate 3D Gaussian element at position y in the neighborhood candidate set. This represents the k nearest neighboring 3D Gaussian elements to the sub-block j of the overlapping 3D Gaussian element at position x within the competition zone, obtained through the KD-Tree spatial indexing algorithm. S44. Input the self-competition score result and the corresponding neighborhood competition score result of each overlapping 3D Gaussian element in the competition region into the probability formula for calculation to obtain the retention probability of each overlapping 3D Gaussian element in the competition region. The probability formula is as follows: ; in, Let x represent the retention probability of an overlapping 3D Gaussian element at position x within the competition region. This represents the sigmoid function. This indicates the boundary sharpness control parameter. This represents the self-competition score of an overlapping 3D Gaussian element at position x within the competition zone. The maximum neighborhood quality score is represented in the neighborhood candidate set of the neighboring sub-block j of the overlapping 3D Gaussian element at position x within the competition zone. S45. Bernoulli sampling is performed based on the retention probability to achieve dynamic filtering of overlapping three-dimensional Gaussian elements within the competing region.

[0044] In this embodiment, as Figure 4As shown, the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region are input into a quality formula for calculation to obtain the self-quality score of each overlapping 3D Gaussian element within the competition region. The self-quality score of each overlapping 3D Gaussian element within the competition region is then input into a first competition formula for competition scoring, resulting in the self-competition score of each overlapping 3D Gaussian element within the competition region. Next, the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region are obtained. The KD-Tree spatial indexing algorithm is used to query the k nearest neighbors of the corresponding overlapping 3D Gaussian element from all original 3D Gaussian elements in the neighboring sub-blocks. Based on these k candidate 3D Gaussian elements, a neighborhood candidate set for the corresponding overlapping 3D Gaussian element is obtained. The neighborhood quality score of each candidate 3D Gaussian element in the neighborhood candidate set of each overlapping 3D Gaussian element is calculated using the aforementioned quality formula. The neighborhood quality score is then input into a second competition formula for competition scoring, resulting in the neighborhood competition score of the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region. The self-competition score of each overlapping 3D Gaussian element in the competition region and the corresponding neighborhood competition score are input into the probability formula for calculation to obtain the retention probability of each overlapping 3D Gaussian element in the competition region. Bernoulli sampling is performed based on the retention probability to achieve dynamic filtering of overlapping 3D Gaussian elements in the competition region. Bernoulli sampling will generate a uniform random number between [0,1]. If the retention probability exceeds the generated uniform random number, the overlapping 3D Gaussian element corresponding to the retention probability is retained; otherwise, it is deleted.

[0045] In this embodiment, an optimization mechanism is also set for the generated 3D reconstruction model of the scene to be reconstructed. The specific steps are as follows: Obtain the opacity and actual scale vector of each reconstructed 3D Gaussian primitive in the 3D reconstruction model, and input the opacity and actual scale vector of each reconstructed 3D Gaussian primitive into optimization conditions in sequence to optimize the 3D reconstruction model and obtain a full-scene model. The optimization conditions are: ; in, Represents the full-scene model. Represents a 3D reconstruction model. Indicates the location in the 3D reconstruction model as Reconstructing three-dimensional Gaussian elements, Represents a logarithm. Indicates the location in the 3D reconstruction model as The opacity of the reconstructed 3D Gaussian elements. This indicates the preset opacity threshold. Indicates the location in the 3D reconstruction model as The reconstructed logarithmic scaling vector of the three-dimensional Gaussian element. Indicates the location in the 3D reconstruction model as The actual scale vector of the reconstructed 3D Gaussian element. The infinite norm of a vector. This indicates the preset scale threshold. This represents the logical AND operator; The optimization of the full-scene model is achieved by freezing the geometric parameters of all full-scene 3D Gaussian primitives in the full-scene model and introducing pseudo-quantization operators and minimum loss functions to the appearance parameters of all full-scene 3D Gaussian primitives in the full-scene model. The geometric parameters include position parameters, rotation parameters, and a logarithmic scaling vector; the appearance parameters include opacity, spherical harmonic DC component, and higher-order residual spherical harmonic coefficients; and the minimum loss function is: ; ; ; in, Let represent the minimum loss function introduced for the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents taking the expected value for all multi-view images. This represents the v-th view in a multi-view image. Represents all multi-view images, Represents the loss function. This represents a differentiable Gaussian splash rendering function. Indicates the position in the full scene model. The opacity of the full-scene 3D high-order primitives. This represents the set of opacities of all 3D Gaussian primitives in the full-scene model. Indicates the position in the full scene model. The spherical harmonic DC component of the full-scene 3D Gaussian element. This represents the set of spherical harmonic DC components of all 3D Gaussian elements in the full-scene model. Indicates the position in the full scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates that the position in the full scene model is The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements are introduced with an 8-bit pseudo-quantization operator. Indicates the position in the full scene model. The high-order residual spherical harmonic coefficients after introducing pseudo-quantization operators in the full-scene 3D Gaussian primitives This represents the set of higher-order residual spherical harmonic coefficients of all full-scene 3D Gaussian elements after introducing pseudo-quantization operators in the full-scene model. Represents the full-scene model. This represents the geometric parameters of all frozen 3D Gaussian primitives in the full-scene model. This represents the real image of the v-th view in a multi-view image.

[0046] In this embodiment, the opacity and actual scale vector of each reconstructed 3D Gaussian primitive in the 3D reconstruction model are sequentially input into optimization conditions. The optimization conditions retain reconstructed 3D Gaussian primitives whose opacity exceeds a preset opacity threshold and whose actual scale vector's infinity norm is lower than a preset scale threshold. Reconstructed 3D Gaussian primitives that do not meet the preset opacity threshold and / or preset scale threshold are deleted, thereby optimizing the 3D reconstruction model and obtaining the full-scene model. The optimization of the full-scene model is achieved by freezing the geometric parameters of all full-scene 3D Gaussian primitives and optimizing the appearance parameters. Specifically, a pseudo-quantization operator and a minimum loss function are introduced into the appearance parameters of all full-scene 3D Gaussian primitives in the full-scene model, that is, minimizing the rendering loss under the condition of fixed geometric parameters.

[0047] In this embodiment, a hierarchical storage mechanism is set up for the optimized full-scene model to achieve lightweight storage. The specific steps are as follows: The higher-order residual spherical harmonic coefficients of each full-scene 3D Gaussian element in the optimized full-scene model are input into the spectrum formula of the third-order spherical harmonic order for calculation, to obtain the third-order spectral energy of each full-scene 3D Gaussian element in the optimized full-scene model. The spectrum formula of the third-order spherical harmonic order is as follows: ; ; in, This indicates the position in the optimized full-scene model. The full-scene 3D Gaussian element's first Spectral energy of spherical harmonic order Let o represent the spherical harmonic order, and o represent the position in the optimized full-scene model. Full-scene 3D Gaussian elements, Indicates the first The set of indices of the order of spherical harmonics. This indicates the position from the optimized full-scene model. The first spherical harmonic coefficient extracted from the high-order residuals of the full-scene 3D Gaussian elements The set of indices of the order of spherical harmonics. The norm of vector L1 is used to represent the vector L1. The third-order spectral energy of each 3D Gaussian element in the optimized full-scene model is aggregated using an aggregation formula to obtain the high-frequency energy of each 3D Gaussian element in the optimized full-scene model. The aggregation formula is as follows: ; in, This indicates the position in the optimized full-scene model. The high-frequency energy of the full-scene three-dimensional Gaussian elements, Indicates the preset first Weights of spherical harmonic orders; The high-frequency energy input gating compression formula of each full-scene 3D Gaussian element in the optimized full-scene model is subjected to spectral gating compression to obtain a spectral-gated full-scene model. The spectral-gated full-scene model is then subjected to hierarchical storage according to a preset hierarchical storage rule to obtain a lightweight full-scene model. The gating compression formula is: ; in, This indicates the position in the full-scene model after spectrum-gated compression. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates the position in the optimized full-scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, Indicates an indicator function, This indicates the preset spectrum gating threshold.

[0048] In this embodiment, the higher-order residual spherical harmonic coefficients of each full-scene 3D Gaussian element in the optimized full-scene model are calculated using a spectral formula for third-order spherical harmonics, where the third-order spherical harmonics are 1st, 2nd, and 3rd, respectively, to obtain the third-order spectral energy of each full-scene 3D Gaussian element in the optimized full-scene model. The third-order spectral energy of each full-scene 3D Gaussian element in the optimized full-scene model is then aggregated using an aggregation formula to obtain the high-frequency energy of each full-scene 3D Gaussian element in the optimized full-scene model. The high-frequency energy of each full-scene 3D Gaussian element in the optimized full-scene model is then input into a gating compression formula to perform spectral gating compression. Spectral gating compression adaptively prunes the higher-order residual coefficients. The meaning of the indicator function in the gating compression formula is: for full-scene 3D Gaussian elements with high-frequency energy below a preset spectral gating threshold, all higher-order residual coefficients are discarded; for full-scene 3D Gaussian elements with high-frequency energy greater than or equal to the preset spectral gating threshold, all higher-order residual coefficients are retained, resulting in a spectral-gated compressed full-scene model. The full-scene model after spectrum gating compression is stored hierarchically according to the preset hierarchical storage rules. The preset hierarchical storage rules are as follows: different storage strategies are adopted according to parameter attributes. Position parameters are stored using FP32, opacity, rotation number, scale and spherical harmonic DC components are stored using FP16, and high-order residual spherical harmonic coefficients are stored using 8-bit quantization, resulting in a lightweight full-scene model.

[0049] Example 2 Please refer to Figure 5 The present invention provides a three-dimensional scene reconstruction system 1 based on three-dimensional Gaussian splashing, including a memory 3, a processor 2, and a computer program stored on the memory 3 and capable of running on the processor 2. When the processor 2 executes the computer program, it implements the steps in Embodiment 1.

[0050] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0051] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0053] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0054] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0055] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0056] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for reconstructing a 3D scene based on 3D Gaussian splashing, characterized in that, include: Acquire multi-view images of the scene to be reconstructed, calculate the sparse point cloud of the scene to be reconstructed based on the multi-view images, perform density adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks, and construct a directed bounding box for each sub-block to form the training domain of each sub-block. Based on the visibility relationship of the sparse point cloud, a corresponding training view set is assigned to the training domain of each sub-block, and view equalization sampling is performed on the training view set corresponding to each sub-block to obtain the equalized view set of each sub-block. Based on the equalized view set of each sub-block, 3D reconstruction is performed on each sub-block to obtain the 3D reconstruction result of each sub-block. The 3D reconstruction result includes a 3D Gaussian set. Based on the set of 3D Gaussian elements of each sub-block, all overlapping 3D Gaussian elements in the overlapping area of ​​adjacent sub-blocks are obtained. The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated. Based on the normalized distance, the corresponding overlapping 3D Gaussian elements are divided into regions to obtain the region division results. The region division results include the core region, the competition region, and the culling region. Overlapping 3D Gaussian elements in the rejection area are deleted, and overlapping 3D Gaussian elements in the competition area are competitively scored to obtain a competition score result. Based on the competition score result, overlapping 3D Gaussian elements in the competition area are dynamically filtered to obtain overlapping 3D Gaussian elements in the competition area after dynamic filtering. Based on the overlapping 3D Gaussian elements in the competition area after dynamic filtering and overlapping 3D Gaussian elements in the core area, cross-block random stitching is performed to generate a 3D reconstruction model of the scene to be reconstructed.

2. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, The step of performing density adaptive segmentation of the scene to be reconstructed based on the sparse point cloud to obtain N sub-blocks includes: The sparse point cloud is divided into voxel grids to obtain M voxel grids. The volume of each voxel grid and the total number of sparse points in each voxel grid are calculated. The local density of each voxel grid is calculated based on the total number of sparse points and the volume. When the local density is lower than the first lower limit density, the voxel mesh corresponding to the local density that is lower than the first lower limit density is merged with the adjacent voxel mesh to obtain a merged voxel mesh. It is determined whether the local density of the merged voxel mesh is lower than the first lower limit density. If so, the merged voxel mesh is merged with the adjacent voxel mesh a second time until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. When the local density exceeds the first upper limit density, the voxel mesh corresponding to the local density that exceeds the first upper limit density is divided into a second voxel mesh to obtain a voxel mesh after the second voxel mesh division. It is determined whether the local density of the voxel mesh after the second voxel mesh division exceeds the first upper limit density. If so, the voxel mesh after the second voxel mesh division is divided into a third voxel mesh until the local density of the final voxel mesh is between the first lower limit density and the first upper limit density. When the local density is between the first lower limit density and the first upper limit density, the voxel mesh is divided into blocks according to a preset size to obtain N sub-blocks.

3. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, The construction of a directed bounding box for each sub-block to form the training domain for each sub-block includes: Based on the multi-view images, calculate the extrinsic and intrinsic parameters of the observation camera corresponding to the scene to be reconstructed. The extrinsic parameters include the camera position, and the intrinsic parameters include the focal length. For each sub-block, a directed bounding box is constructed, and the average distance from the directed bounding box of each sub-block to the camera position is calculated. Based on the focal length, the spatial pixel size corresponding to the imaging resolution of the observation camera at each average distance is calculated. The spatial pixel size is then input into an expansion formula for calculation to obtain the expansion coefficient corresponding to each sub-block. The expansion formula is as follows: ； in, This represents the expansion coefficient of sub-block i. Represents the empirical coefficient. Indicates the spatial pixel size of sub-block i; The directed bounding box of each sub-block is expanded according to the corresponding expansion coefficient to form the training domain of each sub-block.

4. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, The training view set is assigned to the training domain of each sub-block based on the visibility relationship of the sparse point cloud, and viewpoint equalization sampling is performed on the training view set corresponding to each sub-block to obtain the equalized view set of each sub-block. Based on the equalized view set of each sub-block, 3D reconstruction is performed on each sub-block to obtain the 3D reconstruction result of each sub-block, including: Obtain the sparse point set of each sub-block within the training domain, and simultaneously obtain the view point set of all views in the multi-view image. Input the view point set and the sparse point set into the training constraints to allocate a corresponding training view set to the training domain of each sub-block. The training constraints are as follows: ； in, This represents the training view set corresponding to sub-block i. This represents the v-th view in a multi-view image. Represents all multi-view images, Let v represent the set of viewpoints observed by the v-th view in a multi-view image. This represents the sparse point set of sub-block i within the training domain. This indicates the preset view coverage threshold. This indicates the preset sub-block coverage threshold. Represents the logical OR operator; For each view in the training view set corresponding to each sub-block, key features are extracted to obtain the key parameter vector of each view relative to the corresponding sub-block. The corresponding view is then parameterized based on the key parameter vector to obtain the parameterized view. The key parameter vector is the azimuth angle and the center distance. The parameterized view is discretized into bins according to the preset azimuth angle rule and the preset center distance rule to obtain all view bins. A round-robin sampling strategy is used to randomly sample the same number of views from each view bin and merge them to obtain a balanced view set for each sub-block. The sparse point set of each sub-block in the training domain is used as the spatial constraint, and the balanced view set of each sub-block is used as the appearance constraint to perform 3D reconstruction of each sub-block, so as to obtain the 3D reconstruction result of each sub-block.

5. The three-dimensional scene reconstruction method based on three-dimensional Gaussian splashing as described in claim 4, characterized in that, The method of using the sparse point set of each sub-block in the training domain as the spatial constraint and the balanced view set of each sub-block as the appearance constraint to perform 3D reconstruction on each sub-block, and obtaining the 3D reconstruction result of each sub-block includes: The 3D Gaussian primitive parameters of each sub-block are initialized based on the balanced view set of each sub-block and the sparse point set of each sub-block in the training domain. Each sparse point in the sparse point set of each sub-block in the training domain is taken as the center position of the initial Gaussian primitive. The projection coordinates corresponding to the center position of each initial Gaussian primitive are obtained based on the balanced view set of each sub-block. The pixel color of each projection coordinate is obtained by bilinear interpolation, and the average value of all pixel colors is calculated. The average value is taken as the initial spherical harmonic DC component of the corresponding sub-block. The average distance from the center position of each initial Gaussian primitive to the k nearest neighbor sparse points is calculated by the K nearest neighbor search method. The average distance is taken as the initial scale of the initial Gaussian primitive. The center position, the initial spherical harmonic DC component, and the initial scale are combined with preset opacity and preset rotation number to construct the three-dimensional Gaussian element parameters after each sub-block is initialized. The initialized 3D Gaussian meta-parameters of each sub-block are input into a pre-trained Gaussian splash renderer and combined with the reconstruction loss function to independently optimize the initialized 3D Gaussian meta-parameters of each sub-block, thereby obtaining the 3D reconstruction result of each sub-block.

6. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated. Based on the normalized distance, the corresponding overlapping 3D Gaussian elements are divided into regions, and the region division results include: The normalized distance of each overlapping 3D Gaussian element relative to the directed bounding box of its sub-block is calculated using a first distance formula, which is: ； in, Let p represent the normalized distance of the overlapping 3D Gaussian element at position p relative to the directed bounding box of its sub-block i. Indicated by The components are a diagonal matrix with diagonal elements. Let represent the half-side length of the directed bounding box of sub-block i. Let represent the transpose of the rotation matrix of the directed bounding box of sub-block i. This represents an overlapping 3D Gaussian element at position p. Let represent the vector of the center point of the directed bounding box of sub-block i. The infinite norm of a vector; Based on the normalized distance and the region partitioning rule, the corresponding overlapping 3D Gaussian elements are partitioned into regions to obtain the region partitioning results. The region partitioning rule is as follows: 。 7. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, The competition scoring result includes the self-competition scoring result and the neighborhood competition scoring result. Simultaneously performing competition scoring on overlapping 3D Gaussian elements within the competition region to obtain the competition scoring result, and then dynamically filtering the overlapping 3D Gaussian elements within the competition region based on the competition scoring result, includes: Obtain the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region. Input the opacity and logarithmic scaling vector of each overlapping 3D Gaussian element within the competition region into the quality formula for calculation, and obtain the self-quality score of each overlapping 3D Gaussian element within the competition region. The quality formula is: ； in, Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. The opacity of the overlapping 3D Gaussian element at position x within the competition region is represented by . Let x represent the logarithmic scaling vector of overlapping three-dimensional Gaussian elements at position x within the contention zone. This represents the actual scale vector of the overlapping 3D Gaussian elements at position x within the competition zone. This represents the average value. This represents the first positive number that is preset. The self-quality score of each overlapping 3D Gaussian element within the competition region is input into the first competition formula for competition scoring, resulting in the self-competition score of each overlapping 3D Gaussian element within the competition region. The first competition formula is as follows: ； ； in, This represents the self-competition score of the overlapping 3D Gaussian element at position x within the competition zone. Let x represent the self-mass fraction of the overlapping 3D Gaussian element at position x within the competition zone. Indicates the distance decay weight. This represents the normalized distance of the overlapping 3D Gaussian element at position x relative to the directed bounding box of its sub-block i; Obtain the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region. Then, use the KD-Tree spatial indexing algorithm to find the k nearest neighbors of the corresponding overlapping 3D Gaussian element from all original 3D Gaussian elements in the neighboring sub-blocks. This yields a neighborhood candidate set for each overlapping 3D Gaussian element within the competition region. Calculate the neighborhood quality score for each candidate 3D Gaussian element in the neighborhood candidate set using the quality formula. Input the neighborhood quality score into a second competition formula for competition scoring to obtain the neighborhood competition score result for the neighboring sub-blocks of each overlapping 3D Gaussian element within the competition region. The second competition formula is: ； in, This represents the neighborhood competition score of the adjacent sub-block j of the sub-block to which the overlapping 3D Gaussian element at position x belongs within the competition region. Indicates the distance decay weight. This represents the average value. Let represent the neighborhood quality score of the candidate 3D Gaussian element at position y in the neighborhood candidate set. This represents the k nearest neighboring 3D Gaussian elements to the sub-block j of the overlapping 3D Gaussian element at position x within the competition region, obtained through the KD-Tree spatial indexing algorithm. The self-competition score of each overlapping 3D Gaussian element within the competition region and its corresponding neighborhood competition score are input into a probability formula for calculation, yielding the retention probability of each overlapping 3D Gaussian element within the competition region. The probability formula is as follows: ； in, Let x represent the retention probability of an overlapping 3D Gaussian element at position x within the competition region. This represents the sigmoid function. This indicates the boundary sharpness control parameter. This represents the self-competition score of an overlapping 3D Gaussian element at position x within the competition zone. The largest neighborhood quality score in the neighborhood candidate set of the neighboring sub-block j of the overlapping 3D Gaussian element at position x within the competition zone; Bernoulli sampling is performed based on the retention probability to achieve dynamic filtering of overlapping 3D Gaussian elements within the competing region.

8. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 1, characterized in that, Also includes: Obtain the opacity and actual scale vector of each reconstructed 3D Gaussian primitive in the 3D reconstruction model, and input the opacity and actual scale vector of each reconstructed 3D Gaussian primitive into optimization conditions in sequence to optimize the 3D reconstruction model and obtain a full-scene model. The optimization conditions are: ； in, Represents a full-scene model. Represents a 3D reconstruction model. This indicates the location in the 3D reconstruction model. Reconstructing three-dimensional Gaussian elements, Represents a logarithm. This indicates the location in the 3D reconstruction model. The opacity of the reconstructed 3D Gaussian elements. This indicates the preset opacity threshold. This indicates the location in the 3D reconstruction model. The reconstructed logarithmic scaling vector of the three-dimensional Gaussian element. Indicates the location in the 3D reconstruction model as The actual scale vector of the reconstructed 3D Gaussian element. The infinite norm of a vector. This indicates the preset scale threshold. This represents the logical AND operator; The optimization of the full-scene model is achieved by freezing the geometric parameters of all full-scene 3D Gaussian primitives in the full-scene model and introducing pseudo-quantization operators and minimum loss functions to the appearance parameters of all full-scene 3D Gaussian primitives in the full-scene model. The geometric parameters include position parameters, rotation parameters, and a logarithmic scaling vector; the appearance parameters include opacity, spherical harmonic DC component, and higher-order residual spherical harmonic coefficients; and the minimum loss function is: ； ； ； in, Let represent the minimum loss function introduced for the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents the appearance parameters of all 3D Gaussian primitives in the full-scene model. This represents taking the expected value for all multi-view images. This represents the v-th view in a multi-view image. Represents all multi-view images, Represents the loss function. This represents a differentiable Gaussian splash rendering function. This indicates the position in the full scene model. The opacity of the full-scene 3D high-order primitives. This represents the set of opacities of all 3D Gaussian primitives in the full-scene model. This indicates the position in the full scene model. The spherical harmonic DC component of the full-scene 3D Gaussian element. This represents the set of spherical harmonic DC components of all 3D Gaussian elements in the full-scene model. This indicates the position in the full scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates that the position in the full scene model is The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements are introduced with an 8-bit pseudo-quantization operator. This indicates the position in the full scene model. The high-order residual spherical harmonic coefficients after introducing pseudo-quantization operators in the full-scene 3D Gaussian primitives This represents the set of higher-order residual spherical harmonic coefficients of all full-scene 3D Gaussian elements after introducing pseudo-quantization operators in the full-scene model. Represents a full-scene model. This represents the geometric parameters of all frozen 3D Gaussian primitives in the full-scene model. This represents the real image of the v-th view in a multi-view image.

9. The method for reconstructing a 3D scene based on 3D Gaussian splashing as described in claim 8, characterized in that, The optimized full-scene model includes: The higher-order residual spherical harmonic coefficients of each full-scene 3D Gaussian element in the optimized full-scene model are input into the spectrum formula of the third-order spherical harmonic order for calculation, to obtain the third-order spectral energy of each full-scene 3D Gaussian element in the optimized full-scene model. The spectrum formula of the third-order spherical harmonic order is as follows: ； ； in, This indicates the position in the optimized full-scene model. The full-scene 3D Gaussian element's first Spectral energy of spherical harmonic order Let o represent the spherical harmonic order, and o represent the position in the optimized full-scene model. Full-scene 3D Gaussian elements, Indicates the first The set of indices of the order of spherical harmonics. This indicates the position from the optimized full-scene model. The first spherical harmonic coefficient extracted from the high-order residuals of the full-scene 3D Gaussian elements The set of indices of the order of spherical harmonics. Let L1 be the norm of vector L1; The third-order spectral energy of each 3D Gaussian element in the optimized full-scene model is aggregated using an aggregation formula to obtain the high-frequency energy of each 3D Gaussian element in the optimized full-scene model. The aggregation formula is as follows: ； in, This indicates the position in the optimized full-scene model. The high-frequency energy of the full-scene three-dimensional Gaussian elements, Indicates the preset first Weights of spherical harmonic orders; The high-frequency energy input gating compression formula of each full-scene 3D Gaussian element in the optimized full-scene model is subjected to spectral gating compression to obtain a spectral-gated full-scene model. The spectral-gated full-scene model is then subjected to hierarchical storage according to a preset hierarchical storage rule to obtain a lightweight full-scene model. The gating compression formula is: ； in, This indicates the position in the full-scene model after spectrum-gated compression. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, This indicates the position in the optimized full-scene model. The high-order residual spherical harmonic coefficients of the full-scene 3D Gaussian elements, Indicates an indicator function, This indicates the preset spectrum gating threshold.

10. A 3D scene reconstruction system based on 3D Gaussian splashing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

Images