A three-dimensional gaussian spatter method based on adaptive multi-resolution voxel grid

Abstract

The application provides a three-dimensional Gaussian splatting method based on adaptive multi-resolution voxel grids, comprising the following steps: step 1, obtaining a scene sparse point cloud according to multi-view images of a target scene and camera poses corresponding to each image, and constructing a multi-resolution sparse voxel grid; step 2, constructing a neural Gaussian representation based on the multi-resolution sparse voxel grid, and deriving Gaussian points through learnable parameters; step 3, iterative training, decoding attributes of the Gaussian points through a multi-layer perceptron network, optimizing parameters of the multi-layer perceptron network and the learnable parameters in step 2, and performing adaptive active block division, and completing the three-dimensional Gaussian splatting based on the adaptive multi-resolution voxel grids.

Description

Technical Field

[0001] This invention relates to a three-dimensional Gaussian splashing method, and more particularly to a three-dimensional Gaussian splashing method based on an adaptive multi-resolution voxel mesh. Background Technology

[0002] In recent years, with the rapid development of technologies such as virtual reality, augmented reality, and autonomous driving, the real-time reconstruction and rendering of high-quality 3D scenes has become a research hotspot in the fields of computer graphics and computer vision. How to efficiently reconstruct highly realistic 3D scenes from multi-view images is a key core technology driving the implementation of these applications.

[0003] Neural Radiance Fields (NeRF) and its variants have made groundbreaking progress in the field of 3D reconstruction by generating photorealistic new perspective images through encoding scenes as weights of a multilayer perceptron. However, NeRF-like methods require dense sampling of every camera ray, resulting in extremely slow training and rendering speeds, making it difficult to meet the needs of real-time interactive applications. Subsequent research has accelerated this process by introducing explicit data structures, such as existing techniques (Müller T, Evans A, Schied C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM transactions on graphics (TOG), 2022,41(4): 1-15. using multiresolution hash encoding, Fridovich-Keil S, Yu A, Tancik M, et al. Plenoxels: Radiance fields without neural networks[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 5501-5510. and Yu A, Li R, Tancik M, et al. Plenoctrees for real-time rendering of neural radiance fields[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision. 2021: 5752-5761.). In this study, sparse voxel meshes and octrees are used to achieve adaptive spatial partitioning. While these methods significantly improve training and rendering efficiency, it is still difficult to achieve an ideal balance between rendering quality and real-time performance.

[0004] The emergence of 3D Gaussian Splatting (3D GS) provides a novel solution for real-time, high-quality rendering. This method abandons implicit neural representations, instead explicitly representing the scene using a large number of anisotropic 3D Gaussian points, and achieving real-time rendering through a differentiable splatting rendering algorithm. 3D GS surpasses NeRF-based methods in both rendering speed and visual quality, becoming one of the most influential 3D reconstruction technologies in recent years. However, the original 3D GS often generates a large number of redundant Gaussian points in complex scenes, leading to a significant increase in storage overhead; simultaneously, due to the lack of explicit modeling of the scene geometry, artifacts easily appear in textureless areas, and it is difficult to accurately reconstruct geometric details in under-observed areas.

[0005] To address the aforementioned issues, some studies have begun exploring the introduction of hierarchical data structures into 3D GS, organizing and constraining Gaussian points through structures such as regular meshes and octrees. Existing technology (Lu T, Yu M, Xu L, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024: 20654-20664.) is the first to introduce structured thinking into 3D GS, achieving structured organization of Gaussian points by constructing a sparse anchor point mesh and using neural networks to decode and generate Gaussian point attributes. The existing technology (Ren K, Jiang L, Lu T, et al. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians[J]. arXiv preprint arXiv:2403.17898, 2024.) further introduces the concept of Level-of-Detail (LOD), and achieves adaptive adjustment of the number of Gaussian points during the rendering process through octree level selection.

[0006] However, the aforementioned existing technologies still have the following technical defects:

[0007] (1) Static meshes lack adaptability: The anchor mesh remains fixed during training, making it difficult to adaptively adjust the Gaussian point density according to the scene complexity;

[0008] (2) Poor reconstruction quality in areas with insufficient observation: In areas with sparse viewpoints or severe occlusion, relying solely on gradient-based densification strategies is insufficient to effectively cover the scene, resulting in holes or artifacts in the geometric reconstruction.

[0009] (3) Insufficient utilization of depth information: Although existing methods attempt to introduce depth supervision, they are mostly simple loss term superpositions, failing to effectively integrate depth information into the adaptive allocation process of Gaussian points. Summary of the Invention

[0010] Purpose of the invention: The technical problem to be solved by the present invention is to provide a three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh, which addresses the shortcomings of the existing technology.

[0011] To address the aforementioned technical problems, this invention discloses a three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel meshes, comprising the following steps:

[0012] Step 1: Obtain sparse point cloud of the scene based on multi-view images of the target scene and the camera pose corresponding to each image, and construct a multi-resolution sparse voxel mesh.

[0013] Step 2: Construct a neural Gaussian representation based on a multi-resolution sparse voxel grid, and derive Gaussian points through learnable parameters;

[0014] Step 3, iterative training: decode the properties of Gaussian points through a multilayer perceptron network, optimize the parameters of the multilayer perceptron network and the learnable parameters in step 2, and perform adaptive active block division to complete the 3D Gaussian splashing based on an adaptive multi-resolution voxel mesh.

[0015] Furthermore, the construction of the multi-resolution sparse voxel mesh described in step 1 specifically includes:

[0016] Step 1-1: Convert the sparse point cloud of the scene Mapped to voxel size In a sparse voxel mesh, all non-empty voxels are retained as the initial active block, with the midpoint of the initial active block... It is expressed as follows:

[0017]

[0018] in, Indicates the deduplication operation;

[0019] Steps 1-2: Based on the sparse point cloud of the scene Continue constructing voxel sizes respectively , , and The voxel mesh is obtained in a total of 5 layers. The voxel blocks in two adjacent layers are defined as parent blocks and child blocks, respectively. The parent block is composed of 8 child blocks in the next layer of mesh, and the 8 child blocks are sibling blocks.

[0020] Furthermore, step 2, which describes constructing a neural Gaussian representation based on a multi-resolution sparse voxel grid, specifically includes:

[0021] Step 2-1: Use the midpoint of all initial active blocks as anchor points, and for each anchor point... Assign learnable features and scale factor , means as follows:

[0022]

[0023]

[0024] Among them, learnable features Initialize as a zero vector. Represents a 32-dimensional real vector space. Represents a 3-dimensional real vector space;

[0025] Step 2-2, derive each anchor point from Gaussian point , means as follows:

[0026]

[0027] in, express The position of a Gaussian point Anchor point position , This is a learnable bias.

[0028] Furthermore, the iterative training described in step 3 specifically includes:

[0029] Step 3-1: Obtain the attributes of the current Gaussian point other than its position, including color, through a multilayer perceptron network. Opacity ,size and rotation ;

[0030] Step 3-2: Project the current Gaussian point using the splashing algorithm to obtain the rendered image under the current training viewpoint, and calculate the training loss;

[0031] Step 3-3: Optimize anchor point features using the backpropagation algorithm. Learnable bias and the multilayer perceptron network parameters in step 3-1;

[0032] Step 3-4: Repeat steps 3-1 to 3-3 and count the results; each execution... After the round, an adaptive multi-scale Gaussian allocation is performed based on the fitting error within each activity block. The fitting error of the entire scene is modeled as an integer linear programming problem. Based on the solution results, the activity blocks are merged or split, and steps 3-5 are executed.

[0033] Steps 3-5: Calculate depth supervision information and gradient information, optimize the Gaussian point distribution, and perform pruning operations on active blocks in blank areas;

[0034] Step 3-6: Repeat step 3-1 until the preset number of training iterations is reached.

[0035] Further, in step 3-1, the attributes of the current Gaussian point other than its position are obtained through a multilayer perceptron network. Specific methods include:

[0036] Learnable features of anchor points The direction of displacement of the camera position from the current perspective and distance The input is fed into a multilayer perceptron network to obtain the color. Opacity ,size and rotation .

[0037] Furthermore, the calculation of training loss described in step 3-2 specifically includes:

[0038] Step 3-2-1: After filtering out Gaussian points outside the view frustum based on the current training perspective, the Gaussian ellipsoid inside the view frustum is... Projected onto a two-dimensional Gaussian The piecewise differentiable rasterizer sorts the rasterization data in depth order from front to back, and finally calculates the final rendered image, as shown below:

[0039]

[0040] in, This indicates the pixel being queried. Represents the number of 2D Gaussians associated with the query pixel, for the A Gaussian point, This represents the Gaussian point. The color, Represents the Gaussian point Opacity Represents the Gaussian point The contribution weight to this pixel is determined by the opacity of that Gaussian point. and its value in 2D projection Joint decision, Represents the Gaussian point The contribution weight of this pixel, Indicates transmittance. For pixels The color;

[0041] Step 3-2-2: Calculate the rendering depth map from the current training viewpoint. , means as follows:

[0042]

[0043] in, To render depth map medium pixel The depth value, Indicates the first The z-coordinate of a Gaussian point in the camera coordinate system;

[0044] Step 3-2-3: Input the real image from the current training viewpoint into the monocular depth estimator to predict the dense depth map. And perform scale alignment, that is, sparse point cloud of the scene. Projection yields a sparse depth map And through linear transformation, the dense depth map Align to sparse depth map ;

[0045] The specific optimization goals are set as follows:

[0046]

[0047] in, and These are the scale factor and offset to be optimized, respectively. It is a sparse depth map medium pixel The depth value, It is a dense depth map Corresponding pixel The depth value, It is a pixel. The weights;

[0048] Iterative optimization of the scale factor using gradient descent method and offset ;

[0049] Final adjusted dense depth map From scale and offset The calculation yields the following result:

[0050]

[0051] Step 3-2-4: Calculate the loss function based on the rendered image obtained in Step 3-2-1 and the real image from the training viewpoint. , means as follows:

[0052]

[0053] in, , and They represent the weights, The mean absolute error of the color values. For structural similarity loss, For volume regularization loss, The depth regularization loss is derived from the rendering depth map. With dense depth map The Pearson correlation coefficient between them was calculated.

[0054] Furthermore, the depth regularization loss mentioned in step 3-2-4 is calculated using the following method:

[0055]

[0056] in, Describing covariance, Indicates variance.

[0057] Furthermore, steps 3-4 specifically include:

[0058] Step 3-4-1, Calculate The average gradient of the Gaussian points derived from each anchor point in the wheel is used as the fitting error of the active block. ;

[0059] Step 3-4-2, using a binary variable set Indicates active block Changes:

[0060]

[0061] in, This indicates a merge operation. Indicates an operation that remains unchanged. Indicates a splitting operation;

[0062] Step 3-4-3, estimate the volume as Activity block The error under three operations; among which, the error that remains constant is... It is calculated from the current average fitting error, and is specifically expressed as follows:

[0063]

[0064] If the active block The sub-block was previously used as the active block to calculate the fitting error. Error after splitting The sum of the errors of the sub-blocks; otherwise, multiply the current error by 1. The error after splitting into 8 sub-blocks is calculated and expressed as follows:

[0065]

[0066] in, It is an active block The set of sub-blocks, It is a sub-block The error remains constant;

[0067] If the parent block's error has been calculated before, then this error is used as the error after merging with the sibling block. Otherwise, multiply the current error by 8 to calculate the combined error, as shown below:

[0068]

[0069] in, It is an active block The parent block;

[0070] Step 3-4-4: Minimize the fitting error of the entire scene and model it as an integer linear programming problem, as follows:

[0071]

[0072] And it satisfies the following constraints:

[0073]

[0074] in, This indicates the number of active blocks before each optimization.

[0075] Step 3-4-5: Solve the linear programming problem in step 3-4-4, and perform operations on the active block based on the solution results.

[0076] Furthermore, the operations performed on the active block based on the solution results described in steps 3-4-5 are as follows:

[0077] Split operation, for The active block is set, and its eight child blocks are set as new active blocks. The midpoint of each child block is set as a new anchor point. The initial features of the new anchor points directly inherit the features of the parent block. The same method as in step 2-2 is used to derive features from each new anchor point. One Gaussian point;

[0078] The merge operation applies when both sibling blocks are active blocks and The active blocks are merged, and the anchor point feature of the parent block is obtained by taking the average value of the anchor point features of the child blocks;

[0079] The operation remains unchanged for The active block will not be operated.

[0080] Furthermore, steps 3-5 specifically include:

[0081] Step 3-5-1, Calculate the rendering depth map With dense depth map The relative difference between them is used to identify pixel regions with a relative difference greater than a threshold. Pixel regions with a relative difference greater than a threshold are considered as significantly different regions. The pixels in these regions are then back-projected into the three-dimensional space to find the corresponding active block and perform a splitting operation.

[0082] Step 3-5-2: For regions of Gaussian points that are not covered by active blocks but have gradients greater than the threshold, fill in the active blocks; based on the cumulative opacity of Gaussian points within the active blocks, crop active blocks with cumulative opacity less than the threshold.

[0083] Beneficial effects:

[0084] 1. The method of the present invention hierarchically divides the three-dimensional space by constructing a multi-resolution sparse voxel mesh, uses the center point of the voxel block as the anchor point to derive neural Gaussian points, and adaptively splits and merges the voxel blocks based on the fitting error during the training process to achieve dynamic adjustment of the Gaussian point density.

[0085] 2. This invention achieves adaptive allocation of Gaussian points based on the complexity of the scene while controlling the number of Gaussian points.

[0086] 3. This invention introduces a depth supervision mechanism, which uses monocular depth estimation to supplement additional geometric information, achieving more comprehensive Gaussian point coverage and significantly improving the reconstruction quality of textureless regions. Attached Figure Description

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0088] Figure 1 This is a schematic diagram illustrating the initial division of a three-dimensional space into active blocks, using a two-dimensional mesh as an example.

[0089] Figure 2 This is a system framework diagram of the method of the present invention.

[0090] Figure 3 This is a schematic diagram of the network structure for decoding Gaussian point attributes, using a multilayer perceptron for decoding color as an example in this invention.

[0091] Figure 4 This is a schematic diagram showing the correspondence between voxel sizes between two adjacent voxel grids.

[0092] Figure 5 This is a schematic diagram comparing the composite effect rendering of the method of the present invention with other methods from a new perspective.

[0093] Figure 6 This is a schematic diagram comparing the rendering of the new perspective synthesis effect of removing the adaptive multi-scale block optimization module and the complete method in the self-comparison experiment.

[0094] Figure 7 This is a comparative rendering of the synthesis effects from a new perspective, showing the removal of the block splitting module and the deep regularization module based on deep supervision, and the complete method in a self-comparison experiment.

[0095] Figure 8 This is a flowchart of the present invention. Detailed Implementation

[0096] This invention proposes a 3D Gaussian splashing method based on adaptive multi-resolution voxel meshes. It hierarchically divides the 3D space by constructing a multi-resolution sparse voxel mesh, deriving neural Gaussian points from the center points of voxel blocks as anchor points, and adaptively splitting and merging voxel blocks based on fitting errors during training to dynamically adjust the Gaussian point density. Simultaneously, a depth supervision mechanism is introduced, using monocular depth estimation to identify under-reconstructed regions and guide further subdivision of active blocks, significantly improving the reconstruction quality of textureless and under-observed regions. This invention can reduce Gaussian point redundancy, lower storage overhead, and improve rendering efficiency while maintaining rendering visual quality, making it suitable for high-quality real-time rendering of large-scale complex scenes.

[0097] The specific technical solution is as follows: A 3D Gaussian splashing method based on adaptive multi-resolution voxel meshes, which adaptively allocates and optimizes Gaussian points for reconstructing 3D scenes from multi-view images and performing real-time new view synthesis, such as... Figure 8 As shown, it includes the following steps:

[0098] Step 1: Input multi-view images of the scene, the camera pose corresponding to each image, and the sparse scene point cloud obtained by the Structure from Motion (SfM) algorithm, and construct a multi-resolution sparse voxel mesh; including the following steps:

[0099] Step 1-1: The sparse point cloud of the scene obtained by the motion structure recovery algorithm is processed. Mapped to voxel side length is In a sparse voxel mesh, all non-empty voxels are retained as the initial active block, and the midpoint of the active block is represented as:

[0100]

[0101] in, Indicates the deduplication operation;

[0102] Steps 1-2, based on point cloud Continue constructing voxel side lengths as follows , , , The voxel mesh yielded a total of 5 mesh layers, denoted as follows: .

[0103] like Figure 4 As shown, since the side lengths of the voxel blocks in adjacent layers are twice that of each other, the first... Layered voxel mesh Each voxel block in the process can be made by It consists of eight voxel blocks, among which Layers correspond to finer scales of division. Layers correspond to coarser scales of division. To clarify hierarchical relationships, the following terms are defined:

[0104] Parent block: in coarse-scale layer A voxel block whose spatial extent covers fine-scale layers. The sum of eight adjacent voxel blocks;

[0105] Sub-block: in fine-scale layers The eight adjacent voxel blocks in the block together form the spatial extent of a parent block;

[0106] Sibling blocks: Eight child blocks belonging to the same parent block are called sibling blocks.

[0107] This multi-resolution hierarchical structure provides the foundation for subsequent adaptive multi-scale block optimization. When a finer representation is needed, an active block can be split into eight sub-blocks; when a simplified representation is needed, eight sibling blocks can be merged into a single parent block.

[0108] Step 2: Construct a neural Gaussian representation based on a multi-resolution sparse voxel mesh, including the following steps:

[0109] Step 2-1: Use the midpoint of all initial active blocks as anchor points, and for each anchor point... Assign learnable features (Initialized as a zero vector) and scale factor ;

[0110] Step 2-2, derive each anchor point from Gaussian point The position of the Gaussian point is determined by the position of the anchor point. and learnable bias get:

[0111]

[0112] Step 3: Enter the training phase and optimize the learnable parameters from Step 2. , The system includes the following steps: 1) 2) 3) 4) 5) 6) 7) 8 ..." 6)"" 7)" 8)"" 8)"" 9)"" 9)""" 9""" 9—""" """"""""""""""

[0113] Step 3-1: Obtain the Gaussian point's attributes other than its position, including its color. Opacity ,size and rotation This is obtained by decoding the corresponding Multilayer Perceptron (MLP) network. The input to the MLP network, besides the features of the anchor points, is... It also adds the displacement direction of the anchor point relative to the camera position under the current training viewpoint. and distance To enhance the robustness of the perspective.

[0114] Step 3-2: Project the current Gaussian point using the splashing algorithm to obtain the rendered image under the current training viewpoint, and calculate the training loss.

[0115] Step 3-3: Optimize anchor point features using the backpropagation algorithm. Learnable bias And the multilayer perceptron network parameters in step 3-1;

[0116] Steps 3-4, each time passing After training in steps 3-1 to 3-3, an adaptive multi-scale Gaussian allocation is performed based on the fitting error within each active block. The minimum fitting error of the entire scene is modeled as an integer linear programming problem, and the merging or splitting of active blocks is performed based on the solution results.

[0117] Steps 3-5: After each execution of step 3-4, the Gaussian point distribution is optimized using depth supervision information and gradient information, and the active blocks in the blank areas are pruned.

[0118] Step 3-6: Repeat step 3-1 until the preset number of training iterations is reached.

[0119] Step 3-2 includes the following steps:

[0120] Step 3-2-1: After filtering out Gaussian points outside the view frustum based on the viewing angle, the Gaussian ellipsoid inside the view frustum is... Projected onto a two-dimensional Gaussian The piecewise differentiable rasterizer sorts them in depth order from front to back, and finally calculates the final rendered image using the alpha-blending algorithm:

[0121]

[0122] in, This indicates the pixel being queried. Represents the number of 2D Gaussians associated with the query pixel, for the A Gaussian point, This represents the Gaussian point. The color, Represents the Gaussian point Opacity Represents the Gaussian point The contribution weight to this pixel is determined by the opacity of that Gaussian point. and its value in 2D projection Joint decision, This indicates transmittance, i.e., the point at which it is located in Gauss. The product of the "transparency" of all previous Gaussian points, For pixels The color;

[0123] Step 3-2-2, also based on the alpha-blending algorithm, obtains the rendering depth map from this viewpoint. :

[0124]

[0125] in, for medium pixel The depth value, and > 0, Indicates the first The z-coordinate of a Gaussian point in the camera coordinate system, the range of which depends on the distance between the camera and the scene, and is usually... > 0. All other variables are the same as in step 3-2-1.

[0126] Step 3-2-3: Input the real image from the current training viewpoint into the monocular depth estimator to predict the dense depth map. However, since the depth map output by the monocular depth estimator is a relative depth map, it suffers from scale ambiguity and other issues, leading to inconsistencies in depth information between multiple views. Therefore, scale alignment is necessary to align the initial point cloud of the scene. Projection yields a sparse depth map And through linear transformation Align to sparse depth map The specific optimization objective is as follows:

[0127]

[0128] in and These are the scale factor and offset that need to be optimized, respectively. It is a sparse depth map medium pixel The depth value, It is a dense depth map Corresponding pixel depth value It is a pixel. The weights are calculated based on the reprojection error of COLMAP (the smaller the error, the higher the weight). The scale factor is iteratively optimized using gradient descent. and offset Finally, the adjusted dense depth map It can be determined by scale and offset The calculation yielded: ;

[0129] Step 3-2-4: Based on the rendered image obtained in Step 3-2-1 and the real image from this viewpoint, calculate the loss function:

[0130]

[0131] Among them, the first three loss terms are consistent with those in Scaffold GS (reference: Lu T, Yu M, Xu L, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024: 20654-20664). , , They respectively represent , , The weight of the loss term, The mean absolute error between the color values ​​of the rendered image and the real image was calculated. Represents structural similarity loss. The volume regularization loss encourages the minimization of the volume of each Gaussian point, reducing the overlap between Gaussian points. The depth regularization loss is derived from the depth map rendered in step 3-2-2. Compared with the dense depth map obtained in step 3-2-3 The Pearson correlation coefficient between them was calculated as follows:

[0132]

[0133] Describing covariance, Indicates variance.

[0134] Steps 3-4 include the following steps:

[0135] Step 3-4-1, Calculate The average gradient of the Gaussian points derived from each anchor point during training is used as the fitting error of that active block. ;

[0136] Step 3-4-2, using three binary variables This indicates the changes in an active block, specifically representing merging. , remain unchanged and division ;

[0137] Step 3-4-3, estimate the volume as Activity block The error remains constant under the three operations. It can be calculated from the current average fitting error:

[0138]

[0139] If the active block The sub-block was previously used as the active block to calculate the fitting error. Error after splitting It can be estimated as the sum of the errors of the sub-blocks; otherwise, multiply the current error by 1. To approximate the error after splitting into 8 sub-blocks:

[0140]

[0141] in, It is an active block The set of sub-blocks, It is a sub-block The error remains constant.

[0142] If the parent block's error has been calculated before, then this error is used as the error after merging with the sibling block. Otherwise, multiply the current error by 8 to estimate the combined error:

[0143]

[0144] in, It is an active block The parent block;

[0145] Step 3-4-4: To find the partition that minimizes the overall fitting error, the problem is modeled as follows:

[0146]

[0147] And it satisfies the following two constraints:

[0148]

[0149] in, This represents the number of active blocks before each optimization. The first constraint restricts the mutual exclusion of the three operations, while the second constraint controls that the number of active blocks after optimization cannot exceed the number before optimization.

[0150] Step 3-4-5: Solve the linear programming problem in step 3-4-4, and perform merging or splitting operations on the active blocks based on the solution results.

[0151] for The active block is set up, and its eight child blocks are set as new active blocks. The midpoint of each child block is set as a new anchor point. The initial features of the new anchor points directly inherit the features of the parent block, just like in step 2-2. Each new anchor point derives... One Gaussian point;

[0152] For those sibling blocks that are all active blocks and The active blocks are merged, and the anchor point feature of the parent block is obtained by taking the average value of the anchor point features of the child blocks;

[0153] for The active block remains unchanged.

[0154] After the operation, the number of active blocks is greater in areas with large fitting errors, and the density of Gaussian points increases, while the density of Gaussian points decreases in areas with small fitting errors.

[0155] Steps 3-5 include the following steps:

[0156] Step 3-5-1, Calculate With dense depth map The relative difference between them will be considered if the relative difference is greater than a set threshold. The pixel region is considered as a significantly different pixel region. The pixels in this region are projected back into the three-dimensional space to find the corresponding active blocks, and the splitting operation is performed on these active blocks.

[0157] Step 3-5-2: For blocks not covered by active blocks but with gradients greater than the threshold The active block is filled in the region of Gaussian points; based on the cumulative opacity of the Gaussian points within the active block, areas with cumulative opacity less than a threshold are cropped. The activity block.

[0158] This invention aims to improve the 3D Gaussian splashing algorithm, reduce redundant Gaussian points, lower storage overhead, and improve rendering quality. It can be applied to fields such as virtual reality, augmented reality, and autonomous driving, enabling real-time rendering of indoor and outdoor scenes.

[0159] Example:

[0160] The method flow of this embodiment is as follows: Figure 2 As shown, for a given scene dataset, a multi-resolution voxel mesh is first constructed from the input scene point cloud. Then, a multi-scale representation of the scene is achieved through fitting errors, thereby realizing the adaptive allocation of Gaussian points. The attributes of the Gaussian points are decoded by a multilayer perceptron network. Finally, the synthesis of the new perspective image is completed through differentiable rasterization rendering of the original 3DGS. The following describes the various steps of the present invention according to an embodiment:

[0161] Step 1: Input the sparse scene point cloud obtained by the Structure from Motion (SfM) algorithm and construct a multi-resolution sparse voxel mesh; including the following steps:

[0162] Step 1-1, as follows Figure 1 The image shows the sparse point cloud of the scene obtained by the motion structure recovery algorithm. Mapped to voxel side length is In a sparse voxel mesh, all non-empty voxels are retained as the initial active block, and the midpoint of the active block is denoted as... ,in Indicates the deduplication operation;

[0163] Steps 1-2, based on point cloud Continue constructing voxel side lengths as follows , , , The voxel mesh yielded a total of 5 mesh layers, denoted as follows: ,like Figure 4 As shown, the first layer Each voxel block can be made from The layer consists of eight voxel blocks, which form the relationship between parent and child blocks, facilitating the upward merging and downward splitting of active blocks in the subsequent multi-scale block optimization process.

[0164] Step 2: Construct a neural Gaussian representation based on a multi-resolution sparse voxel mesh, including the following steps:

[0165] Step 2-1: Use the midpoint of all initial active blocks as anchor points, and for each anchor point... Assign learnable features (Initialized as a zero vector) and scale factor ;

[0166] Step 2-2, derive each anchor point from Gaussian point The position of the Gaussian point is determined by the position of the anchor point. and learnable bias get: , It is initialized to a zero vector;

[0167] Step 3: Enter the training phase, optimize the learnable parameters from Step 2 and the parameters of the multilayer perceptron network used to decode the Gaussian point attributes, and periodically perform adaptive active block partitioning to optimize the distribution of Gaussian points, including the following steps:

[0168] Step 3-1: Other properties of the Gaussian point besides its position, including color. Opacity ,size and rotation By the corresponding multilayer perceptron Decoded. The input to the multilayer perceptron network, besides the features of the anchor points... It also adds the displacement direction of the anchor point relative to the camera position. and distance To enhance robustness to different viewpoints. Taking color rendering as an example, a multilayer perceptron... Network structure such as Figure 3 As shown, the input fusion features are first mapped through a linear layer, activated by the ReLU function, and then generated through a linear output layer. The color parameters are then applied in 3D, and finally the Sigmoid activation function is used to limit the output values ​​to 3D. Within the range.

[0169] Step 3-2: Project Gaussian points using the splashing algorithm to obtain the rendered image under the current training viewpoint, and calculate the training loss. Step 3-3: Optimize anchor point features using the backpropagation algorithm. Learnable bias And the multilayer perceptron network parameters in step 3-1;

[0170] Steps 3-4, each time passing In each training round, the fitting error of the entire scene is modeled as an integer linear programming problem, and the merging or splitting of activity blocks is performed based on the solution results.

[0171] Steps 3-5: After each execution of step 3-4, the Gaussian point distribution is optimized using depth supervision information and gradient information, and the active blocks in the blank areas are pruned.

[0172] Steps 3-6, repeat steps 3-1 to 3-5, until the training rounds reach 40,000.

[0173] Step 3-2 includes the following steps:

[0174] Step 3-2-1: After filtering out Gaussian points outside the view frustum based on the viewing angle, the Gaussian ellipsoid inside the view frustum is... Projected onto a two-dimensional Gaussian The piecewise differentiable rasterizer sorts them in depth order from front to back, and finally calculates the final rendered image using the alpha-blending algorithm:

[0175]

[0176] in, This indicates the pixel being queried. Represents the number of sorted 2D Gaussians associated with the query pixel, for the A Gaussian point, This represents the Gaussian point. The color, Represents the Gaussian point Opacity Represents the Gaussian point The contribution weight to this pixel is determined by the opacity of that Gaussian point. and its value in 2D projection Joint decision, This indicates transmittance, i.e., the point at which it is located in Gauss. The product of the "transparency" of all previous Gaussian points, For pixels The color;

[0177] Step 3-2-2, also based on the alpha-blending algorithm, obtains the rendering depth map from this viewpoint. :

[0178]

[0179] in, for medium pixel The depth value, Indicates the first The z-buffer has Gaussian points, and the other variables are the same as in step 3-2-1.

[0180] Step 3-2-3: Input the real image from the current training viewpoint into the monocular depth estimator Depth Anything to predict the dense depth map. However, since the depth map output by the monocular depth estimator is a relative depth map, it suffers from scale blurring and other issues, leading to inconsistencies in depth information between multiple views. Therefore, scale alignment is necessary. First, based on the camera pose of the current viewpoint, the initial point cloud of the scene is generated. Projecting all 3D points onto the imaging plane yields a sparse depth map. For each 3D point, its depth value is the point's position in the camera coordinate system. Coordinates. The sparse depth map only has depth values ​​at the point cloud projection locations; depth values ​​at other pixel locations are empty. Then, a linear transformation is performed... Align to sparse depth map To find the optimal scaling factor and offset Construct a weighted least squares optimization problem with the objective function as follows:

[0181]

[0182] in and These are the scale factor and offset that need to be optimized, respectively. It is a sparse depth map medium pixel The depth value, It is a dense depth map Corresponding pixel depth value It is a pixel. The weights are calculated based on the reprojection error of COLMAP (the smaller the error, the higher the weight). The scale factor is iteratively optimized using gradient descent. and offset Finally, the adjusted dense depth map It can be determined by scale and offset The calculation yielded: ;

[0183] Step 3-2-4: Calculate the loss function based on the obtained rendered image and the real image from that viewpoint.

[0184]

[0185] The first three loss terms are consistent with those in Scaffold GS. The mean absolute error between the color values ​​of the rendered image and the real image was calculated. Represents structural similarity loss. The volume regularization loss encourages the minimization of the volume of each Gaussian point, reducing the overlap between Gaussian points. The depth regularization loss is the depth map rendered in step 3-1-2. Compared with the dense depth map obtained in step 3-1-3 The Pearson correlation coefficient between them was calculated as follows:

[0186]

[0187] Describing covariance, Indicates variance.

[0188] Steps 3-4 include the following steps:

[0189] Step 3-4-1, Calculate The average gradient of the Gaussian points derived from each anchor point during training is used as the fitting error of that active block. ;

[0190] Step 3-4-2, using three binary variables This indicates the changes in an active block, specifically representing merging. , remain unchanged and division ;

[0191] Step 3-4-3, estimate the volume as Activity block The error remains constant under the three operations. It can be calculated from the current average fitting error:

[0192]

[0193] If the active block The sub-block was previously used as the active block to calculate the fitting error. Error after splitting It can be estimated as the sum of the errors of the sub-blocks; otherwise, multiply the current error by 1. To approximate the error after splitting:

[0194]

[0195] in, It is an active block The set of sub-blocks, It is a sub-block The error remains constant.

[0196] If the parent block's error has been calculated before, then this error is used as the error after merging with the sibling block. Otherwise, multiply the current error by 8 to estimate the combined error:

[0197]

[0198] in, It is the parent block of m;

[0199] Step 3-4-4: To find the partition that minimizes the overall fitting error, the problem is modeled as follows:

[0200]

[0201] And it satisfies the following two constraints:

[0202]

[0203] in, This represents the number of active blocks before each optimization. The first constraint restricts the mutual exclusion of the three operations, while the second constraint controls that the number of active blocks after optimization cannot exceed the number before optimization.

[0204] Steps 3-4-5 involve using the Gurobi solver to solve the linear programming problem in step 3-4-4, and then merging or splitting the active blocks based on the solution results. For The active block is defined as follows: its eight child blocks are designated as new active blocks, and the midpoint of each child block is designated as a new anchor point. The initial features of the new anchor points directly inherit the features of the parent block. For those sibling blocks that are also active blocks... The active blocks are merged, and the anchor point feature of the parent block is obtained by taking the average value of the anchor point features of the child blocks; The active blocks remain unchanged. After the operation, the number of active blocks increases in regions with large fitting errors, thus increasing the density of Gaussian points, while the density of Gaussian points decreases in regions with small fitting errors.

[0205] Steps 3-5 include the following steps:

[0206] Step 3-5-1, Calculate With dense depth map The relative difference between them is used to identify pixel regions with a relative difference greater than 0.5. These pixel regions are then projected back into the three-dimensional space to find the corresponding active blocks. The same splitting operation as in steps 3-4-5 is then performed on these active blocks.

[0207] Step 3-5-2: For blocks not covered by active blocks but with gradients greater than the threshold In the region of Gaussian points, active blocks are filled, and the side length of the filled active blocks is [value missing]. Based on the cumulative opacity of Gaussian points within the active block, crop out blocks with a cumulative opacity less than a threshold. The activity block.

[0208] Results analysis:

[0209] The experimental environment parameters for this embodiment are as follows:

[0210] The experimental platform parameters for model training and rendering are as follows: Ubuntu 20.04.6 LTS 64-bit operating system, AMD Ryzen 9 5900X CPU (12 cores and 24 threads), NVIDIA RTX A5000 graphics card, CUDA version 12.0, Python programming language, and PyTorch 1.12.1 deep learning framework.

[0211] The comparative experimental results of the present invention with the original 3D GS, Scaffold GS (Lu T, Yu M, Xu L, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024:20654-20664. ), and Octree GS (Ren K, Jiang L, Lu T, et al. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians[J]. arXivpreprint arXiv:2403.17898, 2024. ) are shown in Tables 1 and 2. The analysis is as follows (for fair comparison, the side length of the initial active block in each dataset is...). (The settings are equal to the voxel side length in Scaffold GS).

[0212] Experiments were conducted on six scenes from the publicly available 3D reconstruction standard dataset Mip-NeRF360 and two scenes from Tanks & Temples. The scene names for each dataset are shown in the first column of Table 1, where each category name represents a Bicycle, Garden, Stump, Room, Kitchen, Bonsai, Truck, and Train. New perspective composite renderings are shown below. Figure 5 As shown in the figure; the quantitative evaluation indicators for each scenario are compared in Table 1, and the average indicators for the two datasets are compared in Table 2.

[0213] like Figure 5 As shown, the method of this invention exhibits significant advantages when handling complex geometric structures. For example, in the Room scene, the reconstruction of furniture edges is sharper; in the Train scene, the rendering of complex chains is more accurate.

[0214] As shown in the comparison of metrics in Tables 1 and 2, the method of this invention achieved optimal or near-optimal results in most scenarios. On the six scenarios of the Mip-NeRF360 dataset, the average PSNR of the method of this invention reached 29.69, an improvement of 0.59 compared to the original 3D GS; the average SSIM reached 0.868, an improvement of 0.005 compared to the original 3D GS; and the average LPIPS reached 0.177, a decrease of 0.006 compared to the original 3D GS.

[0215] Table 1

[0216]

[0217] Table 2

[0218]

[0219] In the self-comparison experiment, the adaptive multi-scale block optimization module, the depth-information-based block splitting module, and the depth regularization in the loss function were removed respectively. The results are compared with the final experimental results in Table 3 and... Figure 6 , Figure 7 As shown, it can be seen that optimization based on multi-scale blocks produces better rendering results in areas with complex details. Adding supervision based on depth information can effectively reduce artifacts in the scene and improve rendering quality, demonstrating the effectiveness of these three strategies.

[0220]

[0221] Table 3

[0222] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a three-dimensional Gaussian splashing method based on an adaptive multi-resolution voxel mesh, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0223] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0224] This invention provides a concept and method for a three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel meshes. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh, characterized in that, Includes the following steps: Step 1: Obtain sparse point cloud of the scene based on multi-view images of the target scene and the camera pose corresponding to each image, and construct a multi-resolution sparse voxel mesh. Step 2: Construct a neural Gaussian representation based on a multi-resolution sparse voxel grid, and derive Gaussian points through learnable parameters; Step 3: Iterative training. The attributes of Gaussian points are decoded through a multilayer perceptron network. The parameters of the multilayer perceptron network and the learnable parameters in Step 2 are optimized, and adaptive active block division is performed to complete the 3D Gaussian splashing based on an adaptive multi-resolution voxel mesh.

2. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 1, characterized in that, Step 1, which involves constructing a multi-resolution sparse voxel mesh, specifically includes: Step 1-1: Convert the sparse point cloud of the scene Mapped to voxel size In a sparse voxel mesh, all non-empty voxels are retained as the initial active block, with the midpoint of the initial active block... It is expressed as follows: ； in, This indicates a deduplication operation; Steps 1-2: Based on the sparse point cloud of the scene Continue constructing voxel sizes of respectively , , and The voxel mesh is obtained in a total of 5 layers. The voxel blocks in two adjacent layers are defined as parent blocks and child blocks, respectively. The parent block consists of 8 child blocks in the next layer of mesh, and the 8 child blocks are sibling blocks.

3. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 2, characterized in that, Step 2, which describes constructing a neural Gaussian representation based on a multi-resolution sparse voxel grid, specifically includes: Step 2-1: Use the midpoint of all initial active blocks as anchor points, and for each anchor point... Assign learnable features and scale factor , means as follows: ； ； Among them, learnable features Initialize as a zero vector. Represents a 32-dimensional real vector space. Represents a 3-dimensional real vector space; Step 2-2, derive each anchor point from Gaussian point , means as follows: ； in, express The position of a Gaussian point For anchor point location, This is a learnable bias.

4. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 3, characterized in that, The iterative training described in step 3 specifically includes: Step 3-1: Obtain the attributes of the current Gaussian point other than its position, including color, through a multilayer perceptron network. Opacity ,size and rotation ; Step 3-2: Project the current Gaussian point using the splashing algorithm to obtain the rendered image under the current training viewpoint, and calculate the training loss; Step 3-3: Optimize anchor point features using the backpropagation algorithm. Learnable bias and the multilayer perceptron network parameters in step 3-1; Step 3-4: Repeat steps 3-1 to 3-3 and count the results; each execution... After the round, an adaptive multi-scale Gaussian allocation is performed based on the fitting error within each activity block. The fitting error of the entire scene is modeled as an integer linear programming problem. Based on the solution results, the activity blocks are merged or split, and steps 3-5 are executed. Steps 3-5: Calculate depth supervision information and gradient information, optimize the Gaussian point distribution, and perform pruning operations on active blocks in blank areas; Step 3-6: Repeat step 3-1 until the preset number of training iterations is reached.

5. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 4, characterized in that, Step 3-1: Obtain the attributes of the current Gaussian point other than its position using a multilayer perceptron network. Specific methods include: Learnable features of anchor points The direction of displacement of the camera position from the current perspective and distance The input is fed into a multilayer perceptron network to obtain the color. Opacity ,size and rotation .

6. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 5, characterized in that, Step 3-2, which involves calculating the training loss, specifically includes: Step 3-2-1: After filtering out Gaussian points outside the view frustum based on the current training perspective, the Gaussian ellipsoid inside the view frustum is... Projected onto a two-dimensional Gaussian The piecewise differentiable rasterizer sorts the rasterization data in depth order from front to back, and finally calculates the final rendered image, as shown below: ； in, This indicates the pixel being queried. Represents the number of 2D Gaussians associated with the query pixel, for the A Gaussian point, This represents the Gaussian point. The color, Represents the Gaussian point Opacity Represents the Gaussian point The contribution weight to this pixel is determined by the opacity of that Gaussian point. and its value in 2D projection Joint decision, Represents the Gaussian point The contribution weight of this pixel, Indicates transmittance. For pixels The color; Step 3-2-2: Calculate the rendering depth map from the current training viewpoint. , means as follows: ； in, To render depth map medium pixel The depth value, Indicates the first The z-coordinate of a Gaussian point in the camera coordinate system; Step 3-2-3: Input the real image from the current training viewpoint into the monocular depth estimator to predict the dense depth map. And perform scale alignment, that is, sparse point cloud of the scene. Projection yields a sparse depth map And through linear transformation, the dense depth map Align to sparse depth map ; The specific optimization goals are set as follows: ； in, and These are the scale factor and offset to be optimized, respectively. It is a sparse depth map medium pixel The depth value, It is a dense depth map Corresponding pixel The depth value, It is a pixel. The weights; Iterative optimization of the scale factor using gradient descent method and offset ; Final adjusted dense depth map From scale and offset The calculation yields the following result: ； Step 3-2-4: Calculate the loss function based on the rendered image obtained in Step 3-2-1 and the real image from the training viewpoint. , means as follows: ； in, , and They represent the weights, The mean absolute error of the color values. For structural similarity loss, For volume regularization loss, The depth regularization loss is derived from the rendering depth map. With dense depth map The Pearson correlation coefficient between them was calculated.

7. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 6, characterized in that, The depth regularization loss mentioned in step 3-2-4 is calculated using the following method: ； in, Describing covariance, Indicates variance.

8. The three-dimensional Gaussian splashing method based on adaptive multi-resolution voxel mesh according to claim 7, characterized in that, Steps 3-4 specifically include: Step 3-4-1, Calculate The average gradient of the Gaussian points derived from each anchor point in the wheel is used as the fitting error of the active block. ; Step 3-4-2, using a binary variable set Indicates the active block Changes: ； in, This indicates a merge operation. Indicates an operation that remains unchanged. Indicates a splitting operation; Step 3-4-3, estimate the volume as Activity block The error under three operations; among which, the error that remains constant is... It is calculated from the current average fitting error, and is specifically expressed as follows: ； If the active block The sub-block was previously used as the active block to calculate the fitting error. Error after splitting The sum of the errors of the sub-blocks; otherwise, multiply the current error by 1. The error after splitting into 8 sub-blocks is calculated and expressed as follows: ； in, It is an active block The set of sub-blocks, It is a sub-block The error remains constant; If the parent block's error has been calculated before, then this error is used as the error after merging with the sibling block. Otherwise, multiply the current error by 8 to calculate the combined error, as shown below: ； in, It is an active block The parent block; Step 3-4-4: Minimize the fitting error of the entire scene and model it as an integer linear programming problem, as follows: ； And it satisfies the following constraints: ； in, This indicates the number of active blocks before each optimization. Step 3-4-5: Solve the linear programming problem in step 3-4-4, and perform operations on the active block based on the solution results.

9. A three-dimensional Gaussian splashing method based on an adaptive multi-resolution voxel mesh according to claim 8, characterized in that, The operations performed on the active block based on the solution results in steps 3-4-5 are as follows: Split operation, for The active block is set, and its eight child blocks are set as new active blocks. The midpoint of each child block is set as a new anchor point. The initial features of the new anchor points directly inherit the features of the parent block. The same method as in step 2-2 is used to derive features from each new anchor point. One Gaussian point; The merge operation applies when both sibling blocks are active blocks and The active blocks are merged, and the anchor point feature of the parent block is obtained by taking the average value of the anchor point features of the child blocks; The operation remains unchanged for The active block will not be operated.

10. A three-dimensional Gaussian splashing method based on an adaptive multi-resolution voxel mesh according to claim 9, characterized in that, Steps 3-5 specifically include: Step 3-5-1, Calculate the rendering depth map With dense depth map The relative difference between them is used to identify pixel regions with a relative difference greater than a threshold. Pixel regions with a relative difference greater than a threshold are considered as significantly different regions. The pixels in these regions are then back-projected into the three-dimensional space to find the corresponding active block and perform a splitting operation. Step 3-5-2: For regions of Gaussian points that are not covered by active blocks but have gradients greater than the threshold, fill in the active blocks; based on the cumulative opacity of Gaussian points within the active blocks, crop active blocks with cumulative opacity less than the threshold.

Images