Frequency adaptive compaction method and system for enhancing gaussian splash detail recovery
By evaluating Gaussian contribution and spatial influence in the 2D image domain and quantizing frequency complexity in the 3D domain, adaptive Gaussian splitting is achieved, solving the problem of detail neglect in high-frequency regions caused by Gaussian splashing and improving rendering quality and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2026-03-24
- Publication Date
- 2026-06-09
Smart Images
Figure CN122176191A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer graphics and parallel computing technology, specifically relating to a frequency-adaptive densification method and system for enhancing Gaussian splash detail recovery. Background Technology
[0002] Gaussian splashing initializes the scene with a sparse point cloud generated by a structure of motion reconstruction (SfM) and gradually fills in the blank areas by splitting or cloning existing Gaussians, ultimately covering the entire scene in a compact and accurate manner while achieving fast and high-fidelity rendering. Therefore, Gaussian splashing has been widely used in fields such as simultaneous localization and mapping (SLAM), memory optimization, and dynamic scene modeling.
[0003] In recent years, Gaussian splashing has achieved remarkable success in novel perspective synthesis. However, in the high-frequency region, not all locations can be adequately represented because a mismatch often occurs between the scale of the Gaussian and the complexity of the region: complex details usually require smaller, finer Gaussians, while Gaussian splashing often uses larger and coarser Gaussians, which hinders accurate detail modeling.
[0004] In applications such as virtual reality and robotics, the demand for high-quality 3D scene reconstruction and photorealistic rendering is growing. Neural Radiation Field (NeRF) is a popular solution that has achieved remarkable rendering quality, but its use in real-time scenes is limited by slow inference speeds caused by dense sampling and volumetric rendering. In recent years, Gaussian splashing has achieved a good balance between speed and quality by initializing Gaussians with sparse point clouds obtained from Structure of Motion Reconstruction (SfM) and adaptively refining their properties during training. Despite these advantages, Gaussian splashing still faces difficulties in processing high-frequency regions, often producing blurry artifacts. This problem stems from the mismatch between the Gaussian scale and local frequencies. Specifically, due to the failure to adequately adapt to local frequencies, large and coarse Gaussians blur edges and lose fine details when processing high-frequency regions. This is because existing Gaussian compaction strategies rely primarily on geometric gradient signals. This results in the neglect of regions with inconspicuous geometric features but crucial image details.
[0005] Novel View Synthesis (NVS) aims to generate images from previously unseen perspectives. Traditional methods, such as multi-view stereo vision, often perform poorly in complex scenes. In recent years, Neural Radiation Fields (NeRF), a deep learning-based 3D scene representation method, has achieved photorealistic novel view synthesis through volumetric rendering using light. However, dense sampling and MLP inference lead to extremely high computational costs. To improve efficiency, some methods have replaced MLPs with sparse voxel meshes or hash tables and optimized the sampling process. Advances such as MipNeRF and Instant NGP have improved quality and training speed, but real-time rendering remains challenging. Overall, while NeRF has made significant improvements in efficiency, balancing high fidelity and real-time rendering remains a difficult problem. Summary of the Invention
[0006] To address the aforementioned shortcomings in existing technologies, the frequency-adaptive densification method and system for enhancing Gaussian splash detail recovery provided by this invention solves the problems of existing Gaussian splashing methods being unable to perform fine modeling, easily ignoring important detail areas, and resulting in low rendering quality.
[0007] To achieve the aforementioned objectives, the present invention employs the following technical solution: a frequency-adaptive densification method for enhancing Gaussian splatter detail recovery, comprising: In the 2D image domain, the contribution of each Gaussian to the Gaussian pixel and the spatial influence range of each Gaussian on the image plane are evaluated, and each Gaussian is assigned a splitting priority. In the 3D domain, the degree of mismatch between each Gaussian and the local frequency complexity is quantized by multiple-order moments; For Gaussians whose mismatch with local frequency complexity exceeds a preset threshold, adaptive splitting operations are performed according to splitting priority.
[0008] Furthermore, in the 2D image domain, the contribution of each Gaussian to the Gaussian pixel and the spatial influence range of each Gaussian on the image plane are evaluated, and each Gaussian is assigned a splitting priority, including: The high-frequency contribution of Gaussian gas is calculated using the following expression:
[0009] In the formula, Gauss High-frequency contribution value, This represents the total number of pixels covered by the Gaussian curve. Indicates the first Frequency value of each pixel For Gauss's first The contribution value of opacity per pixel. For Gauss's first The contribution value of opacity per pixel; The expression for calculating the number of pixels covered by the Gaussian conjugate in the rendered image is as follows:
[0010] In the formula, Gauss The number of covered pixels that contribute to the rendered image. This is an indicator function; it takes the value 1 if the condition within the parentheses is met, and 0 otherwise. based on and The frequency coverage factor is calculated using the following expression:
[0011] In the formula, Gauss The corresponding frequency coverage factor, To adjust the index of pixel coverage, It is a constant used to prevent division by zero errors; The frequency coverage factor is normalized to obtain the normalized coverage factor; The original geometric gradient of Gaussian is weighted based on the normalized covering factor, and its expression is as follows:
[0012] In the formula, This represents the weighted gradient value. Gauss Average gradient across multiple views, For normalized coverage factor, To control A coefficient representing the degree of influence on the gradient; Each Gaussian is assigned a splitting priority based on its weighted gradient value.
[0013] Furthermore, in the 3D domain, the degree of mismatch between each Gaussian and the local frequency complexity is quantized using multiple-order moments, including: The frequency response characterizes the energy distribution of a Gaussian, and its expression is:
[0014] In the formula, Gauss of Energy distribution corresponding to the step moment Indicates proportionality by the sign. Represents the entire three-dimensional Euclidean space. Represents the frequency vector. The first digit represents the Euclidean norm of the frequency vector. Power of 1 Gauss Fourier transform; The high-frequency fitness score of Gaussian is calculated based on its energy distribution, and its expression is as follows:
[0015] In the formula, For Gauss The high-frequency adaptability score, , , All are weighting factors; Calculate the frequency complexity of the Gaussian region; The difference between the high-frequency fitness score of each Gaussian and the frequency complexity of the corresponding region is quantified to obtain the degree of mismatch between each Gaussian and the local frequency complexity.
[0016] Furthermore, Gauss of The specific expression for the step moment is:
[0017] In the formula, Gauss of Step moment, Gauss 3D spatial position vector, Gauss The center position vector in three-dimensional space.
[0018] Secondly, the present invention also provides a system for implementing a frequency-adaptive densification method for enhancing Gaussian splatter detail recovery, comprising: The frequency coverage gradient adjustment module is used to evaluate the contribution of each Gaussian to Gaussian pixels and the spatial influence range of each Gaussian on the image plane in the 2D image domain, and to assign a splitting priority to each Gaussian. The multi-moment matched quantization module is used to quantize the degree of mismatch between each Gaussian and the local frequency complexity in the 3D domain using multi-moments. The adaptive densification module is used to perform adaptive splitting operations on Gaussians whose mismatch with local frequency complexity exceeds a preset threshold according to splitting priority.
[0019] Furthermore, the system is trained by minimizing a loss function, the expression of which is:
[0020]
[0021]
[0022]
[0023] In the formula, Represents the loss function. To lose weight, This represents the mean absolute error between the rendered image and the real image. This represents the structural similarity index loss term. Represents the high-frequency loss term. This represents the weighting coefficient of the high-frequency loss term; This represents the total number of pixels in the image. To render an image in pixels The color value at that location, For real images in pixels The color value at that location, Indicates the rendered image. Represents a real image. for The local mean, for The local mean; for The local variance, for Local variance; for and Local covariance; , These are all constants used to maintain computational stability. This represents the high-frequency weight map obtained by applying the Laplacian operator to a grayscale image. This represents the Euclidean norm.
[0024] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of a frequency-adaptive densification method for enhancing Gaussian splatter detail recovery.
[0025] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of a frequency-adaptive densification method for enhancing Gaussian splatter detail recovery.
[0026] The beneficial effects of this invention are as follows: 1. By evaluating the contribution of each Gaussian to Gaussian pixels and the spatial influence range of each Gaussian on the image plane in the 2D image domain, a splitting priority is assigned to each Gaussian, encouraging the preferential densification of Gaussians located in high-frequency regions and with large coverage areas, thereby enhancing the expressive power of local regions.
[0027] 2. By quantizing the mismatch between each Gaussian and the local frequency complexity in the 3D domain through multi-order moment quantization, computational resources are ensured to be precisely allocated to regions that truly require finer-grained representation, while avoiding over-densification in flat regions.
[0028] 3. By introducing a high-frequency loss term, it encourages better reconstruction of high-frequency content such as texture edges and sharp structures, enhances the preservation of details, and improves the reconstruction quality of fine details in the rendered image. Attached Figure Description
[0029] Figure 1 A flowchart of a frequency-adaptive densification method for enhancing Gaussian splash detail recovery provided in this embodiment; Figure 2 Quantitative results from comparative experiments provided for the examples; Figure 3 Qualitative results of comparative experiments provided for the examples. Detailed Implementation
[0030] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0031] like Figure 1 As shown, in one embodiment of the present invention, the frequency-adaptive densification method for enhancing Gaussian splatter detail recovery includes the following steps: S1. In the 2D image domain, the contribution of each Gaussian to the Gaussian pixel and the spatial influence range of each Gaussian on the image plane are evaluated through the frequency coverage gradient adjustment module, and a splitting priority is assigned to each Gaussian.
[0032] In this invention, Gaussian is not a purely mathematical probability distribution function, but rather refers to a three-dimensional Gaussian primitive with physical properties in 3D scene reconstruction. Each Gaussian represents a tiny spatial entity unit in 3D space with a specific shape and transparency, used to characterize the surface geometry and optical features of an object.
[0033] Step S1 specifically includes: The high-frequency contribution of Gaussian gas is calculated using the following expression:
[0034] In the formula, Gauss High-frequency contribution value, This represents the total number of pixels covered by the Gaussian curve. Indicates the first The frequency values of each pixel are calculated by applying the Laplacian operator to the grayscale image. For Gauss's first The contribution value of opacity per pixel. For Gauss's first The contribution value of opacity per pixel; The expression for calculating the number of pixels covered by the Gaussian conjugate in the rendered image is as follows:
[0035] In the formula, Gauss The number of covered pixels that contribute to the rendered image. This is an indicator function; it takes the value 1 if the condition within the parentheses is met, and 0 otherwise. based on and The frequency coverage factor is calculated using the following expression:
[0036] In the formula, Gauss The corresponding frequency coverage factor, To adjust the index of pixel coverage, It is a constant used to prevent division by zero errors; To incorporate this factor into the adaptive splitting process, the frequency coverage factor is normalized to obtain the normalized coverage factor. Through this factor, those with high A Gaussian value will result in an amplified gradient signal, which will be preferentially triggered during the splitting process.
[0037] The original geometric gradient of Gaussian is weighted based on the normalized covering factor, and its expression is as follows:
[0038] In the formula, This represents the weighted gradient value. Gauss Average gradient across multiple views, For normalized coverage factor, To control A coefficient representing the degree of influence on the gradient; Each Gaussian is assigned a splitting priority based on its weighted gradient value.
[0039] This strategy encourages the preferential densification of Gaussians located in high-frequency regions and with large coverage areas, thereby enhancing the expressive power of local regions. This ensures that limited computational resources are precisely allocated to Gaussians that have a significant impact on visual quality but are not fully split.
[0040] S2. In the 3D domain, the degree of mismatch between each Gaussian and the local frequency complexity is quantized using the multi-moment matching quantization module.
[0041] From a 3D perspective, to better adapt the Gaussian in high-frequency regions to the complexity of different areas, this invention introduces multiple moments of the Gaussian. These statistical descriptors are used to guide its adaptive splitting based on the structural properties of the Gaussian.
[0042] Specifically, it includes: For 3D Gaussian ,That The specific steps of the moment are:
[0043] In the formula, Gauss of Step moment, Gauss 3D spatial position vector, Gauss The center position vector in three-dimensional space.
[0044] By utilizing the analytic properties of the Fourier domain, energy can be characterized based on the frequency response, as expressed by:
[0045] In the formula, Gauss of Energy distribution corresponding to the step moment Indicates proportionality by the sign. Represents the entire three-dimensional Euclidean space. Represents the frequency vector. The first digit represents the Euclidean norm of the frequency vector. Power of 1 Gauss Fourier transform.
[0046] Among them, lower-order moments (e.g., k= 1, 2) primarily capture the low-frequency components of Gaussians, where the second-order moments characterize their spatial extent and shape scale. Conversely, higher-order moments (e.g., ) k = 3, 4) Focusing on high-frequency components: The third moment measures the local skewness of the Gaussian, while the fourth moment reflects its sharpness. Therefore, the third and fourth moments of the Gaussian are used to quantitatively assess the level of fine detail it represents, and its spatial scale is estimated by the second moment. This allows us to determine whether a Gaussian is small and fine enough to match the frequency complexity of its local regions. To this end, for Gaussians located in the high-frequency region, the high-frequency fitness score of the Gaussian is calculated based on its energy distribution, expressed as:
[0047] In the formula, For Gauss The high-frequency adaptability score, , , All are weighted factors.
[0048] Each Gaussian is evaluated for its ability to adequately capture local high-frequency details using a high-frequency adaptive score. If a Gaussian's score is lower than the frequency complexity of its corresponding region, it is considered a "frequency-mismatched Gaussian," thus triggering an adaptive splitting operation.
[0049] The frequency complexity of the corresponding Gaussian region is calculated as follows: the reference image is converted to grayscale space, and a second-order differential convolution operation is performed on the image using the Laplacian operator. By calculating the rate of change of brightness between each pixel and its four neighboring pixels, the local high-frequency components of the image are extracted. Subsequently, the absolute value of the convolution result is taken to quantize the energy intensity, and the entire image is normalized to obtain a continuous frequency complexity map representing the image's texture details and edge features.
[0050] The difference between the high-frequency fitness score of each Gaussian and the frequency complexity of the corresponding region is quantified to obtain the degree of mismatch between each Gaussian and the local frequency complexity. When the difference exceeds a preset threshold, the Gaussian is determined to need to be split. This adaptive splitting mechanism ensures that computational resources are accurately allocated to regions that truly require finer-grained representation, while avoiding over-densification in flat regions.
[0051] S3. The adaptive densification module performs adaptive splitting operations on Gaussians whose mismatch with local frequency complexity exceeds a preset threshold according to the splitting priority, thereby completing the frequency adaptive densification to enhance the recovery of Gaussian splash details.
[0052] In 3D scene representation, high-frequency details capture the fine structures that significantly impact visual quality. While current Gaussian splashing methods are optimized for global structure and reconstruction errors, they often neglect high-frequency detail modeling. Meanwhile, steps S1 and S2 guide the densification process in high-frequency regions from two perspectives: 3D geometric properties and 2D projection effects. Specifically, step S1 evaluates the influence of Gaussians on the image plane using a frequency coverage factor, assigning higher splitting priority to "important" Gaussians. Step S2 quantizes the frequency domain characteristics of the Gaussians themselves using multi-moment quantization, identifying Gaussians that are "mismatched" with local frequency complexity and triggering splitting. Together, these two steps ensure that a sufficient number of fine-grained Gaussians are generated in the high-frequency region, providing the necessary representational power for the reconstruction of fine details.
[0053] The frequency-adaptive densification system for enhanced Gaussian splatter detail recovery provided by this invention is trained by minimizing the loss function to encourage better reconstruction of high-frequency content such as texture edges and sharp structures.
[0054] The expression for the loss function is:
[0055]
[0056]
[0057]
[0058] In the formula, Represents the loss function. To lose weight, This represents the mean absolute error between the rendered image and the real image. This represents the structural similarity index loss term. Represents the high-frequency loss term. This represents the weighting coefficient of the high-frequency loss term; This represents the total number of pixels in the image. To render an image in pixels The color value at that location, For real images in pixels The color value at that location, Indicates the rendered image. Represents a real image. for The local mean, for The local mean; for The local variance, for Local variance; for and Local covariance; , These are all constants used to maintain computational stability. This represents the high-frequency weight map obtained by applying the Laplacian operator to a grayscale image. This represents the Euclidean norm.
[0059] This invention introduces multi-order moments to characterize the frequency domain properties of each Gaussian, as these moments capture the statistical properties of the Gaussian to reflect its frequency characteristics. Based on this, a high-frequency adaptive score is defined to evaluate its match with local frequency complexity. An adaptive splitting operation is triggered when the score is significantly lower than the local frequency requirement. Furthermore, in the 2D image domain, a frequency coverage factor is proposed to quantify the contribution of each Gaussian to high-frequency pixels and its spatial coverage. This factor is then combined with conventional geometric gradients to assign weights, thereby prioritizing the densification of Gaussians located in high-frequency regions and having a large spatial influence. Finally, a frequency-weighted term is added to the conventional reconstruction loss to encourage better reconstruction of high-frequency content such as texture edges and sharp structures. In summary, unlike most existing methods that process large Gaussians from only a single 3D or 2D perspective, this invention achieves more accurate refinement by integrating both aspects.
[0060] To verify the effectiveness of this invention, following the Gaussian splashing experimental setup, evaluations were conducted on three representative datasets: nine diverse and complex scenes from Mip-NeRF 360, the "Truck" and "Train" scenes from Tanks and Temples, and two scenes from Deep Blending, Dr. Johnson and Playroom. For quantitative evaluation, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Perceptual Similarity Index (LPIPS) were used.
[0061] The method provided in this invention was implemented in PyTorch, and the default Gaussian splash training parameters were used to ensure consistency. During the experiments, to balance scale estimation and sensitivity to high-frequency details, the weights of the second, third, and fourth moments were set to [values to be inserted here]. = 1.0、 = 0.1 and = 0.5. Frequency coverage spatial weight. It was set to 2.0 to intentionally prioritize splitting larger Gaussians. A small constant was introduced to prevent division by zero errors. = 1 × 10 6 In addition, weight = 0.2 follows the default settings of Gaussian splashing, while the weight of high-frequency terms is 0.2. The value was set to 0.8 to enhance the representation of high-frequency details without compromising training stability. Quantitative results are as follows: Figure 2 As shown, the results demonstrate that, compared to the Gaussian splashed baseline, the present invention achieves superior performance on most metrics, quantitatively validating the effectiveness of our method in improving reconstruction quality. Qualitative results are as follows: Figure 3 As shown, the results indicate that the present invention significantly reduces rendering blur, improves rendering quality in all scenes, and effectively alleviates the blur artifacts observed in some scenes.
[0062] In addition, ablation experiments were conducted on the core modules: Multi-Moment Matched Quantization (MGAS), Frequency Cover Gradient Adjustment (FCGA), and High-Frequency Loss (HFLS) to evaluate their impact on performance. Table 1 shows the quantitative results under different configurations with and without these components. The results show that each individual module improves the results, while combining two or all three modules yields further gains. This demonstrates that MGAS and FCGA enhance rendering through adaptive splitting and gradient adjustment, while HFLS emphasizes high-frequency details through Laplacian weighted feature maps, making the modules complementary and improving reconstruction fidelity.
[0063] Table 1
[0064] In summary, this invention selectively densifies high-frequency regions. Although adding fine-grained Gaussians to these regions increases memory usage, by densifying only frequency-mismatched Gaussians, the overhead is kept within a controllable range, effectively balancing detail preservation and computational efficiency.
Claims
1. A frequency-adaptive densification method for enhancing Gaussian splatter detail recovery, characterized in that, include: In the 2D image domain, the contribution of each Gaussian to the Gaussian pixel and the spatial influence range of each Gaussian on the image plane are evaluated, and each Gaussian is assigned a splitting priority. In the 3D domain, the degree of mismatch between each Gaussian and the local frequency complexity is quantized by multiple-order moments; For Gaussians whose mismatch with local frequency complexity exceeds a preset threshold, adaptive splitting operations are performed according to splitting priority.
2. The method according to claim 1, characterized in that, In the 2D image domain, the contribution of each Gaussian to the Gaussian pixel and the spatial influence range of each Gaussian on the image plane are evaluated, and a splitting priority is assigned to each Gaussian, including: The high-frequency contribution of Gauss is calculated using the following expression: In the formula, Gauss High-frequency contribution value, This represents the total number of pixels covered by the Gaussian curve. Indicates the first Frequency value of each pixel For Gauss's first The contribution value of opacity per pixel. For Gauss's first The contribution value of opacity per pixel; The expression for calculating the number of pixels covered by the Gaussian conjugate in the rendered image is as follows: In the formula, Gauss The number of covered pixels that contribute to the rendered image. This is an indicator function; it takes the value 1 if the condition within the parentheses is met, and 0 otherwise. based on and The frequency coverage factor is calculated using the following expression: In the formula, Gauss The corresponding frequency coverage factor, To adjust the index of pixel coverage, It is a constant used to prevent division by zero errors; The frequency coverage factor is normalized to obtain the normalized coverage factor; The original geometric gradient of Gaussian is weighted based on the normalized covering factor, and its expression is as follows: In the formula, This represents the weighted gradient value. Gauss Average gradient across multiple views, For normalized coverage factor, To control A coefficient representing the degree of influence on the gradient; Each Gaussian is assigned a splitting priority based on its weighted gradient value.
3. The method according to claim 2, characterized in that, In the 3D domain, the degree of mismatch between each Gaussian and the local frequency complexity is quantized using multiple-order moments, including: The frequency response characterizes the energy distribution of a Gaussian, and its expression is: In the formula, Gauss of Energy distribution corresponding to the step moment Indicates proportionality by the sign. Represents the entire three-dimensional Euclidean space. Represents the frequency vector. The first digit represents the Euclidean norm of the frequency vector. Power of 1 Gauss Fourier transform; The high-frequency fitness score of Gaussian is calculated based on its energy distribution, and its expression is as follows: In the formula, For Gauss The high-frequency adaptability score, , , All are weighting factors; Calculate the frequency complexity of the Gaussian region; The difference between the high-frequency fitness score of each Gaussian and the frequency complexity of the corresponding region is quantified to obtain the degree of mismatch between each Gaussian and the local frequency complexity.
4. The method according to claim 3, characterized in that, Gauss of The specific expression for the step moment is: In the formula, Gauss of Step moment, Gauss 3D spatial position vector, Gauss The center position vector in three-dimensional space.
5. A system for implementing the frequency-adaptive densification method for enhanced Gaussian splatter detail recovery as described in any one of claims 1 to 4, characterized in that, include: The frequency coverage gradient adjustment module is used to evaluate the contribution of each Gaussian to Gaussian pixels and the spatial influence range of each Gaussian on the image plane in the 2D image domain, and to assign a splitting priority to each Gaussian. The multi-moment matched quantization module is used to quantize the degree of mismatch between each Gaussian and the local frequency complexity in the 3D domain using multi-moments. The adaptive densification module is used to perform adaptive splitting operations on Gaussians whose mismatch with local frequency complexity exceeds a preset threshold according to splitting priority.
6. The system according to claim 5, characterized in that, The system is trained by minimizing a loss function, the expression of which is: In the formula, Represents the loss function. To lose weight, This represents the mean absolute error between the rendered image and the real image. This represents the structural similarity index loss term. Represents the high-frequency loss term. This represents the weighting coefficient of the high-frequency loss term; This represents the total number of pixels in the image. To render an image in pixels The color value at that location, For real images in pixels The color value at that location, Indicates the rendered image. Represents a real image. for The local mean, for The local mean; for The local variance, for Local variance; for and Local covariance; , These are all constants used to maintain computational stability. This represents the high-frequency weight map obtained by applying the Laplacian operator to a grayscale image. This represents the Euclidean norm.
7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 4.
8. A computer-readable storage medium, characterized in that, The device stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 4.