Dynamic deletion method and system for three-dimensional Gaussian spattering

Abstract

The invention provides a dynamic deletion method and system for three-dimensional Gaussian spatter, and relates to the technical field of computer graphics, the method comprises the following steps: obtaining a Gaussian ellipse set of a three-dimensional scene, the Gaussian ellipse set comprising a plurality of Gaussian ellipses with scale parameters and transparency parameters; performing dynamic density control on each Gaussian ellipse, performing dynamic elimination on the Gaussian ellipse set according to a scale elimination threshold value and a transparency elimination threshold value, and retaining Gaussian ellipses meeting conditions; and performing rasterization rendering on the retained Gaussian ellipse, and introducing a Huber loss function in the rasterization rendering process to optimize a rendered image so as to relieve image artifacts and obtain a final rendered image. According to the method, the problems of transition, insufficiency and artifacts in three-dimensional Gaussian spattering reconstruction are systematically solved, compared with a traditional fixed threshold and linear loss method, the density control precision and the rendering optimization stability are remarkably improved, and high-quality scene reconstruction and vivid image rendering are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics, and in particular to a dynamic deletion method and system for three-dimensional Gaussian splashing. Background Art

[0002] In the fields of computer graphics and computer vision, Novel View Synthesis (NVS) is a core research direction with a wide range of applications, covering multiple fields such as virtual reality, drone navigation, and medical image reconstruction. With the development of technology, NVS has achieved remarkable results in improving image generation quality and rendering efficiency. In 2020, Mildenhall et al. proposed the Neural Radiance Field (NeRF), which uses a multilayer perceptron (MLP) to represent three-dimensional scene information and generates realistic new perspective images through volume rendering technology, which has promoted the development of the NVS field. However, NeRF faces computational efficiency challenges in practical applications, making it difficult to achieve real-time rendering.

[0003] Subsequently, 3D Gaussian Splatting (3DGS) technology emerged. Building on NeRF and combining it with a feature-grid-based representation, 3DGS uses a Gaussian distribution to represent points in a 3D scene, achieving efficient and high-precision rendering. 3DGS represents a complex scene as a set of 3D Gaussian ellipses, rendering them to screen space through splatter-based rasterization. It optimizes the properties of the Gaussian ellipses to accurately reconstruct the scene, and performs a 2D dilation operation to improve rendering. However, 3DGS still suffers from issues such as over-reconstruction, under-reconstruction, and floating artifacts.

[0004] To address these issues, numerous research efforts have been launched. MS-3DGS introduces a multi-scale Gaussian representation, which somewhat alleviates the over-reconstruction problem at low resolutions, but is less effective at medium and high resolutions. 2DGS compresses a three-dimensional Gaussian ellipse into a two-dimensional oriented planar Gaussian ellipse, combining depth distortion and normal consistency to alleviate the under-reconstruction problem. Other research has also introduced filtering mechanisms to suppress artifacts. However, most of these methods only address a specific issue and have limitations.

[0005] In 3DGS and related extension methods, the number of Gaussian ellipses rendered has a significant impact on the reconstruction quality. Too many will lead to over-reconstruction, and too few will result in under-reconstruction. The current Gaussian culling mechanism usually sets two fixed thresholds, which are used to eliminate Gaussian ellipses with too large a scale (greater than 10% of the current scene range) and too small a transparency (less than 0.005). However, this fixed threshold method has defects. It will mistakenly retain some Gaussian ellipses with slightly larger scales, causing them to cover other ellipses and cause over-reconstruction; at the same time, it will also mistakenly eliminate some Gaussian ellipses with slightly smaller transparency that contribute to the reconstruction, resulting in under-reconstruction.

[0006] Furthermore, during the rendering process, most existing methods use mean absolute error (MAE) to optimize images. However, the linear growth of MAE can lead to over-optimization when faced with small outliers, causing large fluctuations in the reconstruction results and the appearance of artifacts. In summary, existing technologies are insufficient in addressing the problems of over-reconstruction, under-reconstruction, and artifacts in 3DGS reconstruction, making it difficult to fully address these issues. A new approach is urgently needed to address these issues and improve the quality and effectiveness of 3D scene reconstruction. Summary of the Invention

[0007] To this end, an embodiment of the present invention provides a dynamic deletion method and system for three-dimensional Gaussian splash, which is used to solve the problems of over-reconstruction, under-reconstruction and artifacts in the existing technology of three-dimensional Gaussian splash when reconstructing scenes, as well as the problems that traditional methods only alleviate a certain problem, fail to solve it comprehensively, and have poor accuracy due to fixed thresholds and linear growth loss functions.

[0008] To solve the above problem, an embodiment of the present invention provides a dynamic deletion method for three-dimensional Gaussian splash, which includes:

[0009] Obtaining a Gaussian ellipse set of a three-dimensional scene, wherein the Gaussian ellipse set includes a plurality of Gaussian ellipses with a scale parameter and a transparency parameter;

[0010] Dynamically control the density of each Gaussian ellipse, dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse, dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse, and dynamically reject the Gaussian ellipse set according to the scale rejection threshold and the transparency rejection threshold, retaining Gaussian ellipses that meet the conditions;

[0011] The retained Gaussian ellipse is rasterized and rendered. The Huber loss function is introduced in the rasterization rendering process to optimize the rendered image to alleviate image artifacts and obtain the final rendered image.

[0012] Preferably, the method for dynamically calculating the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse is:

[0013] T S =(max s -min s )×s×β+β×min s ;

[0014] Among them, T S is the scale removal threshold, min s and max s are the preset minimum and maximum thresholds for the scale, s is the scale value of the current Gaussian ellipse, and β is a calculation parameter set based on experience.

[0015] Preferably, the method for dynamically calculating the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse is:

[0016] T o =(max O -min O )×op×α+α×max O ×min O ;

[0017] Among them, T o is the transparency rejection threshold, min O and max O are the preset minimum and maximum transparency thresholds, op is the transparency of the current Gaussian ellipse, and α is a calculation parameter set based on experience.

[0018] Preferably, the process of dynamically removing the Gaussian ellipse set according to the scale removal threshold and the transparency removal threshold comprises:

[0019] Eliminate Gaussian ellipses whose scale parameters are greater than the corresponding scale elimination threshold to alleviate the problem of over-reconstruction;

[0020] Gaussian ellipses with transparency parameters less than the corresponding transparency culling threshold are eliminated to alleviate the problem of insufficient reconstruction.

[0021] Preferably, the expression of the Huber loss function is:

[0022]

[0023] Among them, L δ Huber loss function with parameter δ as the threshold, y is the actual value, f(x) is the value of the model, and δ is a parameter.

[0024] Preferably, the Gaussian ellipse is defined by position, covariance matrix and transparency.

[0025] An embodiment of the present invention further provides a dynamic deletion system for three-dimensional Gaussian splashes, which is used to implement the above-mentioned dynamic deletion method for three-dimensional Gaussian splashes, specifically comprising:

[0026] A Gaussian parameter acquisition module, configured to acquire a Gaussian ellipse set of a scene, wherein the Gaussian ellipse set includes a plurality of Gaussian ellipses having scale parameters and transparency parameters;

[0027] A dynamic density control module is used to perform dynamic density control on each Gaussian ellipse, dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse, dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse, and dynamically reject the Gaussian ellipse set according to the scale rejection threshold and the transparency rejection threshold, retaining Gaussian ellipses that meet the conditions;

[0028] The loss function optimization module is used to perform rasterization rendering on the retained Gaussian ellipse. During the rasterization rendering process, the Huber loss function is introduced to optimize the rendered image to alleviate image artifacts and obtain the final rendered image.

[0029] Preferably, the dynamic density control module includes:

[0030] A scale threshold calculation unit, used to dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse;

[0031] A transparency threshold calculation unit, used to dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse;

[0032] The culling unit is configured to dynamically cull the Gaussian ellipse set according to the scale culling threshold and the transparency culling threshold, and retain Gaussian ellipses that meet the conditions.

[0033] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned dynamic deletion method for three-dimensional Gaussian splashing.

[0034] An embodiment of the present invention further provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned dynamic deletion method for three-dimensional Gaussian splashing.

[0035] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0036] (1) The present invention proposes a dynamic calculation method for calculating the rejection threshold based on the scale and transparency of Gaussian ellipse. In order to solve the over-reconstruction phenomenon caused by a fixed scale threshold, the present invention calculates the degree to which the Gaussian ellipse covers other Gaussian ellipses during the reconstruction process according to the scale of the current Gaussian ellipse and the maximum and minimum values ​​of the scale in the reconstruction through a preset function, and sets a stricter rejection threshold for Gaussian ellipses with a large coverage degree, so as to more accurately reject Gaussian ellipses with too large scales, thereby alleviating the over-reconstruction phenomenon; in order to solve the problem of insufficient reconstruction caused by a fixed transparency threshold, the present invention passes the transparency of the current Gaussian ellipse and the maximum and minimum values ​​of the transparency in the reconstruction as parameters into a preset function to calculate the contribution of the Gaussian ellipse to the reconstruction result, and sets a looser rejection threshold for Gaussian ellipses with high contribution, so as to more accurately retain useful Gaussian ellipses, thereby alleviating the problem of insufficient reconstruction.

[0037] (2) The present invention designs a loss function based on pixel smoothness. In order to solve the artifact phenomenon in reconstruction caused by the linear growth loss function, the present invention introduces the Huber loss function, which measures the smoothness of the output by the difference between pixels or gradients. When the difference is less than the set threshold, the Huber loss function performs quadratic smoothing by calculating the mean square error. When the difference is greater than the set threshold, the absolute error is calculated for linear smoothing. This flexible loss function brings adaptability to more situations and can handle small outliers more smoothly, thereby achieving the effect of alleviating artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0039] Figure 1 A flow chart of a dynamic deletion method for three-dimensional Gaussian splashing provided by the present invention;

[0040] Figure 2 An overview diagram of a dynamic deletion method for three-dimensional Gaussian splashing provided by the present invention;

[0041] Figure 3 Comparison diagram of dynamic Gaussian culling in the present invention, where (a) is the effect diagram of the existing method (3DGS), and (b) is the effect diagram with dynamic Gaussian culling added;

[0042] Figure 4 The figure shows the comparison before and after the Huber loss function is introduced in the present invention, where (a) is the effect diagram before the introduction and (b) is the effect diagram after the introduction;

[0043] Figure 5 Schematic diagram of the relationship between the value of parameter δ and PSNR in the present invention;

[0044] Figure 6 Schematic diagram of the effect of parameter δ on the reconstruction results in the present invention, where (a) is the effect diagram when the parameter δ is an inappropriate value, and (b) is the effect diagram when the parameter δ is an appropriate value;

[0045] Figure 7 Schematic diagram of the effect of the values ​​of parameters β and α on PSNR in the present invention;

[0046] Figure 8 A comparison diagram of the method of the present invention and the existing method;

[0047] Figure 9 This is a comparison chart on the Synthetic Blender dataset in the present invention;

[0048] Figure 10 A block diagram of a dynamic deletion system for three-dimensional Gaussian splashing provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] Example 1

[0051] In order to solve the problems of over-reconstruction, under-reconstruction and artifacts in the existing three-dimensional Gaussian splashing when reconstructing scenes, as well as the problems that traditional methods only alleviate a certain problem, fail to solve it comprehensively, and have poor accuracy due to fixed thresholds and linear growth loss functions, such as Figure 1 As shown, the present invention proposes a dynamic deletion method for three-dimensional Gaussian splash, which includes:

[0052] S1: Obtain a Gaussian ellipse set of a three-dimensional scene, which contains multiple Gaussian ellipses with scale parameters and transparency parameters;

[0053] S2: Dynamically control the density of each Gaussian ellipse, dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse, dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse, and dynamically reject the Gaussian ellipse set according to the scale rejection threshold and the transparency rejection threshold, and retain the Gaussian ellipses that meet the conditions;

[0054] S3: Perform rasterization rendering on the retained Gaussian ellipse. During the rasterization rendering process, the Huber loss function is introduced to optimize the rendered image to alleviate image artifacts and obtain the final rendered image.

[0055] As can be seen from the above technical solution, the present invention proposes a dynamic deletion method for three-dimensional Gaussian splash. The present invention obtains a set of Gaussian ellipses containing scale and transparency parameters to provide a data basis for subsequent dynamic control; dynamically calculates the scale rejection threshold based on the scale parameter (such as setting a stricter large-scale Gaussian rejection condition through coverage evaluation), and dynamically calculates the transparency rejection threshold based on the transparency parameter (such as retaining small transparency Gaussians that are effective for detail reconstruction through contribution evaluation), changing the "one-size-fits-all" mode of the traditional fixed threshold, accurately controlling the number of Gaussian ellipses, and effectively alleviating the problems of over-reconstruction caused by Gaussian coverage and under-reconstruction caused by information loss; introduces the Huber loss function, and uses quadratic smoothing to optimize small outliers when the error is less than the threshold to avoid excessive fluctuations. When the error is greater than the threshold, linear penalty is used to process large outliers to improve robustness, thereby alleviating the artifact problem caused by the traditional linear loss function. In summary, this method systematically solves the three core problems in three-dimensional Gaussian splash reconstruction through the synergy of dynamic density control and improved loss function, improving scene reconstruction accuracy and image rendering quality.

[0056] Furthermore, the overall process of the method of the present invention is as follows Figure 2 As shown. In order to achieve the effect of alleviating over-reconstruction, under-reconstruction and reducing artifacts while taking into account efficiency, the present invention improves the framework of the existing method starting from the number of rendered Gaussian ellipses and the loss function of training optimization. The improvements over the existing method mainly include two aspects: first, dynamic scale threshold culling and dynamic transparency threshold culling are introduced on its original density control to design a separate culling threshold for each Gaussian ellipse, so as to more accurately eliminate aliased Gaussian ellipses, alleviate the over-reconstruction phenomenon and retain useful Gaussian ellipses to alleviate the problem of under-reconstruction; second, in the training process (rasterization rendering), the present invention adds the Huber loss function to perform quadratic smoothing on small errors to alleviate artifacts. The improvements of the present invention over the prior art are as follows Figure 2 Shown in red box.

[0057] The 3DGS method (existing method) takes as input a set of images of a static scene and a sparse point cloud generated using Structure from Motion (SfM). From these points, the method creates a set of 3D Gaussian ellipses defined by position (mean), covariance matrix, and transparency. The directional appearance component of the radiance field (color) follows the standard practice of representing it using spherical harmonics (SH). Highly anisotropic volume splattering techniques enable a more compact representation of subtle structure in the scene and greater flexibility in the rendering process. Furthermore, an adaptive density control module model allows for pruning and cloning of Gaussian ellipses based on their various parameters. In existing 3DGS methods, the adaptive density control module uses a one-size-fits-all approach to determine whether a Gaussian ellipse is useful for reconstruction. Specifically, the scale culling threshold and transparency culling threshold are fixed at pre-set values, for example, the scale culling threshold is fixed to 10% of the current scene extent and the transparency culling threshold is fixed to 0.005. This does not fully consider the unique influence of different Gaussian ellipses on the reconstruction results during the reconstruction process. Sometimes, a Gaussian ellipse with too large a scale is mistakenly retained to cover other Gaussian ellipses, resulting in over-reconstruction. In addition, some Gaussian ellipses that originally contributed to the reconstruction are mistakenly eliminated, which leads to insufficient reconstruction. To this end, the present invention designs a more flexible dynamic threshold calculation mechanism to replace the original fixed threshold, namely Figure 2 Dynamic scale threshold culling and dynamic transparency culling. During the density control process, the present invention calculates the scale culling threshold and transparency culling threshold for each Gaussian ellipse separately instead of using a fixed threshold. This allows the model to adaptively consider the degree of influence of each Gaussian ellipse on the reconstruction based on its parameters and decide whether to cull it. This mechanism can dynamically adjust the distribution density of the three-dimensional Gaussian ellipse according to the complexity of the scene during the density control process, thereby effectively alleviating the over-reconstruction and under-reconstruction caused by the use of fixed thresholds while ensuring efficiency. In addition, 3DGS optimizes the rendered image by the mean absolute error during the rasterization rendering process, and its linear growth characteristic causes artifacts in the optimized image. To this end, the present invention introduces the Huber loss function to perform a smoother quadratic growth instead of a linear growth on the small errors between the reconstructed image and the real image. This allows the model to avoid the problem of detail loss caused by the filter while alleviating the image artifacts caused by fluctuations due to strict linear penalties for small errors. By introducing these two changes, the motion recovery structure method can simultaneously take into account the problems of over-reconstruction, under-reconstruction, and artifacts while maintaining the efficiency of 3DGS.

[0058] Furthermore, the 3DGS method proposes to represent the 3D scene as a set of scaled 3D Gaussian primitives And use volume splatter to render the image. Each scaled 3D Gaussian The geometry is determined by the transparency (scale) α k ∈[0,1], center and the covariance matrix defined in world space Parameterized, the formula is as follows:

[0059]

[0060] In order to k Constrained to the effective covariance matrix space, 3DGS uses a semi-positive definite parameterization here, is a scaling vector with three rows and one column, is a three-row, three-column rotation matrix parameterized by a quaternion.

[0061] Then, in order to and translation vectors The image is rendered from a specific viewpoint defined by Converted to camera coordinates, the formula is as follows:

[0062] p′ k =Rp k +t,∑′ k =R∑ k R T ;

[0063] Afterwards, they are projected into the ray space via a local affine transformation, which is formulated as follows:

[0064]

[0065] Among them, the Jacobian matrix J k is a 3D Gaussian p′ k Affine approximation of the projective transformation defined by the center.

[0066] Finally, 3DGS uses spherical harmonics to transform the view-dependent color c k Modeling is performed and the image is rendered by alpha blending according to the depth order 1,...,K of the primitive, as follows:

[0067]

[0068] In addition, considering the impact of the number of Gaussian ellipses on the reconstruction results, 3DGS also performs density control during the projection process, but uses two fixed values ​​to screen the scale and transparency of the Gaussian ellipses respectively. This one-size-fits-all approach does not accurately control the density of the Gaussian ellipses, which may lead to over-reconstruction and under-reconstruction.

[0069] Furthermore, during the density control process, 3DGS will eliminate Gaussian ellipses whose scale is greater than a certain threshold. The specific formula is as follows:

[0070] sv>0.1×ex,op <min op ;

[0071] Where sv represents the scale of the current Gaussian ellipse, and ex represents the current scene size. Since the size of each scene is fixed, the scale rejection threshold for that scene is also fixed. While this fixed scale threshold works well in most cases and removes overly large Gaussian ellipses, it inevitably retains some slightly larger ellipses that overlap surrounding Gaussian ellipses, resulting in over-reconstructed Gaussian ellipses.

[0072] To this end, the present invention proposes a method for dynamically calculating the scale rejection threshold of the current Gaussian ellipse based on the scale parameter of the current Gaussian ellipse. The specific formula is as follows:

[0073] T S =(max s -min s )×s×β+β×min s ;

[0074] Among them, T S is the scale removal threshold, min s and max s are the preset minimum and maximum thresholds, s is the scale value of the current Gaussian ellipse, and β is a calculation parameter set based on experience. S First, calculate the coverage of the current Gaussian ellipse to the surrounding Gaussian ellipse (max s -min s )×s×β, and then make fine adjustments on this basis (β×min s Using the above formula, a different scale rejection threshold is calculated for each Gaussian ellipse. This rejection mechanism makes the reconstruction more flexible, allowing for a more tolerant retention of smaller Gaussian ellipses while requiring a more rigorous rejection of larger ones. This dynamic scale control allows for more precise density control, alleviating the over-reconstruction problem caused by the previous fixed threshold.

[0075] In 3DGS, in addition to controlling the density of the Gaussian ellipse by scale, the Gaussian ellipse is also pruned by transparency. When the transparency of the current Gaussian ellipse is less than the transparency culling threshold, it is considered that the Gaussian ellipse will not contribute to the reconstruction and is therefore culled. The specific formula is as follows:

[0076] op <min op ;

[0077] Among them, op is the transparency of the current Gaussian ellipse, min op The transparency culling threshold. The min value of all scenes in the current 3DGS op Both are set to 0.005, but it is worth noting that in some scenes, even if the transparency of the Gaussian ellipse is less than 0.005, it will play a role in the reconstruction of the details of the scene. The original fixed threshold will inevitably lead to the elimination of these Gaussian ellipses, resulting in insufficient reconstruction.

[0078] To this end, the present invention designs a method that can dynamically calculate the transparency rejection threshold based on the current Gaussian ellipse transparency and scene transparency. The specific formula is as follows:

[0079] T o =(max O -min O )×op×α+α×max O ×min O ;

[0080] Among them, T o is the transparency rejection threshold, min O and max O The transparency threshold T of the current Gaussian ellipse is set as the minimum and maximum thresholds, op is the transparency of the current Gaussian ellipse, and α is a calculation parameter set based on experience. o It consists of two parts. The first part (max O -min O )×op×α is the contribution of the current transparency to the reconstruction, and the second part α×max O ×min O This is an empirical fine-tuning of the threshold. Calculating different transparency culling thresholds for different Gaussian ellipses based on the above formula allows the model to completely cull Gaussian ellipses with minimal transparency, partially cull those with moderate transparency, and retain those with greater transparency. This dynamic culling mechanism more accurately retains Gaussian ellipses that contribute to detailed reconstruction based on transparency, thereby alleviating the underreconstruction caused by the previous fixed threshold.

[0081] During the rendering optimization process, 3DGS measures the loss between the rendered image and the real image by the mean absolute error (L1 loss), and its formula is as follows:

[0082]

[0083] Among them, N is the total number of predictions, yi is the true value of the i-th pixel, y′ i is the predicted value of the i-th pixel. It can be seen that the L1 loss is based on linear growth. The accumulation of small errors in the image may cause the optimization to fluctuate too much and produce image artifacts.

[0084] In order to alleviate the artifacts in the rendered image caused by the linear growth characteristic of the mean absolute error, the present invention introduces the Huber loss function based on the original loss function, which is expressed as follows:

[0085]

[0086] Among them, L δ The Huber loss function uses parameter δ as the threshold, where y is the actual value, f(x) is the model value, and δ is a parameter. In practical applications, choosing an appropriate δ value is key. The optimal δ value is usually determined through cross-validation or experimental optimization so that the Huber loss function can perform best when dealing with different types of artifact phenomena. The main function of the Huber loss function is to robustly handle outliers (outliers) in the loss function. Compared with the traditional squared error loss function (L2 loss) and absolute error loss function (L1 loss), the Huber loss function is more sensitive to large outliers but has less impact on small outliers, thus having a unique advantage in balancing the model's ability to handle normal data and abnormal data.

[0087] Specifically, when the error is small, the Huber loss function runs the upper half of the above formula, which takes the form of an average L2 loss function. This ensures smooth processing of normal data points. When the error is large, the lower half of the above formula is run, which takes the form of an average L1 loss function, thereby reducing the excessive impact of outliers on the model. This characteristic enables the Huber loss function to more effectively reduce the deviation caused by outliers when processing images with artifacts, thereby improving the robustness and generalization ability of the model.

[0088] Another major advantage of the Huber loss function introduced in this paper is its high computational efficiency. Compared with directly using the L1 or L2 loss functions, the Huber loss function can provide a more stable and faster convergence process in most cases, thereby accelerating model training. By incorporating the Huber loss function into the loss function, this paper effectively reduces the impact of outliers on the model, thereby providing better detail preservation and structural fidelity, and significantly improving the overall reconstruction effect.

[0089] In order to further illustrate the advantages of the present invention, the following is a description with reference to specific experiments.

[0090] The method was implemented in Python using the PyTorch framework, version 2.0.0, and Python version 3.8. We also used custom CUDA kernels from 3DGS for rasterization, which are extensions of previous methods, and fast radix sorting using the NVIDIA CUB sort routine. All experimental data was obtained using an NVIDIA GeForce RTX3090 server with 24GB of video memory.

[0091] 1. Dataset Selection

[0092] The datasets selected for this paper are largely similar to those used by 3DGS and most of its extensions. They cover scenes with diverse shooting styles, including bounded indoor scenes and large, unbounded outdoor scenes, and include both synthetic and real-world datasets. This diversity ensures that the experimental results of this paper are more generalizable and persuasive. Specific datasets evaluated include the Synthetic Blender dataset, Mip-NeRF360, and Tanks & Temples.

[0093] First, we selected the Synthetic Blender dataset, which contains eight synthetic bounded scenes. Each scene provides a detailed view set and precise camera parameters, which facilitates the generation of a suitable initial SfM point cloud. Furthermore, because the synthetic dataset is accurately labeled and realistically rendered, we can conduct extensive experiments without the cost and limitations of real-world data collection, further validating the model's generalization capabilities.

[0094] Then, the present invention selected the Mip-NeRF360 dataset to evaluate the model of the present invention. The Mip-NeRF360 dataset is an advanced NeRF rendering quality benchmark that provides challenging real-world indoor and outdoor scenes. These scenes contain rich details and complexity and can be used to evaluate the performance of the model in high-quality rendering. The dataset also provides high-quality rendered images and accurate depth information, making it ideal for evaluating the performance of advanced rendering models. The dataset contains a total of 7 real-world scenes, each of which contains a complex central object or area and a detailed background, and the authors tried to prevent photometric changes by fixing the camera exposure settings, minimizing lighting changes, and avoiding moving objects during the shooting process, which helps to improve the stability, generalization ability and comparability of the rendering model, thereby more accurately evaluating the performance of the model.

[0095] Finally, we selected two scenes from the Tanks & Temples dataset to evaluate our model. The Tanks & Temples dataset provides challenging and diverse real-world outdoor scenes. These scenes contain a wide variety of geometries, lighting conditions, and materials, making them useful for evaluating the model's performance and generalization capabilities in complex environments. Furthermore, the dataset provides accurate camera pose and depth information, facilitating a more accurate assessment of the model's rendering quality.

[0096] By selecting these representative datasets, the present invention is able to comprehensively evaluate the performance of the model in different scenarios and conditions.

[0097] In terms of evaluation criteria, in order to clearly compare with 3DGS and its latest methods, the present invention has selected three classic indicators, namely Peak signal-to-noise ratio (PSNR), Structure Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), just like 3DGS. These three evaluation criteria are widely used to evaluate image quality and similarity because they can provide quantitative metrics for different aspects of the image. When evaluating rendering models, PSNR provides a simple and intuitive error metric, SSIM provides a metric for structural and content similarity, and LPIPS is closer to human visual perception and can provide a more accurate image similarity metric. Therefore, these three indicators are usually used simultaneously to evaluate the performance of rendering models to provide a comprehensive evaluation.

[0098] (1) Peak signal-to-noise ratio (PSNR)

[0099] PSNR is a common metric used to measure image quality. It calculates the peak signal-to-noise ratio (PSNR) between the original image and the compressed or processed image. This is the ratio of the maximum possible power of the image to the average power introduced by the distortion. Higher PSNR values ​​indicate better image quality. When evaluating rendering models, PSNR provides a quantitative measure of the difference between generated and real images, and is therefore often used as a metric for evaluating rendering quality. Its calculation is as follows:

[0100]

[0101] Where MAX is the maximum possible value of the image pixel, and MSE (Mean Squared Error) is the mean squared error between the original image and the processed image.

[0102] (2) Structural Similarity Index SSIM

[0103] SSIM is a metric used to compare the similarity of two images. SSIM takes into account the brightness, contrast, and structural information of the image, and measures the similarity between two images by calculating brightness similarity, contrast similarity, and structural similarity. The SSIM value ranges from -1 to 1, where 1 means that the two images are exactly the same. SSIM is often used to evaluate the performance of tasks such as image enhancement, denoising, and compression. When evaluating rendering models, SSIM can provide a more detailed image similarity measure that can capture the similarity in structure and content between rendered images and real images. Its calculation formula is as follows:

[0104]

[0105] Where μ x and μ y is the mean of image x and y, and is the variance of images x and y, σ xy is the covariance of images x and y, c1 and c2 are constants used to stabilize the calculation of the denominator.

[0106] (3) Learning Perceptual Image Patch Similarity LPIPS

[0107] LPIPS is a deep learning-based image similarity metric. It uses a pre-trained deep learning model to learn the perceptual features of images and measures the similarity between images by calculating the distance between them in the feature space. LPIPS better simulates the human visual system's perception of images, and therefore provides a more accurate image similarity metric when evaluating rendering models. Lower LPIPS values ​​indicate greater consistency with human subjective perception. Its calculation formula is as follows:

[0108] LPIPS(x,y)=∑ i w i ||φ i (x)-φ i (x)||;

[0109] Among them, φ i (x) and φ i (x) is the representation of images x and y on the i-th feature channel, w i is the weight of each feature channel.

[0110] 3. Ablation and parameter comparison experiments

[0111] The present invention isolates the improved algorithm modules and conducts a series of experiments to measure their impact on the proposed method. Specifically, the present invention tests the impact of the following aspects on the final reconstruction: dynamic Gaussian culling mechanism, loss function, and related parameter adjustments. The quantitative impact of each module is summarized in Table 1. Here, H represents the Huber loss function; D represents dynamic Gaussian culling; δ is the weight parameter of the Huber loss function; and 7k and 30k represent the number of iterations. The impact of each module is detailed below, in conjunction with Table 1.

[0112] Table 1 PSNR values ​​in ablation experiments

[0113]

[0114] In the present invention's research, the present invention evaluated the importance of dynamic Gaussian culling for over-reconstruction and under-reconstruction. For this ablation experiment, the present invention tested the reconstruction effects with and without the addition of dynamic Gaussian culling. As can be seen from Table 1, the PSNR scores of the method with the addition of the dynamic Gaussian culling mechanism are improved compared to 3DGS, whether it is 7k iterations or 30k iterations. This proves that the improvement of this module is effective for the original method 3DGS. The specific visualization is as follows Figure 3 As shown. Figure 3 It can be seen that when using the original 3DGS culling mechanism, the Gaussian ellipse generated by reconstruction is sometimes too large (not culled) and overlaps with other Gaussian ellipses, resulting in over-reconstruction, and the transparency is too small, resulting in direct culling and insufficient reconstruction. After using the dynamic Gaussian culling mechanism, it can be seen that Figure 3 As can be seen in the image on the right, the Gaussian ellipse generated by the model is more evenly distributed throughout the scene according to scale and transparency, thereby alleviating the phenomenon of over-reconstruction and under-reconstruction, as shown in the red box.

[0115] Next, the present invention will evaluate the impact of the Huber loss function. The present invention will conduct an experimental comparison before and after the introduction of the Huber loss function. In this experiment, the dynamic Gaussian elimination mechanism exists. It can be seen from Table 1 that after adding the new loss function, the PSNR score of this method is also improved compared with 3DGS on this dataset. The specific visualization results are as follows Figure 4 As shown. Figure 4 Before the Huber loss function was introduced, the grass in the upper left image was blurry, but after the Huber loss function was introduced, the texture details of the grass became clearer (upper right image). The Huber loss function also has a certain effect on removing artifacts and blurring, as can be seen in the lower left and right images.

[0116] Finally, since the loss function and dynamic Gaussian elimination of the present invention both have parameters obtained based on experience, the present invention evaluates the impact of different parameter changes on the overall optimization in this ablation experiment. First, from Table 1 and Figure 5 As can be seen from the figure, PSNR reaches its peak when δ=5. The present invention partially visualizes the results obtained by optimizing different parameters, as shown in the figure. Figure 6 The present invention found that when δ is too large (δ = 10, as shown Figure 6 Upper left) or too small (δ=1, such as Figure 6 When the present invention adjusts δ to a suitable value (δ=5), the artifact will be eliminated (as shown in the figure below). Figure 6 upper right and lower right).

[0117] In addition, since the parameters β and α have a direct impact on the calculation of the scale rejection threshold and the transparency rejection threshold of each Gaussian ellipse, and these two rejection thresholds determine the quality of the overall reconstruction, the present invention conducts a series of ablation experiments on these two parameters, as shown in the following example. Figure 7 As shown. Figure 7 It can be seen that when β = 0.01 and α = 0.16, PSNR reaches a maximum value of 23.57.

[0118] 4. Experimental comparison

[0119] The method of the present invention is built on the open source 3DGS code library. Except that the loss function and Gaussian density control strategy are different from 3DGS, the training model of the present invention performs 30K iterations in all scenarios like 3DGS, and uses the same schedule and hyperparameters. The algorithm of the present invention was tested in a total of 9 real-world scenarios and 8 synthetic scenarios. The scenarios selected by the present invention include both bounded indoor environments and unbounded outdoor environments. The complexity and diversity of the scenes can prove that the improvement of the present invention is adaptive and robust. (The present invention uses 3DGS as a quality benchmark to compare with the method of the present invention. In addition, the present invention is also compared with a recent fast NeRF method InstantNGP, Mip-NeRF360 with high-quality effects, and two recent 3DGS improvements, Mip-splatting and MS-3DGS.

[0120] We processed the dataset using the same training / testing split as 3DGS and used the same methodology, selecting one out of every eight images as the test set to generate error metrics for consistent and meaningful comparisons. The experimental results are shown in Table 2. Except for the data for Mip-NeRF360 on its dataset, which we reproduced from the original publication, all other data in the table come from our runs of the author's code for the previous method.

[0121] Table 2 Quantitative evaluation of the method of the present invention and the existing method

[0122]

[0123] The results marked with ^ in Table 2 are directly from the original paper, and all other results are from the present invention's own experiments.

[0124] Table 3 further demonstrates the quantitative comparison of the proposed method with the recent Mip-splatting and MS-3DGS on seven scenes of the Mip-NeRF360 dataset. These data are all obtained on the same computer configuration and in the same environment, and the datasets are all from the 4x resolution of the Mip-NeRF360 dataset.

[0125] Table 3 Comparison of PSNR values ​​on the Mip-NeRF360 dataset

[0126]

[0127]

[0128] The quantitative comparison in Table 3 shows that our method leads the way in the four indoor scenes of counter, room, bonsai, and kitchen. However, it lags slightly behind Mip-splatting in some outdoor scenes. This may be because our method lacks sufficient detail in reconstructing corners in unbounded outdoor scenes.

[0129] The present invention is also Figure 8 The visualization results of the left-out test view of the method of the present invention and the existing rendering method selected for comparison are shown in the figure. For a clearer and more direct comparison, the method of the present invention is also trained for 30K iterations like 3DGS. After the improvement, the present invention found that the over-reconstruction and under-reconstruction caused by the simple and fixed Gaussian culling mechanism of 3DGS have been significantly improved under the modification of the present invention (such as the grass background in the bicycle; the distant hills in the train; the details of the background building in the truck and the window details in the garden background). At the same time, due to the introduction of the Huber loss function, the present invention found that in some cases, the method of the present invention can also achieve a certain deblurring and de-artifacting effect (such as the sofa area in the room, the blur of the vegetation in the stump, and the artifacts in the bonsai). From Figure 8As can be seen from the figure, the improved method of the present invention can maintain good coverage and reconstruct certain details even in distant backgrounds. Furthermore, in areas with monotonous colors and textures, the method of the present invention can also reconstruct more details. Furthermore, the artifacts and blurring that plagued previous methods are also improved in the present method.

[0130] This paper also quantitatively compares the method of the present invention with the above-mentioned methods in the synthetic scene, see Table 4 and Table 5.

[0131] Table 4 PSNR scores of Synthetic Blender dataset

[0132]

[0133]

[0134] Table 5 SSIM scores of Synthetic Blender dataset

[0135]

[0136] From Table 4 and Table 5, we can see that the improved method has a comprehensive improvement on this dataset compared with any previous method. It can also be seen that the improvement in this paper is stable and robust. This paper also conducts a qualitative comparison on this dataset, specifically Figure 9 As shown. Figure 9 It can be seen that the method of the present invention alleviates the artifact problem while avoiding the detail loss problem caused by the filter introduced by Mip-Splatting.

[0137] In summary, the present invention proposes an improved 3DGS splashing technology, the core of which is to introduce a dynamic Gaussian culling mechanism and a new loss function that cooperates with the mechanism on the basis of the original 3DGS, so as to achieve a more accurate and higher-quality reconstruction effect. The dynamic Gaussian culling mechanism more flexibly sets the threshold based on the scale and transparency of the three-dimensional Gaussian ellipse, accurately eliminates Gaussian ellipses with excessive scale, thereby reducing the phenomenon of over-reconstruction, and allows some Gaussian ellipses that should play a role in the reconstruction process to be retained, thereby alleviating the problem of insufficient reconstruction, while the Huber loss function reduces outliers in the training process, thereby alleviating the phenomenon of blurred artifacts in the reconstruction. Experimental results show that when training and testing are carried out at the same scale / sampling rate, the technology of the present invention is competitive with the most advanced methods in terms of performance, while the method of the present invention can more flexibly cope with the reconstruction of different scene environments, and has stronger robustness and adaptability.

[0138] Example 2

[0139] like Figure 10 As shown, the present invention provides a dynamic deletion system for three-dimensional Gaussian splashing, which is used to implement the dynamic deletion method for three-dimensional Gaussian splashing of the above embodiment 1, specifically comprising:

[0140] A Gaussian parameter acquisition module 100 is used to acquire a Gaussian ellipse set of a three-dimensional scene, where the Gaussian ellipse set includes multiple Gaussian ellipses with scale parameters and transparency parameters;

[0141] Dynamic density control module 200, for dynamically controlling the density of each Gaussian ellipse, dynamically calculating the scale rejection threshold of the current Gaussian ellipse based on the scale parameter of the current Gaussian ellipse, dynamically calculating the transparency rejection threshold of the current Gaussian ellipse based on the transparency parameter of the current Gaussian ellipse, and dynamically rejecting the Gaussian ellipse set based on the scale rejection threshold and the transparency rejection threshold, retaining Gaussian ellipses that meet the conditions;

[0142] The loss function optimization module 300 is used to perform rasterization rendering on the retained Gaussian ellipse, and introduce the Huber loss function to optimize the rendered image during the rasterization rendering process to alleviate image artifacts and obtain the final rendered image.

[0143] Furthermore, the dynamic density control module 200 includes: a scale threshold calculation unit, which is used to dynamically calculate the scale rejection threshold of the current Gaussian ellipse based on the scale parameter of the current Gaussian ellipse; a transparency threshold calculation unit, which is used to dynamically calculate the transparency rejection threshold of the current Gaussian ellipse based on the transparency parameter of the current Gaussian ellipse; and a rejection unit, which is used to dynamically reject the Gaussian ellipse set based on the scale rejection threshold and the transparency rejection threshold, and retain the Gaussian ellipses that meet the conditions.

[0144] A dynamic deletion system for three-dimensional Gaussian splashing in this embodiment is used to implement the aforementioned dynamic deletion method for three-dimensional Gaussian splashing. Therefore, the specific implementation method of the dynamic deletion system for three-dimensional Gaussian splashing can be found in the embodiment part of the dynamic deletion method for three-dimensional Gaussian splashing in the previous text. For example, the Gaussian parameter acquisition module 100, the dynamic density control module 200, and the loss function optimization module 300 are respectively used to implement steps S1, S2, and S3 in the above-mentioned dynamic deletion method for three-dimensional Gaussian splashing. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0145] Example 3

[0146] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned dynamic deletion method for three-dimensional Gaussian splashing.

[0147] Example 4

[0148] An embodiment of the present invention provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned dynamic deletion method for three-dimensional Gaussian splash.

[0149] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0152] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A dynamic deletion method for three-dimensional Gaussian splash, characterized in that: include: Obtaining a Gaussian ellipse set of a three-dimensional scene, wherein the Gaussian ellipse set includes a plurality of Gaussian ellipses with scale parameters and transparency parameters; Dynamically control the density of each Gaussian ellipse, dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse, dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse, and dynamically reject the Gaussian ellipse set according to the scale rejection threshold and the transparency rejection threshold, retaining Gaussian ellipses that meet the conditions; The retained Gaussian ellipse is rasterized and rendered. The Huber loss function is introduced in the rasterization rendering process to optimize the rendered image to alleviate image artifacts and obtain the final rendered image.

2. The dynamic deletion method for three-dimensional Gaussian splash according to claim 1, characterized in that: The method for dynamically calculating the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse is: T S =(max s -min s )×s×β+β×min s ; Among them, T S is the scale removal threshold, min s and max s are the preset minimum and maximum thresholds for the scale, s is the scale value of the current Gaussian ellipse, and β is a calculation parameter set based on experience.

3. The dynamic deletion method for three-dimensional Gaussian splash according to claim 1, characterized in that: The method for dynamically calculating the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse is: T o =(max O -min O )×op×α+α×max O ×min O ; Among them, T o is the transparency rejection threshold, min O and max O are the preset minimum and maximum transparency thresholds, op is the transparency of the current Gaussian ellipse, and α is a calculation parameter set based on experience.

4. The dynamic deletion method for three-dimensional Gaussian splash according to claim 1, characterized in that: The process of dynamically eliminating the Gaussian ellipse set according to the scale elimination threshold and the transparency elimination threshold includes: Eliminate Gaussian ellipses whose scale parameters are greater than the corresponding scale elimination threshold to alleviate the problem of over-reconstruction; Gaussian ellipses with transparency parameters less than the corresponding transparency culling threshold are eliminated to alleviate the problem of insufficient reconstruction.

5. The dynamic deletion method for three-dimensional Gaussian splash according to claim 1, characterized in that: The expression of the Huber loss function is: Among them, L δ Huber loss function with parameter δ as the threshold, y is the actual value, f(x) is the value of the model, and δ is a parameter.

6. The dynamic deletion method for three-dimensional Gaussian splash according to claim 1, characterized in that: The Gaussian ellipse is defined by position, covariance matrix and transparency.

7. A dynamic deletion system for three-dimensional Gaussian splashing, characterized in that: The system is used to implement the dynamic deletion method for three-dimensional Gaussian splashing according to any one of claims 1 to 6, specifically comprising: A Gaussian parameter acquisition module, configured to acquire a Gaussian ellipse set of a three-dimensional scene, wherein the Gaussian ellipse set includes a plurality of Gaussian ellipses having scale parameters and transparency parameters; A dynamic density control module is used to perform dynamic density control on each Gaussian ellipse, dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse, dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse, and dynamically reject the Gaussian ellipse set according to the scale rejection threshold and the transparency rejection threshold, retaining Gaussian ellipses that meet the conditions; The loss function optimization module is used to perform rasterization rendering on the retained Gaussian ellipse. During the rasterization rendering process, the Huber loss function is introduced to optimize the rendered image to alleviate image artifacts and obtain the final rendered image.

8. The dynamic deletion system for three-dimensional Gaussian splashing according to claim 7, characterized in that: The dynamic density control module includes: A scale threshold calculation unit, used to dynamically calculate the scale rejection threshold of the current Gaussian ellipse according to the scale parameter of the current Gaussian ellipse; A transparency threshold calculation unit, used to dynamically calculate the transparency rejection threshold of the current Gaussian ellipse according to the transparency parameter of the current Gaussian ellipse; The culling unit is configured to dynamically cull the Gaussian ellipse set according to the scale culling threshold and the transparency culling threshold, and retain Gaussian ellipses that meet the conditions.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the dynamic deletion method for three-dimensional Gaussian splashing as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the dynamic deletion method for three-dimensional Gaussian splashing according to any one of claims 1 to 6.

Images