A view-consistent method and system for three-dimensional gaussian spatter

Abstract

This invention provides a method and system for resisting viewpoint changes in 3D Gaussian splashing, relating to the field of computer graphics technology. The method includes modeling the geometry as a set of 3D Gaussian ellipses without normals; projecting the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space; sorting the 2D Gaussian ellipses overlapping the current pixel region according to depth; calculating the offset between the current pixel center point and the projected coordinates of the 2D Gaussian ellipses, and projecting the offset onto the principal axis of the 2D Gaussian ellipses to obtain the relative coordinates along the principal axis; calculating the integral of the 2D Gaussian ellipses along the principal axis to obtain the directional energy distribution; performing weighted accumulation fusion on the 2D Gaussian ellipses based on the directional energy distribution; calculating the pixel shading value; and obtaining the final rendered image. This invention alleviates the problems of jagged edges, artifacts, and loss of detail caused by "point-to-point" integration.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a method and system for resisting viewpoint changes in three-dimensional Gaussian spraying. Background Technology

[0002] In the field of computer graphics today, novel perspective compositing techniques are developing rapidly. Among them, neural rendering techniques, represented by Neural Radiance Field (NeRF), have made breakthrough progress, driving the development of many advanced applications such as human / virtual modeling and surface reconstruction. However, the back-mapping voxel rendering paradigm used by NeRF has a real-time bottleneck. Although subsequent research has improved it through various optimization strategies, it is still difficult to achieve true real-time rendering in complex scenes.

[0003] To address this challenge, 3D Gaussian Splatting (3DGS) was developed. 3DGS constructs a GPU-accelerated, differentially rasterized pipeline, employs forward mapping technology, and combines explicit point primitive representation to model a scene as a collection of anisotropic 3D Gaussian ellipses carrying physical properties. This has yielded significant results in real-time, high-quality rendering and is widely used in several cutting-edge fields.

[0004] Despite its significant advantages, 3DGS also suffers from a number of problems. In terms of sampling, 3DGS employs a "point-to-point" sampling strategy when shading pixels, treating each pixel as a one-dimensional point sampling a two-dimensional ellipse. This approach violates the Shannon-Nyquist sampling theorem. When the high-frequency components of the scene signal exceed the pixel's Nyquist frequency, jagged edges inevitably appear, severely impacting the quality of the rendered image. While subsequent methods have mitigated the jagged edge problem through filtering mechanisms, overly smooth filtering leads to a loss of detail.

[0005] Furthermore, since the covariance matrix of the 3D Gaussian ellipse projected onto the screen space changes dynamically with the viewing angle, when the principal axis of the projected ellipse is tilted at an angle to the pixel grid, "point-to-point" sampling cannot reflect the directional energy distribution of the ellipse within the pixel window, which leads to insufficient sampling, resulting in inaccurate coloring, artifacts, and loss of detail.

[0006] To address these issues with 3DGS, many studies have attempted to propose solutions. For example, Mip-Splatting proposes a hybrid filtering mechanism for anti-aliasing, but it loses details due to over-filtering; MS3DGS introduces multi-scale Gaussian elliptic representation to alleviate aliasing in low-resolution scenes, but performs poorly in medium- and high-resolution scenes; Pixel-GS calculates the average gradient magnitude of points from a pixel perspective, reducing the dependence on the initial point of Structure from Motion (SfM) and alleviating some artifacts, but increasing memory consumption; VDGS fuses 3DGS and NeRF to alleviate specular reflection artifacts, but the introduction of neural networks reduces rendering efficiency; HO-Gaussian combines mesh-based voxels and display Gaussians to improve the consistency of scene appearance and geometry, but it is not accurate enough in reconstructing fine geometric structures; 2D-Gaussian-Splatting compresses three-dimensional Gaussian ellipses into two-dimensional oriented plane Gaussian ellipses, combining depth distortion and normal consistency principles to capture complex structural details, but it has shortcomings in densification strategies.

[0007] Most of these methods only address a specific problem in 3DGS and still employ the same "point-to-point" integral sampling strategy as 3DGS when shading pixels, failing to fundamentally and systematically solve the problems of aliasing, artifacts, and loss of detail. Therefore, a new method is urgently needed to solve these problems at their root and improve the rendering quality and effects of 3DGS. Summary of the Invention

[0008] To address this issue, this invention provides a method and system for resisting viewpoint changes in 3D Gaussian sputtering, which solves the problems of jagged edges, artifacts, and loss of detail in rendered images caused by the "point-to-point" strategy violating the sampling theorem and not fully considering the impact of viewpoint changes during pixel sampling in the prior art of 3D Gaussian sputtering technology.

[0009] To address the aforementioned problems, embodiments of the present invention provide a method for resisting viewpoint changes in three-dimensional Gaussian spraying, the method comprising:

[0010] The geometry is modeled as a set of 3D Gaussian ellipses that do not require normals. The 3D Gaussian ellipses are projected into 2D Gaussian ellipses in screen space. The 2D Gaussian ellipses that overlap with the current pixel region are sorted according to depth.

[0011] Calculate the offset between the current pixel center point and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the principal axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction;

[0012] The integral of the two-dimensional Gaussian ellipse along the principal axis is calculated using the Abramowitz-Stegun polynomial approximation formula to obtain the directional energy distribution;

[0013] The two-dimensional Gaussian ellipse is weighted and accumulated based on the directional energy distribution to calculate the pixel color values ​​and obtain the final rendered image.

[0014] Preferably, the expression for the three-dimensional Gaussian ellipse is:

[0015]

[0016] In the formula, Let x be the Gaussian energy density value at point x in three-dimensional space. Centered on Gauss, The total covariance matrix defined in the world coordinate system supports anisotropic geometric modeling, Σ -1 Let be the inverse matrix of Σ.

[0017] Preferably, the method for projecting a three-dimensional Gaussian ellipse into a two-dimensional Gaussian ellipse is as follows:

[0018] To project a 3D Gaussian ellipse onto a 2D screen space for pixel shading and rendering, the projection onto the image space is performed using the following formula:

[0019] Σ′=JWΣW T J T ;

[0020] In the formula, W is the observation transformation matrix, W T Let W be the transpose of W, Σ′ be the covariance matrix in the camera coordinate system, and J be the Jacobian matrix of the affine approximation of the projective transformation. T Let J be the transpose of J;

[0021] To avoid the projected 2D Gaussian ellipse becoming too small (less than one pixel) when projected onto the screen space, a dilation operation is performed on the projected 2D Gaussian ellipse. The formula is as follows:

[0022]

[0023] In the formula, Let I be the energy density value of the k-th two-dimensional Gaussian ellipse in screen space at point x, where I is a two-dimensional identity matrix, s is a scalar dilation hyperparameter, and p k Let Σ be the Gaussian center and Σ be the total covariance matrix defined in the world coordinate system. It is a 2×2 symmetric positive definite matrix that describes the shape, size, and orientation of the k-th two-dimensional Gaussian ellipse, and T denotes transpose.

[0024] Preferably, the method for calculating the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is as follows:

[0025] Calculate the difference between the current pixel center coordinates and the center coordinates of the 2D Gaussian ellipse projection on the x and y axes to obtain the 2D offset (d). x ,d y ):

[0026]

[0027] In the formula, x pix x is the x-coordinate of the current pixel. i Let x be the x-axis coordinate of the Gaussian point projection, and y be the y-axis coordinate. pix The y-coordinate of the current pixel. i y-coordinate of the Gaussian point projection.

[0028] Preferably, the method for projecting the offset onto the principal axis direction of a two-dimensional Gaussian ellipse to obtain the relative coordinates along the principal axis direction is as follows:

[0029] The principal axis direction includes two orthogonal eigenvectors (v) of a two-dimensional Gaussian ellipse. 1x ,v 1y ) and (v 2x ,v 2y The relative coordinates (u,v) are calculated using the following formula:

[0030]

[0031] In the formula, v1 and v2 are the eigenvectors of the two-dimensional Gaussian ellipse along the principal axis, and d is the offset. x and d y These are the offsets on the x-axis and y-axis, respectively.

[0032] Preferably, the method for calculating the integral of the two-dimensional Gaussian ellipse along the principal axis using the Abramowitz-Stegun polynomial approximation formula is as follows:

[0033]

[0034] In the formula, Int u Int is the integral along the u-axis. v Let λ be the integral along the v-axis, λ be a constant factor, and σ1 and σ2 be the standard deviations of the Gaussian ellipse. The formula for the polynomial approximation of the Abramowitz-Stegun error function is as follows:

[0035]

[0036] In the formula, sign is used to ensure that the sign of the output is consistent with the sign of the input, a1, a2, a3, a4, a5 and p are all approximation coefficients, k is an adjustable parameter, t is an auxiliary variable, p is the scaling factor in the definition of t, and x is the input variable.

[0037] Preferably, the directional energy distribution is calculated by multiplying the integrals along the principal axis, using the following formula:

[0038] S = M·Int u ·Int v ;

[0039] In the formula, S is the magnitude of the directional energy distribution, M is a constant factor, and Int u and Int v These are the integrals along the u-axis and v-axis, respectively.

[0040] Preferably, the method for calculating pixel color values ​​by weighted accumulation and fusion of the two-dimensional Gaussian ellipse based on the directional energy distribution is as follows:

[0041] Based on the depth sorting results, the color features of each two-dimensional Gaussian ellipse are weighted according to the directional energy distribution S, and the final pixel color C is calculated by weighted α mixing.

[0042]

[0043] In the formula, N is the set of two-dimensional Gaussian ellipses, and c i S represents the color feature of the i-th elliptical region. i Let α be the directional energy distribution of i elliptical regions. i α represents the transparency of this area. j The transparency of the first region is given by T, where T represents the transparency of the current pixel, and bg is the background color.

[0044] This invention also provides a viewpoint variation-resistant system for three-dimensional Gaussian spraying, which implements the viewpoint variation-resistant method for three-dimensional Gaussian spraying described above, specifically including:

[0045] The projection sorting module is used to model the geometry into a set of three-dimensional Gaussian ellipses that do not require normals, project the three-dimensional Gaussian ellipses into two-dimensional Gaussian ellipses in screen space, and sort the two-dimensional Gaussian ellipses that overlap with the current pixel region according to the depth.

[0046] The offset calculation module is used to calculate the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates, and project the offset onto the principal axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction;

[0047] The integral calculation module is used to calculate the integral of the two-dimensional Gaussian ellipse along the principal axis using the Abramowitz-Stegun polynomial approximation formula, and obtain the directional energy distribution.

[0048] The weighted fusion module is used to perform weighted cumulative fusion of the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel color value, and obtain the final rendered image.

[0049] This invention also provides a computer storage medium storing a computer software product, the computer software product including several instructions for causing a computer device to execute the above-described method for resisting viewpoint changes in three-dimensional Gaussian sputtering.

[0050] As can be seen from the above technical solutions, this invention application has the following beneficial effects:

[0051] (1) This invention proposes a sampling calculation paradigm based on pixel regions. Addressing the issues of geometric jagged edges, artifacts, and detail loss caused by pixel coloring deviations resulting from the "point-to-point" strategy of existing methods that use pixel center points to sample 2D Gaussian ellipse points ("point-to-point"), this invention calculates the extent of the 2D Gaussian ellipse's expansion along its principal axis using the standard deviation and eigenvectors of the Gaussian ellipse, thereby determining the directional energy distribution of the Gaussian ellipse corresponding to the pixel region. This transforms the existing "point-to-point" sampling into "face-to-face" sampling. This approach adheres to the Shannon-Nyquist sampling theorem and fully considers the impact of 2D Gaussian ellipse tilt caused by dynamic changes in viewpoint on the pixel region. Then, weighted cumulative fusion is used for more accurate pixel coloring, thus alleviating the problems of jagged edges, artifacts, and detail loss.

[0052] (2) This invention presents a method for efficiently calculating the directional energy distribution of a Gaussian ellipse. To maintain computational efficiency while achieving more accurate calculation of the directional energy distribution, this invention introduces an Abramowitz-Stegun polynomial approximation formula to perform Gaussian integral calculations. This approximation formula approximates the error function using a 5th-order polynomial, requiring only 11 floating-point operations to obtain the calculation result. Furthermore, this function includes an adjustable parameter; adjusting this parameter allows for a more precise determination of the directional energy distribution of the 2D Gaussian ellipse over the pixel region, enabling more accurate weighted cumulative fusion shading and supporting flexibility in choosing efficient computational methods. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:

[0054] Figure 1 A flowchart of a method for resisting viewpoint changes in three-dimensional Gaussian spray provided by the present invention;

[0055] Figure 2 An overview diagram of a method for resisting viewpoint variation in three-dimensional Gaussian spray provided by the present invention;

[0056] Figure 3 The diagram shows the comparison of the two parameters of the present invention on PSNR, where (a) is a schematic diagram of the influence of the Abramowitz-Stegun function parameter k, and (b) is a schematic diagram of the influence of the Gauss-Hermite function parameter β.

[0057] Figure 4 The experimental results of the two functions of this invention are shown in the comparison graphs, where (a) is the result of the Gauss-Hermite function and (b) is the result of the Abramowitz-Stegun function.

[0058] Figure 5 This is a schematic diagram illustrating the effect of parameter k on SSIM in this invention;

[0059] Figure 6 This is a schematic diagram illustrating the influence of parameter k on the reconstruction effect in this invention;

[0060] Figure 7 This is a comparison of experiments using the Synthetic Blender dataset to demonstrate the effectiveness of this invention.

[0061] Figure 8 This is a comparison diagram of the present invention on the Mip-NeRF360;

[0062] Figure 9 This is a comparison chart of the present invention on the Tanks & Temples dataset;

[0063] Figure 10 A block diagram of a viewpoint-variation-resistant system for three-dimensional Gaussian spraying provided by the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example 1

[0066] To address the issues of jagged edges, artifacts, and loss of detail in rendered images caused by the "point-to-point" sampling strategy of existing 3D Gaussian sputtering techniques, which violate the sampling theorem and fail to adequately consider the impact of viewpoint changes during pixel sampling, such as... Figure 1 As shown, this invention proposes a method to resist viewpoint variation for three-dimensional Gaussian spraying, the method comprising:

[0067] S1: Model the geometry as a set of 3D Gaussian ellipses that do not require normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel region according to the depth.

[0068] S2: Calculate the offset between the current pixel center point and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the principal axis of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction;

[0069] S3: Calculate the integral of the two-dimensional Gaussian ellipse along the principal axis using the Abramowitz-Stegun polynomial approximation formula to obtain the directional energy distribution;

[0070] S4: Perform weighted cumulative fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel color value, and obtain the final rendered image.

[0071] As can be seen from the above technical solution, this invention proposes a viewpoint-variation-resistant method for 3D Gaussian splashing. It models the geometry as a 3D Gaussian ellipse without normals and projects and sorts the projections, avoiding the difficulty of estimating the normals of sparse point clouds. Simultaneously, depth sorting ensures the correct occlusion relationship of the Gaussian ellipse, laying the foundation for accurate shading. By projecting the pixel center point offset onto the principal axis, the traditional "point-to-point" sampling is transformed into "face-to-face" region sampling, fully considering the tilt of the 2D Gaussian ellipse caused by viewpoint changes, accurately capturing the directional energy distribution within the pixel region, and solving the aliasing and artifact problems caused by insufficient sampling. The Abramowitz-Stegun polynomial approximation formula is used to efficiently calculate the principal axis integral, accurately quantifying the energy contribution of the Gaussian ellipse to the pixel region with extremely low computational complexity, avoiding over-smoothing or sampling distortion, and preserving high-frequency details. Weighted accumulation fusion based on the directional energy distribution enables the pixel shading value to dynamically adapt to the elliptical shape under viewpoint changes, systematically improving anti-aliasing, artifact removal capabilities, and detail fidelity, ultimately achieving a high-quality, robust new viewpoint compositing rendering effect.

[0072] Furthermore, the overall process of the method of the present invention is as follows: Figure 2 As shown. The improvements of this invention compared to existing methods mainly include two points: First, it transforms the "point-to-point" sampling mechanism into a "face-to-face" sampling mechanism, treating the entire pixel as a continuous region rather than a discrete point. This allows for the calculation of the directional energy distribution of the corresponding 2D Gaussian ellipse within that pixel region. During subsequent pixel shading, this directional energy distribution enables the subsequent task to more accurately determine the impact of the 2D Gaussian ellipse tilt caused by dynamic changes in the viewing angle on the entire pixel window region, after considering the standard deviation of the 2D Gaussian ellipse and its two principal axes. This improves the accuracy of the weighted cumulative fusion shading of the rendered pixels. Second, it introduces the Abramowitz-Stegun polynomial approximation formula. This formula can achieve fast integration calculations with minimal complexity while ensuring accuracy through parameter adjustment. This achieves high-quality shading through weighted cumulative fusion of the directional energy distribution of the 2D Gaussian ellipse on the pixel region, while maintaining flexibility in balancing efficiency. For example... Figure 2 As shown, the red boxes indicate the improvements of this invention compared to existing methods.

[0073] In step S1, firstly, the geometry is modeled as a set of three-dimensional Gaussian ellipses that do not require normals. The expression for the three-dimensional Gaussian ellipse is:

[0074]

[0075] In the formula, Let x be the Gaussian energy density value at point x in three-dimensional space. Centered on Gauss, The total covariance matrix defined in the world coordinate system supports anisotropic geometric modeling, Σ -1 Let be the inverse matrix of Σ.

[0076] Then, the 3D Gaussian ellipse is projected into a 2D Gaussian ellipse in screen space, including:

[0077] To project a 3D Gaussian ellipse onto a 2D screen space for pixel shading and rendering, the projection onto the image space is performed using the following formula:

[0078] Σ′=JWΣW T J T ;

[0079] In the formula, W is the observation transformation matrix, W T Let W be the transpose of W, Σ′ be the covariance matrix in the camera coordinate system, and J be the Jacobian matrix of the affine approximation of the projective transformation. T Let J be the transpose of J.

[0080] To avoid the projected 2D Gaussian ellipse becoming too small (less than one pixel) when projected onto the screen space, a dilation operation is performed on the projected 2D Gaussian ellipse. The formula is as follows:

[0081]

[0082] In the formula, Let I be the energy density value of the k-th two-dimensional Gaussian ellipse in screen space at point x, where I is a two-dimensional identity matrix, s is a scalar dilation hyperparameter, and p k Let Σ be the Gaussian center and Σ be the total covariance matrix defined in the world coordinate system. It is a 2×2 symmetric positive definite matrix that describes the shape, size, and orientation of the k-th two-dimensional Gaussian ellipse, and T denotes transpose.

[0083] It is important to note that this operation adjusts the scale of the two-dimensional Gaussian ellipse while keeping its maximum value unchanged.

[0084] Finally, the two-dimensional Gaussian ellipses that overlap with the current pixel region are sorted according to their depth.

[0085] Because existing point-to-point sampling violates the Shannon-Nyquist sampling theorem and fails to adequately consider the tilt of the 2D Gaussian ellipse relative to the pixel space as the viewing angle dynamically changes, it leads to inaccurate final coloring, causing jagged edges, artifacts, and loss of detail. This invention proposes treating each pixel as a region and sampling the 2D Gaussian ellipse. The directional energy distribution of the 2D Gaussian ellipse over the entire pixel region is calculated using the standard deviation of the 2D Gaussian ellipse and its corresponding principal axis position. This distribution is then weighted, accumulated, and fused to obtain the final pixel color. The following is the face-to-face sampling calculation process.

[0086] In step S2, firstly, the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is calculated, which allows us to obtain the offset d between the two points in two directions. x d y The specific formula is shown below:

[0087]

[0088] In the formula, x pix x is the x-coordinate of the current pixel. i Let x be the x-axis coordinate of the Gaussian point projection, and y be the y-axis coordinate. pix The y-coordinate of the current pixel. i y-coordinate of the Gaussian point projection.

[0089] Then, the offset is projected onto the principal axis of the two-dimensional Gaussian ellipse to obtain the relative coordinates along the principal axis, including:

[0090] The principal axis direction includes the two orthogonal eigenvectors (v) of the two-dimensional Gaussian ellipse. 1x ,v 1y ) and (v 2x ,v 2y The relative coordinates (u,v) are calculated using the following formula:

[0091]

[0092] In the formula, v1 and v2 are the eigenvectors of the two-dimensional Gaussian ellipse along the principal axis, and d is the offset. x and d y These are the offsets on the x-axis and y-axis, respectively. The projected coordinates u and v describe the relative position of the point along the principal axes of the ellipse. They are crucial for calculating the transparency of the Gaussian distribution and its contribution to the pixel, because the influence of the Gaussian distribution decays with increasing distance, and the projected coordinates can effectively help calculate this decay process.

[0093] In step S3, to estimate the directional energy distribution of the 2D Gaussian ellipse over the pixel region, it is necessary to calculate the integral of the Gaussian ellipse along each principal axis (i.e., the probability value of the Gaussian ellipse in each direction). This invention utilizes the Abramowitz-Stegun polynomial approximation formula to calculate the integral of the two-dimensional Gaussian ellipse along the principal axis directions. The calculation formula is as follows:

[0094]

[0095] In the formula, Int u Int is the integral along the u-axis. v Let λ be the integral along the v-axis, λ be a constant factor, and σ1 and σ2 be the standard deviations of the Gaussian ellipse. The formula for the polynomial approximation of the Abramowitz-Stegun error function is as follows:

[0096]

[0097] In the formula, sign is used to ensure that the sign of the output is consistent with the sign of the input, a1, a2, a3, a4, a5 and p are all approximation coefficients, k is an adjustable parameter, by adjusting this parameter, the magnitude of the directional energy distribution of the Gaussian ellipse can be more accurately controlled so as to perform pixel coloring, t is an auxiliary variable, p is the scaling factor in the definition of t, and x is the input variable.

[0098] Finally, by integrating Int in the associative direction of multiplication. u Int v This allows us to obtain the directional energy distribution of the 2D ellipse, using the following formula:

[0099] S = M·Int u ·Int v ;

[0100] In the formula, S is the magnitude of the directional energy distribution, M is a constant factor, and Int u and Int v These are the integrals along the u-axis and v-axis, respectively.

[0101] In step S4, the final rendered image is obtained by weighted accumulation and fusion of the two-dimensional Gaussian ellipse based on the directional energy distribution and calculating the pixel shading values, including:

[0102] Based on the depth sorting results, the color features of each two-dimensional Gaussian ellipse are weighted according to the directional energy distribution S, and the final pixel color C is calculated by weighted α mixing.

[0103]

[0104] In the formula, N is the set of two-dimensional Gaussian ellipses, and c i S represents the color feature of the i-th elliptical region. i Let α be the directional energy distribution of i elliptical regions. i α represents the transparency of this area. j The transparency of the first region is given by T, where T represents the transparency of the current pixel, and bg is the background color.

[0105] By integrating the window signal face-to-face, this invention fully considers the impact of the tilt of the 2D Gaussian ellipse pose caused by dynamic changes in the viewpoint on the pixel window, and quantifies this impact in a weighted manner, enabling the model to complete pixel shading more accurately, thereby alleviating the problems of jagged edges, artifacts and loss of detail caused by "point-to-point" integration.

[0106] To further illustrate the advantages of the present invention, specific experiments are described below.

[0107] This invention's method is implemented in Python using the PyTorch framework, specifically PyTorch version 2.0.0 and Python version 3.8. It also utilizes custom CUDA kernels from 3DGS for rasterization, which are extensions of previous methods, and employs the NVIDIA CUB sorting routine for fast radix sorting. Furthermore, all experimental data for this invention were obtained using an NVIDIA GeForce RTX 3090 server with 24GB of VRAM.

[0108] 1. Dataset Selection

[0109] The datasets selected for this invention are largely the same as those used by 3DGS and most of its extended methods, covering scenes with different shooting styles, including bounded indoor scenes and large unbounded outdoor scenes, and including both synthetic and real-world datasets. This diversity of datasets enhances the generalizability and persuasiveness of the experimental results of this invention. Specific evaluation datasets include Synthetic Blender, Mip-NeRF360, and Tanks & Temples.

[0110] First, this invention selects the Synthetic Blender dataset, which contains eight synthetic bounded scenes. Each scene provides a detailed view set and accurate camera parameters, which helps to generate suitable initial SfM point clouds. Furthermore, since the synthetic dataset has accurate labels and realistic rendering effects, extensive experiments can be conducted without actual data collection costs and limitations to further verify the model's generalization ability.

[0111] Next, this invention selected the Mip-NeRF360 dataset for evaluation. Mip-NeRF360 is a recognized state-of-the-art NeRF rendering quality benchmark, containing challenging real-world indoor and outdoor scenes with rich detail and high complexity, suitable for evaluating the model's performance in high-quality rendering. This dataset provides high-quality rendered images and accurate depth information, making it ideal for evaluating the performance of advanced rendering models. It contains nine real-world scenes, each with a complex central object or region and detailed backgrounds. During filming, the authors used fixed camera exposure settings to minimize lighting variations and avoid object movement, reducing the impact of illumination changes on model evaluation, thereby improving the stability, generalization ability, and comparability of the rendered model.

[0112] Finally, this invention also selected two outdoor scenes from the Tanks & Temples dataset for evaluation. The Tanks & Temples dataset provides diverse and challenging real-world outdoor scenes involving various geometries, lighting conditions, and materials, effectively testing the model's performance and generalization ability in complex environments. Furthermore, the dataset provides accurate camera pose and depth information, which helps to more accurately evaluate the model's rendering quality.

[0113] By selecting these representative datasets, this invention is able to comprehensively evaluate the model's performance under different scenarios and conditions.

[0114] 2. Selection of evaluation criteria

[0115] Regarding the evaluation criteria, in order to clearly compare with 3DGS and its latest methods, this invention, like 3DGS, selected three classic metrics: Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). These three metrics are usually used simultaneously to evaluate the performance of rendering models to provide a more comprehensive evaluation result.

[0116] (1) Peak Signal-to-Noise Ratio (PSNR)

[0117] PSNR is a common metric for measuring image quality. It calculates the peak signal-to-noise ratio between the original image and the compressed or processed image; that is, the ratio of the maximum possible power of the image to the average power introduced by distortion. A higher PSNR value indicates better image quality. When evaluating rendering models, PSNR provides a quantitative measure of the difference between the generated image and the real image, and is therefore often used as a metric for evaluating rendering quality. Its calculation is as follows:

[0118]

[0119] In the formula, MAX is the maximum possible value of the image pixels, and MSE (Mean Squared Error) is the mean squared error between the original image and the processed image.

[0120] (2) Structural Similarity Index (SSIM)

[0121] SSIM is a metric used to compare the similarity between two images. SSIM considers the brightness, contrast, and structural information of an image, measuring the degree of similarity between two images by calculating brightness similarity, contrast similarity, and structural similarity. SSIM values ​​range from -1 to 1, where 1 indicates that the two images are identical. SSIM is commonly used to evaluate the performance of tasks such as image enhancement, denoising, and compression. When evaluating rendering models, SSIM can provide a more granular measure of image similarity, capturing the structural and content similarity between the rendered image and the real image. Its calculation formula is shown below:

[0122]

[0123] In the formula, μ x and μ y It is the mean of the x and y values ​​of the image. and σ is the variance of the image x and y. xy It is the covariance of the image x and y, and c1 and c2 are constants used to stabilize the calculation of the denominator.

[0124] (3) Learning Perceptual Image Patch Similarity LPIPS

[0125] LPIPS is a deep learning-based image similarity metric. It uses a pre-trained deep learning model to learn the perceptual features of images and measures the similarity between images by calculating the distance between them in the feature space. LPIPS can better simulate the human visual system's perception of images; therefore, it can provide a more accurate image similarity metric when evaluating rendering models. A lower LPIPS value indicates a greater consistency with human subjective perception. Its calculation formula is shown below:

[0126] LPIPS(x,y)=∑ i w i ||φ i (x)-φ i (x)||;

[0127] Where, φ i (x) and φ i (x) is the representation of images x and y in the i-th feature channel, w i It represents the weight of each feature channel.

[0128] 3. Ablation and Parameter Comparison Experiment

[0129] This invention primarily measures the impact of improved algorithm modules on the proposed method through a series of ablation experiments. Specifically, this invention tests the effects of the "face-to-face" sampling paradigm, the Abramowitz-Stegun function used to solve the directional energy distribution, and the Gauss-Hermite function on the overall reconstruction performance, as shown in Table 1. Here, G represents the Gauss-Hermite function, defined as follows. Table 1 shows that when the "face-to-face" sampling paradigm is incorporated and the Abramowitz-Stegun function is used to solve the directional energy distribution, the PSNR values ​​for all scenarios are the highest, regardless of whether it's 7k iterations or 30k iterations.

[0130]

[0131] Where θ(x) is the Gauss-Hermite function, which can also approximate the directional energy distribution of a 2D Gaussian ellipse over a pixel region, and β is an adjustable parameter.

[0132] Table 1 PSNR values ​​in ablation experiments

[0133]

[0134] Where P represents the point-to-point sampling paradigm, F represents the face-to-face sampling paradigm; A represents the Abramowitz-Stegun function; G represents the Gauss-Hermite function; 7k and 30k are the number of training iterations.

[0135] Furthermore, since the Abramowitz-Stegun and Gauss-Hermite functions each have adjustable parameters k and β, this invention also conducted experimental comparisons on the impact of these two parameters on the reconstruction results, specifically as follows: Figure 3 As shown. From Figure 3As can be seen, when both "face-to-face" sampling is introduced, the Abramowitz-Stegun function has a higher peak value than the Gauss-Hermite function, reaching 32.69 dB. This is because the Abramowitz-Stegun function can more accurately solve for the directional energy distribution of the 2D Gaussian ellipse, providing more accurate weighting values ​​in the subsequent weighted accumulation fusion, thus enabling precise coloring.

[0136] In addition, this invention also provides a comparison of the visualization effects of these two functions on the reconstruction results, such as... Figure 4 As shown. From Figure 4 As shown in the red box, the Abramowitz-Stegun function performs better than the Gauss-Hermite function in terms of anti-aliasing and is more effective at eliminating image artifacts.

[0137] Furthermore, this invention also conducted a series of experimental comparisons on the influence of the value of k on the Abramowitz-Stegun function on SSIM, specifically as follows: Figure 5 , Figure 6 As shown, the SSIM evaluation metric can reflect the jagged edges and artifacts present in an image to some extent; a higher SSIM indicates fewer jagged edges and artifacts. From... Figure 5 As can be seen, when the k value approaches 0.9 from 0.4, SSIM gradually increases, indicating an improvement in the model's anti-aliasing and artifact removal capabilities. Conversely, when the k value exceeds 0.9, SSIM begins to decrease, and both anti-aliasing and artifact removal capabilities begin to decline. Furthermore, it can also be seen from... Figure 6 It was observed that as the value of k approaches 0.9 from 0.4, the rendered image becomes closer to the ground truth; however, when the value of k is greater than 0.9, the quality of the reconstructed image begins to decline. Therefore, the model performs best when the value of k is 0.9.

[0138] 4. Experimental Comparison

[0139] To verify the advancement and generalization performance of the method in this invention, we conducted comparative experiments with existing state-of-the-art methods based on three datasets: Synthetic Blender, Mip-NeRF360, and Tanks & Temples. The code for this invention is implemented based on 3DGS, and the training parameters and learning rate scheduling are kept consistent with 3DGS to ensure the effectiveness of the scheme.

[0140] Evaluation on Synthetic Blender. This invention is compared with some classic algorithms and state-of-the-art methods such as Mip-Splatting and 2DGS on the Synthetic Blender dataset. These were chosen because of their excellent anti-aliasing capabilities, and comparison with them more effectively demonstrates the superior anti-aliasing performance of the proposed method, as shown in Table 2. Table 2 shows that the proposed method leads in performance on almost all Synthetic Blender datasets, only slightly lagging behind Mip-Splatting in the Mic scene. This is because the complexity of this scene is low, and Mip-Splatting's filtering mechanism does not lose much detail during smoothing. In addition, the average PSNR score of the method of this invention on this dataset is improved by about 0.5dB compared to the original 3DGS, about 0.4dB compared to Mip-Splatting, and about 1.0dB compared to 2DGS. This is because although the filtering mechanism introduced by Mip-Splatting can perform anti-aliasing well, it inevitably loses some details. 2DGS also produces artifacts when the scene viewpoint changes. The anti-viewpoint change brought by the full integration of "face-to-face" can provide better details while performing anti-aliasing and fully consider the impact of viewpoint changes, thereby reducing artifacts.

[0141] Table 2 PSNR scores for the Synthetic Blender Dataset

[0142]

[0143] Furthermore, we performed qualitative comparisons with several state-of-the-art methods on the Synthetic Blender dataset to verify the effectiveness and robustness of our proposed method. The qualitative comparisons are as follows: Figure 7 As shown in the figure, by using the "face-to-face" sampling paradigm, the method of this invention can perform pixel coloring more accurately than Mip-Splatting and 2DGS, thereby achieving anti-aliasing (as seen in the performance on the ficus, hotdog, and ship datasets) and mitigating the problem of detail loss (as seen in the performance on the chair and lego datasets).

[0144] Evaluation on Mip-NeRF360. Similarly, this invention also quantitatively compared itself with some state-of-the-art methods on nine scenes of the Mip-NeRF360 dataset. These methods alleviate the artifact problem of the original 3DGS to some extent. The comparison with them better demonstrates the effectiveness of the invention's method in suppressing artifacts, as shown in Table 3. As can be seen from Table 3, the invention's method outperforms existing methods in indoor scenes (such as counters, rooms, etc.) on this dataset, exhibiting better detail fidelity and artifact removal capabilities. However, in some outdoor scenes (such as bicycles, gardens, etc.), the invention's method is slightly inferior to Mip-Splatting. This is because outdoor scenes have more complex lighting environments, and these scenes are unbounded. In such cases, Mip-Splatting's filtering mechanism can better filter, smooth, and interpolate the scene.

[0145] Table 3 Comparison of PSNR values ​​on the Mip-NeRF360 dataset

[0146]

[0147] Furthermore, we evaluated our method on the Mip-NeRF360 dataset and performed qualitative comparisons with some existing methods, as shown in the qualitative comparisons below. Figure 8 As shown. From Figure 8 As can be seen, the method of the present invention has significantly improved the fidelity of detail (as shown by the red boxes on flowers, counter and room) and the ability to remove artifacts (as shown by the red boxes on flowers, bonsai and kitchen) compared with the previous methods.

[0148] Evaluation on the Tanks & Temples Dataset. The present invention was quantitatively compared with existing methods on two training scenes and on the truck dataset. Specifically, a series of measures were proposed to alleviate the detail loss problem in the original 3DGS. Comparison with these measures verifies the stability of the present invention in restoring reconstructed details, as shown in Table 4. Table 4 shows that by introducing a "face-to-face" sampling strategy, the present invention outperforms previous methods in both scenes.

[0149] Table 4 Comparison of PSNR values ​​on the Tanks & Temples dataset

[0150]

[0151] Furthermore, this invention also qualitatively compares with existing methods in these two scenarios, specifically as follows: Figure 9 As shown. (Through) Figure 9 As can be seen, the method of the present invention can capture details better than the previous method, ensuring that the reconstructed image has higher detail fidelity. For example, it can more clearly reconstruct the background in the distance in the training scene, and can also better reconstruct details such as light strips in the truck scene.

[0152] In summary, this invention proposes a "face-to-face" pixel integration paradigm to replace the original "point-to-point" integration paradigm, effectively supporting the implementation of 3DGS and effectively alleviating the problems of aliasing, artifacts, and detail loss caused by current 3DGS and related methods. Comprehensive experimental results show that the method of this invention is currently the most competitive 3DGS method known to us.

[0153] Example 2

[0154] like Figure 10 As shown, this invention provides a viewpoint-variation-resistant system for three-dimensional Gaussian spray. This system is used to implement the viewpoint-variation-resistant method for three-dimensional Gaussian spray described in Embodiment 1 above, specifically including:

[0155] The projection sorting module 100 is used to model the geometry into a set of three-dimensional Gaussian ellipses that do not require normals, project the three-dimensional Gaussian ellipses into two-dimensional Gaussian ellipses in screen space, and sort the two-dimensional Gaussian ellipses that overlap with the current pixel area according to the depth.

[0156] The offset calculation module 200 is used to calculate the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates, and project the offset onto the principal axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction;

[0157] The integral calculation module 300 is used to calculate the integral of the two-dimensional Gaussian ellipse along the principal axis using the Abramowitz-Stegun polynomial approximation formula, and obtain the directional energy distribution.

[0158] The weighted fusion module 400 is used to perform weighted cumulative fusion of the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel color value, and obtain the final rendered image.

[0159] This embodiment provides a viewpoint change resistance system for three-dimensional Gaussian spray, used to implement the aforementioned viewpoint change resistance method for three-dimensional Gaussian spray. Therefore, the specific implementation of the viewpoint change resistance system for three-dimensional Gaussian spray can be found in the previous embodiment section of the viewpoint change resistance method for three-dimensional Gaussian spray. For example, the projection sorting module 100, offset calculation module 200, integral calculation module 300, and weighted fusion module 400 are respectively used to implement steps S1, S2, S3, and S4 in the aforementioned viewpoint change resistance method for three-dimensional Gaussian spray. Therefore, its specific implementation can be referred to the description of the corresponding embodiments. To avoid redundancy, it will not be repeated here.

[0160] Example 3

[0161] This invention provides a computer storage medium storing a computer software product, which includes several instructions to cause a computer device to execute the above-described method for resisting viewpoint changes in three-dimensional Gaussian sputtering.

[0162] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A view-independent method for three-dimensional Gaussian splatting, characterized in that, include: The geometry is modeled as a set of 3D Gaussian ellipses that do not require normals. The 3D Gaussian ellipses are projected into 2D Gaussian ellipses in screen space. The 2D Gaussian ellipses that overlap with the current pixel region are sorted according to depth. Calculate the offset between the current pixel center point and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the principal axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction; The integral of the two-dimensional Gaussian ellipse along the principal axis is calculated using the Abramowitz-Stegun polynomial approximation formula to obtain the directional energy distribution; The two-dimensional Gaussian ellipse is weighted and accumulated based on the directional energy distribution to calculate the pixel color value and obtain the final rendered image. The method for projecting a three-dimensional Gaussian ellipse into a two-dimensional Gaussian ellipse in screen space is as follows: To project a 3D Gaussian ellipse onto a 2D screen space for pixel shading and rendering, the projection onto the image space is performed using the following formula: ； In the formula, To observe the transformation matrix, for The transpose of the matrix, The covariance matrix in the camera coordinate system. It is the Jacobian matrix of the affine approximation of the projective transformation. for The transpose of the matrix; To avoid the projected 2D Gaussian ellipse becoming too small (less than one pixel) when projected onto the screen space, a dilation operation is performed on the projected 2D Gaussian ellipse. The formula is as follows: ； In the formula, For the first in screen space A two-dimensional Gaussian ellipse at point... The energy density value at that location, It is a two-dimensional identity matrix. It is a scalar expansion hyperparameter. Centered on Gauss, The total covariance matrix defined in the world coordinate system. Let be a 2×2 symmetric positive definite matrix, describing the first... The shape, size, and orientation of a two-dimensional Gaussian ellipse Indicates transpose; The method for calculating the integral of a two-dimensional Gaussian ellipse along its principal axis using the Abramowitz-Stegun polynomial approximation formula is as follows: ； In the formula, for Integral along the axis, for Integral along the axis, A constant factor, , Let be the standard deviation of the Gaussian ellipse. The formula for the polynomial approximation of the Abramowitz-Stegun error function is as follows: ； In the formula, This is used to ensure that the sign of the output is consistent with the sign of the input. , , , , as well as All are approximate coefficients. These are adjustable parameters. As an auxiliary variable, for The scaling factor in the definition For input variables.

2. The method for resisting viewpoint variation in three-dimensional Gaussian spraying according to claim 1, characterized in that, The expression for the three-dimensional Gaussian ellipse is: ； In the formula, For three-dimensional space points The Gaussian energy density value at that location, Centered on Gauss, The covariance matrix is ​​defined in the world coordinate system, supporting anisotropic geometric modeling. for The inverse matrix, This indicates transpose.

3. The method for resisting viewpoint variation in three-dimensional Gaussian spraying according to claim 1, characterized in that, The method for calculating the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is as follows: Calculate the coordinates of the current pixel center point and the coordinates of the center point of the 2D Gaussian ellipse projection. shaft and The difference on the axis yields the two-dimensional offset. , ): ； In the formula, For the current pixel Axis coordinates For Gaussian point projection Axis coordinates For the current pixel Axis coordinates For Gaussian point projection Axis coordinates.

4. The method for resisting viewpoint variation in three-dimensional Gaussian spraying according to claim 1, characterized in that, The method for projecting the offset onto the principal axis direction of a two-dimensional Gaussian ellipse to obtain the relative coordinates along the principal axis direction is as follows: The two orthogonal eigenvectors of the two-dimensional Gaussian ellipse along the principal axis direction ( , )and( , The relative coordinates ( , Calculated using the following formula: ； In the formula, Let be the eigenvectors of a two-dimensional Gaussian ellipse along its principal axes. This is the offset. and They are respectively shaft and Offset on the axis.

5. The method for resisting viewpoint variation in three-dimensional Gaussian spraying according to claim 1, characterized in that, The directional energy distribution is calculated by multiplying the integrals along the principal axis, using the following formula: ； In the formula, The magnitude of the directional energy distribution. A constant factor, and They are respectively shaft and Integral along the axis.

6. The method for resisting viewpoint variation in three-dimensional Gaussian spraying according to claim 1, characterized in that, The method for calculating pixel color values ​​by weighted accumulation and fusion of the two-dimensional Gaussian ellipse based on the directional energy distribution is as follows: Based on the depth sorting results, the color features of each two-dimensional Gaussian ellipse are weighted according to the directional energy distribution S, and the final pixel color is calculated by weighted α mixing. : ； In the formula, Let be a set of two-dimensional Gaussian ellipses. For the first The color characteristics of each elliptical region For the first Directional energy distribution within an elliptical region For the first Transparency of each region For the first Transparency of each region Indicates the transparency of the current pixel. Used as the background color.

7. A system for resisting viewpoint variation in three-dimensional Gaussian spraying, characterized in that, The system is used to implement the viewpoint-variation-resistant method for three-dimensional Gaussian spraying as described in any one of claims 1 to 6, specifically including: The projection sorting module is used to model the geometry into a set of three-dimensional Gaussian ellipses that do not require normals, project the three-dimensional Gaussian ellipses into two-dimensional Gaussian ellipses in screen space, and sort the two-dimensional Gaussian ellipses that overlap with the current pixel region according to the depth. The offset calculation module is used to calculate the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates, and project the offset onto the principal axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the principal axis direction; The integral calculation module is used to calculate the integral of the two-dimensional Gaussian ellipse along the principal axis using the Abramowitz-Stegun polynomial approximation formula, and obtain the directional energy distribution. The weighted fusion module is used to perform weighted cumulative fusion of the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel color value, and obtain the final rendered image.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions for causing a computer device to execute the viewpoint-variation-resistant method for three-dimensional Gaussian sputtering as described in any one of claims 1 to 6.

Images