Three-dimensional Gaussian spattering-oriented viewpoint change resisting method and system

Abstract

The invention provides an anti-viewpoint change method and system for three-dimensional Gaussian spattering, and relates to the technical field of computer graphics, and the method comprises the steps: modeling a geometry into a group of three-dimensional Gaussian ellipses which do not need a normal, projecting the three-dimensional Gaussian ellipses into two-dimensional Gaussian ellipses in a screen space, sorting the two-dimensional Gaussian ellipses overlapped with the current pixel region according to the depth; calculating the offset between the central point of the current pixel and the projection coordinate of the two-dimensional Gaussian ellipse, and projecting the offset to the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinate in the main axis direction; the integral of the two-dimensional Gaussian ellipse in the main axis direction is calculated, and directional energy distribution is obtained; and performing weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution, calculating a pixel coloring value, and obtaining a final rendered image. According to the method, the problems of sawteeth, artifacts and detail loss caused by point-to-point integration are relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics, and in particular to a method and system for resisting viewpoint change for three-dimensional Gaussian splashing. Background Art

[0002] In today's computer graphics field, new perspective synthesis technologies are developing rapidly. Among them, neural rendering technologies, represented by Neural Radiance Field (NeRF), have achieved breakthroughs, driving the development of advanced applications such as human / virtual modeling and surface reconstruction. However, the backward-mapping voxel rendering paradigm employed by NeRF faces real-time bottlenecks. Despite subsequent research and improvements through various optimization strategies, true real-time rendering remains difficult to achieve in complex scenes.

[0003] To address this challenge, 3D Gaussian Splatting (3DGS) was developed. 3DGS builds a differentiable rasterization pipeline suitable for GPU acceleration. Using forward mapping techniques combined with an explicit point primitive representation, it models the scene as a collection of anisotropic 3D Gaussian ellipses with physical properties. This technology has achieved remarkable results in real-time, high-quality rendering and is widely used in multiple cutting-edge fields.

[0004] Despite its obvious advantages, 3DGS also presents a number of challenges. Regarding sampling, 3DGS uses a point-to-point sampling strategy when shading pixels, treating pixels as one-dimensional points and sampling a two-dimensional ellipse. This approach violates the Shannon-Nyquist sampling theorem. When the high-frequency components of the scene signal exceed the pixel's Nyquist frequency, aliasing inevitably occurs, severely impacting the quality of the rendered image. While filtering mechanisms have been developed to mitigate aliasing, overly smooth filtering often results in loss of detail.

[0005] In addition, since the covariance matrix of the three-dimensional Gaussian ellipse projected into the screen space changes dynamically with the viewing angle, when the main axis of the projected ellipse is at an inclined angle to the pixel grid, the "point-to-point" sampling cannot reflect the directional energy distribution of the ellipse within the pixel window, which in turn causes insufficient sampling, resulting in inaccurate shading, artifacts, and loss of details.

[0006] Many studies have attempted to address these issues with 3DGS. For example, Mip-Splatting proposes a hybrid filtering mechanism for anti-aliasing, but loses detail due to excessive filtering. MS3DGS introduces a multi-scale Gaussian ellipse representation to alleviate aliasing in low-resolution scenes, but performs poorly in medium- and high-resolution scenes. Pixel-GS calculates the average gradient magnitude of a point from a pixel perspective, reducing reliance on the initial point of Structure from Motion (SfM) and alleviating some artifacts, but increasing memory consumption. VDGS fuses 3DGS with NeRF to alleviate specular artifacts, but the introduction of a neural network reduces rendering efficiency. HO-Gaussian combines grid-based voxels with display Gaussians to improve scene appearance and geometric consistency, but suffers from inaccurate reconstruction of fine geometric structures. 2D-Gaussian-Splatting compresses a 3D Gaussian ellipse into a 2D oriented planar Gaussian ellipse, combining depth distortion and normal consistency to capture complex structural details, but suffers from deficiencies in its densification strategy.

[0007] Most of these methods only address specific issues of 3DGS and still use the same "point-to-point" integral sampling strategy as 3DGS for pixel shading, failing to fundamentally and systematically address the issues of aliasing, artifacts, and loss of detail. Therefore, a new approach is urgently needed to address these issues at their root and improve the rendering quality and performance of 3DGS. Summary of the Invention

[0008] To this end, an embodiment of the present invention provides a method and system for resisting viewpoint changes for three-dimensional Gaussian splashing, which is used to solve the problem in the existing technology that the three-dimensional Gaussian splashing technology violates the sampling theorem and does not fully consider the impact of viewpoint changes when sampling pixels, resulting in jagged edges, artifacts and loss of details in the rendered image.

[0009] To solve the above problems, an embodiment of the present invention provides a method for resisting viewpoint change for three-dimensional Gaussian splashing, the method comprising:

[0010] Model the geometry as a set of 3D Gaussian ellipses without normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel area according to depth;

[0011] Calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction;

[0012] The Abramowitz-Stegun polynomial approximation formula is used to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis to obtain the directional energy distribution;

[0013] The two-dimensional Gaussian ellipses are weightedly accumulated and fused according to the directional energy distribution, and pixel shading values ​​are calculated to obtain a final rendered image.

[0014] Preferably, the expression of the three-dimensional Gaussian ellipse is:

[0015]

[0016] Where, is the Gaussian energy density value at point x in three-dimensional space, is the Gaussian center, is the full covariance matrix defined in the world coordinate system, supporting anisotropic geometric modeling, Σ -1 is the inverse matrix of Σ.

[0017] Preferably, the method of projecting a three-dimensional Gaussian ellipse into a two-dimensional Gaussian ellipse is:

[0018] In order to project the three-dimensional Gaussian ellipse into the two-dimensional screen space for pixel shading and rendering, the image space is projected, and the formula is:

[0019] Σ′=JWΣW T J T ;

[0020] Where W is the observation transformation matrix, W T is the transposed matrix of W, Σ′ is the covariance matrix in the camera coordinate system, J is the Jacobian matrix of the affine approximation of the projective transformation, J T is the transposed matrix of J;

[0021] In order to avoid the degeneration of the projected two-dimensional Gaussian ellipse being too small when projected into the screen space, that is, smaller than one pixel, the projected two-dimensional Gaussian ellipse is expanded, and the formula is:

[0022]

[0023] Where, is the energy density value of the k-th two-dimensional Gaussian ellipse at point x in screen space, I is a two-dimensional identity matrix, s is a scalar expansion hyperparameter, p k is the Gaussian center, Σ is the full covariance matrix defined in the world coordinate system, is a 2×2 symmetric positive definite matrix that describes the shape, size, and direction of the k-th two-dimensional Gaussian ellipse. T represents the transpose.

[0024] Preferably, the method for calculating the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is:

[0025] Calculate the difference between the coordinates of the center point of the current pixel and the coordinates of the center point of the two-dimensional Gaussian ellipse projection on the x-axis and y-axis to obtain the two-dimensional offset (d x ,d y ):

[0026]

[0027] Where x pix is the x-axis coordinate of the current pixel, x i is the x-axis coordinate of the Gaussian point projection, y pix is the y-axis coordinate of the current pixel, y i is the y-axis coordinate of the Gaussian point projection.

[0028] Preferably, the method of projecting the offset onto the major axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the major axis direction is:

[0029] The main axis direction includes two orthogonal eigenvectors (v 1x ,v 1y ) and (v 2x ,v 2y ), the relative coordinates (u, v) are calculated by the following formula:

[0030]

[0031] Where v1 and v2 are the characteristic vectors of the two-dimensional Gaussian ellipse in the direction of the main axis, d is the offset, and d x and d y The offsets on the x-axis and y-axis respectively.

[0032] Preferably, the method for calculating the integral of the two-dimensional Gaussian ellipse in the direction of the major axis using the Abramowitz-Stegun polynomial approximation formula is:

[0033]

[0034] In the formula, Int u is the integral in the u-axis direction, Int v is the integral in the v-axis direction, λ is a constant factor, σ1 and σ2 are the standard deviations of the Gaussian ellipse, is the Abramowitz-Stegun error function polynomial approximation formula, which is:

[0035]

[0036] Where sign is used to ensure that the sign of the output is consistent with the sign of the input, a1, a2, a3, a4, a5, and p are approximate coefficients, k is an adjustable parameter, t is an auxiliary variable, p is the scaling factor in the definition of t, and x is the input variable.

[0037] Preferably, the directional energy distribution is calculated by multiplying the integrals in the main axis directions, and the calculation formula is:

[0038] S=M·Int u Int v ;

[0039] Where S is the directional energy distribution size, M is the constant factor, Int u and Int v are the integrals along the u-axis and v-axis respectively.

[0040] Preferably, the method of performing weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution to calculate the pixel shading value is:

[0041] Based on the depth sorting result, the color feature of each two-dimensional Gaussian ellipse is weighted according to the directional energy distribution S, and the final pixel color C is calculated by weighted α blending:

[0042]

[0043] Where N is the set of two-dimensional Gaussian ellipses, c i is the color feature of the i-th elliptical area, S i is the directional energy distribution of i elliptical regions, α i is the transparency of the area, α j is the transparency of the first area, T represents the transparency of the current pixel, and bg is the background color.

[0044] An embodiment of the present invention further provides a system for resisting viewpoint changes for three-dimensional Gaussian splashing. The system is used to implement the above-mentioned method for resisting viewpoint changes for three-dimensional Gaussian splashing, and specifically includes:

[0045] A projection sorting module is used to model the geometry as a set of 3D Gaussian ellipses without normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel area according to depth;

[0046] An offset calculation module is used to calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction;

[0047] An integral calculation module is used to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis using the Abramowitz-Stegun polynomial approximation formula to obtain a directional energy distribution;

[0048] The weighted fusion module is used to perform weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel shading value, and obtain the final rendered image.

[0049] An embodiment of the present invention further provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned method for resisting viewpoint change for three-dimensional Gaussian splashing.

[0050] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0051] (1) The present invention proposes a sampling calculation paradigm based on pixel areas. In order to solve the problems of geometric aliasing, artifacts and detail loss caused by pixel shading deviation due to the "point-to-point" strategy of sampling 2D Gaussian ellipse points using the pixel center point in the existing method, the present invention calculates the expansion degree of the 2D Gaussian ellipse in the direction of the main axis through the standard deviation and eigenvector of the Gaussian ellipse, and uses this to determine the directional energy distribution of the Gaussian ellipse corresponding to the pixel area, thereby transforming the "point-to-point" sampling of the existing method into "face-to-face" sampling. This measure not only complies with the Shannon-Nyquist sampling theorem, but also fully considers the impact of the tilt of the 2D Gaussian ellipse on the pixel area due to the dynamic change of the viewing angle, and then performs more accurate pixel shading through weighted accumulation fusion to alleviate the problems of aliasing, artifacts and detail loss.

[0052] (2) The present invention designs a method that can efficiently calculate the directional energy distribution of a Gaussian ellipse. In order to calculate the directional energy distribution more accurately while maintaining the efficiency of the calculation, the present invention introduces the Abramowitz-Stegun polynomial approximation calculation formula to realize the Gaussian integral calculation. The approximate formula approximates the error function through a 5th-order polynomial, and only 11 floating-point operations are required to obtain the calculation result. In addition, the function has an adjustable parameter. By adjusting this parameter, the size of the directional energy distribution of the 2D Gaussian ellipse to the pixel area can be more accurately determined to achieve more accurate weighted cumulative fusion shading, and support the flexibility of efficient calculation selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0054] Figure 1 A flowchart of a method for resisting viewpoint change for three-dimensional Gaussian splashing provided by the present invention;

[0055] Figure 2 An overview diagram of a method for resisting viewpoint changes in three-dimensional Gaussian splashing provided by the present invention;

[0056] Figure 3 The comparison diagram of the two parameters of the present invention on PSNR, wherein (a) is a schematic diagram of the influence of the Abramowitz-Stegun function parameter k, and (b) is a schematic diagram of the influence of the Gauss-Hermite function parameter β;

[0057] Figure 4 The following are comparison diagrams of the experimental effects of the two functions of the present invention, where (a) is the effect diagram under the Gauss-Hermite function and (b) is the effect diagram under the Abramowitz-Stegun function;

[0058] Figure 5 Schematic diagram of the effect of parameter k on SSIM in the present invention;

[0059] Figure 6 Schematic diagram showing the effect of parameter k on the reconstruction effect of the present invention;

[0060] Figure 7 This is an experimental comparison chart of the present invention on the Synthetic Blender dataset;

[0061] Figure 8 A comparison diagram of the present invention on Mip-NeRF360;

[0062] Figure 9 This is a comparison chart of the present invention on the Tanks&Temples dataset;

[0063] Figure 10 A block diagram of a viewpoint change resistance system for three-dimensional Gaussian splashing provided by the present invention. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0065] Example 1

[0066] In order to solve the problem of aliasing, artifacts and loss of details in the rendered image due to the "point-to-point" strategy violating the sampling theorem and not fully considering the impact of perspective changes in pixel sampling in the existing 3D Gaussian splattering technology, such as Figure 1 As shown, the present invention proposes a method for resisting viewpoint change for three-dimensional Gaussian splashing, which includes:

[0067] S1: Model the geometry as a set of 3D Gaussian ellipses without normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel area according to depth;

[0068] S2: Calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction;

[0069] S3: Use the Abramowitz-Stegun polynomial approximation formula to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis to obtain the directional energy distribution;

[0070] S4: Perform weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel shading value, and obtain the final rendered image.

[0071] From the above technical solution, it can be seen that the present invention proposes a method for resisting viewpoint changes for three-dimensional Gaussian splashing, which models the geometric body as a three-dimensional Gaussian ellipse without normals and projects and sorts it, avoiding the difficulty of estimating the normal of sparse point clouds. At the same time, it ensures the correct occlusion relationship of the Gaussian ellipse based on depth sorting, laying the foundation for accurate shading; through the pixel center point offset and the main axis direction projection, the traditional "point-to-point" sampling is converted into "face-to-face" area sampling, fully considering the tilt of the two-dimensional Gaussian ellipse caused by the perspective change, accurately capturing the directional energy distribution in the pixel area, and solving the aliasing and artifact problems caused by insufficient sampling; using the Abramowitz-Stegun polynomial approximation formula to efficiently calculate the main axis integral, accurately quantify the energy contribution of the Gaussian ellipse to the pixel area with extremely low computational complexity, avoid over-smoothing or sampling distortion, and retain high-frequency details; based on the directional energy distribution, weighted accumulation and fusion are performed, so that the pixel shading value can dynamically adapt to the elliptical shape under perspective changes, systematically improving the anti-aliasing, artifact removal capabilities and detail fidelity, and ultimately achieving high-quality, robust new perspective synthesis rendering effects.

[0072] Furthermore, the overall process of the method of the present invention is as follows Figure 2 As shown. The improvements of the present invention compared to the existing methods mainly include two points: the first is to transform the "point-to-point" sampling mechanism into a "face-to-face" sampling mechanism, that is, to regard the entire pixel as a continuous area rather than a discrete point, and then calculate the 2D Gaussian ellipse directional energy distribution corresponding to the pixel area. When the pixel is subsequently colored, this directional energy distribution enables the subsequent task to more accurately determine the impact of the 2D Gaussian ellipse tilt caused by the dynamic change of the viewing angle on the entire pixel window area after considering the standard deviation of the 2D Gaussian ellipse and the corresponding two main axis directions, thereby improving the accuracy of the rendered pixels through weighted accumulation fusion coloring. The second is to introduce the Abramowitz-Stegun polynomial approximation solution formula, which can achieve fast integral calculation with extremely low complexity under the premise of ensuring accuracy through parameter adjustment, and achieve high-quality coloring of the weighted accumulation fusion of the directional energy distribution of the pixel area by the 2D Gaussian ellipse, while achieving flexibility in balancing efficiency. As Figure 2 As shown, the red boxes represent the improvements of the present invention compared with the existing method.

[0073] In step S1, first, the geometric body is modeled as a set of three-dimensional Gaussian ellipses that do not require normals. The expression of the three-dimensional Gaussian ellipse is:

[0074]

[0075] Where, is the Gaussian energy density value at point x in three-dimensional space, is the Gaussian center, is the full covariance matrix defined in the world coordinate system, supporting anisotropic geometric modeling, Σ -1 is the inverse matrix of Σ.

[0076] Then, the three-dimensional Gaussian ellipse is projected into a two-dimensional Gaussian ellipse in screen space, including:

[0077] In order to project the three-dimensional Gaussian ellipse into the two-dimensional screen space for pixel shading and rendering, the image space is projected, and the formula is:

[0078] Σ′=JWΣW T J T ;

[0079] Where W is the observation transformation matrix, W T is the transposed matrix of W, Σ′ is the covariance matrix in the camera coordinate system, J is the Jacobian matrix of the affine approximation of the projective transformation, J T is the transposed matrix of J.

[0080] In order to avoid the degeneration of the projected two-dimensional Gaussian ellipse being too small when projected into the screen space, that is, smaller than one pixel, the projected two-dimensional Gaussian ellipse is expanded, and the formula is:

[0081]

[0082] Where, is the energy density value of the k-th two-dimensional Gaussian ellipse at point x in screen space, I is a two-dimensional identity matrix, s is a scalar expansion hyperparameter, p k is the Gaussian center, Σ is the full covariance matrix defined in the world coordinate system, is a 2×2 symmetric positive definite matrix that describes the shape, size, and direction of the k-th two-dimensional Gaussian ellipse. T represents the transpose.

[0083] It is important to note that this operation adjusts the scale of the two-dimensional Gaussian ellipse while keeping its maximum value unchanged.

[0084] Finally, the two-dimensional Gaussian ellipses overlapping with the current pixel area are sorted according to depth.

[0085] Because the existing "point-to-point" sampling violates the Shannon-Nyquist sampling theorem and does not fully consider the tilt of the 2D Gaussian ellipse relative to the pixel space as the viewing angle changes dynamically, this will lead to inaccurate final shading and cause problems such as aliasing, artifacts, and loss of details. The present invention proposes to sample the 2D Gaussian ellipse by treating the pixel as an area, and calculate the directional energy distribution of the 2D Gaussian ellipse for the entire pixel area through the standard deviation of the 2D Gaussian ellipse and the corresponding principal axis position, and then obtain the final pixel color by weighted accumulation and fusion of this distribution. The following is the face-to-face sampling calculation process.

[0086] In step S2, first, the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is calculated, and the offset d between the two points in two directions can be obtained respectively. x d y , the specific formula is as follows:

[0087]

[0088] Where x pix is the x-axis coordinate of the current pixel, x i is the x-axis coordinate of the Gaussian point projection, y pix is the y-axis coordinate of the current pixel, y i is the y-axis coordinate of the Gaussian point projection.

[0089] Then, the offset is projected onto the major axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the major axis direction, including:

[0090] The principal axis direction includes the two orthogonal eigenvectors of the two-dimensional Gaussian ellipse (v 1x ,v 1y ) and (v 2x ,v 2y ), the relative coordinates (u, v) are calculated using the following formula:

[0091]

[0092] Where v1 and v2 are the characteristic vectors of the two-dimensional Gaussian ellipse in the direction of the main axis, d is the offset, and d x and d y are the offsets on the x and y axes, respectively. The projected coordinates u and v describe the point's relative position along the major axis of the ellipse. They are crucial for calculating the transparency and contribution of the Gaussian distribution to the pixel, as the effect of the Gaussian distribution decays with distance, and projected coordinates effectively account for this decay.

[0093] In step S3, in order to estimate the directional energy distribution of the 2D Gaussian ellipse to the pixel area, it is necessary to calculate the integral of the Gaussian ellipse on each major axis (that is, the probability value of the Gaussian ellipse in each direction). The present invention uses the Abramowitz-Stegun polynomial approximation formula to calculate the integral of the two-dimensional Gaussian ellipse in the major axis direction. The calculation formula is:

[0094]

[0095] In the formula, Int u is the integral in the u-axis direction, Int v is the integral in the v-axis direction, λ is a constant factor, σ1 and σ2 are the standard deviations of the Gaussian ellipse, is the Abramowitz-Stegun error function polynomial approximation formula, which is:

[0096]

[0097] Where sign is used to ensure that the sign of the output is consistent with the sign of the input, a1, a2, a3, a4, a5, and p are all approximate coefficients, k is an adjustable parameter. By adjusting this parameter, the directional energy distribution of the Gaussian ellipse can be more accurately controlled for pixel coloring, t is an auxiliary variable, p is the scaling factor in the definition of t, and x is the input variable.

[0098] Finally, combine the integrals in the directions Int by multiplication u 、Int v , we can get the directional energy distribution of the 2D ellipse. The specific formula is:

[0099] S=M·Int u Int v ;

[0100] Where S is the directional energy distribution size, M is the constant factor, Int u and Int v are the integrals along the u-axis and v-axis respectively.

[0101] In step S4, a method of performing weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution and calculating the pixel shading value to obtain the final rendered image includes:

[0102] Based on the depth sorting result, the color feature of each two-dimensional Gaussian ellipse is weighted according to the directional energy distribution S, and the final pixel color C is calculated by weighted α blending:

[0103]

[0104] Where N is the set of two-dimensional Gaussian ellipses, c i is the color feature of the i-th elliptical area, S i is the directional energy distribution of i elliptical regions, α i is the transparency of the area, α j is the transparency of the first area, T represents the transparency of the current pixel, and bg is the background color.

[0105] Through face-to-face window signal integration, the present invention fully considers the impact of 2D Gaussian ellipse posture tilt on the pixel window due to dynamic changes in viewing angle, and quantifies this impact in a weighted manner, so that the model can complete pixel shading more accurately, thereby alleviating the problems of aliasing, artifacts and detail loss caused by "point-to-point" integration.

[0106] In order to further illustrate the advantages of the present invention, the following is a description with reference to specific experiments.

[0107] The method was implemented in Python using the PyTorch framework, version 2.0.0, and Python version 3.8. We also used custom CUDA kernels from 3DGS for rasterization, which are extensions of previous methods, and fast radix sorting using the NVIDIA CUB sort routine. All experimental data was obtained using an NVIDIA GeForce RTX3090 server with 24GB of video memory.

[0108] 1. Dataset Selection

[0109] The datasets selected for this paper are largely similar to those used by 3DGS and most of its extensions. They cover scenes with diverse shooting styles, including bounded indoor scenes and large, unbounded outdoor scenes, and include both synthetic and real-world datasets. This diversity ensures that the experimental results of this paper are more generalizable and persuasive. Specific datasets evaluated include Synthetic Blender, Mip-NeRF360, and Tanks & Temples.

[0110] First, we selected the Synthetic Blender dataset, which contains eight synthetic bounded scenes. Each scene provides a detailed view set and precise camera parameters, which facilitates the generation of a suitable initial SfM point cloud. Furthermore, the precise labeling and realistic rendering of this synthetic dataset allow for extensive experiments without the cost and limitations of real-world data collection, further validating the model's generalization capabilities.

[0111] Next, the present invention selected the Mip-NeRF360 dataset for evaluation. Mip-NeRF360 is a recognized advanced NeRF rendering quality benchmark that contains challenging real-world indoor and outdoor scenes. These scenes are rich in details and high in complexity, and are suitable for evaluating the performance of models in high-quality rendering. This dataset provides high-quality rendered images and accurate depth information, making it an ideal choice for evaluating the performance of advanced rendering models. It contains a total of 9 real-world scenes, each with a complex central object or area and a detailed background. During the shooting process, the authors adopted a fixed camera exposure setting to minimize lighting changes and avoid object movement, reducing the impact of photometric changes on model evaluation, thereby improving the stability, generalization ability and comparability of the rendering model.

[0112] Finally, we selected two outdoor scenes from the Tanks & Temples dataset for evaluation. The Tanks & Temples dataset provides diverse and challenging real-world outdoor scenes with a wide range of geometries, lighting conditions, and materials, effectively testing the model's performance and generalization capabilities in complex environments. Furthermore, the dataset provides accurate camera pose and depth information, enabling more precise evaluation of the model's rendering quality.

[0113] By selecting these representative datasets, the present invention is able to comprehensively evaluate the performance of the model in different scenarios and conditions.

[0114] 2. Selection of evaluation criteria

[0115] In terms of evaluation criteria, in order to clearly compare with 3DGS and its latest methods, this paper selects three classic indicators, namely Peak signal-to-noise ratio (PSNR), Structure Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), just like 3DGS. These three indicators are usually used simultaneously to evaluate the performance of rendering models to provide a more comprehensive evaluation result.

[0116] (1) Peak signal-to-noise ratio (PSNR)

[0117] PSNR is a common metric used to measure image quality. It calculates the peak signal-to-noise ratio (PSNR) between the original image and the compressed or processed image. This is the ratio of the maximum possible power of the image to the average power introduced by the distortion. Higher PSNR values ​​indicate better image quality. When evaluating rendering models, PSNR provides a quantitative measure of the difference between generated and real images, and is therefore often used as a metric for evaluating rendering quality. Its calculation is as follows:

[0118]

[0119] Where MAX is the maximum possible value of the image pixel, and MSE (Mean Squared Error) is the mean squared error between the original image and the processed image.

[0120] (2) Structural Similarity Index SSIM

[0121] SSIM is a metric used to compare the similarity of two images. SSIM takes into account the brightness, contrast, and structural information of the image, and measures the similarity between two images by calculating brightness similarity, contrast similarity, and structural similarity. The SSIM value ranges from -1 to 1, where 1 means that the two images are exactly the same. SSIM is often used to evaluate the performance of tasks such as image enhancement, denoising, and compression. When evaluating rendering models, SSIM can provide a more detailed image similarity measure that can capture the similarity in structure and content between rendered images and real images. Its calculation formula is as follows:

[0122]

[0123] Where μ x and μ y is the mean of image x and y, and is the variance of images x and y, σ xy is the covariance of images x and y, c1 and c2 are constants used to stabilize the calculation of the denominator.

[0124] (3) Learning Perceptual Image Patch Similarity LPIPS

[0125] LPIPS is a deep learning-based image similarity metric. It uses a pre-trained deep learning model to learn the perceptual features of images and measures the similarity between images by calculating the distance between them in the feature space. LPIPS better simulates the human visual system's perception of images, and therefore provides a more accurate image similarity metric when evaluating rendering models. Lower LPIPS values ​​indicate greater consistency with human subjective perception. Its calculation formula is as follows:

[0126] LPIPS(x,y)=∑ i w i ||φ i (x)-φ i (x)||;

[0127] Among them, φ i (x) and φ i (x) is the representation of images x and y on the i-th feature channel, w i is the weight of each feature channel.

[0128] 3. Ablation and parameter comparison experiments

[0129] The present invention mainly measures the impact of the improved modules of the algorithm on the method of the present invention by constructing a series of ablation experiments. Specifically, the present invention tests the impact of the "face-to-face" sampling paradigm, the Abramowitz-Stegun function for solving the directional energy distribution, and the Gauss-Hermite function on the overall reconstruction effect, as shown in Table 1, where G represents the Gauss-Hermite function, and its definition formula is shown below. As can be seen from Table 1, when the "face-to-face" sampling paradigm is added and the Abramowitz-Stegun function is used to solve the directional energy distribution, the PSNR value of each scene is the highest regardless of whether it is 7k iterations or 30k iterations.

[0130]

[0131] Among them, θ(x) is the Gauss-Hermite function, which can also approximately calculate the directional energy distribution of the 2D Gaussian ellipse to the pixel area, and β is an adjustable parameter.

[0132] Table 1 PSNR values ​​in ablation experiments

[0133]

[0134] Among them, P represents the point-to-point sampling paradigm, F represents the face-to-face sampling paradigm; A represents the Abramowitz-Stegun function; G represents the Gauss-Hermite function; 7k and 30k are the number of training iterations.

[0135] In addition, since the Abramowitz-Stegun function and the Gauss-Hermite function have adjustable parameters k and β respectively, the present invention also conducts an experimental comparison on the influence of these two parameters on the reconstruction results. Figure 3 As shown. Figure 3As can be seen in the figure, when both face-to-face sampling are introduced, the Abramowitz-Stegun function has a higher peak value of 32.69dB than the Gauss-Hermite function. This is because the Abramowitz-Stegun function can more accurately solve the directional energy distribution of the 2D Gaussian ellipse, providing more precise weighting values ​​in the subsequent weighted accumulation fusion, thereby achieving accurate shading.

[0136] In addition, the present invention also provides a comparison of the visualization effects of these two functions on the reconstruction results, such as Figure 4 As shown. Figure 4 As can be seen from the red box in the figure, the Abramowitz-Stegun function performs better in anti-aliasing and is more capable of eliminating image artifacts than the Gauss-Hermite function.

[0137] In addition, the present invention also conducts a series of experimental comparisons on the effect of the value of k on the Abramowitz-Stegun function on SSIM. Figure 5 、 Figure 6 As shown. The SSIM evaluation index can reflect the aliasing and artifacts in the image to a certain extent. The larger the SSIM, the less aliasing and artifacts there are in the image. Figure 5 As can be seen from the figure, when the k value approaches 0.9 from 0.4, SSIM gradually increases, which also means that the model's anti-aliasing effect and artifact removal capabilities are improved; when the k value is greater than 0.9, SSIM begins to decrease, and the anti-aliasing and artifact removal capabilities begin to decline. In addition, it can also be seen from Figure 6 It is observed that when the k value approaches 0.4 from 0.9, the rendered image becomes closer to the true value; when the k value is greater than 0.9, the quality of the reconstructed image begins to deteriorate. Therefore, when the k value is 0.9, the model performs best.

[0138] 4. Experimental comparison

[0139] To validate the advanced nature and generalization performance of our method, we conducted comparative experiments with state-of-the-art methods on the Synthetic Blender, Mip-NeRF360, and Tanks & Temples datasets. Our code is based on 3DGS, with consistent training parameters and learning rate scheduling to ensure the effectiveness of our approach.

[0140] Evaluation on Synthetic Blender. This paper compares the proposed method with several classic algorithms and the latest existing methods, Mip-Splatting and 2DGS, on the Synthetic Blender dataset. These algorithms are chosen because they have excellent anti-aliasing capabilities, and comparison with them effectively demonstrates the advanced anti-aliasing performance of the proposed method. This is shown in Table 2. As can be seen from Table 2, the proposed method leads the way on almost all Synthetic Blender datasets, only slightly lagging behind Mip-Splatting in the Mic scene. This is because the complexity of this scene is relatively low, and Mip-Splatting's filtering mechanism does not lose much detail during smoothing. In addition, the average PSNR score of the method of the present invention on this data set is improved by about 0.5dB compared with the original 3DGS, about 0.4dB compared with Mip-Splatting, and about 1.0dB compared with 2DGS. This is because although the filtering mechanism introduced by Mip-Splatting can perform anti-aliasing well, it will inevitably lose some details. 2DGS will also produce artifacts when the scene perspective changes. The anti-viewpoint change brought by the "face-to-face" full integration can provide better details while performing anti-aliasing and fully consider the impact of perspective changes, thereby reducing artifacts.

[0141] Table 2 PSNR scores of the Synthetic Blender Dataset

[0142]

[0143] In addition, we also conduct qualitative comparisons with several state-of-the-art methods on the Synthetic Blender dataset to verify the effectiveness and robustness of the proposed method. Figure 7 As shown in the figure, through the "face-to-face" sampling paradigm, the method of the present invention can perform pixel shading more accurately than Mip-Splatting and 2DGS, thereby performing anti-aliasing (as shown in the ficus, hotdog, and ship datasets) and alleviating the problem of detail loss (as shown in the chair and lego datasets).

[0144] Evaluation on Mip-NeRF360. Similarly, the present invention also makes a quantitative comparison with some of the latest existing methods in nine scenes on the Mip-NeRF360 dataset. These methods have alleviated the artifact problem of the original 3DGS to a certain extent. By comparing with them, the effectiveness of the method of the present invention in suppressing artifacts can be better reflected, as shown in Table 3. It can be seen from Table 3 that the method of the present invention performs better than the existing methods in indoor scenes (such as counter, room, etc.) on this dataset, and has better detail fidelity and artifact removal capabilities. However, in some outdoor scenes (such as bicycle, garden, etc.), the method of the present invention is slightly worse than Mip-Splatting. This is because outdoor scenes have more complex lighting environments, and these scenes are unbounded scenes. In this case, the filtering mechanism of Mip-Splatting can better filter, smooth and interpolate the scenes.

[0145] Table 3 Comparison of PSNR values ​​on the Mip-NeRF360 dataset

[0146]

[0147] In addition, we also evaluated the proposed method on the Mip-NeRF360 dataset and made qualitative comparisons with some existing methods. Figure 8 As shown. Figure 8 As can be seen from the figure, the method of the present invention has significantly improved detail fidelity (as shown by the red boxes on flowers, counter, and room) and artifact removal capability (as shown by the red boxes on flowers, bonsai, and kitchen) compared to the previous methods.

[0148] Evaluation on the Tanks & Temples Dataset. This method quantitatively compares itself with existing methods in two scenarios on this dataset: train and truck. A series of measures are proposed to mitigate the detail loss issue in the original 3DGS. This comparison verifies the stability of the proposed method in recovering reconstructed details, as shown in Table 4. As can be seen from Table 4, by introducing a "face-to-face" sampling strategy, the proposed method outperforms existing methods in both scenarios.

[0149] Table 4 Comparison of PSNR values ​​on the Tanks&Temples dataset

[0150]

[0151] In addition, the present invention also makes a qualitative comparison with the existing methods in these two scenarios. Figure 9 As shown. Figure 9 It can be seen that compared with previous methods, the method of the present invention can better capture details and ensure that the reconstructed image has higher detail fidelity. For example, in the train scene, the distant background can be reconstructed more clearly, and in the truck scene, details such as the light strip can also be better reconstructed.

[0152] In summary, this paper proposes a "face-to-face" pixel integration paradigm to replace the existing "point-to-point" integration paradigm, effectively supporting the implementation of 3DGS and mitigating the aliasing, artifacts, and detail loss issues associated with existing 3DGS and related methods. Comprehensive experimental results demonstrate that this method is the most competitive 3DGS method known to date.

[0153] Example 2

[0154] like Figure 10 As shown, the present invention provides a viewpoint change resistance system for three-dimensional Gaussian splashing, which is used to implement the viewpoint change resistance method for three-dimensional Gaussian splashing of the above embodiment 1, specifically comprising:

[0155] A projection sorting module 100 is configured to model the geometric body as a set of three-dimensional Gaussian ellipses without normals, project the three-dimensional Gaussian ellipses into two-dimensional Gaussian ellipses in screen space, and sort the two-dimensional Gaussian ellipses that overlap with the current pixel area according to depth;

[0156] An offset calculation module 200 is used to calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction;

[0157] An integral calculation module 300 is used to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis using the Abramowitz-Stegun polynomial approximation formula to obtain a directional energy distribution;

[0158] The weighted fusion module 400 is used to perform weighted accumulation fusion on the two-dimensional Gaussian ellipses according to the directional energy distribution, calculate the pixel shading value, and obtain the final rendered image.

[0159] The present embodiment provides a viewpoint change resistance system for three-dimensional Gaussian splashing, which is used to implement the aforementioned viewpoint change resistance method for three-dimensional Gaussian splashing. Therefore, the specific implementation method of the viewpoint change resistance system for three-dimensional Gaussian splashing can be seen in the embodiment part of the viewpoint change resistance method for three-dimensional Gaussian splashing mentioned above. For example, the projection sorting module 100, the offset calculation module 200, the integral calculation module 300, and the weighted fusion module 400 are respectively used to implement steps S1, S2, S3, and S4 in the aforementioned viewpoint change resistance method for three-dimensional Gaussian splashing. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0160] Example 3

[0161] An embodiment of the present invention provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned method for resisting viewpoint change for three-dimensional Gaussian splashing.

[0162] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0165] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A method for resisting viewpoint change for three-dimensional Gaussian splashing, characterized in that: include: Model the geometry as a set of 3D Gaussian ellipses without normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel area according to depth; Calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction; The Abramowitz-Stegun polynomial approximation formula is used to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis to obtain the directional energy distribution; The two-dimensional Gaussian ellipses are weightedly accumulated and fused according to the directional energy distribution, and pixel shading values ​​are calculated to obtain a final rendered image.

2. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The expression of the three-dimensional Gaussian ellipse is: Where, is the Gaussian energy density value at point x in three-dimensional space, is the Gaussian center, is the full covariance matrix defined in the world coordinate system, supporting anisotropic geometric modeling, Σ -1 is the inverse matrix of Σ.

3. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The method for projecting a three-dimensional Gaussian ellipse into a two-dimensional Gaussian ellipse is: In order to project the three-dimensional Gaussian ellipse into the two-dimensional screen space for pixel shading and rendering, the image space is projected, and the formula is: Σ′=JWΣW T J T ; Where W is the observation transformation matrix, W T is the transposed matrix of W, Σ′ is the covariance matrix in the camera coordinate system, J is the Jacobian matrix of the affine approximation of the projective transformation, J T is the transposed matrix of J; In order to avoid the degeneration of the projected two-dimensional Gaussian ellipse being too small when projected into the screen space, that is, smaller than one pixel, the projected two-dimensional Gaussian ellipse is expanded, and the formula is: Where, is the energy density value of the k-th two-dimensional Gaussian ellipse at point x in screen space, I is a two-dimensional identity matrix, s is a scalar expansion hyperparameter, p k is the Gaussian center, Σ is the full covariance matrix defined in the world coordinate system, is a 2×2 symmetric positive definite matrix that describes the shape, size, and direction of the k-th two-dimensional Gaussian ellipse. T represents the transpose.

4. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The method for calculating the offset between the current pixel center point and the two-dimensional Gaussian ellipse projection coordinates is: Calculate the difference between the coordinates of the center point of the current pixel and the coordinates of the center point of the two-dimensional Gaussian ellipse projection on the x-axis and y-axis to obtain the two-dimensional offset (d x ,d y ): Where x pix is the x-axis coordinate of the current pixel, x i is the x-axis coordinate of the Gaussian point projection, y pix is the y-axis coordinate of the current pixel, y i is the y-axis coordinate of the Gaussian point projection.

5. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The method of projecting the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction is: The two orthogonal eigenvectors (v 1x ,z 1y ) and (v 2x ,v 2y ), the relative coordinates (u, v) are calculated by the following formula: Where v1 and v2 are the characteristic vectors of the two-dimensional Gaussian ellipse in the direction of the main axis, d is the offset, and d x and d y The offsets on the x-axis and y-axis respectively.

6. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The method for calculating the integral of the two-dimensional Gaussian ellipse in the direction of the major axis using the Abramowitz-Stegun polynomial approximation formula is: In the formula, Int u is the integral in the u-axis direction, Int v is the integral in the v-axis direction, λ is a constant factor, σ1 and σ2 are the standard deviations of the Gaussian ellipse, is the Abramowitz-Stegun error function polynomial approximation formula, which is: Where sign is used to ensure that the sign of the output is consistent with the sign of the input, a1, a2, a3, a4, a5, and p are approximate coefficients, k is an adjustable parameter, t is an auxiliary variable, p is the scaling factor in the definition of t, and x is the input variable.

7. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The directional energy distribution is calculated by multiplying the integrals in the principal axis direction using the following formula: s=M·Int u ·Int v ; Where s is the directional energy distribution size, m is the constant factor, Int u and Int v are the integrals along the u-axis and v-axis respectively.

8. The method for resisting viewpoint change for three-dimensional Gaussian splashing according to claim 1, characterized in that: The method for calculating the pixel shading value by weighted accumulation fusion of the two-dimensional Gaussian ellipse according to the directional energy distribution is: Based on the depth sorting result, the color feature of each two-dimensional Gaussian ellipse is weighted according to the directional energy distribution S, and the final pixel color C is calculated by weighted α blending: Where N is the set of two-dimensional Gaussian ellipses, c i is the color feature of the i-th elliptical area, S i is the directional energy distribution of i elliptical regions, α i is the transparency of the area, α j is the transparency of the first area, T represents the transparency of the current pixel, and bg is the background color.

9. A system for resisting viewpoint change for three-dimensional Gaussian splashing, characterized in that: The system is used to implement the method for resisting viewpoint change for three-dimensional Gaussian splashing according to any one of claims 1 to 8, specifically comprising: A projection sorting module is used to model the geometry as a set of 3D Gaussian ellipses without normals, project the 3D Gaussian ellipses into 2D Gaussian ellipses in screen space, and sort the 2D Gaussian ellipses that overlap with the current pixel area according to depth; An offset calculation module is used to calculate the offset between the center point of the current pixel and the projection coordinates of the two-dimensional Gaussian ellipse, and project the offset onto the main axis direction of the two-dimensional Gaussian ellipse to obtain the relative coordinates in the main axis direction; An integral calculation module is used to calculate the integral of the two-dimensional Gaussian ellipse in the direction of the principal axis using the Abramowitz-Stegun polynomial approximation formula to obtain a directional energy distribution; The weighted fusion module is used to perform weighted accumulation fusion on the two-dimensional Gaussian ellipse according to the directional energy distribution, calculate the pixel shading value, and obtain the final rendered image.

10. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the anti-viewpoint change method for three-dimensional Gaussian splashing according to any one of claims 1 to 8.

Images