Planar Gaussian-based coloring model construction method, system and device and medium

Abstract

The invention discloses a coloring model construction method, system and device based on plane Gaussian, and a medium. The method comprises the following steps: initializing a position parameter, a depth sequence, a shape attribute, a coloring attribute and opacity of a Gaussian ball; determining the shortest axis of the Gaussian ball according to the shape attribute, and further calculating flattening loss; obtaining a local normal of the Gaussian sphere according to the shortest axis and the observation direction, and further calculating normal alignment loss; calculating sparsity loss according to the opacity; and coloring the Gaussian ball and calculating color loss, constructing a final loss function according to flattening loss, normal alignment loss, sparsity loss and color loss, and updating the shape attribute, local normal, opacity and coloring attribute through a back propagation algorithm. According to the method, the normal estimation precision is improved through the flattened Gaussian ball, so that the coloring quality is improved, high-quality rendering of various materials is realized through the combined coloring model, and the method can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a method, system, device and medium for constructing a shading model based on planar Gaussian. Background Art

[0002] In recent years, significant progress has been made in the field of 3D computer vision, especially in the reconstruction and visualization of complex 3D scenes. Neural Radiance Field (NeRF) enables highly realistic 3D view synthesis, but its optimization process is time-consuming and the rendering speed is slow, which limits its use in real-time applications. To address these issues, 3D Gaussian Splatting (3DGS) has emerged. 3DGS combines an efficient 3D Gaussian representation and tile-based splatting, making neural rendering more suitable for real-time scenes. However, despite some success, 3DGS performs poorly in handling surface scenes with reflective and specular properties. Its general approach to appearance modeling cannot capture complex view-dependent changes, which are crucial for achieving realistic rendering of materials with strong reflective properties.

[0003] Some advanced NeRF methods introduce a shading function into the implicit scene representation to effectively model reflective surfaces. However, due to the inherent limitations of implicit representation, these methods are time-consuming in the optimization process and difficult to apply to real-time rendering scenes. Based on the 3DGS framework, GaussianShader addresses this limitation by associating material properties with Gaussians, thus enabling scene modeling using a simplified shading function while improving the optimization and rendering speed while maintaining high-quality rendering.

[0004] Despite the above progress, accurately estimating surface normals remains a major challenge. This is due to the discrete nature of Gaussians. When the surface normals cannot be precisely aligned with the actual geometry of the scene, it leads to a series of problems that seriously affect the rendering quality. Inaccurate normals result in incorrect shading calculations, generating unrealistic lighting effects, especially for materials with strong specular reflections. Such misalignments can trigger artifacts such as blurry reflections and incorrect light bounces, overall reducing the realism of the rendered scene. For reflective surfaces, the light interaction highly depends on the precise direction of the surface normal, and even a slight inaccuracy can significantly weaken the fidelity of the final output. At the same time, the weak constraints of Gaussians and the limitations of the simplified shading model may lead to suboptimal geometric and material decomposition effects. Summary of the Invention

[0005] The purpose of the present invention is to solve at least to some extent one of the technical problems existing in the prior art.

[0006] To this end, an object of an embodiment of the present invention is to provide a method for constructing a shading model based on planar Gaussians, which improves the accuracy of normal estimation by flattening Gaussian spheres, thereby enhancing the shading quality and geometric representation, realizes robust material decomposition by combining shading models, and improves the rendering performance on various material properties.

[0007] Another object of an embodiment of the present invention is to provide a system for constructing a shading model based on planar Gaussians.

[0008] To achieve the above technical object, the technical solutions adopted in the embodiments of the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for constructing a shading model based on planar Gaussians, including:

[0010] Obtain a depth map and point cloud data according to the input image and camera parameters, and initialize the position parameters and depth sorting of the Gaussian sphere according to the depth map and the point cloud data;

[0011] Initialize the shape attributes, shading attributes, and opacity of the Gaussian sphere;

[0012] Determine the shortest axis of the Gaussian sphere according to the shape attributes, and calculate the flattening loss according to the shortest axis;

[0013] Obtain the local normal of the Gaussian sphere according to the shortest axis and the viewing direction;

[0014] Obtain the global normal according to the depth map, and then calculate the normal alignment loss according to the global normal and the local normal;

[0015] Calculate the sparsity loss according to the opacity;

[0016] Calculate the color of the Gaussian sphere according to the shading attributes, the viewing direction, and the local normal;

[0017] Obtain the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane according to the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere;

[0018] Calculate the color loss according to the final color and the true color of the input image;

[0019] Construct a final loss function according to the flattening loss, the normal alignment loss, the sparsity loss, and the color loss, and update the shape attributes, the local normal, the opacity, and the shading attributes through the backpropagation algorithm.

[0020] Further, obtaining a depth map and point cloud data according to the input image and camera parameters, and initializing the position parameters and depth sorting of the Gaussian sphere according to the depth map and the point cloud data, includes:

[0021] Obtaining the point cloud data through a structure from motion algorithm according to the input image and the camera parameters;

[0022] Obtaining the depth map according to the point cloud data;

[0023] Initializing the position parameters according to the point cloud data;

[0024] Initializing the depth sorting according to the depth map.

[0025] Further, the shape attribute includes a covariance matrix, and determining the shortest axis of the Gaussian sphere according to the shape attribute, and calculating a flattening loss according to the shortest axis, includes:

[0026] Decomposing the covariance matrix to obtain a scale matrix;

[0027] Determining the shortest axis according to the scale matrix;

[0028] Calculating the flattening loss according to the shortest axis.

[0029] Further, obtaining the local normal of the Gaussian sphere according to the shortest axis and the viewing direction, includes:

[0030] Judging whether the direction of the shortest axis is consistent with the viewing direction;

[0031] When the direction of the shortest axis is consistent with the viewing direction, taking the direction of the shortest axis as the direction of the local normal to obtain the local normal.

[0032] Further, the coloring attribute includes an original diffuse color, a diffuse attribute, a metallic attribute, an occlusion attribute, a specular reflection attribute, and a roughness attribute, and calculating the color of the Gaussian sphere through the following formula:

[0033] c(ω o )=γ(c d +c s +c r (ω o ))

[0034] λ d =c raw +g s ☉(1 - g m )

[0035] λ s =0.04(1 - g m )+gs ☉g m

[0036] c d = λ d ☉g o ☉L d (n)

[0037] c s = λ s ☉g o ☉L s (ω o , n, ρ)

[0038] r = 2(ω o ·n)n - ω o

[0039] L d (n) = ∫ Ω L(ω i )(ω i ·n)dω i

[0040] L s (ω o , n, ρ) = ∫ Ω L(ω i )D(r, ρ)(ω i ·n)dω i

[0041] Wherein, c is the color of the Gaussian sphere, ωo is the viewing direction, γ is the gamma adjustment mapping function for color rendering, c d is the diffuse color, c s is the specular color, c r is the indirect illumination simulated by spherical harmonics, ⊙ represents element-by-element multiplication, λ d and λ s are preset intermediate terms, g o is the occlusion property, L d is the direct diffuse light, L s is the direct specular light, n is the local normal, ρ is the roughness property, c raw is the original diffuse color, g s is the specular property, g m is the metallic property, r is the reflection direction, Ω is the upper half of the Gaussian sphere, ω i is the incident light radiation direction, L(ω i ) is the trainable environment light map, D(r, ρ) is the normal distribution function.

[0042] Further, obtaining the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane according to the position parameter, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere includes:

[0043] Projecting the Gaussian sphere onto a two-dimensional plane according to the position parameter and the viewing direction;

[0044] Calculating the final color according to the depth sorting, the opacity, and the color of the Gaussian sphere.

[0045] Further, the final loss function is:

[0046]

[0047] where L total is the total loss function, L color is the color loss, L geometry is the geometric loss, L auxiliary is the auxiliary loss, L normal is the normal alignment loss, L flatten is the flattening loss, L sparse is the sparsity loss, λ n is the training weight of the normal alignment loss, λ f is the training weight of the flattening loss, λ s is the training weight of the sparsity loss.

[0048] In a second aspect, an embodiment of the present invention provides a planar Gaussian-based shading model construction system, including:

[0049] An input processing module, configured to obtain a depth map and point cloud data according to an input image and camera parameters, and initialize the position parameter and depth sorting of the Gaussian sphere according to the depth map and the point cloud data;

[0050] An initialization module, configured to initialize the shape attribute, shading attribute, and opacity of the Gaussian sphere;

[0051] A flattening loss module, configured to determine the shortest axis of the Gaussian sphere according to the shape attribute, and calculate the flattening loss according to the shortest axis;

[0052] A normal estimation module, configured to obtain the local normal of the Gaussian sphere according to the shortest axis and the viewing direction;

[0053] A normal alignment loss module, configured to obtain the global normal according to the depth map, and further calculate the normal alignment loss according to the global normal and the local normal;

[0054] A sparsity loss module, which is used to calculate the sparsity loss according to the opacity;

[0055] A coloring module, which is used to calculate the color of the Gaussian sphere according to the coloring attribute, the viewing direction, and the local normal;

[0056] A rendering module, which is used to obtain the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane according to the position parameter, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere;

[0057] A color loss module, which is used to calculate the color loss according to the final color and the true color of the input image;

[0058] A total loss module, which is used to construct a final loss function according to the flattening loss, the normal alignment loss, the sparsity loss, and the color loss, and update the shape attribute, the local normal, the opacity, and the coloring attribute through the backpropagation algorithm.

[0059] In a third aspect, an embodiment of the present invention provides an apparatus, including:

[0060] At least one processor;

[0061] At least one memory for storing at least one program;

[0062] When the at least one program is executed by the at least one processor, the at least one processor implements a method for constructing a coloring model based on planar Gaussian as described above.

[0063] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to execute the method for constructing a coloring model based on planar Gaussian as described above when executed by the processor.

[0064] The advantages and beneficial effects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention:

[0065] The present invention effectively imposes a flattening constraint on the shape of the Gaussian sphere by introducing a flattening loss, enabling the Gaussian sphere to more precisely fit the object surface; estimates the local normal through the shortest axis of the flattened Gaussian sphere, and can obtain a high-precision normal estimation result without additional correction measures, thereby improving the shading quality and geometric representation accuracy; improves the existing simplified shading model, effectively improves the model's ability to decompose the diffuse texture by introducing the original diffuse color term, and combines the advantages of the metallic property and the occlusion term, can effectively handle high-metallic objects, and significantly improves the ability to render reflective surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a step diagram of a method for constructing a shading model based on planar Gaussian provided by an embodiment of the present invention;

[0067] Figure 2 It is a schematic diagram of a system for constructing a shading model based on planar Gaussian provided by an embodiment of the present invention;

[0068] Figure 3 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0070] In the description of the present invention, the meaning of "a plurality of" is two or more. If the first and second are described, it is only for the purpose of distinguishing technical features and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of this technical field.

[0071] For the convenience of understanding, the nouns and technical terms that may appear in this specification are explained as follows:

[0072] 3D Gaussian Splatter (3DGS): 3DGS is a modeling technique that represents a scene using 3D Gaussian spheres, each of which includes a set of property characterizations. These properties include the position p, opacity α, covariance matrix Σ for characterizing shape properties, and spherical harmonics (SH) coefficients (i.e., shading properties) for color representation. The covariance matrix Σ is further decomposed into a rotation matrix R and a scale matrix S, expressed as Σ = RSSTRT. In 3DGS, rendering starts from projecting Gaussian spheres into the two-dimensional normalized device coordinate (NDC) space through the camera view. Subsequently, these Gaussian spheres are processed by a fast and differentiable tile-based rasterizer and sorted according to their projected depths. Through sorting, alpha blending is performed on the two-dimensional Gaussians according to their opacity values to calculate the final color of each pixel. The difference between the final color and the real image can be used to construct a loss function, and thus optimize the properties of Gaussian spheres according to backpropagation.

[0073] Figure 1 The step diagram of a method for constructing a shading model based on planar Gaussians provided by an embodiment of the present invention is referred to Figure 1 An embodiment of the present invention provides a method for constructing a shading model based on planar Gaussians, including:

[0074] Obtain a depth map and point cloud data according to the input image and camera parameters, and initialize the position parameters and depth sorting of Gaussian spheres according to the depth map and point cloud data;

[0075] Initialize the shape properties, shading properties, and opacity of Gaussian spheres;

[0076] Determine the shortest axis of the Gaussian sphere according to the shape property, and calculate the flattening loss according to the shortest axis;

[0077] Obtain the local normal of the Gaussian sphere according to the shortest axis and the viewing direction;

[0078] Obtain the global normal according to the depth map, and then calculate the normal alignment loss according to the global normal and the local normal;

[0079] Calculate the sparsity loss according to the opacity;

[0080] Calculate the color of the Gaussian sphere according to the shading property, viewing direction, and local normal;

[0081] Obtain the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane according to the position parameters, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere;

[0082] Calculate the color loss according to the final color and the real color of the input image;

[0083] Construct the final loss function based on the flattening loss, normal alignment loss, sparsity loss, and color loss, and update the shape attributes, local normals, opacity, and shading attributes through the backpropagation algorithm.

[0084] In some alternative embodiments, obtain the depth map and point cloud data according to the input image and camera parameters, and initialize the position parameters and depth sorting of the Gaussian sphere based on the depth map and point cloud data, including:

[0085] Obtain the point cloud data through the Structure-from-Motion (SfM) algorithm according to the input image and camera parameters;

[0086] Obtain the depth map according to the point cloud data;

[0087] Initialize the position parameters according to the point cloud data;

[0088] Initialize the depth sorting according to the depth map.

[0089] Specifically, in this embodiment, the input image is a multi-view image of the object to be modeled. The sparse point cloud is reconstructed from the multi-view image and camera parameters through the Structure-from-Motion (SfM) algorithm to obtain the point cloud data. The depth value of each pixel is obtained by projecting the point cloud onto the camera plane, thereby constructing the depth map, which is used to provide the depth sorting of the Gaussian sphere and estimate the global normal, so as to determine the occlusion relationship of the Gaussian sphere and facilitate the subsequent calculation of the final color after the Gaussian sphere is projected onto the plane.

[0090] In addition, the point cloud data is also used to initialize the position parameters of the Gaussian sphere. In the embodiments of the present invention, each point cloud point is initialized as a Gaussian sphere, and the position of the point cloud point determines the three-dimensional position of the Gaussian sphere. Other initial parameters of the Gaussian sphere, such as shape attributes, shading attributes, and opacity, are initialized to fixed values and gradually optimized in the subsequent backpropagation.

[0091] In some alternative embodiments, the shape attribute includes a covariance matrix. Determine the shortest axis of the Gaussian sphere according to the shape attribute, and calculate the flattening loss according to the shortest axis, including:

[0092] Decompose the covariance matrix to obtain the scale matrix;

[0093] Determine the shortest axis according to the scale matrix;

[0094] Calculate the flattening loss according to the shortest axis.

[0095] In some alternative embodiments, obtain the local normal of the Gaussian sphere according to the shortest axis and the viewing direction, including:

[0096] Judge whether the direction of the shortest axis is consistent with the viewing direction;

[0097] When the direction of the shortest axis is consistent with the viewing direction, the direction of the shortest axis is taken as the direction of the local normal to obtain the local normal.

[0098] Specifically, related research on 3DGS has shown that flattened Gaussian spheres can capture the object surface more accurately and align with the underlying geometry, while the accurate shading of Gaussian spheres depends on accurate normal estimation. Based on these insights, in this embodiment, flattened Gaussian spheres are integrated into the rendering pipeline, and a specific constraint is imposed on the shape of the Gaussian spheres by introducing a flattening loss, thereby achieving more accurate normal estimation on these flattened Gaussian spheres and improving the shading effect.

[0099] In this embodiment, the shape attributes of the Gaussian sphere can be described by the covariance matrix Σ, which determines the distribution shape of the Gaussian sphere in three-dimensional space. Through eigenvalue decomposition, the covariance matrix can be decomposed into a rotation matrix R and a scale matrix S, expressed as Σ = RSS T R T . Among them, the diagonal elements g scale = [s1, s2, s3] represent the scale sizes of the Gaussian sphere in the three principal axis directions. According to the minimum eigenvalue of g scale , the shortest axis can be determined, and then the direction corresponding to the shortest axis can be determined according to the eigenvector of the minimum eigenvalue. Therefore, in this embodiment, the flattening loss L flatten is defined as:

[0100]

[0101] Among them, min axis (g scale ) represents the axis corresponding to the minimum eigenvalue of g scale , that is, the shortest axis. Taking this axis as the target to be flattened, by imposing this loss, the Gaussian sphere is encouraged to compress along its shortest axis, thereby enhancing its matching ability with the underlying surface of the scene.

[0102] After the Gaussian is flattened, in this embodiment, the normal vector of the Gaussian sphere can be directly defined by its shortest axis. This is a significant improvement. Compared with other normal estimation methods, in this embodiment, more accurate shading results can be achieved without additional correction measures (such as normal offset). The normal of each Gaussian sphere is selected according to the shortest axis, and this axis naturally aligns with the surface geometry. Considering that the shortest axis may have two directions, in this embodiment, the correct normal direction is determined by aligning this axis with the viewing direction, and its formal definition is as follows:

[0103]

[0104] Among them, n represents the normal of the Gaussian sphere, v represents the positive direction of the shortest axis of the Gaussian sphere, and ω oThe viewing perspective. By selecting the normal direction consistent with the perspective direction, the shading calculation in the scene is ensured to be consistent and accurate on the reflective surface. This method not only simplifies the process of normal estimation but also significantly improves the overall rendering quality by maintaining the consistency of light interaction on the reflective surface. To further address the problem that the unstructured nature of the Gaussian makes it difficult to precisely define the surface normal of the scene, this embodiment defines the normal alignment loss L through the following formula normal :

[0105]

[0106] where represents the normal rendered from the Gaussian sphere, represents the normal obtained from the rendered unbiased depth map through a Sobel-like operation. This alignment ensures that the local Gaussian normals are consistent with the global scene geometry, further improving the accuracy of the global normals when dealing with flattened Gaussian spheres, thereby enhancing the overall geometric accuracy and improving the shading effect.

[0107] In some alternative embodiments, the shading attributes include the original diffuse color, diffuse attributes, metallic attributes, occlusion attributes, specular reflection attributes, and roughness attributes. The color of the Gaussian sphere is calculated through the following formula

[0108] c(ω o ) = γ(c d + c s + c r (ω o ))

[0109] λ d = c raw + g s ☉(1 - g m )

[0110] λ s = 0.04(1 - g m ) + g s ☉g m

[0111] c d = λ d ☉g o ☉L d (n)

[0112] c s = λ s ☉g o ☉L s (ω o , n, ρ)

[0113] r = 2(ω o ·n)n - ωo

[0114] L d (n) = ∫ Ω L(ω i )(ω i ·n)dω i

[0115] L s (ω o , n, ρ) = ∫ Ω L(ω i )D(r, ρ)(ω i ·n)dω i

[0116] Where c is the color of the Gaussian sphere, ω o is the viewing direction, γ is the gamma correction mapping function for color rendering, c d is the diffuse color, c s is the specular color, c r is the indirect illumination simulated by spherical harmonics, ⊙ represents element-wise product, λ d and λ s are preset intermediate terms, g o is the occlusion property, L d is the direct diffuse light, L s is the direct specular light, n is the local normal, ρ is the roughness property, c raw is the original diffuse color, g s is the specular property, g m is the metallic property, r is the reflection direction, Ω is the upper half of the Gaussian sphere, ω i is the incident light radiation direction, L(ω i ) is the trainable environment light map, D(r, ρ) is the normal distribution function.

[0117] Specifically, to enhance the rendering quality of reflective surfaces and facilitate relighting of the scene, the traditional GaussianShader introduces a simplified Bidirectional Reflectance Distribution Function (BRDF) and colors the 3D Gaussian sphere in combination with a learnable environment light map. Specifically, each Gaussian sphere is augmented with shading attributes including diffuse, roughness, specular, and a residual color term to represent indirect illumination that cannot be captured by the simplified shading function. Given the viewing direction ω o , the shading color of the Gaussian sphere can be calculated by the following formula:

[0118] c(ω o ) = γ(c d + cs +c r (ω o ))

[0119] where c is the color of the Gaussian sphere, ω o is the viewing direction, γ is the gamma adjustment mapping function for color rendering, c d is the diffuse color, c s is the specular color, c r is the indirect illumination simulated by spherical harmonics.

[0120] In the shading model of GaussianShader, c d and c r have the following expansion forms:

[0121] c d = c raw + (1 - g s ) ⊙ L d (n)

[0122] c s = g s ⊙ L s (ω o , n, ρ)

[0123] where c raw is the original diffuse color, which is directly added to the final diffuse color, g s is the specular reflection property, ⊙ represents element-wise multiplication, n is the normal, L d (n) is the direct diffuse light calculated through the ambient light map, ρ is the roughness property, L s (ω o , n, ρ) is the direct specular light calculated through the same ambient light map.

[0124] The calculation formulas for the direct diffuse light component L d (n) and the specular light component L s (ω o , n, ρ) are as follows:

[0125] r = 2(ω o · n)n - ω o

[0126] L d (n) = ∫ Ω L(ω i )(ω i · n)dω i

[0127] L s (ω o , n, ρ) = ∫Ω L(ω i )D(r,ρ)(ω i ·n)dω i

[0128] where r is the reflection direction, Ω is the upper half of the Gaussian sphere, ω i is the incident light radiation direction, L(ω i ) is a trainable environment light map, represented by a 6×64×64 cube map, and D(r,ρ) is the SpecularGGX normal distribution function, which is used to define the distribution range of specular reflection.

[0129] To achieve efficient shading, multi-level mipmaps are pre-computed for different reflection directions and roughness levels. During rendering, the result of L is obtained by interpolation from these pre-computed mipmaps, ensuring a balance between computational efficiency and rendering accuracy. s In contrast, the shading model of NvDiffRec models c

[0130] and c d in the following form: r

[0131] λ d = g d ⊙(1 - g m )

[0132] λ s = 0.04(1 - g m ) + g d ⊙g m

[0133] c d = λ d ☉g o ☉L a (n)

[0134] c s = λ s ☉g o ☉L s (ω o , n, ρ)

[0135] where g o is the occlusion property, g d is the diffuse property, and g m is the metallic property.

[0136] The shading model of NvDiffRec includes an explicit occlusion term g o and an explicit metallicity term g m , while the model of GaussianShader uses a special c​raw terms, but does not include occlusion or metallicity. After integrating the shading model of NvDiffRec into GaussianShader, it can be observed that the model performs well on high-metallicity objects, but performs poorly on low-metallicity objects compared to the model of GaussianShader. By analyzing these results, it can be speculated that the explicit handling of metallicity and occlusion in the NvDiffRec model enables it to effectively model high-metallicity materials, but the lack of the original diffuse color term limits its performance on low-metallicity materials. On the contrary, the shading model of GaussianShader performs excellently on low-metallicity materials due to the inclusion of the original diffuse color term, but performs poorly on high-metallicity materials.

[0137] In response to these findings, this embodiment proposes a novel combined shading model that combines the advantages of the two methods. Specifically, based on NvDiffRec, a c raw term is added to its model to characterize the original diffuse color. Since the newly added term already models the diffuse texture, the original g d term is renamed to g s to better control the specular reflection component, similar to the model of GaussianShader. The improved formula is as follows:

[0138] λ d = c raw + g s ⊙ (1 - g m )

[0139] λ s = 0.04(1 - g m ) + g s ⊙ g m

[0140] Compared with the shading model of GaussianShader, the new model introduces metallicity and occlusion terms, enabling it to handle high-metallicity objects more effectively. Compared with the model of NvDiffRec, the c raw term improves the decomposition ability of the diffuse texture, thus performing better on low-metallicity materials. This combined model effectively integrates the advantages of the two shading models while alleviating their respective limitations.

[0141] In some alternative embodiments, the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane is obtained according to the position parameter, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere, including:

[0142] Project the Gaussian sphere onto a two-dimensional plane according to the position parameter and the viewing direction;

[0143] Calculate the final color based on depth sorting, opacity, and the color of the Gaussian spheres.

[0144] Specifically, in this embodiment, the Gaussian sphere position parameters are projected onto a two-dimensional plane through the viewing direction to generate their two-dimensional coordinates in the normalized device coordinate (NDC) space. Subsequently, these Gaussian spheres are processed by a fast and differentiable tile-based rasterizer and sorted according to their projected depths. Through sorting, alpha blending is performed on the two-dimensional Gaussians according to their opacity values to calculate the final color of each pixel. The calculation formula is as follows:

[0145]

[0146] where c i is the color of each Gaussian sphere calculated from the previous shading equation, and the transparency, α i of each Gaussian sphere is determined by weighing the opacity of the Gaussian with its power (i.e., its exponential decay function from the mean), and the cumulative transparency represents the degree to which the current Gaussian sphere is occluded by the previous Gaussian spheres.

[0147] After the color of each pixel has been accumulated in the rendered image, the difference between the rendered image and the input real image is minimized through color loss. The color loss L color is defined as follows:

[0148]

[0149] where G gt is the color of the input real image.

[0150] This loss function optimizes the Gaussian parameters in the scene through backpropagation, thereby gradually improving the scene representation to achieve a more accurate rendering effect.

[0151] In some alternative embodiments, the final loss function L total is:

[0152]

[0153] where L total is the total loss function, L color is the color loss, L geometry is the geometric loss, L auxiliary is the auxiliary loss, L normal is the normal alignment loss, L flatten is the flattening loss, L sparse is the sparsity loss, λ n 、λ f and λ s are all training weights.

[0154] Specifically, in this embodiment, the total loss function L total is defined as follows:

[0155]

[0156] where L total is the total loss function, L color is the color loss, L geometry is the geometric loss, L auxiliary is the auxiliary loss, L normal is the normal alignment loss, L flatten is the flattening loss, L sparse is the sparsity loss, λ n is the training weight of the normal alignment loss, λ f is the training weight of the flattening loss, λ s is the training weight of the sparsity loss.

[0157] The definitions of the color loss L color , the normal alignment loss L normal , and the flattening loss L flatten have been given above, and the definition of the sparsity loss L sparse is as follows:

[0158]

[0159] where α i represents the opacity value of each Gaussian sphere, and |α| is the number of total opacity values. By introducing the sparsity loss to drive the opacity values of the Gaussian spheres closer to a binary state (i.e., completely opaque or completely transparent), the sparsity loss encourages the Gaussian spheres to converge into a more coherent and thinner geometric representation, which helps the flattened Gaussians to form clear and well-defined surface segments and more closely follow the underlying scene structure.

[0160] The geometric loss L geometry handles the structural accuracy of the scene, and the auxiliary loss L auxiliary includes L sparse , which enhances the clarity of the surface representation by encouraging the sparsity of the opacity values. The total loss function L total combines these loss functions with the main color loss L color , and their weights are set experimentally to λ s = 0.001, λ n = 0.015, and λ f = 100.

[0161] In summary, the workflow of this embodiment is as follows: Receive multi-view input images and camera parameters, generate point cloud data through the structure-from-motion algorithm, initialize the position parameters of the Gaussian sphere through the point cloud data, and generate a depth map. The depth map is used to provide depth sorting information and the estimation of global normals. Subsequently, initialize the shape attributes, shading attributes, and opacity of the Gaussian sphere, obtain the scale matrix of the Gaussian sphere by decomposing the covariance matrix, thereby determining the shortest axis direction, calculate the flattening loss, then estimate the local normal according to the shortest axis direction and the viewing angle, and further calculate the normal alignment loss. After that, project and render the Gaussian sphere to obtain the final color of the rendered image, and then calculate the color loss. Finally, construct a total loss function by combining the flattening loss, normal alignment loss, sparsity loss, and color alignment loss, and perform backpropagation through the Adam optimizer to optimize the shape attributes, local normals, shading attributes, and opacity of the Gaussian sphere.

[0162] It can be recognized that in the embodiment of the present invention, by introducing the flattening loss, a flattening constraint is effectively imposed on the shape of the Gaussian sphere, enabling the Gaussian sphere to more precisely fit the object surface; estimating the local normal through the shortest axis of the flattened Gaussian sphere, a high-precision normal estimation result can be obtained without additional correction measures, thereby improving the shading quality and geometric representation accuracy; by improving the existing simplified shading model, the ability of the model to decompose the diffuse texture is effectively improved by introducing the original diffuse color term. At the same time, by combining the advantages of the metallic property and the occlusion term, high-metallic objects can be effectively processed, and the ability to render reflective surfaces is significantly improved.

[0163] Referring to Figure 2 , the embodiment of the present invention provides a shading model construction system based on planar Gaussian, including:

[0164] An input processing module, which is used to obtain a depth map and point cloud data according to the input image and camera parameters, and initialize the position parameters and depth sorting of the Gaussian sphere according to the depth map and point cloud data;

[0165] An initialization module, which is used to initialize the shape attributes, shading attributes, and opacity of the Gaussian sphere;

[0166] A flattening loss module, which is used to determine the shortest axis of the Gaussian sphere according to the shape attributes and calculate the flattening loss according to the shortest axis;

[0167] A normal estimation module, which is used to obtain the local normal of the Gaussian sphere according to the shortest axis and the viewing direction;

[0168] A normal alignment loss module, which is used to obtain the global normal according to the depth map, and then calculate the normal alignment loss according to the global normal and the local normal;

[0169] A sparsity loss module, which is used to calculate the sparsity loss according to the opacity;

[0170] A coloring module, which is used to calculate the color of the Gaussian sphere according to the coloring attribute, the viewing direction, and the local normal;

[0171] A rendering module, which is used to obtain the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane according to the position parameter, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere;

[0172] A color loss module, which is used to calculate the color loss according to the final color and the true color of the input image;

[0173] A total loss module, which is used to construct a final loss function according to the flattened loss, normal alignment loss, sparsity loss, and color loss, and update the shape attribute, local normal, opacity, and coloring attribute through the backpropagation algorithm.

[0174] Refer to Figure 3 , an embodiment of the present invention provides a computer device, including:

[0175] At least one processor;

[0176] At least one memory, which is used to store at least one program;

[0177] When the above at least one program is executed by the above at least one processor, the above at least one processor implements the above method for constructing a coloring model based on planar Gaussian.

[0178] The content in the above method embodiment is applicable to the device embodiment of the present invention. The functions specifically implemented by the device embodiment of the present invention are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those of the above method embodiment.

[0179] An embodiment of the present invention also provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the above method for constructing a coloring model based on planar Gaussian.

[0180] A computer-readable storage medium according to an embodiment of the present invention can execute a method for constructing a coloring model based on planar Gaussian provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0181] An embodiment of the present invention also discloses a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 the method for constructing a shading model based on planar Gaussian shown in

[0182] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the above-mentioned blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are foreseeable, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0183] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the above functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0184] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs, etc., which can store program codes of various kinds.

[0185] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0186] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fiber devices, and portable compact disc read-only memories (CDROMs). Additionally, the computer-readable medium can even be paper or other suitable media on which the above program can be printed, because the above program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or processing it in other suitable ways when necessary, and then storing it in a computer memory.

[0187] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0188] In the foregoing description of this specification, descriptions with reference to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0189] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0190] The above is a specific description of the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for constructing a shading model based on a plane Gaussian, characterized in that: include: Acquire a depth map and point cloud data according to an input image and camera parameters, and initialize position parameters and depth sorting of a Gaussian sphere according to the depth map and the point cloud data; Initializing the shape attributes, coloring attributes and opacity of the Gaussian sphere; Determine the shortest axis of the Gaussian sphere according to the shape attribute, and calculate the flattening loss according to the shortest axis; Obtaining a local normal of the Gaussian sphere according to the shortest axis and the viewing direction; Acquire a global normal according to the depth map, and then calculate a normal alignment loss according to the global normal and the local normal; Calculating a sparsity loss based on the opacity; Calculating the color of the Gaussian sphere according to the shading attribute, the viewing direction, and the local normal; Obtaining a final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane according to the position parameter, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere; Calculating a color loss based on the final color and a true color of the input image; A final loss function is constructed according to the flattening loss, the normal alignment loss, the sparsity loss and the color loss, and the shape attributes, the local normal, the opacity and the shading attributes are updated through a back-propagation algorithm.

2. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: The step of acquiring a depth map and point cloud data according to an input image and camera parameters, and initializing position parameters and depth sorting of a Gaussian sphere according to the depth map and the point cloud data includes: Acquire the point cloud data by using a structure-from-motion algorithm according to the input image and the camera parameters; Acquire the depth map according to the point cloud data; Initializing the position parameters according to the point cloud data; The depth sorting is initialized according to the depth map.

3. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: The shape attribute includes a covariance matrix, and determining the shortest axis of the Gaussian sphere according to the shape attribute, and calculating the flattening loss according to the shortest axis include: Decomposing the covariance matrix to obtain a scale matrix; determining the shortest axis according to the scale matrix; The flattening loss is calculated based on the shortest axis.

4. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: The obtaining the local normal of the Gaussian sphere according to the shortest axis and the observation direction includes: Determining whether the direction of the shortest axis is consistent with the observation direction; When the direction of the shortest axis is consistent with the observation direction, the direction of the shortest axis is used as the direction of the local normal to obtain the local normal.

5. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: The shading attributes include original diffuse color, diffuse attribute, metal attribute, occlusion attribute, specular attribute and roughness attribute. The color of the Gaussian sphere is calculated by the following formula: c(ω o )=γ(c d +c s +c r (oh o )) l d =c raw +g s ⊙(1-g m ) l s =0.04(1-g m )+g s ⊙g m c d =λ d ⊙g o ⊙L d (n) c s =λ s ⊙g o ⊙L s (oh o ,n,p) r=2(ω o ·n)n-ω o L d (n)=∫ Ω L(ω i )(oh i ·n)dω i L s (oh o ,n,ρ)=∫ Ω L(ω i )D(r,ρ)(ω i ·n)dω i Where c is the color of the Gaussian ball, ω o is the viewing direction, γ is the gamma adjustment mapping function for color rendering, c d is the diffuse color, c s is the specular color, c r is the indirect lighting simulated by spherical harmonics, ⊙ represents the element-by-element product, λ d and λ s is the preset intermediate term, g o is the occlusion attribute, L d is the direct diffuse reflected light, L s is the direct specular reflection light, n is the local normal, ρ is the roughness attribute, c raw is the original diffuse color, g s is the specular reflection property, g m is the metal property, r is the reflection direction, Ω is the upper half of the Gaussian sphere, ω i is the radiation direction of the incident light, L(ω i ) is a trainable ambient light map, and D(r, ρ) is the normal distribution function.

6. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: Obtaining a final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane according to the position parameter, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere, including: Projecting the Gaussian sphere onto a two-dimensional plane according to the position parameter and the observation direction; The final color is calculated according to the depth order, the opacity, and the color of the Gaussian sphere.

7. The method for constructing a shading model based on a plane Gaussian according to claim 1, characterized in that: The final loss function is: Among them, L total is the total loss function, L color is the color loss, L geometry is the geometric loss, L auxiliary is the auxiliary loss, L normal is the normal alignment loss, L flatten is the flattening loss, L sparse is the sparsity loss, λ n is the training weight of the normal alignment loss, λ f is the training weight of the flattened loss, λ s is the training weight of the sparsity loss.

8. A shading model construction system based on plane Gaussian, characterized in that: include: An input processing module, the input processing module is used to obtain a depth map and point cloud data according to an input image and camera parameters, and initialize position parameters and depth sorting of a Gaussian sphere according to the depth map and the point cloud data; An initialization module, the initialization module is used to initialize the shape attribute, coloring attribute and opacity of the Gaussian sphere; A flattening loss module, the flattening loss module is used to determine the shortest axis of the Gaussian sphere according to the shape attribute, and calculate the flattening loss according to the shortest axis; A normal estimation module, the normal estimation module is used to obtain a local normal of the Gaussian sphere according to the shortest axis and the observation direction; A normal alignment loss module, the normal alignment loss module is used to obtain a global normal according to the depth map, and then calculate a normal alignment loss according to the global normal and the local normal; A sparseness loss module, the sparseness loss module is used to calculate the sparseness loss according to the opacity; A coloring module, the coloring module being used to calculate the color of the Gaussian sphere according to the coloring attribute, the viewing direction and the local normal; A rendering module, the rendering module is used to obtain the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane according to the position parameter, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere; A color loss module, the color loss module is used to calculate the color loss according to the final color and the true color of the input image; A total loss module, wherein the total loss module is used to construct a final loss function according to the flattening loss, the normal alignment loss, the sparsity loss and the color loss, and to update the shape attributes, the local normal, the opacity and the shading attributes through a back-propagation algorithm.

9. A device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a planar Gaussian-based shading model construction method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute a planar Gaussian-based shading model construction method as described in any one of claims 1 to 7 when executed by the processor.

Images