A planar Gaussian-based shading model construction method, system, device and medium

Abstract

The application discloses a kind of based on plane Gauss's color model construction method, system, device and medium, method includes the position parameter of initialization Gauss ball, depth order, shape attribute, color attribute and opacity;According to shape attribute, the shortest axis of Gauss ball is determined, and then flat loss is calculated;According to the shortest axis and observation direction, the local normal of Gauss ball is obtained and then normal alignment loss is calculated;According to opacity, sparsity loss is calculated;Gauss ball is colored and color loss is calculated, according to flat loss, normal alignment loss, sparsity loss and color loss, construct final loss function, and shape attribute, local normal, opacity and color attribute are updated by back propagation algorithm.The application improves normal estimation accuracy by flattening Gauss ball, and then improves the color quality, by combining color model, high-quality rendering of a variety of materials is realized, and can be widely applied in computer vision technical field.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device and medium for constructing a shading model based on planar Gaussian. Background Technology

[0002] Significant progress has been made in the field of 3D computer vision in recent years, particularly in the reconstruction and visualization of complex 3D scenes. Neural Radiance Field (NeRF) has achieved highly realistic 3D view synthesis, but its optimization process is time-consuming and rendering is slow, limiting its use in real-time applications. To address these issues, 3D Gaussian Splatting (3DGS) has emerged. 3DGS combines efficient 3D Gaussian representation with tile-based splatting, making neural rendering more suitable for real-time scenes. However, despite some success, 3DGS falls short in handling surfaces with reflective and specular properties. Its general approach to appearance modeling fails to capture complex viewpoint-dependent variations, which are crucial for realistic rendering of materials with strong reflective properties.

[0003] Some advanced NeRF methods incorporate shading functions into implicit scene representations to efficiently model reflective surfaces. However, due to the inherent limitations of implicit representations, these methods are time-consuming to optimize and difficult to apply to real-time rendering scenes. Building upon the 3DGS framework, GaussianShader addresses this limitation by associating material properties with Gaussians, enabling scene modeling using simplified shading functions while improving optimization and rendering speed while maintaining high-quality rendering.

[0004] Despite the progress made, accurately estimating surface normals remains a significant challenge. This is due to the inherent discreteness of Gaussians; when surface normals fail to align precisely with the actual geometry of the scene, a series of problems arise, severely impacting rendering quality. Inaccurate normals lead to incorrect shading calculations, resulting in unrealistic lighting effects, especially for materials with strong specular reflections. This misalignment can cause artifacts such as blurred reflections and incorrect ray bounces, reducing the overall realism of the rendered scene. For reflective surfaces, ray interaction is highly dependent on the precise direction of the surface normals; even slight inaccuracies can significantly reduce the realism of the final output. Furthermore, the weak constraints of Gaussians and the limitations of simplified shading models can lead to suboptimal geometry and material decomposition results. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, one objective of this invention is to provide a method for constructing a shading model based on a planar Gaussian sphere, which improves the accuracy of normal estimation by flattening the Gaussian sphere, thereby enhancing shading quality and geometric representation. It also achieves robust material decomposition by combining shading models and improves rendering performance across various material properties.

[0007] Another objective of this invention is to provide a shading model construction system based on planar Gaussian.

[0008] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0009] In a first aspect, embodiments of the present invention provide a method for constructing a shading model based on planar Gaussian, comprising:

[0010] Depth map and point cloud data are obtained based on the input image and camera parameters. The position parameters and depth sorting of the Gaussian sphere are initialized based on the depth map and point cloud data.

[0011] Initialize the shape, shading, and opacity of the Gaussian sphere;

[0012] The shortest axis of the Gaussian sphere is determined based on the shape properties, and the flattening loss is calculated based on the shortest axis.

[0013] The local normal of the Gaussian sphere is obtained based on the shortest axis and the viewing direction;

[0014] The global normal is obtained from the depth map, and then the normal alignment loss is calculated based on the global normal and the local normal.

[0015] The sparsity loss is calculated based on the opacity.

[0016] The color of the Gaussian sphere is calculated based on the shading attribute, the viewing direction, and the local normal.

[0017] The final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane is obtained based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere.

[0018] Calculate the color loss based on the final color and the true color of the input image;

[0019] The final loss function is constructed based on the flattening loss, the normal alignment loss, the sparsity loss, and the color loss. The shape attribute, the local normal, the opacity, and the shading attribute are then updated using the backpropagation algorithm.

[0020] Further, the step of obtaining depth maps and point cloud data based on the input image and camera parameters, and initializing the position parameters and depth sorting of the Gaussian sphere based on the depth maps and point cloud data, includes:

[0021] Based on the input image and the camera parameters, the point cloud data is obtained using a motion reconstruction structure algorithm.

[0022] The depth map is obtained based on the point cloud data;

[0023] Initialize the position parameters based on the point cloud data;

[0024] The depth sorting is initialized based on the depth map.

[0025] Furthermore, the shape attribute includes a covariance matrix, and the step of determining the shortest axis of the Gaussian sphere based on the shape attribute and calculating the flattening loss based on the shortest axis includes:

[0026] Decompose the covariance matrix to obtain the scaling matrix;

[0027] The shortest axis is determined based on the scale matrix;

[0028] The flattening loss is calculated based on the shortest axis.

[0029] Furthermore, obtaining the local normal of the Gaussian sphere based on the shortest axis and the viewing direction includes:

[0030] Determine whether the direction of the shortest axis is consistent with the observation direction;

[0031] When the direction of the shortest axis is consistent with the observation direction, the direction of the shortest axis is taken as the direction of the local normal, and the local normal is obtained.

[0032] Furthermore, the shading attributes include the original diffuse color, diffuse properties, metallic properties, occlusion properties, specular properties, and roughness properties, and the color of the Gaussian sphere is calculated using the following formula:

[0033] c(ω o )=γ(c d +c s +c r (ω o ))

[0034] λ d =c raw +g s ☉(1-g m )

[0035] λ s =0.04(1-g m )+gs ☉g m

[0036] c d =λ d ☉g o ☉L d (n)

[0037] c s =λ s ☉g o ☉L s (ω o ,n,ρ)

[0038] r=2(ω o ·n)n-ω o

[0039] L d (n)=∫ Ω L(ω i (ω) i ·n)dω i

[0040] L s (ω o ,n,ρ)=∫ Ω L(ω i )D(r,ρ)(ω i ·n)dω i

[0041] Where c is the color of the Gaussian sphere, ωo is the viewing direction, γ is the gamma adjustment mapping function used for color rendering, and c d For diffuse color, c s For specular reflection color, c r Let ⊙ represent indirect illumination simulated by spherical harmonics, where ⊙ denotes the element-wise product, and λ represents the indirect illumination. d and λ s For the preset intermediate term, g o For the occlusion attribute, L d For direct diffuse reflection light, L s For direct specular reflection, n is the local normal, ρ is the roughness attribute, and c is the local normal. raw For the original diffuse color, g s For the specular reflection property, g m The metallic property is r, the reflection direction is r, Ω is the upper half of the Gaussian sphere, and ω is ω. i Let L(ω) be the direction of incident light radiation. i ) is a trainable ambient ray map, and D(r,ρ) is the normal distribution function.

[0042] Further, obtaining the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere includes:

[0043] The Gaussian sphere is projected onto a two-dimensional plane according to the position parameters and the observation direction.

[0044] The final color is calculated based on the depth sort, the opacity, and the color of the Gaussian sphere.

[0045] Furthermore, the final loss function is:

[0046]

[0047] Among them, L total Let L be the total loss function. color For color loss, L geometry For geometric loss, L auxiliary To mitigate losses, L normal For normal alignment loss, L flatten For flattening loss, L sparse For sparsity loss, λ n λ is the training weight for the normal alignment loss. f λ represents the training weights of the flattening loss. s The training weights are those for the sparse loss.

[0048] Secondly, an embodiment of the present invention provides a shading model construction system based on planar Gaussian, comprising:

[0049] An input processing module is used to obtain depth maps and point cloud data based on input images and camera parameters, and to initialize the position parameters and depth sorting of a Gaussian sphere based on the depth maps and point cloud data.

[0050] An initialization module is used to initialize the shape attributes, shading attributes, and opacity of the Gaussian sphere.

[0051] A flattening loss module is used to determine the shortest axis of the Gaussian sphere based on the shape attribute, and to calculate the flattening loss based on the shortest axis.

[0052] A normal estimation module is used to obtain the local normal of the Gaussian sphere based on the shortest axis and the viewing direction.

[0053] A normal alignment loss module is used to obtain global normals based on the depth map, and then calculate normal alignment loss based on the global normals and the local normals.

[0054] A sparsity loss module, which is used to calculate sparsity loss based on the opacity;

[0055] A coloring module is used to calculate the color of the Gaussian sphere based on the coloring attributes, the viewing direction, and the local normal.

[0056] A rendering module is used to obtain the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere.

[0057] A color loss module is used to calculate color loss based on the final color and the true color of the input image;

[0058] The total loss module is used to construct a final loss function based on the flattening loss, the normal alignment loss, the sparsity loss, and the color loss, and to update the shape attribute, the local normal, the opacity, and the shading attribute through a backpropagation algorithm.

[0059] Thirdly, embodiments of the present invention provide an apparatus, comprising:

[0060] At least one processor;

[0061] At least one memory for storing at least one program;

[0062] When the at least one program is executed by the at least one processor, the at least one processor implements a planar Gaussian-based shading model construction method as described above.

[0063] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method for constructing a plane Gaussian-based shading model.

[0064] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0065] This invention effectively imposes a flattening constraint on the shape of a Gaussian sphere by introducing a flattening loss, enabling the Gaussian sphere to fit the object surface more accurately. Local normals are estimated using the shortest axis of the flattened Gaussian sphere, achieving high-precision normal estimation results without additional correction measures, thereby improving shading quality and geometric representation accuracy. Furthermore, by improving existing simplified shading models and introducing a raw diffuse color term, the model's ability to decompose diffuse maps is effectively enhanced. Combined with the advantages of metallic properties and occlusion terms, it can effectively handle highly metallic objects, significantly improving the ability to render reflective surfaces. Attached Figure Description

[0066] Figure 1 A step diagram illustrating a method for constructing a shading model based on planar Gaussian shading, provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a plane Gaussian-based shading model construction system provided in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0069] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0070] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0071] To facilitate understanding, the following explanations of any terms and technical jargon that may appear in this manual are provided:

[0072] 3D Gaussian Splashing (3DGS): 3DGS is a modeling technique that uses 3D Gaussian spheres to represent scenes, each of which includes a set of attribute representations. These attributes include position p, opacity α, a covariance matrix Σ representing shape attributes, and spherical harmonics (SH) coefficients (i.e., shading attributes) representing color. The covariance matrix Σ is further decomposed into a rotation matrix R and a scale matrix S, denoted as Σ = RSSTRT. In 3DGS, rendering begins by projecting the Gaussian spheres through the camera's viewpoint into a two-dimensional normalized device coordinate (NDC) space. These Gaussian spheres are then processed by a fast and differentiable tile-based rasterizer and sorted according to their projection depth. This sorting, combined with alpha blending of the 2D Gaussians based on their opacity values, calculates the final color for each pixel. The difference between this final color and the real image can be used to construct a loss function, thereby optimizing the properties of the Gaussian spheres through backpropagation.

[0073] Figure 1 A step diagram illustrating a method for constructing a shading model based on planar Gaussian shading provided in an embodiment of the present invention is shown below. Figure 1 This invention provides a method for constructing a shading model based on planar Gaussian, comprising:

[0074] Obtain depth map and point cloud data based on the input image and camera parameters, and initialize the position parameters and depth sorting of the Gaussian sphere based on the depth map and point cloud data;

[0075] Initialize the shape, shading, and opacity of the Gaussian sphere;

[0076] Determine the shortest axis of the Gaussian sphere based on its shape properties, and calculate the flattening loss based on the shortest axis.

[0077] The local normal of the Gaussian sphere is obtained based on the shortest axis and the direction of observation;

[0078] Obtain global normals from the depth map, and then calculate normal alignment loss based on global and local normals;

[0079] Calculate the sparsity loss based on the opacity;

[0080] The color of the Gaussian sphere is calculated based on the shading properties, the viewing direction, and the local normal.

[0081] The final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane is obtained based on the position parameters, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere.

[0082] Calculate the color loss based on the final color and the true color of the input image;

[0083] The final loss function is constructed based on the flattening loss, normal alignment loss, sparsity loss, and color loss. The shape attributes, local normals, opacity, and shading attributes are then updated using the backpropagation algorithm.

[0084] In some alternative embodiments, depth map and point cloud data are obtained based on the input image and camera parameters, and the position parameters and depth sorting of the Gaussian sphere are initialized based on the depth map and point cloud data, including:

[0085] Based on the input image and camera parameters, point cloud data is obtained using the structure-of-motion reconstruction algorithm.

[0086] Obtain a depth map from point cloud data;

[0087] Initialize position parameters based on point cloud data;

[0088] Initialize depth sorting based on the depth map.

[0089] Specifically, in this implementation, the input image is a multi-view image of the object to be modeled. The sparse point cloud is reconstructed from the multi-view image and camera parameters using the Structure-from-Motion (SfM) algorithm to obtain point cloud data. The depth value of each pixel is obtained by projecting the point cloud onto the camera plane, thereby constructing a depth map. This map is used to provide depth sorting of the Gaussian sphere and estimate the global normal, thereby determining the occlusion relationship of the Gaussian sphere and facilitating the subsequent calculation of the final color of the Gaussian sphere after projection onto the plane.

[0090] Furthermore, point cloud data is also used to initialize the position parameters of the Gaussian sphere. In this embodiment of the invention, each point cloud point is initialized as a Gaussian sphere, and the position of the point cloud point determines the three-dimensional position of the Gaussian sphere. Other initial parameters of the Gaussian sphere, such as shape attributes, shading attributes, and opacity, are initialized to fixed values ​​and will be gradually optimized during subsequent backpropagation.

[0091] In some alternative embodiments, the shape attribute includes a covariance matrix, the shortest axis of the Gaussian sphere is determined based on the shape attribute, and a flattening loss is calculated based on the shortest axis, including:

[0092] Decompose the covariance matrix to obtain the scale matrix;

[0093] Determine the shortest axis based on the scaling matrix;

[0094] Calculate the flattening loss based on the shortest axis.

[0095] In some alternative embodiments, the local normal to the Gaussian sphere is obtained based on the shortest axis and the viewing direction, including:

[0096] Determine whether the direction of the shortest axis is consistent with the direction of observation;

[0097] When the direction of the shortest axis is consistent with the observation direction, the direction of the shortest axis is taken as the direction of the local normal, and the local normal is obtained.

[0098] Specifically, research by 3DGS indicates that flattened Gaussian spheres can more accurately capture object surfaces and align with underlying geometry. The accurate shading of Gaussian spheres depends on accurate normal estimation. Based on these insights, this embodiment integrates flattened Gaussian spheres into the rendering pipeline. By introducing a flattening loss, specific constraints are imposed on the shape of the Gaussian spheres, thereby achieving more accurate normal estimation on these flattened Gaussian spheres and improving the shading effect.

[0099] In this embodiment, the shape properties of the Gaussian sphere can be described by the covariance matrix Σ, which determines the distribution shape of the Gaussian sphere in three-dimensional space. Through eigenvalue decomposition, the covariance matrix can be decomposed into a rotation matrix R and a scaling matrix S, represented as Σ = RSS. T R T Among them, the diagonal element g of S scale =[s1,s2,s3] represents the scale of the Gaussian sphere along the three principal axes, according to g scale The minimum eigenvalue can determine the shortest axis, and then the direction corresponding to the shortest axis can be determined based on the eigenvector of the minimum eigenvalue. Therefore, in this embodiment, the flattening loss L is used. flatten Defined as:

[0100]

[0101] Where, min axis (g scale ) represents g scale The axis corresponding to the smallest eigenvalue, i.e. the shortest axis, is used as the target for flattening. By applying this loss, the Gaussian sphere is encouraged to compress along its shortest axis, thereby enhancing its ability to match the underlying surface of the scene.

[0102] After the Gaussian sphere is flattened, this embodiment can directly define its normal vector through the shortest axis of the Gaussian sphere. This is a significant improvement; compared to other normal estimation methods, this embodiment achieves more accurate shading results without additional correction measures (such as normal offset). The normal of each Gaussian sphere is selected based on its shortest axis, which is naturally aligned with the surface geometry. Considering that the shortest axis may have two directions, this embodiment determines the correct normal direction by aligning this axis with the view direction, the formal definition of which is as follows:

[0103]

[0104] Where n represents the normal to the Gaussian sphere, v represents the positive direction of the shortest axis of the Gaussian sphere, and ω oFor the viewing angle, by selecting a normal direction consistent with the viewing direction, the consistency and accuracy of shading calculations on the reflective surface in the scene are ensured. This method not only simplifies the normal estimation process but also significantly improves the overall rendering quality by maintaining the consistency of light interaction on the reflective surface. To further address the problem that the unstructured nature of Gaussians makes it difficult to accurately define the surface normals of the scene, this embodiment defines the normal alignment loss L using the following formula. normal :

[0105]

[0106] in, This represents the normal rendered from the Gaussian sphere. This represents the normal obtained from the rendered unbiased depth map through a Sobel-like operation. This alignment ensures that the local Gaussian normals are consistent with the global scene geometry, further improving the accuracy of global normals when dealing with flattened Gaussian spheres, thereby enhancing overall geometric precision and improving shading effects.

[0107] In some alternative embodiments, the shading properties include the original diffuse color, diffuse properties, metallic properties, occlusion properties, specular properties, and roughness properties, and the color of the Gaussian sphere is calculated using the following formula:

[0108] c(ω o )=γ(c d +c s +c r (ω o ))

[0109] λ d =c raw +g s ☉(1-g m )

[0110] λ s =0.04(1-g m )+g s ☉g m

[0111] c d =λ d ☉g o ☉L d (n)

[0112] c s =λ s ☉g o ☉L s (ω o ,n,ρ)

[0113] r=2(ω o ·n)n-ωo

[0114] L d (n)=∫ Ω L(ω i (ω) i ·n)dω i

[0115] L s (ω o ,n,ρ)=∫ Ω L(ω i )D(r,ρ)(ω i ·n)dω i

[0116] Where c is the color of the Gaussian sphere, ω o For the viewing direction, γ is the gamma adjustment mapping function used for color rendering, and c d For diffuse color, c s For specular reflection color, c r Let ⊙ represent indirect illumination simulated by spherical harmonics, where ⊙ denotes the element-wise product, and λ represents the indirect illumination. d and λ s For the preset intermediate term, g o For occlusion properties, L d For direct diffuse reflection light, L s For direct specular reflection, n is the local normal, ρ is the roughness property, and c is the surface roughness. raw For the original diffuse color, g s For specular reflection properties, g m It has metallic properties, r is the direction of reflection, Ω is the upper half of the Gaussian sphere, and ω i Let L(ω) be the direction of incident light radiation. i ) is a trainable ambient ray map, and D(r,ρ) is the normal distribution function.

[0117] Specifically, to enhance the rendering quality of reflective surfaces and facilitate scene relighting, the traditional Gaussian Shader introduces a simplified Bidirectional Reflectance Distribution Function (BRDF) combined with a learnable ambient map to shade 3D Gaussian spheres. Specifically, each Gaussian sphere is augmented with shading attributes including diffuse, roughness, specular, and a residual color term to represent indirect lighting that the simplified shading function cannot capture. Given the viewing direction ω... o The color of a Gaussian sphere can be calculated using the following formula:

[0118] c(ω o )=γ(c d +cs +c r (ω o ))

[0119] Where c is the color of the Gaussian sphere, ω o For the viewing direction, γ is the gamma adjustment mapping function used for color rendering, and c d For diffuse color, c s For specular reflection color, c r This represents indirect lighting simulated using spherical harmonic functions.

[0120] In the Gaussian Shader shading model, c d and c r The expanded form is as follows:

[0121] c d =c raw +(1-g s )☉L d (n)

[0122] c s =g s ☉L s (ω o ,n,ρ)

[0123] Among them, c raw This is the original diffuse color, which is directly added to the final diffuse color. s It represents the specular reflection property, ⊙ represents the element-wise product, n is the normal, and L is the reflection property. d (n) represents the direct diffuse light calculated using the ambient light map, ρ is the roughness attribute, and L... s (ω o ,n,ρ) is the direct specular reflection light calculated using the same ambient light map.

[0124] Direct diffuse reflection component L d (n) and specular reflection light component L s (ω o The formula for calculating (n, ρ) is as follows:

[0125] r=2(ω o ·n)n-ω o

[0126] L d (n)=∫ Ω L(ω i (ω) i ·n)dω i

[0127] L s (ω o ,n,ρ)=∫Ω L(ω i )D(r,ρ)(ω i ·n)dω i

[0128] Where r is the direction of reflection, Ω is the upper half of the Gaussian sphere, and ω i Let L(ω) be the direction of incident light radiation. i ) is a trainable ambient occlusion map, represented by a 6×64×64 cubic map, and D(r,ρ) is the SpecularGGX normal distribution function, used to define the distribution range of specular reflections.

[0129] To achieve efficient shading, multi-level mipmaps are pre-computed for different reflection directions and roughness levels. During rendering, L is obtained from these pre-computed mipmaps through interpolation. s The result ensures a balance between computational efficiency and rendering accuracy.

[0130] In contrast, NvDiffRec's shading model will... d and c r The model is in the following form:

[0131] λ d =g d ⊙(1-g m )

[0132] λ s =0.04(1-g m )+g d ⊙g m

[0133] c d =λ d ☉g o ☉L a (n)

[0134] c s =λ s ⊙g o ⊙L s (ω o ,n,ρ)

[0135] Where g o It's an occlusion property, g d It is a diffuse reflection property, g m It has metallic properties.

[0136] The NvDiffRec shading model includes an explicit occlusion term g. o and the clearly defined metallic term g m The GaussianShader model uses a special craw The NvDiffRec shading model, while containing a primary diffuse color term, does not include occlusion or metallicity. After integrating the NvDiffRec shading model into the GaussianShader, it was observed that the model performs well on highly metallic objects but performs worse than the GaussianShader model on low-metallic objects. Analysis of these results suggests that the explicit handling of metallicity and occlusion in the NvDiffRec model enables it to effectively model highly metallic materials, but the lack of a primary diffuse color term limits its performance on low-metallic materials. Conversely, the GaussianShader shading model, due to its inclusion of a primary diffuse color term, performs well on low-metallic materials but poorly on highly metallic materials.

[0137] To address these findings, this embodiment proposes a novel combined shading model that integrates the advantages of both methods. Specifically, based on NvDiffRec, it adds c to its model. raw This term represents the original diffuse color. Since the newly added term already models the diffuse map, the original g... d Item renamed to g s This allows for better control of the specular reflection component, similar to the Gaussian Shader model. The improved formula is as follows:

[0138] λ d =c raw +g s ⊙(1-g m )

[0139] λ s =0.04(1-g m )+g s ⊙g m

[0140] Compared to GaussianShader's shading model, the new model introduces metallic and occlusion terms, making it more effective at handling highly metallic objects. And compared to the NvDiffRec model, c raw This improvement enhances the decomposition capabilities of diffuse maps, resulting in superior performance on low-metallic materials. This combined model effectively integrates the advantages of both shading models while mitigating their respective limitations.

[0141] In some alternative embodiments, the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane is obtained based on position parameters, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere, including:

[0142] The Gaussian sphere is projected onto a two-dimensional plane based on the position parameters and the viewing direction.

[0143] The final color is calculated based on depth sorting, opacity, and the color of the Gaussian sphere.

[0144] Specifically, in this embodiment, the position parameters of the Gaussian spheres are projected onto a two-dimensional plane through the viewing direction to generate their two-dimensional coordinates in the Normalized Device Coordinate (NDC) space. These Gaussian spheres are then processed by a fast and differentiable tile-based rasterizer and sorted according to their projection depth. Based on this sorting, the two-dimensional Gaussian spheres are alpha-blended according to their opacity values ​​to calculate the final color of each pixel, using the following formula:

[0145]

[0146] Among them, c i Let be the color of each Gaussian sphere calculated from the shading equation above, and let be the transparency and α of each Gaussian sphere. i The opacity of the Gaussian sphere is determined by balancing its power (i.e., its exponential decay function from the mean), and the degree to which the current Gaussian sphere is occluded by the preceding Gaussian sphere is represented by the cumulative opacity.

[0147] After the color of each pixel has accumulated in the rendered image, a color loss is used to minimize the difference between the rendered image and the input real image. The color loss L color The definition is as follows:

[0148]

[0149] Among them, G gt The color of the input real image.

[0150] This loss function optimizes the Gaussian parameters in the scene through backpropagation, thereby gradually improving the scene representation to achieve more accurate rendering results.

[0151] In some alternative embodiments, the final loss function L total for:

[0152]

[0153] Among them, L total Let L be the total loss function. color For color loss, L geometry For geometric loss, L auxiliary To mitigate losses, L normal For normal alignment loss, L flatten For flattening loss, L sparse For sparsity loss, λ n , λ f and λ s All of these are training weights.

[0154] Specifically, in this embodiment, the total loss function L total The definition is as follows:

[0155]

[0156] Among them, L total Let L be the total loss function. color For color loss, L geometry For geometric loss, L auxiliary To mitigate losses, L normal For normal alignment loss, L flatten For flattening loss, L sparse For sparsity loss, λ n λ is the training weight for the normal alignment loss. f For the training weights of the flattening loss, λ s The training weights are for sparse loss.

[0157] Color loss L color Normal alignment loss L normal Flattening loss L flatten The definition of sparsity loss L has been given above, and the definition of sparsity loss L is given above. sparse The definition is as follows:

[0158]

[0159] Where, α i Let |α| represent the opacity value of each Gaussian sphere, and |α| be the total number of opacity values. By introducing a sparsity loss to push the opacity values ​​of the Gaussian spheres closer to a binary state (i.e., completely opaque or completely transparent), the sparsity loss encourages the Gaussian spheres to converge into a more coherent and thinner geometric representation, which helps to flatten the Gaussian spheres into clear and well-defined surface fragments that more closely follow the underlying scene structure.

[0160] Geometric loss L geometry Structural accuracy of the processing scene, auxiliary loss L auxiliary Including L sparse It enhances the clarity of surface representation by encouraging sparsity of opacity values. The total loss function L... total These loss functions are compared with the main color loss L color Combined, its weights are experimentally set to λ. s =0.001, λ n =0.015 and λ f =100.

[0161] In summary, the workflow of this embodiment is as follows: receiving multi-view input images and camera parameters, generating point cloud data through the structure-of-motion (SOG) algorithm, initializing the position parameters of the Gaussian sphere using the point cloud data, and generating a depth map. The depth map is used to provide depth sorting information and estimation of global normals. Subsequently, initializing the shape attributes, shading attributes, and opacity of the Gaussian sphere, obtaining the scale matrix of the Gaussian sphere by decomposing the covariance matrix to determine the shortest axis direction, calculating the flattening loss, estimating local normals based on the shortest axis direction and viewing angle, and then calculating the normal alignment loss. Afterward, projecting and rendering the Gaussian sphere to obtain the final color of the rendered image, and then calculating the color loss. Finally, combining the flattening loss, normal alignment loss, sparsity loss, and color alignment loss to construct the total loss function, and performing backpropagation through the Adam optimizer to optimize the shape attributes, local normals, shading attributes, and opacity of the Gaussian sphere.

[0162] It can be recognized that, by introducing a flattening loss, the embodiments of the present invention effectively impose a flattening constraint on the shape of the Gaussian sphere, enabling the Gaussian sphere to fit the object surface more accurately; by estimating the local normals through the shortest axis of the flattened Gaussian sphere, high-precision normal estimation results can be obtained without additional correction measures, thereby improving the shading quality and geometric representation accuracy; by improving the existing simplified shading model, the introduction of the original diffuse color term effectively improves the model's ability to decompose diffuse maps, and by combining the advantages of metallic properties and occlusion terms, it can effectively handle highly metallic objects and significantly improve the ability to render reflective surfaces.

[0163] Reference Figure 2 This invention provides a shading model construction system based on planar Gaussian, comprising:

[0164] The input processing module is used to obtain depth map and point cloud data based on the input image and camera parameters, and initialize the position parameters and depth sorting of the Gaussian sphere based on the depth map and point cloud data.

[0165] The initialization module is used to initialize the shape properties, shading properties, and opacity of the Gaussian sphere.

[0166] The flattening loss module is used to determine the shortest axis of the Gaussian sphere based on its shape properties, and then calculate the flattening loss based on the shortest axis.

[0167] The normal estimation module is used to obtain the local normals of the Gaussian sphere based on the shortest axis and the viewing direction.

[0168] The normal alignment loss module is used to obtain global normals from the depth map, and then calculate the normal alignment loss based on the global normals and local normals.

[0169] The sparsity loss module is used to calculate the sparsity loss based on the opacity.

[0170] The shading module is used to calculate the color of the Gaussian sphere based on shading attributes, viewing direction, and local normals.

[0171] The rendering module is used to obtain the final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane based on position parameters, depth sorting, opacity, viewing direction, and the color of the Gaussian sphere.

[0172] The color loss module is used to calculate the color loss based on the final color and the true color of the input image.

[0173] The total loss module is used to construct the final loss function based on the flattening loss, normal alignment loss, sparsity loss, and color loss, and to update the shape attributes, local normals, opacity, and shading attributes through the backpropagation algorithm.

[0174] Reference Figure 3 This invention provides a computer device, including:

[0175] At least one processor;

[0176] At least one memory for storing at least one program;

[0177] When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned method for constructing a plane Gaussian-based shading model.

[0178] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0179] This invention also provides a computer-readable storage medium storing a processor-executable program that, when executed by a processor, performs the aforementioned method for constructing a plane Gaussian-based shading model.

[0180] This invention provides a computer-readable storage medium that can execute a method for constructing a plane Gaussian-based shading model provided in the method embodiments of this invention. It can execute any combination of the implementation steps of the method embodiments and has the corresponding functions and beneficial effects of the method.

[0181] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method for constructing a shading model based on planar Gaussians is shown.

[0182] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0183] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0184] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0186] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0187] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0188] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0189] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0190] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for constructing a shading model based on planar Gaussian, characterized in that, include: Depth map and point cloud data are obtained based on the input image and camera parameters. The position parameters and depth sorting of the Gaussian sphere are initialized based on the depth map and point cloud data. Initialize the shape, shading, and opacity of the Gaussian sphere; The shortest axis of the Gaussian sphere is determined based on the shape properties, and the flattening loss is calculated based on the shortest axis. The local normal of the Gaussian sphere is obtained based on the shortest axis and the viewing direction; The global normal is obtained from the depth map, and then the normal alignment loss is calculated based on the global normal and the local normal. The sparsity loss is calculated based on the opacity. The color of the Gaussian sphere is calculated based on the shading attribute, the viewing direction, and the local normal. The final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane is obtained based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere. Calculate the color loss based on the final color and the true color of the input image; The final loss function is constructed based on the flattening loss, the normal alignment loss, the sparsity loss, and the color loss. The shape attribute, the local normal, the opacity, and the shading attribute are then updated using the backpropagation algorithm. The final loss function is: Among them, L total Let L be the total loss function. color For color loss, L geometry For geometric loss, L auxiliary To mitigate losses, L normal For normal alignment loss, L flatten For flattening loss, L sparse For sparsity loss, λ n λ is the training weight for the normal alignment loss. f λ represents the training weights of the flattening loss. s For the training weights of the sparse loss, g scale The diagonal elements of the scaling matrix of the Gaussian sphere are... For g scale The axis corresponding to the smallest eigenvalue.

2. The method for constructing a shading model based on planar Gaussian as described in claim 1, characterized in that, The step of obtaining depth maps and point cloud data based on input images and camera parameters, and initializing the position parameters and depth sorting of a Gaussian sphere based on the depth maps and point cloud data, includes: Based on the input image and the camera parameters, the point cloud data is obtained using a motion reconstruction structure algorithm. The depth map is obtained based on the point cloud data; Initialize the position parameters based on the point cloud data; The depth sorting is initialized based on the depth map.

3. The method for constructing a shading model based on planar Gaussian as described in claim 1, characterized in that, The shape attribute includes a covariance matrix. Determining the shortest axis of the Gaussian sphere based on the shape attribute and calculating the flattening loss based on the shortest axis includes: Decompose the covariance matrix to obtain the scaling matrix; The shortest axis is determined based on the scale matrix; The flattening loss is calculated based on the shortest axis.

4. The method for constructing a shading model based on planar Gaussian as described in claim 1, characterized in that, The process of obtaining the local normal of the Gaussian sphere based on the shortest axis and the viewing direction includes: Determine whether the direction of the shortest axis is consistent with the observation direction; When the direction of the shortest axis is consistent with the observation direction, the direction of the shortest axis is taken as the direction of the local normal, and the local normal is obtained.

5. The method for constructing a shading model based on planar Gaussian as described in claim 1, characterized in that, The shading attributes include the original diffuse color, diffuse properties, metallic properties, occlusion properties, specular properties, and roughness properties. The color of the Gaussian sphere is calculated using the following formula: c(ω o )=γ(c d +c s +c r (oh o )) l d =c raw +g s ☉(1-g m ) l s =0.04(1-g m )+g s ☉g m c d =λ d ☉g o ☉L d (n) c s =λ s ☉g o ☉L s (oh o ,n,p) r=2(ω o ·n)n-ω o L d (n)=∫ Ω L(ω i )(oh i ·n)dω i L s (oh o ,n,ρ)=∫ Ω L(ω i )D(r,ρ)(ω i ·n)dω i Where c is the color of the Gaussian sphere, ω o Where γ is the viewing direction, c is the gamma adjustment mapping function used for color rendering, and γ is the viewing direction. d For diffuse color, c s For specular reflection color, c r Let ⊙ represent indirect illumination simulated by spherical harmonics, where ⊙ denotes the element-wise product, and λ represents the indirect illumination. d and λ s For the preset intermediate term, g o For the occlusion attribute, L d For direct diffuse reflection light, L s For direct specular reflection, n is the local normal, ρ is the roughness attribute, and c is the local normal. raw For the original diffuse color, g s For the specular reflection property, g m The metallic property is r, the reflection direction is r, Ω is the upper half of the Gaussian sphere, and ω is ω. i Let L(ω) be the direction of incident light radiation. i ) is a trainable ambient ray map, and D(r,ρ) is the normal distribution function.

6. The method for constructing a shading model based on planar Gaussian as described in claim 1, characterized in that, The final color of each pixel after the Gaussian sphere is projected onto the two-dimensional plane is obtained based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere, including: The Gaussian sphere is projected onto a two-dimensional plane according to the position parameters and the observation direction. The final color is calculated based on the depth sort, the opacity, and the color of the Gaussian sphere.

7. A shading model construction system based on planar Gaussian, characterized in that, include: An input processing module is used to obtain depth maps and point cloud data based on input images and camera parameters, and to initialize the position parameters and depth sorting of a Gaussian sphere based on the depth maps and point cloud data. An initialization module is used to initialize the shape attributes, shading attributes, and opacity of the Gaussian sphere. A flattening loss module is used to determine the shortest axis of the Gaussian sphere based on the shape attribute, and to calculate the flattening loss based on the shortest axis. A normal estimation module is used to obtain the local normal of the Gaussian sphere based on the shortest axis and the viewing direction. A normal alignment loss module is used to obtain global normals based on the depth map, and then calculate normal alignment loss based on the global normals and the local normals. A sparsity loss module, which is used to calculate sparsity loss based on the opacity; A coloring module is used to calculate the color of the Gaussian sphere based on the coloring attributes, the viewing direction, and the local normal. A rendering module is used to obtain the final color of each pixel after the Gaussian sphere is projected onto a two-dimensional plane based on the position parameters, the depth sorting, the opacity, the viewing direction, and the color of the Gaussian sphere. A color loss module is used to calculate color loss based on the final color and the true color of the input image; The total loss module is used to construct a final loss function based on the flattening loss, the normal alignment loss, the sparsity loss, and the color loss, and to update the shape attribute, the local normal, the opacity, and the shading attribute through the backpropagation algorithm. The final loss function is: Among them, L total Let L be the total loss function. color For color loss, L geometry For geometric loss, L auxiliary To mitigate losses, L normal For normal alignment loss, L flatten For flattening loss, L sparse For sparsity loss, λ n λ is the training weight for the normal alignment loss. f λ represents the training weights of the flattening loss. s For the training weights of the sparse loss, g scale The diagonal elements of the scaling matrix of the Gaussian sphere are... For g scale The axis corresponding to the smallest eigenvalue.

8. An apparatus, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a planar Gaussian-based shading model construction method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform a planar Gaussian-based shading model construction method as described in any one of claims 1-6.

Images