Field scene illumination decomposition method based on 3D Gaussian splashing

Abstract

The invention discloses a field scene illumination decomposition method based on 3D Gaussian splashing, and relates to the technical field of 3D Gaussian splashing. The method comprises the following steps: acquiring a data set containing an RGB image and an HDR image of a real field scene; the method comprises the following steps: carrying out geometric reconstruction on a scene by using 3DGaussian Splitting based on RGB images in a data set: endowing each Gaussian distribution generated by sampling the RGB images in the data set by using the traditional 3DGaussian Splitting with a related parameter, introducing normal estimation, and adding a normal parameter for each Gaussian distribution; designing a multi-layer perceptron; using an HDR image in the data set as ambient light, and predicting a coefficient of a spherical harmonic function of the ambient light by using a multi-layer perceptron; according to a coefficient of a spherical harmonic function of ambient light, an unshaded illumination model and a shaded illumination model are established, and then the shaded illumination model is optimized to complete effective separation of illumination and shadow information from a scene. According to the invention, the calculation cost of the illumination decomposition method based on 3D Gaussian splashing is reduced, and the illumination decomposition precision is improved at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D Gaussian splashing, and in particular to a field scene illumination decomposition method based on 3D Gaussian splashing. Background Art

[0002] 3D reconstruction has long been a core task in fields such as computer graphics and autonomous driving, pursuing high-precision scene restoration. 3D Gaussian Slatting is a revolutionary technique that achieves real-time, high-quality 3D reconstruction through explicit Gaussian distribution modeling. Its principle is to decompose a scene into multiple Gaussian "particles," each carrying attributes such as position, covariance (controlling shape and orientation), color, and opacity. These parameters are optimized to achieve detailed capture and realistic rendering. However, in this process, lighting information is coupled into the Gaussian distribution's properties, such as color and opacity, resulting in an ineffective separation of the resulting surface properties from the scene's ambient lighting. For example, shadow and highlight information is fixed into the Gaussian geometry representation, resulting in material distortions such as oversmoothing of reflective areas or shadow artifacts. This problem is particularly prominent in outdoor scenes, where the complex interaction of ambient light can significantly distort geometric details, such as the loss of fine structure in vegetation in shadows. This illumination decomposition challenge not only exists within the Gaussian modeling framework but also more generally challenges mainstream 3D reconstruction paradigms, such as the Neural Radiance Fields (Nerf) illumination decomposition method and alternatives based on spherical harmonics (e.g., NeRF-OSR, NeRF for Outdoor Scene Relighting). The former sacrifices real-time performance by implicitly representing continuous fields, while the latter still does not explicitly model the radiance transfer function (RTF). The RTF describes the transmission of illumination from ambient light to a specific point on the surface of an object. It accounts for complex interactions such as reflection, refraction, and shadowing of light on the surface. By combining the spherical harmonic coefficients of ambient light with the geometric and material properties of the surface, it can efficiently simulate the reflection and shadowing effects of light on the surface. Compared to alternatives such as Neural Radiance Fields and spherical harmonics-based illumination decomposition methods, 3D Gaussian splattering uses an explicit representation, which provides researchers with the opportunity to directly access illumination information and enables the decoupling of illumination from elements such as scene geometry.

[0003] Currently, using 3D Gaussian splatter as a basic representation for lighting decomposition is a very novel research topic, with only a few researchers conducting research in this area: Liang et al. first proposed a deep gradient derivation regularization method to optimize normal estimation, and used spherical harmonics to bake occlusion information to model indirect lighting. Spherical harmonics (SH) are a set of orthogonal basis functions used to represent functions on a sphere. For ambient light, its lighting distribution on a sphere can be expressed as a linear combination of a series of spherical harmonics. The orthogonality of spherical harmonics allows complex lighting functions to be decomposed into a set of coefficients that can be efficiently stored and manipulated; Du et al. innovatively introduced a diffusion model as a prior for geometry and material estimation, and combined it with parameterized light source optimization technology to improve the accuracy of lighting decomposition; Chen et al. constructed a lighting decomposition model based on Gaussian splatter by deeply fusing deferred shading with lightweight path tracing technology. The above-mentioned illumination decomposition methods based on 3D Gaussian splatting often have the following problems: 1. These methods rely on the stability of pre-trained models, which have the disadvantages of weak generalization ability and high training cost. 2. Regardless of whether these methods are based on lightweight path tracing technology or estimate indirect illumination by baking simulated occlusion information, they will result in low reconstruction accuracy and poor illumination decomposition effect. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a field scene illumination decomposition method based on 3D Gaussian splashing, aiming to reduce the computational cost of the illumination decomposition method based on 3D Gaussian splashing while improving the illumination decomposition accuracy.

[0005] The technical solution of the present invention is:

[0006] A method for decomposing outdoor scene illumination based on 3D Gaussian splashing, the method comprising the following steps:

[0007] Step 1: Obtain a dataset containing RGB images and HDR images of real wild scenes;

[0008] Step 2: Based on the RGB images in the dataset, use 3D Gaussian Splatting to geometrically reconstruct the scene: Use traditional 3D Gaussian Splatting to assign relevant parameters to each Gaussian distribution generated by sampling the RGB images in the dataset, and introduce normal estimation to add normal parameters to each Gaussian distribution;

[0009] Step 3: Design a multi-layer perceptron; use the HDR image in the dataset as the ambient light and use the multi-layer perceptron to predict the coefficients of the spherical harmonic function of the ambient light;

[0010] Step 4: Based on the coefficients of the spherical harmonics of the ambient light predicted by the multi-layer perceptron in step 3, establish an unshaded lighting model and a shadowed lighting model, and then optimize the shadowed lighting model to effectively separate the lighting and shadow information from the scene.

[0011] Furthermore, according to the outdoor scene illumination decomposition method, step 2 further includes the following steps:

[0012] Step 2.1: Generate a Gaussian distribution by randomly sampling the RGB images in the dataset and optimize each parameter in the Gaussian distribution;

[0013] Step 2.2: A normal map is obtained by rendering the normal vector of each pixel in the RGB image in the dataset; a depth map is obtained by rendering the depth value of each pixel in the RGB image in the dataset; a pseudo-normal map is derived by using the depth map, and the pseudo-normal map is used to supervise the optimization process of the normal map. The optimized normal map adds normal parameters to each Gaussian distribution generated in step 2.1 to improve the accuracy of the Gaussian distribution;

[0014] Furthermore, according to the outdoor scene illumination decomposition method, a stochastic gradient descent algorithm is used in step 2.1 to optimize each parameter in the Gaussian distribution.

[0015] Furthermore, according to the outdoor scene illumination decomposition method, step 3 further includes the following steps:

[0016] Step 3.1: Set the degree of the spherical harmonics; initialize the multilayer perceptron: set the input dimension, depth, and width, and use the degree of the spherical harmonics as the dimension of the multilayer perceptron output layer;

[0017] Step 3.2: Construct the encoding layer of the multi-layer perceptron: The output of each layer i that makes up the encoding layer is processed through a fully connected layer and a ReLU activation function according to the following formula:

[0018] h i =ReLU(W i h i-1 +b i ) (3)

[0019] Where W i and b i is the weight and bias of layer i, h i-1 is the output of the previous layer of the current i layer, h i is the output of the current i layer;

[0020] Step 3.3: Construct the output layer of the multilayer perceptron to map the output of the hidden layer to the dimension of the coefficients of the spherical harmonic function, that is, 3*(self.features_dc_dim+self.features_rest_dim), where 3 represents the three RGB color channels, self.features_dc_dim and self.features_rest_dim represent the dimensions of the zero-order coefficient and high-order coefficient of the spherical harmonic function respectively; the output layer is calculated by the following formula:

[0021] y=W out h D +b out (4)

[0022] Where W out is the weight of the output layer, b out is the bias of the output layer, h D is the output of the last hidden layer, and y is the output of the output layer;

[0023] Step 3.4: Perform forward propagation on the multi-layer perceptron constructed in the above steps to obtain the coefficients of the spherical harmonic function of the ambient light, namely (3, (self.features_dc_dim+self.features_rest_dim)).

[0024] Furthermore, according to the outdoor scene illumination decomposition method, step 4 includes the following steps:

[0025] Step 4.1: Use spherical harmonics to represent the ambient light in the scene;

[0026] Step 4.2: Using the coefficients L of the spherical harmonics of ambient light i Modeling to obtain an unshadowed lighting model;

[0027] Step 4.3: Use the coefficients L of the spherical harmonics of the ambient light i Modeling to obtain a shadowed lighting model;

[0028] Step 4.4: Introduce the shadow loss function and regularization term to supervise the optimization of the shadowed lighting model, and effectively separate the lighting and shadow information from the scene under the premise of complying with physical laws.

[0029] Furthermore, according to the outdoor scene illumination decomposition method, in step 4.1, based on the ambient light function L(ω) given by the HDR image, the coefficient of the spherical harmonic function L i It can be expressed as the ambient light function L(ω) and the spherical harmonic basis function Y i The projection of (ω)) on the spherical domain Ω:

[0030] Li =∫ Ω L(ω)Y i (ω)dω (5)

[0031] Furthermore, according to the outdoor scene illumination decomposition method, the coefficient L of the spherical harmonic function of the ambient light is used in step 4.2. i The unshaded illumination model obtained by modeling is: the color response c of each Gaussian basis element k Calculated by the following formula:

[0032]

[0033] where ρ k is the intrinsic color of the Gaussian element; n k is the surface normal; ω i is the incident light direction; M(L i ) is a matrix derived from equation (5) and is used to calculate the irradiance under unshaded illumination; the matrix M(L i ) is calculated by the following formula:

[0034]

[0035] Among them, M ij Denotes the matrix M(L i ) in the i-th row and j-th column, Y i (ω)Y j (ω) represents the spherical harmonics of the corresponding rows and columns respectively.

[0036] Furthermore, according to the outdoor scene illumination decomposition method, the coefficient L of the spherical harmonic function of the ambient light is used in step 4.3. i The model of the shadowed lighting is: by combining the radiation transfer function with the coefficients of the spherical harmonics of the ambient light, the color response after considering the shadow is calculated.

[0037]

[0038] Among them, ρ k is the intrinsic color of the Gaussian element; d k,i is the coefficient of the spherical harmonic function of Gaussian basis element k, which is used to represent shadow information; n is the degree of the spherical harmonic function; (n+1) 2 represents the spherical harmonic coefficient; l k The coefficients of the spherical harmonics of the ambient light at the location of Gaussian basis k.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) This invention uses 3D Gaussian splatting as a basic representation to achieve the task of illumination decomposition. 3D Gaussian splatting uses an explicit Gaussian point cloud to represent the scene, reducing the computational effort during rendering and achieving fast rendering speed. It does not rely on a large amount of pre-training data to learn a general scene representation, and does not require pre-training for specific scenes, resulting in strong generalization capabilities. This method utilizes pseudo-normal map supervision and total variation loss optimization to ensure high-precision acquisition of scene geometry information, reduce training costs, and provide a solid foundation for illumination modeling.

[0041] (2) The present invention designs a multilayer perceptron (MLP) to predict the coefficients of the spherical harmonics of ambient light, and introduces the radiation transfer function to model the shadow to achieve the estimation of indirect lighting, thereby improving the accuracy of lighting estimation and the precision of reconstruction results, and enhancing the effect of lighting decomposition.

[0042] (3) In the illumination model processing process, the present invention introduces a shadow modeling and constraint mechanism to ensure the physical rationality and stability of the illumination decomposition results. Specifically, a dedicated shadow loss function and multiple regularization terms are used to strictly constrain the radiation transfer function, thereby achieving better decomposition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an overall flow chart of the outdoor scene illumination decomposition method based on 3D Gaussian splashing of the present invention;

[0044] Figure 2 Schematic diagram of the structure of the illumination decomposition model based on 3D Gaussian splashing in the present invention;

[0045] Figure 3 The comparison diagram of experimental results, where (a) is the result without shadow after reconstruction; (b) is the result with shadow after reconstruction. DETAILED DESCRIPTION

[0046] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0047] Figure 1 This is the overall flow chart of the field scene illumination decomposition method based on 3D Gaussian splashing of the present invention. Figure 2 This is a schematic diagram of the structure of the illumination decomposition model based on 3D Gaussian splashing in the present invention. Figure 1 and Figure 2 Describe the method in detail, such as Figure 1 and Figure 2 As shown, the method includes the following steps:

[0048] Step 1: Obtain a dataset containing RGB images and HDR images of real wild scenes;

[0049] The publicly available NERF-OSR dataset is used in this embodiment. The dataset contains RGB images and HDR (High Dynamic Range) images of real outdoor scenes.

[0050] Step 2: Based on the RGB images in the dataset, use 3D Gaussian Splatting to geometrically reconstruct the scene: Use traditional 3D Gaussian Splatting to assign relevant parameters to each Gaussian distribution generated by sampling the RGB images in the dataset. In addition, normal estimation is introduced to add normal parameters to each Gaussian distribution.

[0051] Step 2.1: Generate a Gaussian distribution by randomly sampling the RGB images in the dataset and optimize each parameter in the Gaussian distribution.

[0052] By randomly sampling the RGB images in the dataset, a series of initial Gaussian distributions covering the entire scene are generated. Each Gaussian distribution is defined by a set of parameters, including position, orientation, scale, color, and opacity. Summarizing the properties of these Gaussian distributions, the mathematical expression of each Gaussian distribution G(x) is obtained as:

[0053]

[0054] Where x represents a point in the scene space; μ is the mean vector of the Gaussian distribution, μ∈R 3 , determines the center position of the Gaussian distribution; Σ is the covariance matrix, Σ∈R 3*3 , which is used to control the shape and direction of the Gaussian distribution. In addition, each Gaussian distribution is also assigned a color and opacity.

[0055] To ensure that these Gaussian distributions can better fit the geometric structure of the scene, the Stochastic Gradient Descent (SGD) algorithm is used to optimize the parameters of these Gaussian distributions to minimize the difference between the Gaussian distributions and the actual geometric structure of the scene while maintaining the sparsity and efficiency of the Gaussian distributions.

[0056] Step 2.2: Obtain a normal map by rendering the normal vector of each pixel in the RGB image in the dataset; Obtain a depth map by rendering the depth value of each pixel in the RGB image in the dataset; Derived a pseudo-normal map by using the depth map, and used the pseudo-normal map to supervise the optimization process of the normal map. The optimized normal map adds normal parameters to each Gaussian distribution generated in step 2.1 to improve the accuracy of the Gaussian distribution;

[0057] First, we use a depth map and a normal map to obtain the scene's geometric information. The depth map is obtained by rendering the depth value of each pixel in the RGB image in the dataset, reflecting the scene's depth structure. The normal map, on the other hand, is obtained by rendering the normal vector of each pixel in the RGB image in the dataset, describing the surface orientation of the scene. To improve rendering accuracy and efficiency, we use a pseudo-normal map to supervise the optimization of the rendered normals. The pseudo-normal map is derived from the depth map and serves as reference information for normal map optimization.

[0058] In the process of optimizing the normal map, the total variation loss (TV loss) is introduced to achieve smoother results. The TV loss function encourages the normal differences between adjacent pixels to be small, thereby reducing noise and discontinuities in the normal map. The loss function in the optimization process includes pseudo-normal loss and total variation loss. The mathematical expressions of the three are:

[0059] L n =L n-p +λ n-TV L TVnormal (2)

[0060] Among them, L n is the loss function in the optimization process, L n-p is the pseudo-normal loss, L TVnormal is the total variation loss, λ n-TV is a weight coefficient used to balance the contribution between pseudo-normal loss and total variation loss.

[0061] Step 3: Design a multi-layer perceptron; use the HDR image in the dataset as the ambient light and use the multi-layer perceptron to predict the coefficients of the spherical harmonics (SH) of the ambient light.

[0062] Step 3.1: Set the degree of the spherical harmonics function; use the HDR image in the dataset as the ambient light, initialize the multilayer perceptron, and use the degree of the spherical harmonics function as the dimension of the multilayer perceptron output layer during initialization.

[0063] Before initializing the multi-layer perceptron (MLP), you first need to set the degree of the spherical harmonics (sh_degree), which determines the complexity of the spherical harmonics. The higher the degree of the spherical harmonics, the richer the lighting details that can be represented, but the computational complexity will also increase accordingly. Next, initialize the multi-layer perceptron (MLP) and set the depth (mlp_D) and width (mlp_W) of the MLP. These two parameters determine the capacity and expressiveness of the MLP. The depth represents the number of layers of the multi-layer perceptron, and the width represents the number of neurons in each layer of the multi-layer perceptron. Then set the input dimension (N_a) of the multi-layer perceptron, which is usually the dimension of the appearance embedding vector. In this embodiment, the HDR image in the data set is used as the ambient light, and the appearance embedding vector contains the appearance feature information of the HDR image. Finally, the degree of the spherical harmonics is used as the dimension of the output layer of the multi-layer perceptron.

[0064] Step 3.2: Construct the encoding layer of the multi-layer perceptron.

[0065] The encoding layer of a multilayer perceptron consists of multiple fully connected layers, each followed by a ReLU activation function. The ReLU activation function introduces nonlinearity, enabling the network to learn complex feature representations. The first layer maps the input dimensions to the hidden layer dimensions, converting the appearance feature information of the original input, the HDR image, into a higher-level feature representation. Subsequent hidden layers map the hidden layer dimensions of the previous layer to the hidden layer dimensions of the current layer, further extracting and converting features. The output of each layer i is processed through a fully connected layer and a ReLU activation function:

[0066] h i =ReLU(W i h i-1 +b i ) (3)

[0067] Where W i and b i is the weight and bias of layer i, h i-1 is the output of the previous layer of the current i layer, h i is the output of the current i layer.

[0068] Step 3.3: Construct the output layer of the multilayer perceptron.

[0069] The output layer is the last layer of the MLP, and its function is to map the output of the hidden layer to the dimension of the coefficients of the spherical harmonics. The output dimension is 3*(self.features_dc_dim+self.features_rest_dim), where 3 represents the three RGB color channels, and self.features_dc_dim and self.features_rest_dim represent the dimensions of the zero-order coefficient (DC component) and high-order coefficient (AC component) of the spherical harmonics, respectively. Specifically, the output layer is calculated using the following formula:

[0070] y=W out h D +b out (4)

[0071] Where W out is the weight of the output layer, b out is the bias of the output layer, h D is the output of the last hidden layer, and y is the output of the output layer.

[0072] Step 3.4: Perform forward propagation on the multi-layer perceptron constructed in the above steps to obtain the coefficients of the spherical harmonic function of the ambient light.

[0073] During the forward propagation, the input appearance embedding vector first passes through each encoding layer, where a ReLU activation function is applied. The ReLU activation function ensures that the MLP can learn nonlinear features, thereby improving its expressiveness. Finally, the coefficients of the spherical harmonics of the ambient light (3, (self.features_dc_dim + self.features_rest_dim)) are obtained through the output layer for subsequent processing.

[0074] Step 4: Based on the coefficients of the spherical harmonics of the ambient light predicted by the multi-layer perceptron in step 3, establish an unshaded lighting model and a shadowed lighting model, and then optimize the shadowed lighting model to effectively separate the lighting and shadow information from the scene.

[0075] Step 4.1: Use spherical harmonics to represent the ambient light in the scene.

[0076] Based on the ambient light function L(ω) given by the HDR image, the coefficient of its spherical harmonic function L i It can be expressed as the ambient light function L(ω) and the spherical harmonic basis function Y i The projection of (ω) onto the spherical domain Ω:

[0077] L i =∫ Ω L(ω)Y i (ω)dω (5)

[0078] The given ambient light function L(ω) is obtained from the HDR image in the data set. Specifically, the HDR image is converted into a cube map as an environment map. The environment map is a polyhedral texture used to represent ambient light. Each face is a two-dimensional texture that represents the intensity and color of the ambient light in six main directions.

[0079] Step 4.2: Using the coefficients L of the spherical harmonics of ambient light i Modeling to obtain an unshadowed lighting model.

[0080] Without considering shadows, the illumination of the object surface can be calculated by the dot product of the spherical harmonics and the surface normal. For each Gaussian basis element, its color response c k It can be calculated by the following formula:

[0081]

[0082] The physical meaning of color response is the final color performance of the object surface under specific lighting conditions, ρ k is the intrinsic color of the Gaussian element, n k is the surface normal, ω i is the incident light direction, M(L i ) is a matrix derived from equation (5) and is used to calculate the irradiance under unshaded illumination. i ) can be calculated by triple product of spherical harmonics as follows:

[0083] M ij =∫ Ω Y i (ω)Y j (ω)dω (7)

[0084] Among them, M ij Denotes the matrix M(L i ) in the i-th row and j-th column, Y i (ω)Y j (ω) represents the spherical harmonic functions of the corresponding rows and columns respectively. The calculation of this matrix utilizes the orthogonality of the spherical harmonic functions to ensure the efficiency and accuracy of the lighting calculation.

[0085] Step 4.3: Use the coefficients L of the spherical harmonics of the ambient light i Modeling obtains the shadowed lighting model.

[0086] Considering the shadow, the radiation transfer function D is introduced. k(ω), which represents the light transfer from the ambient light to a specific Gaussian basis. The shaded lighting model combines the radiative transfer function with the coefficients of the spherical harmonics of the ambient light to calculate the color response after taking shadows into account.

[0087]

[0088] Among them, d k,i is the coefficient of the spherical harmonic function of Gaussian basis element k, which is used to represent shadow information. n is the degree of the spherical harmonic function, which determines the order of the spherical harmonic function expansion. In this embodiment, the order of the spherical harmonic function is 3, and the corresponding spherical harmonic coefficient is (n+1) 2 The number of l is 16, k The coefficients of the spherical harmonics of the ambient light at the location of Gaussian basis k.

[0089] Step 4.4: Introduce the shadow loss function and regularization term to supervise the optimization of the shadowed lighting model, and effectively separate the lighting and shadow information from the scene under the premise of complying with physical laws.

[0090] Step 4.4.1: Establish a shadow loss function to constrain the value of the radiation transfer function to be within a reasonable range.

[0091] In this embodiment, the shadow loss function will ensure that the radiation transfer function D k (ω i ) has a value between 0 and 1, where 0 represents full shadow and 1 represents full exposure to light. The shadow loss function is defined as:

[0092]

[0093] where ω i Indicates the direction of light; is the expected value over all Gaussian basis points and lighting directions, used to calculate the average value of the loss function;

[0094] This loss function penalizes D k (ω i ) exceeds the value of [0,1] to ensure the physical rationality of the radiation transfer function. Specific penalty measures:

[0095]

[0096] Step 4.4.2: Introduce multiple regularization terms to further constrain the radiative transfer function.

[0097] This step's 0-1 That is, l in step 4.1 shadow , used to ensure D k (ω i) has a value between 0 and 1:

[0098]

[0099] l + Ensure that the ambient light intensity is non-negative:

[0100]

[0101] Among them L c (ω i ) represents the light direction ω i The ambient light intensity on the

[0102] Make sure the radiative transfer function is aligned with the direction of the surface normal:

[0103]

[0104] where n k Represents the surface normal.

[0105] l ↓ Make sure the irradiance in the shadowed area does not exceed that in the unshaded area:

[0106]

[0107] Combined with each regularization term l 0-1 , l + , After adjusting the weights, we get the final regularization term R(G):

[0108]

[0109] The regularization term R(G) is a comprehensive constraint term, obtained by integrating each of the above regularization terms. It is used to impose multiple constraints on the radiation transfer function in the illumination decomposition task, ensuring that the scene after illumination decomposition conforms to physical laws and improving the stability and generalization ability of the model. The weight values λ1, λ2, λ3, and λ4 are obtained based on the Bayesian optimization method based on the probability model. The specific process is to construct a surrogate model of the objective function, such as Gaussian process regression, and gradually explore the parameter space to find the optimal value. This is a step-by-step iterative optimization process. The specific values of λ1, λ2, λ3, and λ4 need to depend on different validation sets.

[0110] Figure 3The following is a comparison chart of the results of this experiment, where Figure (a) shows the reconstruction result after the method of the present invention is added, and Figure (b) shows the reconstruction result without the method of the present invention. It can be clearly seen from the annotations that in places where shadow occlusion is more obvious due to the lighting in the scene, Figure (a) effectively decomposes the shadow occlusion caused by the lighting influence compared to Figure (b). This means that the method of the present invention can more effectively complete the lighting decomposition in the scene. Compared with other reconstruction schemes that do not introduce the method of the present invention, it has a great advantage in terms of the quality of the lighting decomposition effect. In addition, based on 3D Gaussian splashing, it has strong generalization ability.

[0111] It should be understood that, inspired by the technical concept of the present invention, those skilled in the art may make various improvements or changes based on the above content without departing from the content of the present invention, which still fall within the scope of protection of the present invention.

Claims

1. A method for outdoor scene illumination decomposition based on 3D Gaussian splashing, characterized in that: The method comprises the following steps: Step 1: Obtain a dataset containing RGB images and HDR images of real wild scenes; Step 2: Based on the RGB images in the dataset, use 3D Gaussian Splatting to geometrically reconstruct the scene: Use traditional 3D Gaussian Splatting to assign relevant parameters to each Gaussian distribution generated by sampling the RGB images in the dataset, and introduce normal estimation to add normal parameters to each Gaussian distribution; Step 3: Design a multi-layer perceptron; use the HDR image in the dataset as the ambient light and use the multi-layer perceptron to predict the coefficients of the spherical harmonic function of the ambient light; Step 4: Based on the coefficients of the spherical harmonics of the ambient light predicted by the multi-layer perceptron in step 3, establish an unshaded lighting model and a shadowed lighting model, and then optimize the shadowed lighting model to effectively separate the lighting and shadow information from the scene.

2. The outdoor scene illumination decomposition method according to claim 1, characterized in that: The step 2 further comprises the following steps: Step 2.1: Generate a Gaussian distribution by randomly sampling the RGB images in the dataset and optimize each parameter in the Gaussian distribution; Step 2.2: Obtain a normal map by rendering the normal vector of each pixel of the RGB image in the dataset; Obtain a depth map by rendering the depth value of each pixel of the RGB image in the dataset; A pseudo normal map is derived by using the depth map, and the pseudo normal map is used to supervise the optimization process of the normal map. The optimized normal map adds normal parameters to each Gaussian distribution generated in step 2.1 to improve the accuracy of the Gaussian distribution.

3. The outdoor scene illumination decomposition method according to claim 2, characterized in that: In step 2.1, the stochastic gradient descent algorithm is used to optimize each parameter in the Gaussian distribution.

4. The outdoor scene illumination decomposition method according to claim 2, characterized in that: The step 3 further comprises the following steps: Step 3.1: Set the degree of the spherical harmonics; initialize the multilayer perceptron: set the input dimension, depth, and width, and use the degree of the spherical harmonics as the dimension of the multilayer perceptron output layer; Step 3.2: Construct the encoding layer of the multi-layer perceptron: The output of each layer i that makes up the encoding layer is processed through a fully connected layer and a ReLU activation function according to the following formula: h i =ReLU(W i h i-1 +b i ) (3) Where W i and b i is the weight and bias of layer i, h i-1 is the output of the previous layer of the current i layer, h i is the output of the current i layer; Step 3.3: Construct the output layer of the multilayer perceptron to map the output of the hidden layer to the dimension of the coefficients of the spherical harmonic function, that is, 3*(self.features_dc_dim+self.features_rest_dim), where 3 represents the three RGB color channels, self.features_dc_dim and self.features_rest_dim represent the dimensions of the zero-order coefficient and high-order coefficient of the spherical harmonic function respectively; the output layer is calculated by the following formula: y=W out h D +b out (4) Where W out is the weight of the output layer, b out is the bias of the output layer, h D is the output of the last hidden layer, and y is the output of the output layer; Step 3.4: Perform forward propagation on the multi-layer perceptron constructed in the above steps to obtain the coefficients of the spherical harmonic function of the ambient light, namely (3, (self.features_dc_dim+self.features_rest_dim)).

5. The outdoor scene illumination decomposition method according to claim 4, characterized in that: The step 4 comprises the following steps: Step 4.1: Use spherical harmonics to represent the ambient light in the scene; Step 4.2: Using the coefficients L of the spherical harmonics of ambient light i Modeling to obtain an unshadowed lighting model; Step 4.3: Use the coefficients L of the spherical harmonics of the ambient light i Modeling to obtain a shadowed lighting model; Step 4.4: Introduce the shadow loss function and regularization term to supervise the optimization of the shadowed lighting model, and effectively separate the lighting and shadow information from the scene under the premise of complying with physical laws.

6. The outdoor scene illumination decomposition method according to claim 5, characterized in that: In step 4.1, based on the ambient light function L(ω) given by the HDR image, the coefficient of its spherical harmonic function L i It can be expressed as the ambient light function L(ω) and the spherical harmonic basis function Y i The projection of (ω) onto the spherical domain Ω: L i =∫ Ω L(ω)Y i (ω)dω (5).

7. The outdoor scene illumination decomposition method according to claim 6, characterized in that: The coefficient L of the spherical harmonic function of ambient light used in step 4.2 i The unshaded illumination model obtained by modeling is: the color response c of each Gaussian basis element k Calculated by the following formula: where ρ k is the intrinsic color of the Gaussian element; n k is the surface normal; ω i is the incident light direction; M(L i ) is a matrix derived from equation (5) and is used to calculate the irradiance under unshaded illumination; the matrix M(L i ) is calculated by the following formula: M ij =∫ Ω Y i (ω)Y j (ω)dω (7) Among them, M ij Denotes the matrix M(L i ) in the i-th row and j-th column, Y i (ω)Y j (ω) represents the spherical harmonics of the corresponding rows and columns respectively.

8. The outdoor scene illumination decomposition method according to claim 6, characterized in that: The coefficient L of the spherical harmonic function of ambient light used in step 4.3 i The model of the shadowed lighting is: by combining the radiation transfer function with the coefficients of the spherical harmonics of the ambient light, the color response after considering the shadow is calculated. Among them, ρ k is the intrinsic color of the Gaussian element; d k,i is the coefficient of the spherical harmonic function of Gaussian basis element k, which is used to represent shadow information; n is the degree of the spherical harmonic function; (n+1) 2 represents the spherical harmonic coefficient; l k The coefficients of the spherical harmonics of the ambient light at the location of Gaussian basis k.

Images