High-quality inverse rendering method based on 2D diffusion model and 3D Gaussian splash

Abstract

The invention discloses a high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing, belongs to the technical field of inverse rendering, and solves the problem that the reconstruction precision of an existing inverse rendering method in terms of material attributes is insufficient. The 2D diffusion model and the distillation learning method are introduced into the 3D Gaussian splash model, excellent material priori knowledge learned by the 2D diffusion model is applied to the rendering method of the 3D Gaussian splash model, and the loss function in the 3D Gaussian splash model is guided and optimized through distillation learning, so that the accuracy of the reconstructed and generated material attribute is improved, and the reconstruction efficiency of the 3D Gaussian splash model is improved. The 2D diffusion model is trained in advance, so that excessive time overhead is not brought in actual use, and the time advantage of an original 3D Gaussian method is reserved.

Description

Technical Field

[0001] This invention relates to the field of reverse rendering technology, specifically to a high-quality reverse rendering method based on a 2D diffusion model and 3D Gaussian splashing. Background Technology

[0002] With the rapid popularization of digital twins, virtual reality (VR), and augmented reality (AR) technologies, high-quality, highly realistic 3D scene reconstruction and dynamic lighting simulation have become important research directions in computer vision and graphics. Against this backdrop, inverse rendering, as a key bridge connecting the physical world and digital space, is increasingly demonstrating its core value. Inverse rendering can recover the physical properties of a scene from multi-view images, including geometric structure, material reflection properties, and lighting distribution. It not only enables realistic new perspective compositing but also supports dynamic relighting and scene physics editing, thus finding wide applications in various fields such as film and television special effects, autonomous driving simulation, industrial design, and metaverse content generation.

[0003] However, traditional inverse rendering methods still face many challenges. Most methods rely on strong simplification assumptions, such as using simplified lighting models or assuming that object surfaces are ideal Lambertian reflectors, while ignoring the reflective properties of complex materials such as metal, cloth, and skin. In terms of lighting modeling, fixed light sources or single ambient lights are often used, making it difficult to realistically reflect the dynamic light and shadow interactions in actual scenes. In addition, existing technologies mostly use offline calculations and rely on pre-generated lightmaps, which cannot meet the needs of real-time interactive applications.

[0004] To address the aforementioned issues, the existing technology "GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering" proposes a 3D Gaussian splatting method based on decomposed global illumination. This method generates G-buffer data and achieves high-quality inverse rendering reconstruction and relighting results by decomposing global illumination (direct and indirect lighting rendering). While retaining the time advantage of the 3D Gaussian method, it makes significant breakthroughs in decoupling material properties from lighting. However, the decomposition of material properties (metallicity, roughness) relies on a supervisory signal, resulting in insufficient handling of complex materials (such as anisotropy). The accuracy in material properties (metallicity, roughness) is still far inferior to methods based on neural radiation fields. Therefore, this method does not perform ideally in relighting applications and still suffers from insufficient accuracy in reconstructing material properties. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing, which solves the problem of insufficient accuracy in reconstructing material properties in existing inverse rendering methods.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing is provided, including the following steps: S1. Input multi-view images into the 3D Gaussian splash model for scene reconstruction to generate G-Buffer data containing geometric and material properties; among which, material properties include reflectance maps and ORM maps; S2. Perform direct lighting rendering and indirect lighting rendering on the G-Buffer data, fuse the rendering results of the two to obtain the predicted rendering image, and compare the predicted rendering image with the multi-view image to obtain the first loss. S3. Input the predicted rendered image and material properties into the pre-trained 2D diffusion model to obtain the predicted optimized material properties, and calculate the second loss between the optimized material properties and the material properties through the distillation learning method. S4. Combine the first loss and the second loss to obtain the joint loss, and iteratively optimize the geometric material property parameters in the 3D Gaussian splash model through backpropagation of the joint loss. S5. Repeat steps S1 to S4 for a preset number of times, and then output the final predicted rendering image.

[0007] Furthermore, the training method for the 2D diffusion model includes the following steps: SA, obtain the dataset based on the PBR model; the dataset includes each training sample. ,in, , , These are the input viewpoint images and... Reflectance map and ORM map from the corresponding viewpoint; SB, take a training sample Inputting into the VAE encoder, the conditional implicit variables are obtained respectively. Reflectance latent variable and ORM hidden variables and will and Concatenate them into the target latent variable Z; SC, random sampling time step and noise By applying noise perturbation to Z, the noise latent variable is obtained. and calculate with The corresponding target velocity vector , The expression is: in, and The cumulative noise scheduling parameters and noise standard deviation parameters during the diffusion process are respectively used. SD, will , and The common input is the prediction U-Net network in the 2D diffusion model, and the output is... Predicted value ; SE, Calculation and The mean squared error between them is used as the training loss. and based on Perform backpropagation to update and predict the U-Net network. The expression is: in, For input variables , , ), , The expected value of the calculation; Repeat steps SF, SB~SE, until the training loss converges, completing the training of the 2D diffusion model.

[0008] Furthermore, the method for obtaining the dataset is as follows: SA1. Obtain multiple 3D models, all of which include PBR textures; SA2. Determine if each 3D model has a glass surface. If so, replace the glass surface on the corresponding 3D model with a black surface; otherwise, do nothing. SA3: Randomly select multiple viewpoints for rendering each 3D model, and obtain the input viewpoint image, reflectance map and ORM map under the corresponding viewpoint.

[0009] Furthermore, the roughness of the black surface is 0.25, and the metallicity is 0.

[0010] Furthermore, the expression for the second loss is: in, This is the second loss; This refers to the input variable being , and The expected value of the calculation; For perspective variables; and , respectively, are the loss weights for latent space and RGB space; z is the latent space result of the reflectance map and ORM map after being encoded by the VAE encoder; Latent space results predicted by a pre-trained 2D diffusion model; This is the decoder for a pre-trained 2D diffusion model.

[0011] Furthermore, the expression for the first loss is: in, This is the first loss; , and These are color loss, material loss, and illumination loss, respectively. and These are the weights for material loss and illumination loss, respectively.

[0012] Furthermore, the expression for the joint loss is: in, For joint losses; for The loss weight.

[0013] This invention discloses a high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing, the beneficial effects of which are: 1. This invention introduces a 2D diffusion model and distillation learning method into a 3D Gaussian splash model. The excellent material prior knowledge learned by the 2D diffusion model is applied to the rendering method of the 3D Gaussian splash model. Distillation learning is used to guide and optimize the loss function in the 3D Gaussian splash model, which improves the accuracy of the reconstructed material properties and significantly enhances the accuracy of relighting. At the same time, since the 2D diffusion model is pre-trained, it does not incur much time overhead in actual use, retaining the time advantage of the original 3D Gaussian method and solving the problem of insufficient accuracy in material property reconstruction in existing inverse rendering methods.

[0014] 2. This invention ensures that the 2D diffusion model used has accurate material understanding capabilities through a diffusion model training process customized for material property estimation tasks. It is fine-tuned using a high-quality dataset containing accurate physical properties (PBR), enabling the model to effectively decouple and predict key physical properties of reflectivity (albedo) and ORM (occlusion, roughness, metallicity) from a single object image. This provides a reliable and high-quality source of material prior signals for the entire inverse rendering method and significantly improves the discriminability and robustness of the estimation.

[0015] 3. Considering that the glass surface has complex optical properties and is prone to interfering with material estimation, this invention effectively improves the purity and quality of training data by automatically identifying and replacing the glass surface in the 3D model, providing a solid data guarantee for training a robust and generalized 2D diffusion model.

[0016] 4. This invention clarifies the specific physical parameters of the black surface used to replace the glass surface. The surface defined by this specific parameter combination approximates an ideal diffuse reflector, which can minimize the complex simulation of lighting interactions and simplify the model learning process. This standardized approach not only ensures data consistency but also improves training efficiency and the stability of the model after convergence.

[0017] 5. The second loss expression provided by this invention innovatively combines dual constraints of the latent space and the RGB pixel space. By comparing the encoded material properties in the latent space, the consistency of high-level semantic features is guaranteed; simultaneously, comparison in the decoded RGB space ensures the visual fidelity of the final output material map. This dual supervision mechanism can more comprehensively and precisely guide the optimization direction of the 3D Gaussian model, thereby more effectively improving the reconstruction accuracy of material properties.

[0018] 6. The first loss in this invention comprehensively considers reconstruction errors from multiple physical dimensions, including color, material, and lighting. It retains the advantages of existing methods such as GI-GS in geometric reconstruction and new perspective compositing, ensuring the stability of basic rendering quality and providing a reliable and balanced optimization foundation for introducing diffusion model priors for specific optimization.

[0019] 7. The joint loss function of this invention achieves an organic combination of "experience" and "prior knowledge" by weighting and combining the traditional inverse rendering loss (first loss) with the prior knowledge loss from the 2D diffusion model (second loss). This joint loss function enables the model optimization process to not only fit the specific content of the input image but also conform to the general laws of natural material distribution. This collaborative optimization strategy can improve material accuracy while maintaining good overall coordination and visual realism. Attached Figure Description

[0020] Figure 1 This is a flowchart of a high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing. Detailed Implementation

[0021] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0022] refer to Figure 1 This embodiment provides a high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing, including the following steps: S1. Input multi-view images into the 3D Gaussian splash model for scene reconstruction, generating G-Buffer data containing geometric and material properties; among which, material properties include reflectance maps and ORM maps. In the ORM map, ORM is a combination channel of occlusion, roughness, and metallicity.

[0023] S2. Perform direct lighting rendering and indirect lighting rendering on the G-Buffer data, fuse the rendering results of the two to obtain the predicted rendering image, and compare the predicted rendering image with the multi-view image to obtain the first loss.

[0024] Specifically, the expression for the first loss is: in, This is the first loss; , and These are color loss, material loss, and illumination loss, respectively. and These are the weights for material loss and illumination loss, respectively.

[0025] S3. Input the predicted rendered image and material properties into the pre-trained 2D diffusion model to obtain the predicted optimized material properties, and calculate the second loss between the optimized material properties and the material properties through the distillation learning method.

[0026] In the 2D diffusion model, the time step is set to [0.02, 0.98], and 5 steps of DDIM sampling are performed to calculate the second loss. DDIM (Denoising Diffusion Implicit Models) is a deterministic sampler whose sampling path is uniquely determined. This characteristic provides a stable and consistent guiding signal for gradient descent-based 3D Gaussian parameter optimization. During the inverse rendering optimization process, for the same set of inputs, the direction of the calculated SDS loss gradient is repeatable each time. This effectively avoids optimization path oscillation and non-convergence problems caused by sampling randomness, thus significantly improving the stability and reliability of the entire training process.

[0027] Specifically, the expression for the second loss is: in, This is the second loss; This refers to the input variable being , and The expected value of the calculation; For perspective variables; and , respectively, are the loss weights for latent space and RGB space; z is the latent space result of the reflectance map and ORM map after being encoded by the VAE encoder; Latent space results predicted by a pre-trained 2D diffusion model; This is the decoder for a pre-trained 2D diffusion model.

[0028] S4. Combine the first loss and the second loss to obtain the joint loss, and iteratively optimize the geometric material property parameters in the 3D Gaussian splash model through backpropagation of the joint loss.

[0029] Specifically, the expression for the joint loss is: in, For joint losses; for The loss weight.

[0030] S5. Repeat steps S1 to S4 for a preset number of times, and then output the final predicted rendering image.

[0031] In this embodiment, the training method for the 2D diffusion model includes the following steps: SA, obtain the dataset based on the PBR model; the dataset includes each training sample. ,in, , , These are the input viewpoint images and... Reflectance map and ORM map at the corresponding viewpoint.

[0032] SB, take a training sample Inputting into the VAE encoder, the conditional implicit variables are obtained respectively. Reflectance latent variable and ORM hidden variables and will and The concatenation is used to form the target latent variable Z.

[0033] SC, random sampling time step and noise By applying noise perturbation to Z, the noise latent variable is obtained. and calculate with The corresponding target velocity vector , The expression is: in, and The cumulative noise scheduling parameters and noise standard deviation parameters during the diffusion process are respectively.

[0034] SD, will , and The common input is the prediction U-Net network in the 2D diffusion model, and the output is... Predicted value .

[0035] SE, Calculation and The mean squared error between them is used as the training loss. and based on Perform backpropagation to update and predict the U-Net network. The expression is: in, For input variables , , ), , The expected value of the calculation.

[0036] Repeat steps SF, SB~SE, until the training loss converges, completing the training of the 2D diffusion model.

[0037] In this embodiment, the 2D diffusion model is an improvement on the existing Stable Diffusion pre-trained model. The text input condition is replaced with the object image x, and the output target is locked as the material property map Id (reflectivity) and Iorm (occlusion, roughness, metallicity). This transformation changes the model from "imagining the scene based on the text" to "analyzing the physical properties based on the image", accurately positioning the needs of the inverse rendering task.

[0038] As a further solution in this embodiment, the method for obtaining the dataset is as follows: SA1. Obtain multiple 3D models, each containing PBR textures. PBR textures are a collective term for texture maps used in Physically Based Rendering.

[0039] Specifically, the 3D model in this embodiment first starts with existing datasets, such as the Objaverse dataset, which is a large and diverse pre-processed dataset. However, because it contains many low-quality, textureless multi-view images of objects, these low-quality and non-PBR texture object data are filtered and removed, resulting in approximately 7,000 usable data points. Considering that this dataset is not diverse enough, to further improve diversity, Blender is used to create additional high-quality PBR texture object data.

[0040] SA2. Determine if each 3D model has a glass surface. If so, replace the glass surface on the corresponding 3D model with a black surface with a roughness of 0.25 and a metallicity of 0; otherwise, do nothing.

[0041] SA3: Randomly select multiple viewpoints for rendering each 3D model, and obtain the input viewpoint image, reflectance map and ORM map under the corresponding viewpoint.

[0042] In this embodiment, for each 3D model, multiple viewpoints are randomly selected on a hemisphere with a radius of 1.5-2 for rendering images. Each model renders 30 x-data images from different viewpoints and two corresponding Id and Iorm material attribute images. Ambient lighting is also randomly selected. Based on this, the 2D diffusion model training dataset in this embodiment consists of a high-quality PBR model with multi-view image datasets and corresponding material attribute images, totaling approximately 20k images, providing sufficient prior knowledge for subsequent 2D diffusion model training.

[0043] In summary, the beneficial effects of this solution are as follows: This solution introduces a 2D diffusion model and a distillation learning method, successfully applying the excellent material prior knowledge learned by the diffusion model to the 3D Gaussian splash inverse rendering method. This improves the accuracy of the reconstructed material properties and significantly enhances the accuracy of relighting. At the same time, since the 2D diffusion model is pre-trained, it does not incur excessive time overhead in actual use, thus retaining the time advantage of the original 3D Gaussian method.

[0044] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this invention.

Claims

1. A high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing, characterized in that, Includes the following steps: S1. Input multi-view images into a 3D Gaussian splash model for scene reconstruction to generate G-Buffer data containing geometric and material properties; wherein, the material properties include reflectance maps and ORM maps; S2. Perform direct lighting rendering and indirect lighting rendering on the G-Buffer data, fuse the rendering results of the two to obtain a predicted rendering image, and compare the predicted rendering image with the multi-view image to obtain a first loss. S3. Input the predicted rendered image and the material properties into the pre-trained 2D diffusion model to obtain the predicted optimized material properties, and calculate the second loss between the optimized material properties and the material properties through the distillation learning method. S4. Combine the first loss and the second loss to obtain the joint loss, and iteratively optimize the geometric material property parameters in the 3D Gaussian splash model through backpropagation of the joint loss. S5. Repeat steps S1 to S4 for a preset number of times, and then output the final predicted rendering image.

2. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 1, characterized in that, The training method for the 2D diffusion model includes the following steps: SA, obtain the dataset based on the PBR model; the dataset includes each training sample containing ,in, , , These are the input viewpoint images and... Reflectance map and ORM map from the corresponding viewpoint; SB, take a training sample Inputting into the VAE encoder, the conditional implicit variables are obtained respectively. Reflectance latent variable and ORM hidden variables and will and Concatenate them into the target latent variable Z; SC, random sampling time step and noise By applying noise perturbation to Z, the noise latent variable is obtained. and calculate with The corresponding target velocity vector , The expression is: in, and The cumulative noise scheduling parameters and noise standard deviation parameters during the diffusion process are respectively used. SD, will , and The common input is the prediction U-Net network in the 2D diffusion model, and the output is... Predicted value ; SE, Calculation and The mean squared error between them is used as the training loss. and based on Perform backpropagation to update and predict the U-Net network. The expression is: in, For input variables , , ), , The expected value of the calculation; SF, repeat steps SB~SE until the training loss converges, and the training of the 2D diffusion model is completed.

3. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 2, characterized in that, The method for obtaining the dataset is as follows: SA1. Obtain multiple 3D models, all of which include PBR textures; SA2. Determine if each 3D model has a glass surface. If so, replace the glass surface on the corresponding 3D model with a black surface; otherwise, do nothing. SA3: Randomly select multiple viewpoints for rendering each 3D model, and obtain the input viewpoint image, reflectance map and ORM map under the corresponding viewpoint.

4. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 3, is characterized in that, The roughness of the black surface is 0.25, and the metallicity is 0.

5. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 2, characterized in that, The expression for the second loss is: in, This is the second loss; This refers to the input variable being , and The expected value of the calculation; For perspective variables; and , respectively, are the loss weights for latent space and RGB space; z is the latent space result of the reflectance map and ORM map after being encoded by the VAE encoder; Latent space results predicted by a pre-trained 2D diffusion model; This is the decoder for a pre-trained 2D diffusion model.

6. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 5, characterized in that, The expression for the first loss is: in, This is the first loss; , and These are color loss, material loss, and illumination loss, respectively. and These are the weights for material loss and illumination loss, respectively.

7. The high-quality inverse rendering method based on a 2D diffusion model and 3D Gaussian splashing as described in claim 6, characterized in that, The expression for the joint loss is: in, For joint losses; for The loss weight.

Images