Method for generating harmonious object in three-dimensional scene based on physical rendering and uncertainty estimation

Abstract

The invention provides a method for generating a harmonious object in a three-dimensional scene based on physical rendering and uncertainty estimation, and the method comprises the steps: giving a three-dimensional scene, and generating the text description of a target object and a rectangular frame of the position of the target object in a 2D rendering view of the three-dimensional scene; according to the text description of the generated target object, a multi-view image generation diffusion model is utilized to construct a deformable tetrahedral mesh of the geometric appearance of the target object; constructing a feature network comprising a material item prediction module and an uncertainty item prediction module; obtaining a rendered picture of the target object by using a physical rendering technology, and performing training optimization on the feature network; and obtaining a multi-view scene rendering dense view containing the target object and a 3D model file according to the three-dimensional scene, the deformable tetrahedral mesh and a material item output by the trained feature network. According to the method, the problem that the target object and the scene are not harmonious in fusion is effectively solved, and the reality sense and the visual effect of the target object in the scene are enhanced.

Description

Technical Field

[0001] The invention relates to the field of computer vision and generation models, and relates to a method for generating harmonious objects in a three-dimensional scene, and in particular to a method for generating harmonious objects in a three-dimensional scene based on physical rendering and uncertainty estimation. Background Art

[0002] In recent years, with the development of computer vision and generative modeling, technologies such as text-driven 3D object generation and 3D scene editing have emerged and are widely used in film, television, games, architectural design, VR / AR and other fields. In these applications, users can input specific text prompts and other information to generate the 3D object they want at the specified location of the scene, greatly improving modeling efficiency and reducing costs. However, although the current research on generating 3D objects in three-dimensional scenes has achieved certain results, it is still difficult to make the generated 3D objects visually natural and harmonious with the scene. Some existing methods, such as: A 2024 article, InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes, proposed a method for inserting 3D objects generated by text into a 3D scene reconstructed by a neural radiance field (NeRF). The method first uses the user's given input rectangular box and prompt text in a 3D scene as the basis, and obtains a single image of the object in harmony with the scene through an image completion prior model. Then, the 3D object is obtained using the existing method of reconstructing 3D objects from a single image. Subsequently, the object is rescaled and placed in the corresponding position in the scene according to the estimated object scale and depth. However, although this method uses the image completion method to generate a 2D object that is in harmony with the scene from a single perspective, the subsequent process of reconstructing the 3D object from a single image cannot ensure that the object is in harmony with the scene from multiple other perspectives, resulting in the final generated 3D object being not harmonious and natural with the scene.

[0003] A 2024 article GO-NeRF: Generating Virtual Objects in Neural Radiance Fields proposed a method for generating 3D objects in a 3D scene that are in harmony with the scene under multi-view conditions. This method replaces the fractional distillation sampling loss (SDS Loss) commonly used in the 3D process with the image completion fractional distillation sampling loss (Inpainting SDS Loss), and uses the multi-view image completion prior model to gradually optimize the generated objects in the scene. At the same time, some traditional image harmonization loss functions, such as saturation loss and style consistency loss, are used to optimize the harmony between objects and scenes. Although this method considers the harmony between the generated objects and the scene under multiple viewpoints, it ignores the ambient lighting of the scene, and the traditional image harmonization loss has the disadvantages of reduced color accuracy, poor generalization ability, and possible loss of details, which can easily lead to suboptimal or low-quality generation results. Summary of the invention

[0004] In view of the problems that the existing technology does not fully utilize the scene semantic information and ignores the ambient lighting when generating objects in a three-dimensional scene, resulting in the inharmonious fusion of the object and the scene and low generation quality, the present invention aims to provide a new method that can fully utilize the scene semantic information and the ambient lighting, and integrate the scene semantic information and the ambient lighting into the generated 3D object through estimated uncertainty, so as to ensure that the object is naturally integrated into the scene and maintains harmony with the scene from multiple perspectives, while improving the quality and fidelity of the generated object.

[0005] The present invention proposes a method for generating harmonious objects in three-dimensional scenes based on physical rendering and uncertainty estimation. The method mainly models the geometry and material of the generated 3D objects based on physical rendering, then relights the 3D objects according to a given scene, and optimizes the harmony of each perspective of the 3D objects combined with the scene in combination with the estimated uncertainty, ensuring that the generated objects are naturally harmonious with the scene and improving the quality and fidelity of the generated objects.

[0006] The technical solution adopted by the present invention is as follows: A method for generating harmonious objects in a three-dimensional scene based on physical rendering and uncertainty estimation comprises the following steps: Given a three-dimensional scene, generate a text description of a target object and a rectangular box of the location of the target object in a 2D rendering view of the three-dimensional scene; Based on the generated text description of the target object, a deformable tetrahedral mesh of the geometric appearance of the target object is constructed based on a two-stage method using a pre-trained multi-view image generation diffusion model; Constructing a feature network, the feature network comprising a material item prediction module and an uncertainty item prediction module; the input of the feature network is the sampling points of the deformable tetrahedral mesh surface, and the output is the material item and uncertainty item of the object; Using physical rendering technology to obtain a rendered image of the target object, and training and optimizing the feature network; A multi-view scene rendering dense view and a 3D model file containing a target object are obtained according to the three-dimensional scene, the deformable tetrahedral mesh and the material items output by the trained feature network.

[0007] Furthermore, the three-dimensional scene is a pre-trained three-dimensional scene represented by a neural radiation field NeRF.

[0008] Furthermore, according to the text description of the generated target object, a pre-trained multi-view image generation diffusion model is used to construct a deformable tetrahedral mesh of the geometric appearance of the target object based on a two-stage method, and the specific steps are: Construct a 3D target object represented by NeRF; Taking the camera parameters as input, using NeRF's multi-layer perceptron to process the input, outputting a color field and a density field, and obtaining a 2D image img1 of the 3D object represented by NeRF under the camera parameter conditions through volume rendering technology according to the color field and the density field; Taking the text description and camera parameters of the generated target object as input, a pre-trained multi-view image generation diffusion model is used to output a 2D image img0 of the target object that meets the text description under the camera parameter conditions; Using the images img1 and img0, SDS loss (Score Distillation Sampling Loss) is calculated to train the 3D target object represented by NeRF, so as to obtain the 3D target object represented by NeRF with a rough geometric appearance; Extracting an initial deformable tetrahedral mesh from the 3D target object represented by NeRF having a rough geometric appearance; Obtaining a normal map normal1 of the initial deformable tetrahedral mesh under the camera parameter condition and a normal map normal2 of the 3D target object represented by NeRF with a rough geometric appearance; The SDS loss is calculated using the image img0 and the normal map normal1, and the MSE loss (Mean Squared Error Loss) is calculated using the normal maps normal1 and normal2, and the initial deformable tetrahedral mesh is trained to obtain a deformable tetrahedral mesh including a refined geometric appearance of the target object.

[0009] Furthermore, the specific steps of extracting an initial deformable tetrahedral mesh from the 3D target object represented by NeRF with a rough geometric appearance are: The density field of the 3D target object represented by NeRF with a rough geometric appearance is converted into a signed distance function SDF (Signed Distance Function), the signed distance function SDF is subtracted from a non-zero constant to obtain an initial SDF, and an initial deformable tetrahedral mesh is obtained according to the initial SDF using a differentiable tetrahedron marching algorithm (Deep Marching Tetrahedra, DMTet).

[0010] Furthermore, the feature network includes an input layer, a hash network coding layer, a linear layer, an activation function layer and an output layer.

[0011] Furthermore, the material items include albedo, metallicity and roughness.

[0012] Furthermore, the method of obtaining a rendered image of a target object by using physical rendering technology specifically includes: Image-based lighting technology (IBL) calculates the ambient light on the surface of the target object according to the environment map, combines the material items and the ambient light, and obtains the rendered image of the target object through physical rendering technology (PBR).

[0013] Furthermore, training and optimizing the feature network specifically includes: Randomly initialize the environment map, obtain an initial rendered image of the target object according to the initial environment map and the initial material item output by the feature network, and use the initial rendered image to train the material item prediction module of the feature network through a material item loss function; Acquire the spatial coordinates of the target object in the three-dimensional scene according to the three-dimensional scene, the initial rendering image of the target object, and the rectangular frame of the position of the target object in the 2D rendering view of the three-dimensional scene; Acquire a scene environment map according to the spatial coordinates, and acquire a new rendered image of the target object according to the scene environment map and the material item output by the trained feature network; Use the pre-trained 2D image harmonization network to obtain the resulting image after the target object and the 3D scene are harmonized; The uncertainty term loss and the harmonization loss are calculated according to the new rendered image of the target object and the harmonized result image, and the albedo parts of the uncertainty term prediction module and the material term prediction module of the feature network are trained.

[0014] Further, obtaining the spatial coordinates of the target object in the three-dimensional scene according to the three-dimensional scene, the rendered image of the target object and the rectangular frame of the position of the target object in the 2D rendering view of the three-dimensional scene specifically includes: Obtaining a 2D rendered image and a scene depth map of the three-dimensional scene; According to the rectangular frame of the position of the target object in the 2D rendering view of the 3D scene, a fused image is constructed with the 2D rendering image of the 3D scene as the background and the rendering image of the target object as the foreground; Estimate the depth map of the fused image using a monocular depth estimation method, align the depth map of the fused image with the scene depth map using a least squares method, and determine the depth of the foreground target object; The center point of the rectangular frame is reversely projected into the three-dimensional scene, and the coordinates of the center point are calculated according to the depth of the foreground target object as the spatial coordinates of the target object in the three-dimensional scene.

[0015] Furthermore, the method of using the pre-trained 2D image harmonization network to obtain the result image after the target object and the three-dimensional scene are harmonized includes: inputting the new rendered image of the target object and the 2D rendered image of the three-dimensional scene under the same camera parameter conditions into the pre-trained 2D image harmonization network, and outputting the harmonized result image of the target object.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The method proposed in the present invention makes full use of scene semantic information and ambient lighting, and integrates the two into the generated 3D object through estimated uncertainty, which can effectively solve the problem of inharmonious fusion of target objects and scenes in the prior art, ensure that the generated target objects are naturally and seamlessly integrated into the scene, and enhance the realism and visual effect of the target objects in the scene.

[0017] 2. The technical means adopted in the present invention can decouple the geometry and material modeling of the target object. During the generation process, a carefully designed optimization method is used to ensure the generation of high-quality, high-fidelity and multi-perspective consistent geometry and material information, thus overcoming the problems of low quality and poor fidelity of the target objects generated in the prior art.

[0018] 3. The present invention realizes a simple user interaction design. The user only needs to provide a text description and a rectangular frame of the target object's location in the 2D picture of the scene rendering view. After calculating the spatial position and scale, the target object can be accurately placed in the 3D space. The occlusion relationship is calculated according to the depth, and the inharmonious occlusion caused by perspective switching is reduced. The adaptability and stability of the object in complex 3D scenes are enhanced, and the flexibility and creativity of scene editing are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a general flow chart of the generation method in an embodiment of the present invention.

[0020] Figure 2 It is a sub-flow chart of the geometric learning phase of the generation method in an embodiment of the present invention.

[0021] Figure 3 It is a sub-flow chart of the material modeling stage of the generation method in the embodiment of the present invention.

[0022] Figure 4 It is a characteristic network structure diagram of predicted material and uncertainty of the generation method in an embodiment of the present invention.

[0023] Figure 5 It is a sub-flow chart of positioning an object in a scene in a generation method in an embodiment of the present invention.

[0024] Figure 6 It is a sub-flow chart of the harmonization stage of the generation method in an embodiment of the present invention.

[0025] Figure 7 This is the test result diagram (1) of generating a “pumpkin head” in an embodiment of the present invention.

[0026] Figure 8 This is the test result diagram (2) of generating a “pumpkin head” in an embodiment of the present invention.

[0027] Fig. 9 This is the test result diagram (3) of the “Pumpkin Head” generated in the embodiment of the present invention.

[0028] Fig.10 This is the result diagram (1) of the test generation of “Charmander” in the embodiment of the present invention.

[0029] Fig.11 This is the result diagram (2) of the test generation of “Charmander” in the embodiment of the present invention.

[0030] Fig.12 This is the result diagram (3) of the test generation of “Charmander” in the embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0032] This embodiment provides a method for generating a harmonized object in a three-dimensional scene based on physical rendering and uncertainty estimation, comprising the following steps: S1. Given a three-dimensional scene, generate a text description of a target object and a rectangular box of the position of the target object in a 2D rendering view of the three-dimensional scene; the three-dimensional scene is a pre-trained three-dimensional scene represented by a neural radiation field NeRF.

[0033] S2. Based on the generated text description of the target object, a pre-trained multi-view image generation diffusion model is used to construct a deformable tetrahedral mesh of the geometric appearance of the target object based on a two-stage method; the specific steps are: S2.1. Construct a 3D target object represented by NeRF; S2.2. Take the camera parameters as input, use NeRF's multi-layer perceptron to process the input, output the color field and density field, and obtain the 2D image img1 of the 3D object represented by NeRF under the camera parameter conditions through volume rendering technology according to the color field and density field; S2.3. Taking the text description and camera parameters of the generated target object as input, a 2D image img0 of the target object that meets the text description under the camera parameter conditions is output using a pre-trained multi-view image generation diffusion model; S2.4. Using the images img1 and img0, calculate the SDS loss (Score Distillation Sampling Loss), train the 3D target object represented by the NeRF, and obtain the 3D target object represented by the NeRF with a rough geometric appearance; Extracting an initial deformable tetrahedral mesh from the 3D target object represented by NeRF with a rough geometric appearance; specifically: converting the density field of the 3D target object represented by NeRF with a rough geometric appearance into a signed distance function (SDF), subtracting the signed distance function (SDF) from a non-zero constant to obtain an initial SDF, and obtaining an initial deformable tetrahedral mesh according to the initial SDF using a differentiable tetrahedron marching algorithm (Deep Marching Tetrahedra, DMTet); Obtaining a normal map normal1 of the initial deformable tetrahedral mesh under the camera parameter condition and a normal map normal2 of the 3D target object represented by NeRF with a rough geometric appearance; The SDS loss is calculated using the image img0 and the normal map normal1, and the MSE loss (Mean Squared Error Loss) is calculated using the normal maps normal1 and normal2, and the initial deformable tetrahedral mesh is trained to obtain a deformable tetrahedral mesh including a refined geometric appearance of the target object.

[0034] S3. Construct a feature network including an input layer, a hash network coding layer, a linear layer, an activation function layer and an output layer, wherein the feature network includes a material item prediction module and an uncertainty item prediction module; the input of the feature network is the sampling points of the deformable tetrahedral mesh surface, and the output is the material item and uncertainty item of the object, wherein the material item includes albedo, metallicity and roughness.

[0035] S4. Using physical rendering technology to obtain a rendered image of the target object, and train and optimize the feature network.

[0036] The method of obtaining a rendered image of a target object using physical rendering technology specifically includes: calculating the ambient light on the surface of the target object according to an environment map using image-based lighting technology (IBL), combining material items and ambient light, and obtaining a rendered image of the target object using physical rendering technology (PBR).

[0037] Training and optimizing the feature network specifically includes: Randomly initialize the environment map, obtain an initial rendered image of the target object according to the initial environment map and the initial material item output by the feature network, and use the initial rendered image to train the material item prediction module of the feature network through a material item loss function; The method comprises the following steps: obtaining the spatial coordinates of the target object in the three-dimensional scene according to the three-dimensional scene, the initial rendered image of the target object and the rectangular box of the target object's position in the 2D rendered view of the three-dimensional scene; obtaining the 2D rendered image of the three-dimensional scene and the scene depth map; constructing a fused image according to the rectangular box of the target object's position in the 2D rendered view of the three-dimensional scene, taking the 2D rendered image of the three-dimensional scene as the background and the rendered image of the target object as the foreground; estimating the depth map of the fused image by using a monocular depth estimation method, aligning the depth map of the fused image with the scene depth map by using the least squares method, and determining the depth of the foreground target object; reversely projecting the center point of the rectangular box into the three-dimensional scene, and calculating the coordinates of the center point according to the depth of the foreground target object as the spatial coordinates of the target object in the three-dimensional scene.

[0038] A scene environment map is obtained according to the spatial coordinates, a new rendered image of the target object is obtained according to the scene environment map and the material items output by the trained feature network, a pre-trained 2D image harmonization network is used to obtain a result image after harmonization of the target object and the three-dimensional scene, uncertainty item loss and harmonization loss are calculated according to the new rendered image of the target object and the harmonized result image, and the albedo parts of the uncertainty item prediction module and the material item prediction module of the feature network are trained.

[0039] S5. Obtain a multi-view scene rendering dense view and a 3D model file containing a target object according to the three-dimensional scene, the deformable tetrahedral mesh and the material items output by the trained feature network.

[0040] In another embodiment of the present invention, Figure 1 As shown, a method for generating harmonious objects in a three-dimensional scene based on physical rendering and uncertainty estimation comprises the following steps: Step 1, obtaining input: the input is a given three-dimensional scene, a text description of a target object, and a rectangular box of the position of the target object in a 2D rendering view of any viewing angle of the three-dimensional scene.

[0041] (1.1) The three-dimensional scene is a pre-trained three-dimensional scene represented by NeRF. According to the characteristics of NeRF, the sampled spatial coordinates, azimuth and tilt angles can be used to calculate the 2D rendering view and depth map of any perspective of the scene through the volume rendering integral equation, so as to be used in the subsequent harmonization stage to calculate the spatial position and scale of the object in the scene; (1.2) Generate a text description of the target object, which will be used as the input of the image generation diffusion model to calculate the SDS loss and guide the generation of 3D objects. It should be noted that in order to avoid the common "multi-head" problem of 3D images, that is, due to factors such as model instability or improper parameter settings, the generation results are complicated and inconsistent. The image generation model used in all subsequent processes is the existing multi-view image generation diffusion model (such as MVDream) pre-trained on a large-scale 3D shape dataset; (1.3) The user needs to draw a rectangular box in the 2D rendering view of a certain perspective of the 3D scene to mark the location where the target object is to be placed. When calculating the spatial position and scale of the target object in the scene, the object's position will be converted into 3D space coordinates through reverse projection. This interactive design in the 2D view enables users to complete complex 3D object generation and positioning tasks in a more intuitive and simple way, bringing users a smoother and more efficient operation experience.

[0042] Step 2, geometric learning stage: According to the text description of the generated target object, a pre-trained multi-view image generation diffusion model is used to construct a deformable tetrahedral mesh of the geometric appearance of the object based on a two-stage method. Among them, the first stage uses SDS loss to guide the generation of 3D objects represented by NeRF. The second stage extracts the deformable mesh from NeRF and fine-tunes the mesh to improve the geometric accuracy of the generated object. The two-stage geometric learning method adopted by the present invention is similar to the existing method. The first stage generates a rough geometric appearance, and the second stage refines the rough geometry. However, compared with the existing method, the present invention adopts a carefully designed optimization method, which can improve the geometric accuracy of the generated object on the one hand, and speed up the training process on the other hand to achieve rapid convergence.

[0043] Reference Figure 2 , the implementation of this step includes the following: (2.1) The first stage of geometric learning is to generate a rough geometric appearance: (2.1.1) When training a 3D object represented by NeRF, first sample the camera parameter c as input. The camera parameter represents the viewing angle (denoted by the symbol v) of the 3D target object, which is a 5-dimensional vector consisting of spatial coordinates, azimuth, and tilt angle. Then use NeRF's multi-layer perceptron (MLP) to process the input and output the color field and density field, which respectively represent the color information and density distribution of the object in space. Through the volume rendering technology, the 2D image img1 of the 3D target object represented by NeRF under the viewing angle v can be calculated; (2.1.2) The text description of the target object and the camera parameters c are used as inputs, and the pre-trained multi-view image generation diffusion model can be used to generate an image img0 that meets the text description under the viewing angle v. The multi-view image generation diffusion model gradually adds noise to the original image, and then learns to restore the data from the noise. In this process, the time step represents the degree of noise addition. In the early stage of training, the sampling time step range is set to U(0.02, 0.98), and then reduced to U(0.02, 0.5) in the later stage. This is because img1 is obtained by randomly initialized NeRF in the early stage of training, which does not meet the text description. A larger time step is conducive to NeRF learning to generate the general features of the object. Setting a smaller time step in the later stage is conducive to NeRF fine-tuning; (2.1.3) Use img0 and img1 to calculate SDS_Loss(img0, img1), train NeRF, and get the rough geometric appearance of the first stage; (2.2) The second stage of geometric learning: fine-tuning the rough geometry: (2.2.1) Extract the deformable mesh from the NeRF representation generated in the first stage. First, subtract the density field of NeRF from a non-zero constant to obtain the initial SDF. Then, use the differentiable tetrahedron marching algorithm to obtain the initial deformable tetrahedral mesh. (2.2.2) Obtain the normal map normal1 of the initial deformable tetrahedral mesh and the normal map normal2 of NeRF under the viewing angle v, and directly use the gradient of the NeRF density field to approximate the normal; (2.2.3) Using the multi-view diffusion model and normal2 as supervision, SDS_Loss(img0,normal1) and MSE_Loss(normal1, normal2) are calculated respectively for training the deformable tetrahedral mesh. In the second stage, the normal map obtained by calculating the gradient of the NeRF density field is used as supervision, which helps to speed up the convergence of the mesh and preserve the rich geometric details of NeRF.

[0044] Through the two-stage geometry modeling approach, we are able to obtain high-quality 3D objects in the geometry learning stage. These objects not only accurately reflect the input text description, but are also reasonable and accurate in geometry, laying a solid foundation for the subsequent material modeling and harmonization stages.

[0045] Step 3, material modeling phase: construct a feature network F including a material item prediction module and an uncertainty item prediction module to predict the material items of the object surface, including albedo, metallicity, and roughness. Render the 3D object appearance based on physical rendering (PBR) and image-based lighting (IBL), and use SDS loss and other loss functions to optimize the material item prediction module.

[0046] Reference Figure 3 , the implementation of this step includes the following: (3.1) To predict the material item of the object surface, first sample the points on the grid surface according to the camera parameter c, and then input them into the feature network F. The structure of the feature network F refers to Figure 4 As shown in the figure, the input data is first processed by hash grid encoding, and then passes through a linear layer to map the input to a feature space with a dimension of 64 and apply the rectified linear unit (ReLU), and then passes through a linear layer to map the input to a feature space with a dimension of 5, and further transforms the data. The left branch applies the Sigmoid Function to the output of the linear layer to obtain the output of the material item, and the right branch first passes through a linear layer to map the input to a feature space with a dimension of 1, and then applies the softplus function to obtain the output of the uncertainty item.

[0047] By using hash grid coding and MLP with fewer channels and layers, the generation speed can be greatly improved while ensuring the generation quality. The material items and uncertainty items of the target object surface are output after being processed by the feature network F. The feature network for generating the uncertainty items will be trained and used in the harmonization stage; (3.2) Physically based rendering (PBR) is used to obtain the initial rendered image img2. Specifically, the classic Cook-Torrance bidirectional reflectance distribution function (Cook-Torrance BRDF) model is used to divide the surface rendering of the target object into two independent parts: diffuse reflection and specular reflection. The diffuse reflection term and specular reflection term are calculated based on the realistic shading technology in Unreal Engine 4 combined with the predicted material terms, and finally the initial rendered image img2 of the target object under the viewing angle v is obtained.

[0048] It should be noted that in order to decouple the material and ambient lighting of the generated object so that the predicted material is more accurate, the environment map in the IBL process uses a trainable environment map tensor with a random initialization range of [0.25, 0.75]; (3.3) Loss calculation and network parameter update, using the multi-view diffusion model to generate the result image img0 as supervision, calculate SDS_Loss (img0, img2), which is used to train the feature network F and the environment map tensor for the material item. In addition, since there is no material prior information, it is challenging to directly decouple the material of the object from the ambient lighting. The present invention uses material smoothing loss and illumination loss to guide the optimization to obtain a good separation result of the material and ambient lighting: (3.3.1) Calculate the material smoothness loss. Taking albedo a as an example, the material smoothness loss Defined as: , Where a(x) represents the albedo parameter at the surface x of the object, is a random three-dimensional displacement vector of x, which follows a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. The number of dimensions that represent the albedo. Using material smoothing loss can prevent local mutations or unreasonable fluctuations, which helps to ensure that the transition of the material on the surface of the object is natural and continuous, making the generated 3D object more realistic in appearance; (3.3.2) Calculating the illumination loss: In order to reduce the illumination loss caused by separating high-intensity illumination and sharp shadows into albedo and separating the base color of the object into ambient illumination during the process of decoupling the material and ambient illumination of the object, the illumination loss designed by the present invention is Defined as: ,

[0049] in and are the diffuse and specular radiances sampled from the environment map, is a simple brightness operator that is calculated by averaging the three color channels of the input light, while is the brightness calculation formula in the Hue-Saturation-Value (HSV) model. The first term of the illumination loss approximates the illumination with the brightness of the PBR result image img2, and then propagates the gradient to and , which favors baking high-intensity lighting and sharp shadows into the environment map; while the second term penalizes variations in light color, making the environment map tend towards black and white light.

[0050] Step 4, object positioning in the scene: Based on the input 3D scene and the rectangular box where the target object is located, the target object is regarded as the foreground and the 3D scene is used as the background to construct a fused image. The depth information of the foreground object is determined by monocular depth estimation and least squares method, and then the spatial coordinates of the target object in 3D space are calculated by back projection.

[0051] Reference Figure 5 , the implementation of this step is as follows: (4.1) Construct the fused image. First, render a 2D image of the given 3D scene based on the camera parameter c. and the scene depth map, and then render the rendered image of the target object obtained in step 3 , and then the rendered image of the target object is drawn according to the rectangular frame where the target object is located. 2D image as foreground, 3D scene As the background, construct the fusion picture ; (4.2) Calculate the depth of the foreground object and use common monocular depth estimation methods (such as ominidata) to estimate the fused image Since the estimated depth is non-metric, the depth map can be aligned with the scene depth map by the least squares method. Through this alignment, the depth of the foreground object can be determined; (4.3) Calculate the spatial coordinates of the target object in the scene, and reversely project the center point p of the rectangular box into the 3D space as the spatial coordinates of the generated object. The projection formula is: ,

[0052] in Represents the coordinates of a point in three-dimensional space; is the inverse matrix of the camera extrinsic parameter matrix, where is the rotation matrix, describing the rotation attitude of the camera, is the translation vector, describing the translation position of the camera; represents the depth value of point p in the depth map; is the inverse matrix of the camera's intrinsic parameter matrix, which is used to transform the homogeneous coordinate points on the image plane into normalized coordinates in the camera coordinate system. Represents a point on the image plane The homogeneous coordinates of and is the pixel coordinate of the point on the image plane, and the 1 in the last dimension is to satisfy the representation of homogeneous coordinates.

[0053] Step 5, harmonization stage: The generated object is re-illuminated by obtaining the scene environment map according to the spatial coordinates of the object, and the harmonization between the generated object and the scene is optimized by using the estimated uncertainty and the pre-trained 2D image harmonization network to ensure that the generated object is naturally integrated into the scene.

[0054] Reference Figure 6 , the implementation of this step includes the following: (5.1) Obtain the scene environment map according to the spatial coordinates of the object, use the generated spatial coordinates of the object as the position of the camera, take the directions of the six faces of the cube as the viewing angle, render six scene images to form the environment cube map, and then construct these six images into a panoramic image, which is the scene environment map; (5.2) Randomly sample the camera parameters and obtain a new rendering result image img3 of the target object under the camera parameter condition based on physical rendering (PBR). The process is similar to (3.2), except that the scene environment map is used to replace the previous trainable environment map tensor in the IBL process, and the target object is re-illuminated to integrate the scene's ambient light into the target object; (5.3) Harmonization optimization: According to the estimated uncertainty u and the pre-trained 2D image harmonization network, the target object and the scene are harmonized and optimized. The uncertainty loss and harmonization loss are calculated respectively, and the albedo part of the uncertainty item prediction module and the material item prediction module of the feature network F are trained: (5.3.1) Calculate the uncertainty loss, which is used to train the feature network F to estimate the uncertainty. In the process of Vincent 3D, its supervision mechanism is not based on images rendered by real 3D objects, but relies on a multi-view diffusion model to calculate SDS_Loss for control. This unique supervision method may cause many unexpected situations, which in turn causes the generated objects to have poor quality and low sense of reality, and there will be problems such as disharmony in the overall presentation. The present invention introduces uncertainty to transform the original loss function. When the local loss of the calculation is large due to model instability, the estimated uncertainty will also increase. The transformed loss will make this part of the loss less affected, and when calculating the harmonized loss, local harmonization optimization can be performed according to the estimated uncertainty. The uncertainty loss function Defined as: , in represents the uncertainty term of the estimate, Indicates The first term is to transform the original ,The second term is the uncertainty regularization term, which prevents predicting infinite uncertainty, thus leading to zero loss; (5.3.2) Calculate the harmonization loss, which is used to train the feature network F to predict the albedo. Since the traditional harmonization loss has problems such as low color accuracy and poor generalization ability, the present invention uses a pre-trained 2D image harmonization network (such as PCTNet) to guide the harmonization optimization process. First, the new rendered image of the target object and the 2D rendered image of the three-dimensional scene under the same camera parameter conditions are input into the pre-trained 2D image harmonization network, and the target object and the scene harmonized result image img4 is output. Then, by calculating the difference between the new rendering result image img3 and the image img4 as a penalty, and combining the estimated uncertainty u, the harmony between the target object and the scene can be further optimized for local details. The harmonization loss function Defined as: ,

[0055] in represents the uncertainty term of the estimate, represents the mean square error loss function.

[0056] The rendering result image img3 under different camera parameter conditions and the harmonized result image img4 are used for multiple iterative training to obtain the final trained feature network.

[0057] Step 6, get output: The output is a multi-view scene rendering dense view containing the target object and a 3D model file of the target object. In these views, the occlusion relationship between the target object and the three-dimensional scene is calculated through the depth, and harmony can be maintained when switching different perspectives. Some test experimental results can be referred to Figure 7-Figure 12 As shown, various forms of 3D representations can be easily reconstructed based on these dense views. When generating the 3D model file of the target object, the deformable tetrahedral mesh is first parameterized by UV unfolding, and then the material properties are predicted based on the sampling points in the texture space using the trained feature network. Then, a seamless texture is generated by adaptive image restoration, and finally the 3D model file of the object is output.

[0058] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation, characterized in that: The following steps are involved: Given a three-dimensional scene, generate a text description of a target object and a rectangular box of the location of the target object in a 2D rendering view of the three-dimensional scene; Based on the generated text description of the target object, a deformable tetrahedral mesh of the geometric appearance of the target object is constructed based on a two-stage method using a pre-trained multi-view image generation diffusion model; Constructing a feature network, wherein the feature network includes a material item prediction module and an uncertainty item prediction module; The input of the feature network is the sampling points of the deformable tetrahedral mesh surface, and the output is the material item and uncertainty item of the object; Using physical rendering technology to obtain a rendered image of the target object, and training and optimizing the feature network; A multi-view scene rendering dense view and a 3D model file containing a target object are obtained according to the three-dimensional scene, the deformable tetrahedral mesh and the material items output by the trained feature network.

2. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 1, characterized in that: The three-dimensional scene is a pre-trained three-dimensional scene represented by a neural radiation field NeRF.

3. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 1, characterized in that: The step of generating a text description of the target object and using a pre-trained multi-view image generation diffusion model to construct a deformable tetrahedral mesh of the geometric appearance of the target object based on a two-stage method is as follows: Construct a 3D target object represented by NeRF; Taking the camera parameters as input, using NeRF's multi-layer perceptron to process the input, outputting a color field and a density field, and obtaining a 2D image img1 of the 3D object represented by NeRF under the camera parameter conditions through volume rendering technology according to the color field and the density field; Taking the text description and camera parameters of the generated target object as input, a pre-trained multi-view image generation diffusion model is used to output a 2D image img0 of the target object that meets the text description under the camera parameter conditions; Using the image img1 and the image img0 to calculate the SDS loss, training the 3D target object represented by the NeRF, and obtaining the 3D target object represented by the NeRF with a rough geometric appearance; Extracting an initial deformable tetrahedral mesh from the 3D target object represented by NeRF having a rough geometric appearance; Obtaining a normal map normal1 of the initial deformable tetrahedral mesh under the camera parameter condition and a normal map normal2 of the 3D target object represented by NeRF with a rough geometric appearance; The SDS loss is calculated using the image img0 and the normal map normal1, the MSE loss is calculated using the normal map normal1 and the normal map normal2, and the initial deformable tetrahedral mesh is trained to obtain a deformable tetrahedral mesh including a refined geometric appearance of the target object.

4. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 3, characterized in that: The specific steps of extracting the initial deformable tetrahedral mesh from the 3D target object represented by NeRF with a rough geometric appearance are: The density field of the 3D target object represented by NeRF with a rough geometric appearance is converted into a signed distance function SDF, the signed distance function SDF is subtracted from a non-zero constant to obtain an initial SDF, and an initial deformable tetrahedral mesh is obtained according to the initial SDF using a differentiable tetrahedron marching algorithm.

5. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 1, characterized in that: The feature network includes an input layer, a hash network coding layer, a linear layer, an activation function layer and an output layer. 6 . The method for generating harmonious objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 1 , wherein the material items include albedo, metallicity and roughness.

7. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 1, characterized in that: The method of obtaining a rendered image of a target object by using physical rendering technology specifically includes: Image-based lighting technology calculates the ambient light on the surface of the target object according to the environment map, combines the material items and the ambient light, and obtains the rendered image of the target object through physical rendering technology.

8. According to the method for generating harmonious objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 6, training and optimizing the feature network specifically comprises: Randomly initialize the environment map, obtain an initial rendered image of the target object according to the initial environment map and the initial material item output by the feature network, and use the initial rendered image to train the material item prediction module of the feature network through a material item loss function; Acquire the spatial coordinates of the target object in the three-dimensional scene according to the three-dimensional scene, the initial rendering image of the target object, and the rectangular frame of the position of the target object in the 2D rendering view of the three-dimensional scene; Acquire a scene environment map according to the spatial coordinates, and acquire a new rendered image of the target object according to the scene environment map and the material item output by the trained feature network; Use the pre-trained 2D image harmonization network to obtain the resulting image after the target object and the 3D scene are harmonized; The uncertainty term loss and the harmonization loss are calculated according to the new rendered image of the target object and the harmonized result image, and the albedo parts of the uncertainty term prediction module and the material term prediction module of the feature network are trained.

9. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 8, characterized in that: Acquiring the spatial coordinates of the target object in the three-dimensional scene according to the three-dimensional scene, the rendered image of the target object, and the rectangular frame of the position of the target object in the 2D rendering view of the three-dimensional scene specifically includes: Obtaining a 2D rendered image and a scene depth map of the three-dimensional scene; According to the rectangular frame of the position of the target object in the 2D rendering view of the 3D scene, a fused image is constructed with the 2D rendering image of the 3D scene as the background and the rendering image of the target object as the foreground; Estimate the depth map of the fused image using a monocular depth estimation method, align the depth map of the fused image with the scene depth map using a least squares method, and determine the depth of the foreground target object; The center point of the rectangular frame is reversely projected into the three-dimensional scene, and the coordinates of the center point are calculated according to the depth of the foreground target object as the spatial coordinates of the target object in the three-dimensional scene.

10. The method for generating harmonized objects in a three-dimensional scene based on physical rendering and uncertainty estimation according to claim 8, characterized in that: The method of using the pre-trained 2D image harmonization network to obtain a result image after the target object and the three-dimensional scene are harmonized includes the following specific steps: The newly rendered image of the target object and the 2D rendered image of the 3D scene under the same camera parameter conditions are input into the pre-trained 2D image harmonization network, and the harmonized result image of the target object is output.

Images