Method for generating hierarchical 3D scene from text, electronic equipment and storage medium

Abstract

The invention discloses a method for generating a hierarchical 3D scene through a text, electronic equipment and a storage medium, and the method comprises the following steps: initializing a multi-level 3D Gaussian, randomly selecting a camera angle to carry out micro rendering, and obtaining a multi-level Gaussian rendering graph; adding random noise into the multi-level Gaussian rendering graph, and inputting a prompt text of a to-be-generated 3D scene and the multi-level Gaussian rendering graph after noise addition into a pre-trained multi-level image diffusion model to predict noise; and iteratively optimizing the predicted noise and the random noise by using the fractional distillation loss, and reversely transmitting the gradient of the fractional distillation loss to the multi-level 3D Gaussian to obtain a final multi-level 3D scene. And complex hierarchical scene-level 3D generation can be realized.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and graphics, and in particular to a method, electronic device and storage medium for generating a hierarchical 3D scene from text. Background Art

[0002] The technology of generating 3D scenes from text has great potential in fields such as games, robotics, and VR / AR. Current 3D creation research can be divided into two major categories: reasoning-only 3D native methods and optimization-based 2D lifting methods. In theory, 3D native methods have the potential to generate 3D consistent assets in seconds, but at the cost of extensive training on large-scale 3D datasets. Creating such datasets requires a lot of manpower, and even so, these datasets still suffer from limited diversity and authenticity.

[0003] On the other hand, Score Distillation Sampling (SDS) addresses the limitations of 3D data by extracting 3D geometry and appearance from a powerful 2D diffusion model, which has inspired the development of 2D lifting methods. To deal with the inconsistency and ambiguity brought by SDS supervision, Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) are often adopted, which has strong capabilities in modeling rich 3D information. Despite the continuous improvement in generation quality, the optimization time of these methods often takes hours due to the high cost of NeRF rendering, limiting their large-scale application in the real world. The occupancy clipping technique used to accelerate NeRF does not work well in the generation setting supervised by the ambiguous SDS loss, but in the reconstruction setting.

[0004] The Chinese patent application with publication number CN117953180A proposes an efficient and consistent text-to-3D dual-mode multi-view latent variable diffusion generation method, which uses a two-dimensional latent variable diffusion model to jointly train a dual-mode multi-view latent variable diffusion model; obtains the text of the three-dimensional object to be generated, and based on the efficiency and consistency trade-off strategy, uses the dual-mode multi-view latent variable diffusion model to infer the text to generate a first three-dimensional object; finally, the surface texture of the first three-dimensional object is optimized to generate a second three-dimensional object. However, this method can only generate object-level 3D products well, and is not suitable for complex hierarchical scene-level 3D generation. Summary of the invention

[0005] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and to propose a method, electronic device and storage medium for generating a hierarchical 3D scene from text, which can realize complex hierarchical scene-level 3D generation.

[0006] The present invention adopts the following technical solution:

[0007] A method for generating a hierarchical 3D scene from text comprises the following steps:

[0008] Initialize a multi-level 3D Gaussian, randomly select a camera angle for differentiable rendering, and obtain a multi-level Gaussian rendering image;

[0009] Adding random noise to the multi-level Gaussian rendering image, and inputting the prompt text of the 3D scene to be generated and the multi-level Gaussian rendering image after adding noise into a pre-trained multi-level image diffusion model to predict the noise;

[0010] The predicted noise and the random noise are iteratively optimized using fractional distillation loss, and the gradient of the fractional distillation loss is back-transferred to the multi-level 3D Gaussian to obtain a final multi-level 3D scene.

[0011] Furthermore, the multi-level 3D Gaussian is initialized to a randomly sampled position within a sphere and scaled by unit; the multi-level 3D Gaussian includes a foreground 3D Gaussian and a background 3D Gaussian, and the foreground 3D Gaussian and the background 3D Gaussian are initialized to randomly sampled positions within spheres of different sizes.

[0012] Furthermore, the foreground 3D Gaussian is subjected to differentiable rendering to obtain a foreground Gaussian rendering image, the background 3D Gaussian is subjected to differentiable rendering to obtain a background Gaussian rendering image, and the foreground 3D Gaussian and the background 3D Gaussian are subjected to differentiable rendering together to obtain the global Gaussian rendering image.

[0013] Furthermore, the optimizable parameters of the multi-level 3D Gaussian are Θ, Θ i ={x i ,s i ,q i ,α i ,c i} is the optimizable parameter of the i-th multi-level 3D Gaussian, x i Indicates the center, s i represents the scaling factor, q i represents the rotation quaternion, α i Indicates opacity, c i represents the color characteristics used for volume rendering, represents a set of real numbers; the gradient of the fractional distillation loss is back-transferred to the multi-level 3D Gaussian to optimize the optimizable parameters.

[0014] Furthermore, the differentiable rendering is performed by projecting the multi-level 3D Gaussian onto a graphics plane as a 2D Gaussian, and then performing a depth sequential volume rendering from front to back for each pixel on the image plane.

[0015] Furthermore, in the iterative optimization process, each time the 3D Gaussian of the corresponding level is subjected to differentiable rendering to obtain a Gaussian rendered image, and at the same time the time step t is linearly reduced, and random noise is weightedly added to the Gaussian rendered image of the corresponding level to obtain a noisy Gaussian rendered image, and the noisy Gaussian rendered image is used as the input of the multi-layer image diffusion model to predict the potential image corresponding to the random noise.

[0016] Furthermore, the gradient of the fractional distillation loss It is expressed as:

[0017]

[0018] Θ is the optimizable parameter of the multi-level 3D Gaussian, E t,p,ε is the expected value of t, p, ε, t is the time step, p is the camera posture corresponding to the differentiable rendering, ε is random noise, and ω(t) is the weight function that varies with t; ε φ is the pre-trained multi-level image diffusion model, is the Gaussian rendering image after adding noise to the corresponding level l, y l is the prompt text output by the CLIP text encoder of the corresponding level l, ε is random noise, and the corresponding level l contains foreground f, background b, and global bl.

[0019] Further, the gradient of the fractional distillation loss is calculated The optimizable parameter θ is updated by back propagation to the multi-level 3D Gaussian. The updated multi-level 3D Gaussian is used as the initial scene for the next update, and it is continuously iterated until the preset number of updates is completed to obtain the final optimized multi-level 3D Gaussian.

[0020] An electronic device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for generating a 3D scene from text.

[0021] A storage medium stores a computer program / instruction thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of a method for generating a 3D scene from text as described above.

[0022] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention initializes a multi-level 3D Gaussian, randomly selects camera angles for differentiable rendering, obtains a multi-level Gaussian rendering image; adds random noise to the multi-level Gaussian rendering image, inputs the prompt text of the 3D scene to be generated and the multi-level Gaussian rendering image after noise into a pre-trained multi-level image diffusion model to predict noise; uses fractional distillation loss to iteratively optimize the predicted noise and random noise, and reversely transfers the gradient of the fractional distillation loss to the multi-level 3D Gaussian to obtain the final multi-level 3D scene. The automation and optimization of generating hierarchical 3D scenes from text is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of the present invention;

[0025] Figure 2 The main principle diagram of the method of the present invention is shown in FIG.

[0026] Figure 3 This is the schematic diagram of the multi-level diffusion model.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0028] The present invention is further described below through specific implementation modes.

[0029] See also Figure 1 and Figure 2 , a method for generating a hierarchical 3D scene from text, comprising the following steps:

[0030] 1) Initialize a multi-level 3D Gaussian and randomly select camera angles for differentiable rendering to obtain a multi-level Gaussian rendering image.

[0031] Among them, the multi-level 3D Gaussian includes a foreground 3D Gaussian and a background 3D Gaussian. The foreground 3D Gaussian is subjected to differentiable rendering to obtain a foreground Gaussian rendering image, the background 3D Gaussian is subjected to differentiable rendering to obtain a background Gaussian rendering image, and the foreground 3D Gaussian and the background 3D Gaussian are subjected to differentiable rendering together to obtain a global Gaussian rendering image. That is, the multi-level Gaussian rendering image includes a foreground Gaussian rendering image, a background Gaussian rendering image, and a global Gaussian rendering image.

[0032] Initialization refers to initializing the multi-level 3D Gaussian to randomly sample positions within the sphere and scale by unit without rotation. Specifically, sample from random camera poses p around the center of the object and the scene, and render the RGB image i of the current view p The multi-level 3D Gaussian includes a foreground 3D Gaussian and a background 3D Gaussian. The foreground 3D Gaussian and the background 3D Gaussian are initialized at randomly sampled positions in spheres of different sizes. The foreground 3D Gaussian is initialized in a smaller sphere, and the background 3D Gaussian is initialized in a larger sphere.

[0033] Differentiable rendering is performed by projecting a multi-level 3D Gaussian onto the graphics plane as a 2D Gaussian, and then performing a front-to-back depth-sequential volume rendering for each pixel on the image plane to evaluate the final color and alpha.

[0034] In the present invention, the optimizable parameters of the multi-level 3D Gaussian are Θ, Θ i ={x i ,s i ,q i ,α i ,c i} is the parameter of the i-th foreground 3D Gaussian or background 3D Gaussian, x i Indicates the center, s i represents the scaling factor, q i represents the rotation quaternion, α i Indicates opacity, c i represents the color characteristics used for volume rendering, represents a set of real numbers; the optimizable parameters are optimized during the iterative optimization process. These optimizable parameters of the 3D Gaussian are periodically densified during each iterative optimization process.

[0035] 2) Adding random noise to the multi-level Gaussian rendering, the prompt text of the 3D scene to be generated and the noisy multi-level Gaussian rendering are input into a pre-trained multi-level image diffusion model to predict the noise.

[0036] Among them, see Figure 3 The multi-level image diffusion model uses the Unet neural network, which uses the attention sharing mechanism and LoRAs to expand the basic image diffusion model into a multi-layer model. The potential image of the foreground is represented as x f , the background latent image is represented as x b , the global latent image is represented as x bl For each attention layer in the base image diffusion model, the three channels are merged into a jointly optimized large model by concatenating the {key, query, value} vectors of all activated three latent images. The combined random noise is expressed as ε = [ε f ,ε b ,ε bl ], the merged CLIP text embedding is represented as y = [y f ,y b ,y bl ], the combined multi-level diffusion model is expressed as ε φ, then its loss is expressed as:

[0037]

[0038] Training loss Until the multi-layer image diffusion model converges, Indicates x f ,x b ,x bl ,t,ε takes the expected value and t is the time step.

[0039] Add random noise to the multi-level Gaussian rendering of step 1) to obtain a noisy multi-level Gaussian rendering, including a noisy foreground Gaussian rendering, a noisy background Gaussian rendering, and a noisy global Gaussian rendering; input the text of the 3D scene to be generated into the CLIP text encoder, output the text feature vector of the prompt text, input the text feature vector and the noisy multi-level Gaussian rendering into the multi-layer image diffusion model, and obtain the predicted noise corresponding to the foreground latent image, the background latent image, and the global latent image.

[0040] 3) The predicted noise and random noise are iteratively optimized using the fractional distillation loss, and the gradient of the fractional distillation loss is back-propagated to the multi-level 3D Gaussian to obtain the final multi-level 3D scene.

[0041] In the iterative optimization process, steps 1) and 2) are repeated. Each time, the 3D Gaussian of the corresponding level is rendered differently to obtain a Gaussian rendered image, and the time step t is linearly reduced. The random noise is weightedly added to the Gaussian rendered image of the corresponding level to obtain a noisy Gaussian rendered image. The noisy Gaussian rendered image is used as the input of the multi-layer image diffusion model to predict the potential image corresponding to the random noise.

[0042] Gradient of fractional distillation loss It is expressed as:

[0043]

[0044] Θ is the optimizable parameter of the multi-level 3D Gaussian, E t,p,ε is the expected value of t, p, ε, t is the time step, p is the camera posture corresponding to the differentiable rendering, ε is random noise, and ω(t) is the weight function that varies with t; ε φ is a pre-trained multi-level image diffusion model, is the Gaussian rendering image after adding noise to the corresponding level l, y l is the prompt text output by the CLIP text encoder of the corresponding level l, ε is random noise, and the corresponding level l contains foreground f, background b, and global bl.

[0045] The gradient of the fractional distillation loss will be calculated The parameter Θ can be updated by back-propagating to the multi-level 3D Gaussian, foreground rendering Gradient optimization of foreground 3D Gaussian optimizable parameters, background rendering Gradient optimized background 3D Gaussian optimizable parameters, global rendering The gradient optimizes the optimizable parameters of the foreground and background 3D Gaussians; the updated multi-level 3D Gaussian is used as the initial scene for the next update, and it is continuously iterated until the preset number of updates is completed to obtain the final optimized multi-level 3D Gaussian, and the optimized multi-level 3D Gaussian constitutes the final multi-level 3D scene.

[0046] The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement steps of a method for generating a 3D scene from text.

[0047] The present invention also provides a storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of a method for generating a 3D scene from text are implemented.

[0048] The storage medium of the present invention may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0049] The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method in the above embodiment.

[0050] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0051] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0052] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A method for generating a hierarchical 3D scene from text, characterized in that: The following steps are involved: Initialize a multi-level 3D Gaussian, randomly select a camera angle for differentiable rendering, and obtain a multi-level Gaussian rendering image; Adding random noise to the multi-level Gaussian rendering image, and inputting the prompt text of the 3D scene to be generated and the multi-level Gaussian rendering image after adding noise into a pre-trained multi-level image diffusion model to predict the noise; The predicted noise and the random noise are iteratively optimized using fractional distillation loss, and the gradient of the fractional distillation loss is back-transferred to the multi-level 3D Gaussian to obtain a final multi-level 3D scene.

2. A method for generating a hierarchical 3D scene from text according to claim 1, characterized in that: The multi-level 3D Gaussian is initialized to a randomly sampled position in a sphere and scaled by unit; the multi-level 3D Gaussian includes a foreground 3D Gaussian and a background 3D Gaussian, and the foreground 3D Gaussian and the background 3D Gaussian are initialized to randomly sampled positions in spheres of different sizes.

3. A method for generating a hierarchical 3D scene from text as claimed in claim 2, characterized in that: The foreground 3D Gaussian is subjected to differentiable rendering to obtain a foreground Gaussian rendering image, the background 3D Gaussian is subjected to differentiable rendering to obtain a background Gaussian rendering image, and the foreground 3D Gaussian and the background 3D Gaussian are subjected to differentiable rendering together to obtain the global Gaussian rendering image.

4. The method for generating a hierarchical 3D scene from text according to claim 1, characterized in that: The optimizable parameters of the multi-level 3D Gaussian are Θ, Θ i ={x i ,s i ,q i ,α i ,c i } is the optimizable parameter of the i-th multi-level 3D Gaussian, x i Indicates the center, s i represents the scaling factor, q i represents the rotation quaternion, α i Indicates opacity, c i represents the color characteristics used for volume rendering, represents a set of real numbers; the gradient of the fractional distillation loss is back-transferred to the multi-level 3D Gaussian to optimize the optimizable parameters.

5. The method for generating a hierarchical 3D scene from text according to claim 1, characterized in that: The differentiable rendering is performed by projecting the multi-level 3D Gaussian onto a graphics plane as a 2D Gaussian, and then performing a front-to-back depth sequential volume rendering on each pixel on the image plane.

6. The method for generating a hierarchical 3D scene from text according to claim 1, characterized in that: During the iterative optimization process, each time the 3D Gaussian of the corresponding level is rendered differently to obtain a Gaussian rendered image, and the time step t is linearly reduced. The random noise is weightedly added to the Gaussian rendered image of the corresponding level to obtain a noisy Gaussian rendered image, and the noisy Gaussian rendered image is used as the input of the multi-layer image diffusion model to predict the potential image corresponding to the random noise.

7. The method for generating a hierarchical 3D scene from text according to claim 6, characterized in that: The gradient of the fractional distillation loss It is expressed as: Θ is the optimizable parameter of the multi-level 3D Gaussian, E t,p,ε is the expected value of t, p, ε, t is the time step, p is the camera posture corresponding to the differentiable rendering, ε is random noise, and ω(t) is the weight function that varies with t; ε φ is the pre-trained multi-level image diffusion model, is the Gaussian rendering image after adding noise to the corresponding level l, y l is the prompt text output by the CLIP text encoder of the corresponding level l, ε is random noise, and the corresponding level l contains foreground f, background b, and global bl.

8. The method for generating a hierarchical 3D scene from text according to claim 7, characterized in that: The gradient of the fractional distillation loss will be calculated The optimizable parameter θ is updated by back propagation to the multi-level 3D Gaussian. The updated multi-level 3D Gaussian is used as the initial scene for the next update, and it is continuously iterated until the preset number of updates is completed to obtain the final optimized multi-level 3D Gaussian.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for generating a 3D scene from text according to any one of claims 1 to 8.

10. A storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of a method for generating a 3D scene from text as claimed in any one of claims 1 to 8 are implemented.

Images