Method and system for generating multiple 3d objects using text and 2d bounding boxes

Abstract

The application discloses a method and system for simultaneously generating multiple 3D objects by using text and 2D bounding boxes, comprising a 3D content enhancement module, a layout control module, a multi-view consistency control module and a layout and multi-view knowledge integration module. The 3D content enhancement module is used to improve the quality of the generated 3D content, the layout control module is used to control the position between the multiple 3D objects simultaneously generated, the multi-view consistency control module is used to maintain the multi-view consistency of the multiple objects generated, and the layout and multi-view knowledge integration module is used to unify the multiple distilled knowledge and further enhance the quality of the generated objects. The application does not need to change the structure and parameters of the pre-trained model, and simultaneously solves the problems that the existing method cannot understand the complex logical relationship in the input text and can only generate a single object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method and system for simultaneously generating multiple 3D objects using text and 2D bounding boxes. BACKGROUND

[0002] With the continuous development of the gaming, VR and AR industries, the demand for 3D content is increasing. However, manually creating 3D content using manpower is a very expensive task, so there is an urgent need to solve the problem of quickly generating high-quality 3D content that meets people's needs using artificial intelligence technology.

[0003] Current methods for generating 3D content using text are mainly divided into optimization-based methods and feedforward-based methods. The feedforward-based method usually uses fractional distillation sampling to extract prior knowledge from a pre-trained 2D diffusion model to a 3D implicit representation, but its 3D content generation is slow and also faces the problem of multiple faces. With the emergence of large 3D datasets, the feedforward-based method uses an improved Transformer architecture and a feedforward training method to achieve end-to-end text-to-3D content generation, but this method requires a large amount of computing resources for model training in 3D generation, which is costly.

[0004] More importantly, existing work on text-to-three-dimensional object generation mainly focuses on generating a single object, and lacks control over the final result of the generated object. For example, when a text prompt containing multiple objects is provided (such as "three donuts"), current text-to-three-dimensional content generation methods have difficulty producing satisfactory results. The position of the generated 3D object exhibits various possibilities, and the position of the single object does not meet the desired requirements. In addition, these methods often encounter challenges such as object fusion and omission. The final count of the generated 3D object is often incorrect, and the characteristics of multiple objects are often mixed together. SUMMARY

[0005] The purpose of the present application is to provide a method and system for simultaneously generating multiple 3D objects using text and 2D bounding boxes, which solves the problem that existing methods cannot understand complex logical relationships in input text and can only generate a single object.

[0006] The technical solution to achieve the purpose of the present application is:

[0007] A method for simultaneously generating multiple 3D objects using text and 2D bounding boxes, comprising:

[0008] Step 1, obtain input text y and bounding box l, extract input text and bounding box features (y, l), and generate a 2D picture through a pre-trained layout control diffusion model Initialize multiple 3D objects g(θ) through a multi-layer perception model;

[0009] Step 2, render a 2D picture x = g(θ, c) from g(θ) at a view angle c and add normal distribution noise;

[0010] Step 3, based on the input text y and the bounding box l, extract layout prior knowledge from the pre-trained layout control diffusion model using distillation, and optimize the generated multiple 3D objects;

[0011] Step 4, render a 2D picture x = g(θ, c) from g(θ) at a view angle c and add normal distribution noise; As a pre-trained multi-view Figure 1 diffusion model, extract multi-view prior knowledge using distillation, and optimize the generated multiple 3D objects;

[0012] Step 5, update the low-rank matrix in the pre-trained layout diffusion model using the mean square error loss function to obtain the 3D object distribution;

[0013] Step 6, use layout multi-view variational distillation to unify layout prior knowledge and multi-view prior knowledge based on the 3D object distribution, and optimize the multiple 3D objects;

[0014] Step 7, repeat steps 2-6, based on the 2D noise-added rendering picture design and the calculation of the layout control diffusion model, multi-view Figure 1 diffusion model and layout diffusion model loss function, optimize the multiple 3D objects.

[0015] Further, the layout diffusion model is adjusted through cola.

[0016] Further, the layout control diffusion model takes the input text y and the bounding box l feature as guidance, denoises the 2D noise-added rendering picture in step 2, calculates the loss value between the removed noise and the added noise, and distills the knowledge, the loss function and the loss gradient of the layout control diffusion model are designed as:

[0017]

[0018] where w(t) is the guidance weight of the t-th step of stable diffusion, ∈ layout is the denoiser of the pre-trained layout control diffusion model, x t is the encoding added with noise in the t-th step of stable diffusion, t is the number of steps of stable diffusion in the optimization process, E t∈，c represents expectation, ∈ is the added noise.

[0019] Further, the multi-view Figure 1The consistency diffusion model takes the 2D picture generated in step 1 as a guide, denoises the 2D picture with added noise obtained in step 2 at the corresponding view, calculates a loss value between the removed noise and the added noise, distills knowledge, and optimizes the plurality of 3D objects Figure 1 The loss function and loss gradient of the consistency diffusion model are designed as:

[0020]

[0021] Wherein, ∈ 0123 is a denoiser of the pre-trained multi-view Figure 1 The denoiser of the consistency diffusion model.

[0022] Further, the update target of the layout diffusion model is:

[0023]

[0024] Wherein, ∈ sola is a denoiser of the pre-trained layout diffusion model.

[0025] Further, layout multi-view variational distillation is used to unify layout prior knowledge and multi-view prior knowledge based on 3D object distribution, specifically including:

[0026]

[0027] A system for simultaneously generating multiple 3D objects using text and 2D bounding boxes includes a 3D content enhancement module, a layout control module, a multi-view control module, a layout and multi-view knowledge integration module, and a 3D content description module; wherein:

[0028] The 3D content description module is used to describe the plurality of 3D objects to be generated, and the module initializes a plurality of 3D objects g(θ) through a multi-layer perception model, and renders a 2D picture x from them at a view c and adds normally distributed noise;

[0029] The layout control module uses distillation to extract layout prior knowledge from a pre-trained layout control diffusion model based on the input text y and the bounding box l, and generates a 2D picture and participates in optimizing the plurality of 3D objects;

[0030] The multi-view control module distills the 2D picture as a pre-trained multi-view Figure 1 consistency diffusion model to extract multi-view prior knowledge and optimize the plurality of 3D objects;

[0031] The 3D content enhancement module uses a mean square error loss function to update a low-rank matrix in the pre-trained layout diffusion model; and obtains a 3D object distribution;

[0032] The layout and multi-view knowledge integration module uses layout multi-view variational distillation, unifies layout prior knowledge and multi-view prior knowledge based on 3D object distribution, and optimizes multiple 3D objects.

[0033] Compared with the prior art, the beneficial effects of the present application are that the present application adds a layout control module, a consistency control module, a 3D content enhancement module and a layout and multi-view knowledge unification module in the existing optimization-based 3D object generation system to generate 3D content conforming to the text description. Figure 1 The generation system and method do not need to change the structure and parameters of the pre-trained model, and at the same time solve the problems that the existing method cannot understand the complex logical relationship in the input text and can only generate a single object. BRIEF DESCRIPTION OF DRAWINGS

[0034] To further clarify the above and other advantages and features of the embodiments of the present application, more particular description of the embodiments of the present application will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the application and are therefore not to be considered limiting of its scope. In the drawings, the same or corresponding elements are denoted by the same or similar reference signs.

[0035] Figure 1 A structural schematic diagram of an existing tuning system is shown;

[0036] Figure 2 A structural schematic diagram of a system for simultaneously generating multiple 3D objects using text and 2D bounding boxes according to an embodiment of the present application is shown;

[0037] Figure 3 A flowchart of a generation method for simultaneously generating multiple 3D objects using text and 2D bounding boxes according to an embodiment of the present application is shown;

[0038] Figure 4 A qualitative comparison experiment diagram of a generation method for simultaneously generating multiple 3D objects using text and 2D bounding boxes according to an embodiment of the present application and an existing method is shown. DETAILED DESCRIPTION

[0039] In the following description, the present application is described with reference to various embodiments. Those skilled in the art will recognize, however, that the various embodiments can be practiced without one or more of the specific details, or with other replacement and / or additional methods or components, and in some instances, some well-known structures or operations are not shown or described in detail in order to avoid obscuring the inventive aspects. Similarly, for purposes of explanation, specific numbers and configurations are set forth in order to provide a thorough understanding of the embodiments of the present application. However, the present application can be practiced without the specific details (e.g., the specific number of components and configurations) and in some instances, some well-known structures or operations are not shown or described in detail in order to avoid obscuring the inventive aspects. Furthermore, it is to be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0040] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0041] It should be noted that the steps of the methods described in the embodiments of the present application are performed in a particular order, but this is merely for the purpose of illustrating the specific embodiments, and does not limit the order of the steps. Conversely, in different embodiments of the present application, the order of the steps can be adjusted according to the actual needs of the adjustment.

[0042] The scheme of the present application will be further described below in conjunction with the embodiment drawings.

[0043] Figure 1 A structural schematic diagram of an existing optimization-based generation system is shown. In the figure, the gray modules represent modules or parameters that need to be adjusted. As shown, the existing generation system includes a pre-trained model 101 and a 3D content description module 102. The pre-trained model 101 usually adopts a stable diffusion model, and the 3D content description module 102 usually adopts a structure based on a multi-layer perception machine. For different input texts, the existing optimization-based text generation 3D object system needs to distill the pre-training knowledge in the pre-trained model to the 3D content description module through a distillation method. However, this method cannot solve the multi-face consistency problem of the generated content, and when the input text contains multiple objects, the existing method cannot understand the logical relationship contained therein, and the generated multiple 3D objects cannot meet people's expectations. Figure 1

[0044] A structural schematic diagram of a system for simultaneously generating multiple 3D objects using text and 2D bounding boxes according to an embodiment of the present application is shown. In the figure, the gray modules represent modules or parameters that need to be adjusted. As shown, the system includes a pre-trained model 101, a 3D content description module 102, and a 2D content description module 103. The pre-trained model 101 usually adopts a stable diffusion model, the 3D content description module 102 usually adopts a structure based on a multi-layer perception machine, and the 2D content description module 103 usually adopts a structure based on a multi-layer perception machine. For different input texts, the system needs to distill the pre-training knowledge in the pre-trained model to the 3D content description module and the 2D content description module through a distillation method. However, this method cannot solve the multi-face consistency problem of the generated content, and when the input text contains multiple objects, the existing method cannot understand the logical relationship contained therein, and the generated multiple 3D objects cannot meet people's expectations. Figure 2 Figure 2 ​As shown, a system for generating multiple 3D objects from text and 2D bounding boxes simultaneously includes a 3D content enhancement module 200, a layout control module 201, a multi-view control module 202, a layout and multi-view knowledge integration module 203, and a 3D content description module 204. Compared with the existing generation system for generating 3D objects from text, the system for generating multiple 3D objects from text and 2D bounding boxes simultaneously provided by the present application removes the module 101 and adds the 3D content enhancement module 200, the layout control module 201, the multi-view control module 202, the layout and multi-view knowledge integration module 203, and the 3D content description module 204. This makes it possible to generate multiple 3D objects from specified text and 2D bounding boxes simultaneously, and solves the problems of the existing method that cannot understand complex object relationships in the text, that the generated 3D content has inconsistent multiple faces, and that objects are fused.

[0045] The 3D content description module 204 contains a neural radiance field for describing the generated 3D objects. The description structure and / or method used by the 3D content description module 204 is basically the same as that of the prior art (a commonly used multi-layer perception model), and will not be described here.

[0046] The 3D content enhancement module 200 is used to represent the distribution of the generated 3D content. Since the fine-tuned diffusion model can be used to fit the distribution of the 3D content corresponding to the input text and 2D bounding boxes, in an embodiment of the present application, the 3D content enhancement module includes a fine-tuned layout diffusion model. The layout diffusion model uses the GLIGEN model, and the layout diffusion model is fine-tuned by cola, which can gradually approach the distribution of the generated 3D objects.

[0047] The layout control module 201 is used to control the position of the generated 3D content. The layout control diffusion model uses the GLIGEN model. Since the layout control diffusion model is trained using a large amount of data and contains powerful layout control prior knowledge, in an embodiment of the present application, the layout control module includes a pre-trained layout control diffusion model. By distilling the prior knowledge of the layout control diffusion model, we can simultaneously generate multiple 3D objects that conform to the input bounding boxes. The loss function and gradient of the layout control diffusion model are as follows:

[0048]

[0049] where w(t) is the guide weight of the t-th step of stable diffusion, ∈layoutis the denoiser of the pre-trained layout control diffusion model, x tTo stabilize the diffusion-added noise encoding in the t-th step, t is the number of steps of the stable diffusion in the optimization process. Specifically, guided by the text and bounding box features, the 2D noisy rendering picture rendered from the 3D representation module is denoised, and the loss value is calculated by the difference between the removed noise and the added noise to distill knowledge.

[0050] The multi-view Figure 1 consistency control module 202 is used to control the multi-view Figure 1 consistency of the generated plurality of 3D objects. Since the diffusion model is trained using a large amount of training data, it already contains a large amount of 3D prior knowledge about the visual world, including various images from different perspectives. Therefore, the multi-view diffusion model injects camera viewpoint information into the input by fine-tuning the diffusion model. Thus, images with multi-view consistency can be generated. Therefore, in an embodiment of the present application, the multi-view Figure 1 consistency control module includes a pre-trained multi-view Figure 1 consistency diffusion model, by distilling the prior knowledge of the multi-view Figure 1 consistency control model, we can generate multiple multi-view Figure 1 consistent 3D objects at the same time, solving the multi-face problem.

[0051] The multi-view Figure 1 consistency diffusion model adopts stab1e-zero123, a pre-trained multi-view Figure 1 consistency diffusion model, the loss function and gradient are designed as:

[0052]

[0053] Where, ∈ 0123 is the denoiser of the pre-trained multi-view Figure 1 consistency diffusion model, and the 2D prior picture generated by the layout control module is used as guidance to denoise the 2D noisy rendering picture rendered from the 3D representation module in the corresponding view, and the loss value is calculated by the difference between the removed noise and the added noise to distill knowledge.

[0054] The layout and multi-view knowledge integration module 203 is used to integrate the layout prior knowledge and multi-view Figure 1 consistency prior knowledge distilled from the layout control diffusion model and the multi-view Figure 1 consistency control module. Since there is a difference between the layout distribution and the multi-view Figure 3 consistency distribution, the present application considers using a loss function to describe the distance between the two and the distribution of the generated 3D content, therefore, in an embodiment of the present application, the layout and multi-view knowledge integration module contains a loss function. By unifying the layout prior knowledge and the multi-view prior knowledge, the present application further enhances the quality of the generated 3D content.

[0055] based on the generation system as described above, Figure 3 A flowchart of a method for simultaneously generating multiple 3D objects using text and 2D bounding boxes according to an embodiment of the present application is shown. As shown in Figure 1 A method for simultaneously generating multiple 3D objects using text and 2D bounding boxes, according to an embodiment of the present application, comprises:

[0056] Step 300, initialize a neural radiance field g(θ) with parameter θ.

[0057] Step 301, render a 2D image x = g(θ, c) from g(θ) at a view angle c, and add Gaussian noise ∈ to the image.

[0058] Step 302, use the pre-trained layout-controlled diffusion model in module 201 to generate a 2D image as the 2D prior of module 202. Next, use layout score distillation to extract layout prior knowledge from the pre-trained layout-controlled diffusion model in module 201. The formula of layout score distillation is as follows:

[0059]

[0060] where (y, l) represents the input text and 2D bounding box, ∈ layout (x t , t, (y, l)) is the noise removed by the denoiser of the pre-trained layout-controlled diffusion model in module 201 for the noisy image.

[0061] Step 303, use multi-view Figure 4 consistency distillation to use the view angle information c and the pre-trained multi-view diffusion model in module 202 to denoise the noisy rendered image x at the corresponding view angle, and the formula of multi-view score distillation is as follows:

[0062]

[0063] where is the noise removed by the denoiser of the pre-trained multi-view diffusion model in module 201 for the noisy image at view angle c.

[0064] Step 304, use the mean square error loss function to update the low-rank matrix in module 200, and at every thousand steps, weight fusion is performed between the weights of the low-rank matrix and the weights of the original model, and the optimization objective is:

[0065]

[0066] wherein, ∈ cola (x t , t, (y, l, c)) is the noise removed by the denoiser using the diffusion model being fine-tuned in the module 200.

[0067] Step 305, using layout multi-view variational distillation, unifies layout prior knowledge and multi-view prior knowledge, further enhancing the quality of the generated 3D content:

[0068]

[0069] Repeat the above steps 301 to 305, calculate the multi-loss function to optimize the quality of the generated 3D multi-object in the 3D content description module.

[0070] As shown in Table 4, after testing on the multi-object generation dataset on T3Bench, the performance of the generation system and method provided by the application in various tasks such as generation quality, text alignment and user research is far superior to the classic text-based 3D object generation method, and the qualitative experimental results are as shown in ​ However, the generation system and method provided by the application have a significantly higher understanding of complex relationships in text than the classic text-based generation method.

[0071] Table 4 Qualitative comparison of experimental results of the generation method of the application and the existing method

[0072]

[0073] A computer readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, performs the steps of the method.

[0074] Although the above describes embodiments of the application, it should be understood that they are presented only as examples, not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made without departing from the spirit and scope of the application. Therefore, the width and scope of the application disclosed herein should not be limited by the above disclosed exemplary embodiments, but should only be defined according to the appended claims and their equivalent replacements.

Claims

1. A method for simultaneously generating multiple 3D objects using text and 2D bounding boxes, characterized in that: include: Step 1: Get the input text y and bounding box l, extract the input text and bounding box features (y, l), and generate a 2D image through the pre-trained layout control diffusion model Initialize multiple 3D objects g(θ) through the multi-layer perceptron model; Step 2: Render a 2D image x = g(θ, c) from g(θ) at a viewing angle c and add normally distributed noise. Step 3: Based on the input text y and bounding box l, distillation is used to extract layout prior knowledge from the pre-trained layout control diffusion model to optimize the generated multiple 3D objects; Step 4: Convert the 2D image As a priori for the pre-trained multi-view consistency diffusion model, distillation is used to extract multi-view prior knowledge and optimize the generated multiple 3D objects; Step 5: Use the mean square error loss function to update the low-rank matrix in the pre-trained layout diffusion model to obtain the 3D object distribution; Step 6: Use layout multi-view variational distillation to optimize multiple 3D objects based on 3D object distribution, unify layout prior knowledge and multi-view prior knowledge; Step 7: Repeat steps 2-6 to design and calculate the loss functions of the layout control diffusion model and the multi-view consistency diffusion model based on the 2D noisy rendered image in step 2, and optimize multiple 3D objects. Using layout multi-view variational distillation, based on 3D object distribution, we unify layout prior knowledge and multi-view prior knowledge, specifically including: Among them, E t,∈,c represents the expectation, w(t) is the guide weight of the t-th step of stable diffusion, ∈ layout is the denoiser of the pre-trained layout control diffusion model, ∈ 0123 is the denoiser of the pre-trained multi-view consistent diffusion model, ∈ cola is the denoiser of the pre-trained layout diffusion model, x t is the code for adding noise in the t-th step of stable diffusion, and t is the number of stable diffusion steps in the optimization process.

2. The method of claim 1, wherein: The layout diffusion model is adjusted by COLA.

3. The method of simultaneously generating multiple 3D objects using text and 2D bounding boxes according to claim 1, characterized in that: The layout control diffusion model uses the input text y and bounding box l features as guidance to denoise the 2D noisy rendered image in step 2. The loss value distillation knowledge is calculated by calculating the difference between the removed noise and the added noise. The loss function and loss gradient of the layout control diffusion model are designed as: Where w(t) is the guide weight of the t-th step of stable diffusion, ∈ layout is a denoiser for the pre-trained layout control diffusion model, x t is the code for adding noise in the tth step of stable diffusion, t is the number of stable diffusion steps in the optimization process, E t,∈,c represents the expectation, and ε is the added noise.

4. The method of simultaneously generating multiple 3D objects using text and 2D bounding boxes according to claim 3, wherein: The multi-view consistent diffusion model uses the 2D image generated in step 1 as a guide, denoises the 2D noisy rendered image obtained in step 2 at the corresponding viewing angle, and calculates the loss value distillation knowledge based on the difference between the removed noise and the added noise. The loss function and loss gradient of the multi-view consistent diffusion model are designed as follows: Among them, ε 0123 Denoiser for pre-trained multi-view consistent diffusion models.

5. The method of simultaneously generating multiple 3D objects using text and 2D bounding boxes according to claim 4, characterized in that: The update target of the layout diffusion model is: Among them, ∈ cola Denoiser for the pre-trained layout diffusion model.

6. A system for implementing the method of simultaneously generating multiple 3D objects using text and 2D bounding boxes as described in any one of claims 1 to 5, characterized in that: It includes a 3D content enhancement module, a layout control module, a multi-view control module, a layout and multi-view knowledge integration module, and a 3D content description module; wherein: The 3D content description module is used to describe multiple 3D objects to be generated. This module initializes multiple 3D objects g(θ) through a multi-layer perceptron model, and renders a 2D image x from it at a viewing angle c and adds normally distributed noise; The layout control module uses distillation to extract layout prior knowledge from the pre-trained layout control diffusion model based on the input text y and bounding box l to generate a 2D image and participated in optimizing multiple 3D objects; The multi-view control module converts the 2D image Distill the multi-view prior knowledge as the prior of the pre-trained multi-view consistency diffusion model to optimize multiple 3D objects; The 3D content enhancement module uses a mean square error loss function to update the low-rank matrix in the pre-trained layout diffusion model to obtain a 3D object distribution. The layout and multi-view knowledge integration module uses layout multi-view variational distillation to unify layout prior knowledge and multi-view prior knowledge based on 3D object distribution and optimize multiple 3D objects.

7. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 5.

Images