Method and device for generating 3D from text, storage medium and electronic equipment
By generating an albedo map based on text description and combining 3D reconstruction and SDS optimization technology, the problems of non-re-lighting rendering and geometric defects in the existing technology are solved, and a more complete geometric features and a stable generation process are achieved.
Patent Information
- Application Number
- CN202411539560.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-11
AI Technical Summary
In the prior art, when using RGB data to generate 3D assets, highlights and shadows are bake onto the 3D model texture surface, resulting in the generated assets being re-illuminated and rendered, and the generated 3D model may have geometric defects such as voids.
By generating an albedo map based on text description, four albedo maps are generated using the pre-trained albedo-normal alignment multi-view diffusion model, and then fine 3D objects are obtained through 3D reconstruction model and SDS optimization technology, and finally generated using a stable diffusion model. 3D model including geometric models and PBR materials.
The generated 3D model geometric features are more complete and the process is more stable. It can be re-rendered and illuminated under different lighting conditions, improving the quality and generation efficiency of the 3D model.
Smart Images
Figure CN119418010A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of three-dimensional modeling, and in particular to a method, storage medium, electronic device and apparatus for generating 3D text. Background Art
[0002] At present, in the task of generating 3D from text, the use of models trained with RGB data for 3D asset generation will result in highlights and shadows being baked into the 3D model texture surface, causing the generated assets to be unable to be re-lit and rendered; and the generated 3D model will have defects such as holes in its geometry due to the influence of highlights and shadows.
[0003] Therefore, in order to solve the above technical problems and meet practical needs, a three-dimensional modeling technology is now provided. Summary of the invention
[0004] The present application provides a method, storage medium, electronic device and apparatus for generating 3D from text, which generates an albedo map based on text description, thereby obtaining a fine 3D object and a final 3D model. Compared with traditional technical means, the geometric features are more complete and the generation process is more stable.
[0005] In a first aspect, the present application provides a method for generating 3D text, the method comprising the following steps: Input text description and generate four albedo maps based on the pre-trained albedo-normal aligned multi-view diffusion model; Inputting the four albedo maps into a preset 3D reconstruction model to obtain an initial prior model; Based on the initial prior model, a refined 3D object is obtained by using SDS optimization technology; Based on the refined 3D object, the albedo and the geometric shape are set, and a stable diffusion model is used to obtain the final output 3D model.
[0006] Based on the above technical solution, the visual of the four albedo maps Figure 1 To.
[0007] Based on the above technical solution, the four albedo maps include albedo and normal.
[0008] Based on the above technical solution, the fine 3D object has a detailed mesh and albedo texture.
[0009] Based on the above technical solution, the final output 3D model includes a geometric model and a PBR material.
[0010] In a second aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating 3D text mentioned in the first aspect is implemented.
[0011] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, the method for generating 3D text mentioned in the first aspect is implemented.
[0012] In a fourth aspect, the present application provides a device for generating 3D text, the device comprising: The albedo map generation module is used to input a text description and generate four albedo maps based on the pre-trained albedo-normal aligned multi-view diffusion model; An initial priori construction module, which is used to input the four albedo maps into a preset 3D reconstruction model to obtain an initial priori model; A fine 3D construction module, which is used to obtain a fine 3D object based on the initial prior model by using SDS optimization technology; The final 3D construction module is used to obtain a final output 3D model based on the fine 3D object, by setting the albedo and the geometric shape and using a stable diffusion model.
[0013] Based on the above technical solution, the visual of the four albedo maps Figure 1 To; The four albedo maps include albedo and normal.
[0014] Based on the above technical solution, the fine 3D object has a detailed mesh and albedo texture; The final output 3D model includes a geometric model and a PBR material.
[0015] The beneficial effects of the technical solution provided by this application include: This application generates an albedo map based on text descriptions to obtain a fine 3D object and a final 3D model. Compared with traditional technical means, the geometric features are more complete and the generation process is more stable. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1A flowchart of the steps of the 3D task generation method provided in an embodiment of the present application; Figure 2 A technical schematic diagram of a 3D task generation method provided in an embodiment of the present application; Figure 3 This is a structural block diagram of a 3D task generation device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0019] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0020] The embodiments of the present application provide a method, storage medium, electronic device and apparatus for generating 3D from text, which generates an albedo map based on text description, thereby obtaining a fine 3D object and a final 3D model. Compared with traditional technical means, the geometric features are more complete and the generation process is more stable.
[0021] In order to achieve the above technical effects, the overall idea of this application is as follows: A method for generating 3D text, the method comprising the following steps: S1, input text description, generate four albedo maps based on the pre-trained albedo-normal alignment multi-view diffusion model; S2, inputting the four albedo maps into a preset 3D reconstruction model to obtain an initial prior model; S3, based on the initial prior model, the SDS optimization technique is used to obtain a fine 3D object; S4. Based on the fine 3D object, set the albedo and geometry, use the stable diffusion model, and obtain the final output 3D model.
[0022] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0023] First, see Figure 1~2 As shown, the embodiment of the present application provides a method for generating 3D text, the method comprising the following steps: S1, input text description, generate four albedo maps based on the pre-trained albedo-normal alignment multi-view diffusion model; S2, inputting the four albedo maps into a preset 3D reconstruction model to obtain an initial prior model; S3, based on the initial prior model, the SDS optimization technique is used to obtain a fine 3D object; S4. Based on the fine 3D object, set the albedo and geometry, use the stable diffusion model, and obtain the final output 3D model.
[0024] It should be noted that the terms in the embodiments of the present application are as follows: AN-MVM: albedo-normal aligned multi-view diffusion model, albedo-normal aligned multi-view consistent diffusion model; Normal: normal; Albedo: albedo; SDS: score distillation sample, score distillation sampling; TRM: transformer-based reconstruction diffusionmodel transformer-based reconstruction model; PBR: Physically-Based Rendering, physically based rendering; Stable-diffusionmodel, stable diffusion model.
[0025] The technical principle of the embodiment of this application: The open source diffusion model has a good ability to generate a single image based on text descriptions. The technical solution of the embodiment of the present application is to expand on this and design and develop an albedo and normal map that can generate consistent albedo and normal maps based on text. Then, a coarse-to-fine optimization technique is used to first reconstruct the albedo map, and then perform refined SDS optimization based on the reconstruction result. Finally, the open source diffusion model is used to generate PBR materials.
[0026] Specifically in the embodiment of the present application, UniDream is proposed, which is a novel framework that allows relightable objects to be generated from text descriptions. The key idea of the technical solution is to train a diffusion model that can provide physically based rendering (PBR) material priors and multi-view geometry priors. Specifically, an albedo normal aligned multi-view diffusion model (AN-MVM) for consistent multi-view image generation is first developed, which is trained on a set of paired albedo and normal data rendered from a 3D object dataset. Then, according to the simplified BRDF model, a 3D representation is defined, which includes albedo, normal, roughness, and metal properties, which are optimized based on the trained diffusion model and stable diffusion model. Compared with previous text-to-3D methods, the method of the embodiment of the present application can disentangle lighting and PBR materials, and achieve high-quality relightable objects under different environmental lighting conditions. In order to robustly obtain 3D objects using PBR from a 2D diffusion model, a three-stage generation pipeline is further developed. Initially, the albedo-normal aligned diffusion model (AN-MVM) is used to generate multi-view albedo and normal images. Subsequently, a transformer-based reconstruction model (TRM) is employed to convert the multi-view albedo images into a 3D coarse model, and SDS refinement is performed based on AN-MVM. Finally, the albedo and normal properties of the 3D model are repaired, and its roughness and metallic properties are optimized using a stable diffusion model.
[0027] As shown in the attached figure of the specification Figure 2 As shown, UniDream can be divided into three stages and four modules: Left: The multi-view diffusion model generates multi-view images based on the input text.
[0028] Middle part: First, the four-view albedo map obtains the 3D prior through the reconstruction model, and then the multi-view diffusion model performs SDS optimization based on the 3D prior to generate a 3D object with albedo texture.
[0029] Right: Generating a PBR material using the stable diffusion model.
[0030] In the embodiment of the present application, an albedo map is generated based on the text description, thereby obtaining a fine 3D object and a final 3D model. Compared with traditional technical means, the geometric features are more complete and the generation process is more stable.
[0031] Furthermore, the four albedo maps are Figure 1 To.
[0032] Furthermore, the four albedo maps include albedo and normal.
[0033] Furthermore, the fine 3D object has a detailed mesh and an albedo texture.
[0034] Furthermore, the final output 3D model includes a geometric model and a PBR material.
[0035] It should be noted that the problems of traditional technical means are as follows: (1) The generation speed is relatively slow, and it often takes a lot of iterations to optimize a 3D model with good geometry and texture; (2) The generated 3D models often have the problem of highlights and shadows being baked into the texture surface, resulting in the 3D models being unable to be re-rendered and re-lit in the 3D production pipeline.
[0036] The technical advantages of the embodiments of the present application are: (1) The multi-stage generation process can significantly optimize the speed of text-to-3D generation. When the quality is aligned with the existing solution or even better, the generation speed can be greatly improved, thereby improving the efficiency of 3D model generation. (2) Decouple highlight shadows and textures during the generation process, so that the generated 3D model will not produce highlight shadows when there is no light source, and can show the texture of the corresponding real highlights and shadows when re-rendering and lighting.
[0037] The main process of the technical solution of the embodiment of this application is as follows: (1) Rendering albedo and normal data for supervised training of a multi-view diffusion model to decouple the effects of highlights and shadows; (2) Use the multi-view reconstruction model as the 3D prior model to accelerate the generation process; (3) Based on the 3D prior, use the multi-view diffusion model trained in step (1) and use SDS to generate a 3D model with good albedo and normal quality; (4) Fixed the generated albedo and normal, and used the SD model to generate high-quality PBR materials. The final 3D model can be re-lit for rendering.
[0038] The specific operation process is: (1) First, a text description is given as input, and then the pre-trained albedo-normal alignment multi-view diffusion model generates four view images based on this text description. Figure 1 consistent albedo and normal maps; (2) The four albedo maps obtained in step (1) are input into the transformer-based 3D reconstruction model, which reconstructs and outputs a rough 3D model as the initial prior model for the next process; (3) Based on the rough initial prior model obtained in step (2), the SDS optimization technique is used to refine and generate a fine 3D object with detailed mesh and albedo texture as output using the multi-view diffusion model with albedo normal alignment; (4) Based on the output of step (3), we fix the albedo and geometry, and then use the stable diffusion model to generate the corresponding PBR material. The geometry model and PBR material are combined as the final output.
[0039] In summary, the technical solution of the embodiment of the present application proposes a novel text-to-3D generation framework, which can stably generate high-quality 3D objects by utilizing a multi-stage generation strategy based on SDS refinement of multi-view diffusion and reconstruction models. A large number of experiments have demonstrated the superiority of UniDream in three key areas as follows:
[0040] (1) Realistic Materials: By separating lighting from textures, UniDream accurately generates PBR materials that approximate real-world textures and can be re-lit under various lighting conditions, greatly enhancing realism.
[0041] (2) Complete Geometry: By incorporating normal supervision into our optimization process, UniDream excels in generating more comprehensive geometric details than other existing methods, resulting in more geometrically complete 3D objects.
[0042] (3) Stable generation: Due to the introduction of 3D priors in the reconstruction model and the normal supervision in the SDS process, the effectiveness of UniDream in generating 3D objects is ahead of other methods.
[0043] In a second aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating 3D text mentioned in the first aspect is implemented.
[0044] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, the method for generating 3D text mentioned in the first aspect is implemented.
[0045] Fourthly, see Figure 3 As shown, an embodiment of the present application provides a device for generating 3D text, the device comprising: The albedo map generation module is used to input a text description and generate four albedo maps based on the pre-trained albedo-normal aligned multi-view diffusion model; An initial priori construction module, which is used to input the four albedo maps into a preset 3D reconstruction model to obtain an initial priori model; A fine 3D construction module, which is used to obtain a fine 3D object based on the initial prior model by using SDS optimization technology; The final 3D construction module is used to obtain a final output 3D model based on the fine 3D object, by setting the albedo and the geometric shape and using a stable diffusion model.
[0046] In the embodiment of the present application, an albedo map is generated based on the text description, thereby obtaining a fine 3D object and a final 3D model. Compared with traditional technical means, the geometric features are more complete and the generation process is more stable.
[0047] Furthermore, the four albedo maps are Figure 1 To; The four albedo maps include albedo and normal.
[0048] Furthermore, the fine 3D object has a detailed mesh and albedo texture; The final output 3D model includes a geometric model and a PBR material.
[0049] It should be noted that the device for generating 3D text provided in the embodiment of the present application, and its corresponding technical problems, technical means and technical effects are similar to the principles of the method for generating 3D text in terms of principle.
[0050] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0051] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A method for generating 3D text, characterized in that: The method comprises the following steps: Input text description and generate four albedo maps based on the pre-trained albedo-normal aligned multi-view diffusion model; Inputting the four albedo maps into a preset 3D reconstruction model to obtain an initial prior model; Based on the initial prior model, a refined 3D object is obtained by using SDS optimization technology; Based on the refined 3D object, the albedo and the geometric shape are set, and a stable diffusion model is used to obtain the final output 3D model.
2. The method for generating 3D text according to claim 1, characterized in that: The four albedo maps are consistent in view.
3. The method for generating 3D text according to claim 1, characterized in that: The four albedo maps include albedo and normal.
4. The method for generating 3D text according to claim 1, characterized in that: The fine 3D object has a detailed mesh and albedo texture.
5. The method for generating 3D text according to claim 1, characterized in that: The final output 3D model includes a geometric model and a PBR material.
6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program running on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
8. A device for generating 3D text, characterized in that: The device comprises: The albedo map generation module is used to input a text description and generate four albedo maps based on the pre-trained albedo-normal aligned multi-view diffusion model; An initial priori construction module, which is used to input the four albedo maps into a preset 3D reconstruction model to obtain an initial priori model; A fine 3D construction module, which is used to obtain a fine 3D object based on the initial prior model by using SDS optimization technology; The final 3D construction module is used to obtain a final output 3D model based on the fine 3D object, by setting the albedo and the geometric shape and using a stable diffusion model.
9. The device for generating 3D text according to claim 8, characterized in that: The views of the four said albedo maps are consistent; The four albedo maps include albedo and normal.
10. The device for generating 3D text according to claim 8, characterized in that: The fine 3D object has a detailed mesh and an albedo texture; The final output 3D model includes a geometric model and a PBR material.