A 3D task generation method, device, storage medium and electronic device
By optimizing NeRF and using Shap-E and Prior technologies, the problems of slow generation speed, uncontrollable process and multi-head phenomena in 3D task generation are solved, achieving more efficient and accurate 3D task generation.
Patent Information
- Application Number
- CN202410933037.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2025-04-29
- Estimated Expiration
- 2044-07-12
AI Technical Summary
At this stage, there are problems such as slow generation speed, uncontrollable generation process and prone to long generation results in 3D task generation, mainly due to the lack of large-scale text-3D pairing data.
By optimizing NeRF, including initialization, constraint training and optimized texture representation, Shap-E is used to generate a rough 3D model and Prior rendered depth as Condition to guide ControlNet to optimize NeRF models.
It improves the accuracy and efficiency of 3D task generation, solves the problems of slow generation speed and uncontrollable process, and reduces the occurrence of multi-head phenomena.
Smart Images
Figure CN118799493B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of 3D task generation algorithms, and in particular to a 3D task generation method, device, storage medium and electronic device. Background Art
[0002] Producing high-quality 3D content with rich geometric and texture details is crucial for many fields such as gaming, film, and AR / VR industries. However, this task requires a huge workload from experts in related fields. The impressive success of large-scale text-to-image diffusion models in image content generation has inspired researchers to explore generating 3D assets from text descriptions, which has the potential to reduce the cost of 3D content creation.
[0003] A major bottleneck in directly training 3D diffusion models is the lack of large-scale text-3D paired data. The largest public text-3D dataset, Objaverse, contains only 800,000 text-3D pairs, while image-text datasets (e.g., Laion-5B) contain billions of text-image pairs. Therefore, current 3D task generation tasks suffer from slow generation speed, uncontrollable generation process, and multiple generation results.
[0004] Therefore, in order to meet actual needs, a 3D task generation technology is now provided. Summary of the invention
[0005] The present application provides a 3D task generation method, device, storage medium and electronic device, which optimize NeRF to improve the 3D task generation accuracy and generation efficiency to meet corresponding work requirements.
[0006] In a first aspect, the present application provides a 3D task generation method, the method comprising the following steps:
[0007] Initializing the geometric occupancy of NeRF and optimizing the NeRF by adaptive shape-guided constraint training;
[0008] Based on the optimized NeRF, the geometric shape is obtained;
[0009] Performing RGB refinement based on the geometric shape to obtain a preliminary texture of the geometric shape;
[0010] The preliminary texture is optimized by DMTet to obtain an optimized texture of the geometric shape.
[0011] Furthermore, the geometric occupancy of NeRF is initialized, including the following steps:
[0012] Use Shap-E to generate a rough 3D model and initialize the geometric occupancy of NeRF.
[0013] Further, the geometric occupancy of NeRF is initialized, and constraint training is performed by adaptive shape guidance to optimize the NeRF, including the following steps:
[0014] The geometric occupancy of NeRF is initialized using the point cloud generated by Prior in latent space, constraint training is performed based on adaptive shape guidance, and the depth rendered by Prior is used as a Condition to guide ControlNet to optimize NeRF.
[0015] Further, performing DMTet optimization on the preliminary texture to obtain the optimized texture of the geometric shape comprises the following steps:
[0016] The NeRF representation is replaced by the DMTet representation, and the preliminary texture is optimized by DMTet to obtain the optimized texture of the geometric shape.
[0017] In a second aspect, the present application provides a 3D task generation device, the device comprising:
[0018] A first optimization module is used to initialize the geometric occupancy of NeRF and perform constraint training based on adaptive shape guidance to optimize the NeRF;
[0019] A geometry acquisition module, which is used to obtain the geometric shape based on the optimized NeRF;
[0020] A texture generation module, which is used to perform RGB refinement based on the geometric shape to obtain a preliminary texture of the geometric shape;
[0021] The texture optimization module is used to perform DMTet optimization on the preliminary texture to obtain the optimized texture of the geometric shape.
[0022] Furthermore, the first optimization module is also used to generate a rough 3D model using Shap-E to initialize the geometric occupancy of NeRF.
[0023] Furthermore, the first optimization module is also used to initialize the geometric occupancy of NeRF using the point cloud generated by Prior in latent space, perform constraint training based on adaptive shape guidance, and use the depth rendered by Prior as Condition to guide ControlNet to optimize NeRF.
[0024] Furthermore, the texture optimization module is also used to use DMTet representation to replace NeRF representation, perform DMTet optimization on the preliminary texture, and obtain the optimized texture of the geometric shape.
[0025] In a third aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method mentioned in the first aspect is implemented.
[0026] In a fourth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program running on the processor, and the processor implements the method mentioned in the first aspect when executing the computer program.
[0027] The beneficial effects of the technical solution provided by this application include:
[0028] This application optimizes NeRF to improve the accuracy and efficiency of 3D task generation to meet corresponding work requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0030] Figure 1 A flowchart of the steps of the 3D task generation method provided in an embodiment of the present application;
[0031] Figure 2 A technical schematic diagram of a 3D task generation method provided in an embodiment of the present application;
[0032] Figure 3 RGB optimization principle diagram of the 3D task generation method provided in the embodiment of the present application;
[0033] Figure 4 A DMTet optimization principle diagram of the 3D task generation method provided in an embodiment of the present application;
[0034] Figure 5 This is a structural block diagram of a 3D task generation device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0036] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0037] The embodiments of the present application provide a 3D task generation method, device, storage medium and electronic device, which optimize NeRF to improve the 3D task generation accuracy and generation efficiency to meet corresponding work requirements.
[0038] In order to achieve the above technical effects, the overall idea of this application is as follows:
[0039] A 3D task generation method, the method comprising the following steps:
[0040] S1. Initialize the geometric occupancy of NeRF, and use adaptive shape guidance to perform constraint training to optimize NeRF.
[0041] S2, based on the optimized NeRF, obtain the geometric shape;
[0042] S3, performing RGB refinement based on geometric shapes to obtain preliminary textures of geometric shapes;
[0043] S4. Perform DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape.
[0044] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0045] First, see Figures 1 to 4 As shown, the embodiment of the present application provides a 3D task generation method, which includes the following steps:
[0046] S1. Initialize the geometric occupancy of NeRF, and use adaptive shape guidance to perform constraint training to optimize NeRF.
[0047] S2, based on the optimized NeRF, obtain the geometric shape;
[0048] S3, performing RGB refinement based on geometric shapes to obtain preliminary textures of geometric shapes;
[0049] S4. Perform DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape.
[0050] In the embodiment of the present application, NeRF is optimized to improve the 3D task generation accuracy and efficiency to meet the corresponding work requirements.
[0051] It should be noted that the embodiments of the present application can be based on FastDream, which is a machine learning model, and can also be used as a hierarchical framework for efficient text-to-3D generation. FastDream did not initially optimize 3D geometry and texture at the same time, but instead focused on generating 3D geometry and then merging detailed texture information in a progressive manner. The decoupled design of the embodiments of the present application can not only produce better generation quality, but also higher training efficiency. In addition, it enables image-driven controllable 3D generation, and can solve 3D geometric distortion caused by fuzzy geometric clues supervised by 2D.
[0052] In particular, 3D geometry is first generated using the Latent-NeRF framework. Unlike previous work that starts from scratch or uses limited sketch shapes, a case-specific coarse point cloud is adopted, which can be immediately obtained through the 3D generative model Shap-E. This point cloud provides a rich 3D geometric prior that can be used to accelerate NeRF training. However, challenges arise in balancing Prior utilization and NeRF expressiveness. The goal of this application is to align the NeRF output with the basic skeleton of the point cloud while retaining its freedom to express text-hinted geometry. To achieve this goal, this application uses the generated point cloud to initialize the geometric occupancy of NeRF, uses adaptive shape guidance to constrain its training, and uses the depth of Prior rendering as a Condition to guide ControlNet to optimize the NeRF model. More specifically, the technical solution of the present application encourages the occupancy of NeRF to match the surface distribution of the coarse point cloud, while gradually reducing the constraint weights near the surface of the point cloud, thereby retaining the expressiveness of NeRF for geometric modeling; wherein,
[0053] Latent: Latent Diffusion is an open source training framework for AI painting developed based on the latent diffusion model (LDMs) algorithm. The framework can be used for various image synthesis tasks such as text and image generation, unconditional image generation, image restoration, layout image generation, semantic image generation and image super-resolution.
[0054] Furthermore, the geometric occupancy of NeRF is initialized, including the following steps:
[0055] Use Shap-E to generate a rough 3D model and initialize the geometric occupancy of NeRF.
[0056] Further, the geometric occupancy of NeRF is initialized, and constraint training is performed by adaptive shape guidance to optimize the NeRF, including the following steps:
[0057] The geometric occupancy of NeRF is initialized using the point cloud generated by Prior in latent space, constraint training is performed based on adaptive shape guidance, and the depth rendered by Prior is used as a Condition to guide ControlNet to optimize NeRF.
[0058] Further, performing DMTet optimization on the preliminary texture to obtain the optimized texture of the geometric shape comprises the following steps:
[0059] The NeRF representation is replaced by the DMTet representation, and the preliminary texture is optimized by DMTet to obtain the optimized texture of the geometric shape.
[0060] In summary, the technical solution of the embodiment of the present application is implemented by utilizing 3D Prior;
[0061] First, initialize NeRF;
[0062] Secondly, the shape of the generated 3D model is controlled by inputting constraints during the NERF generation process;
[0063] Then, use the rendering depth as a condition to guide the training;
[0064] Finally, a 3D model with consistent geometric direction and good effect is quickly generated;
[0065] 3D Prior: 3D prior inspection is to construct a rough 3D model by roughly representing the target to be generated during the 3D model generation process;
[0066] NeRF: Neural Radiance Fields, a neural radiation field model, generates new views of a 3D scene given an image, conditioned on the image and precise pose. It also explicitly defines the 3D shape and appearance of the scene as a continuous function, and can generate a 3D mesh by marching cubes. Before use, the parameters must be initialized and adjusted according to the image to be generated.
[0067] Depth: Depth is a parameter used in rendering models to highlight the sense of space, depth of field and other three-dimensional effects;
[0068] Condition: Guiding conditions are the control conditions that control the generation effect of 3D models during the generation process.
[0069] It should be noted that the drawings in the specification Figure 2 This is a technical schematic diagram of an embodiment of the present application. Figure 2F in it is the symbol for MLP, which is a multi-layer perceptron. Appearance refers to the texture part, Geometry refers to the geometry part, Initialization refers to the initialization part, and Shap-E is a pre-trained model used to provide rough 3D geometry prior.
[0070] in addition, Figure 2 The operations corresponding to the numeric symbols in are:
[0071] (1) The corresponding operations are: the rough model provided by Shap-E is first used as the geometry initialization;
[0072] (2) The corresponding operations are: when training NeRF, it also serves as a geometric constraint for shape control;
[0073] (3) The corresponding operation is: perform RGB texture optimization based on the obtained NeRF geometry.
[0074] It should be noted that the drawings in the specification Figure 2 This is a technical schematic diagram of an embodiment of the present application.
[0075] The drawings in the specification Figure 3 This is the RGB optimization principle diagram of the embodiment of the present application. Figure 3 RGB refine in it means RGB texture optimization, F represents the symbol of MLP, which is a multi-layer perceptron, Decoder represents decoding the RGB image from the NeRF feature, Encoder represents encoding the RGB image using an encoder of another diffusion model, and LSDS represents the SDS loss function.
[0076] In addition, the drawings of the specification Figure 3 This describes the operation process of the RGB optimization stage:
[0077] First, the RGB image is decoded from the NeRF feature, and then the RGB image is encoded using an encoder of another diffusion model, and finally the SDS loss function is calculated for optimization.
[0078] It should be noted that the drawings in the specification Figure 4 This is the DMTet optimization principle diagram of the embodiment of this application. Figure 4 In which DMTet refine represents the DMTet optimization stage, Render represents rendering, Image represents image, Encoder represents encoder, and LSDS represents SDS loss function.
[0079] In addition, the drawings of the specification Figure 3 Describes the operation process of the DMTet optimization phase:
[0080] First, an RGB image is rendered from the DMTet feature, and then the RGB image is encoded using an encoder of another diffusion model, and finally the SDS loss function is calculated for optimization.
[0081] The overall process of the technical solution of the embodiment of the present application is as follows:
[0082] In the first step, the point cloud generated by Shap-E is used in latent space to initialize the geometric occupancy of NeRF, adaptive shape guidance is used to constrain its training, and the depth of Prior rendering is used as a Condition to guide ControlNet to optimize the NeRF model to obtain better geometry.
[0083] The second step is to perform RGBrefine based on the first step to get the preliminary texture:
[0084] Similar to processes such as DreamFusion, the RGB representation of a view is obtained by the decoder after rendering from (x, y, z, θ), and the image encoder is used to convert the RGB space to the latent space, and then the same diffusion process as the geometry stage is performed to optimize the texture representation of the object in NeRF. The MLP of NeRF in the RGB refinement stage is initialized with the weights of the MLP in the geometry stage.
[0085] The third step is to perform DMTet optimization based on the second step to get a better texture:
[0086] Although RGB space can represent texture well, it lacks the ability to capture finer details. Following Magic3D, DMTet optimization is combined to enhance the quality of texture representation. DMTet training starts with a 3D shape represented by NeRF. A deformable tetrahedral mesh (VT, T) is used for representation, where VT represents a vertex in the mesh T. Each vertex Contains a signed distance field (SDF) value si∈R and a deformation Δvi∈R3 relative to its initial canonical coordinates. DMTet refinement is performed in three steps. First, a surface mesh is extracted from the SDF using a differentiable marching tetrahedron algorithm; second, a differentiable rasterizer is used to render the extracted surface mesh into a high-resolution image; finally, the SDS gradient is used to optimize si and Δvi for each vi.
[0087] Based on the above technical solution, the key points in the technical solution of the embodiment of the present application are: using Prior to initialize NeRF, constrain the production process and guide the training process.
[0088] The technical solution of the embodiment of the present application is used to gradually optimize the geometric shape and texture of a 3D object:
[0089] First, the sparse point cloud generated by Shap-E is used as the prior for initializing the geometric space occupancy of Latent-NeRF;
[0090] In addition, an adaptive shape guidance is proposed to balance the use of 3D priors and the expressiveness of NeRF, and the rendered depth is used as the condition of the controlnet to train NeRF;
[0091] Once valid geometry is obtained, it is given accurate color information through RGB refinement, and the surface is optimized using DMTet to obtain finer details;
[0092] In summary, the FastDream model has achieved significant improvements over existing methods in terms of efficiency, controllability, and geometric stability.
[0093] Second, see Figure 5 As shown, an embodiment of the present application provides a 3D task generating device, the device comprising:
[0094] A first optimization module is used to initialize the geometric occupancy of NeRF and perform constraint training based on adaptive shape guidance to optimize the NeRF;
[0095] A geometry acquisition module, which is used to obtain the geometric shape based on the optimized NeRF;
[0096] A texture generation module, which is used to perform RGB refinement based on the geometric shape to obtain a preliminary texture of the geometric shape;
[0097] The texture optimization module is used to perform DMTet optimization on the preliminary texture to obtain the optimized texture of the geometric shape.
[0098] In the embodiment of the present application, NeRF is optimized to improve the 3D task generation accuracy and efficiency to meet the corresponding work requirements.
[0099] It should be noted that the embodiments of the present application can be based on FastDream, which is a machine learning model, and can also be used as a hierarchical framework for efficient text-to-3D generation. FastDream did not initially optimize 3D geometry and texture at the same time, but instead focused on generating 3D geometry and then merging detailed texture information in a progressive manner. The decoupled design of the embodiments of the present application can not only produce better generation quality, but also higher training efficiency. In addition, it enables image-driven controllable 3D generation, and can solve 3D geometric distortion caused by fuzzy geometric clues supervised by 2D.
[0100] In particular, 3D geometry is first generated using the Latent-NeRF framework. Unlike previous work that starts from scratch or uses limited sketch shapes, a case-specific coarse point cloud is adopted, which can be immediately obtained through the 3D generative model Shap-E. This point cloud provides a rich 3D geometric prior that can be used to accelerate NeRF training. However, challenges arise in balancing Prior utilization and NeRF expressiveness. The goal of this application is to align the NeRF output with the basic skeleton of the point cloud while retaining its freedom to express text-hinted geometry. To achieve this goal, this application uses the generated point cloud to initialize the geometric occupancy of NeRF, uses adaptive shape guidance to constrain its training, and uses the depth of Prior rendering as a Condition to guide ControlNet to optimize the NeRF model. More specifically, the technical solution of the present application encourages the occupancy of NeRF to match the surface distribution of the coarse point cloud, while gradually reducing the constraint weights near the surface of the point cloud, thereby retaining the expressiveness of NeRF for geometric modeling; wherein,
[0101] Latent: Latent Diffusion is an open source training framework for AI painting developed based on the latent diffusion model (LDMs) algorithm. The framework can be used for various image synthesis tasks such as text and image generation, unconditional image generation, image restoration, layout image generation, semantic image generation and image super-resolution.
[0102] Furthermore, the first optimization module is also used to generate a rough 3D model using Shap-E to initialize the geometric occupancy of NeRF.
[0103] Furthermore, the first optimization module is also used to initialize the geometric occupancy of NeRF using the point cloud generated by Prior in latent space, perform constraint training based on adaptive shape guidance, and use the depth rendered by Prior as Condition to guide ControlNet to optimize NeRF.
[0104] Furthermore, the texture optimization module is also used to use DMTet representation to replace NeRF representation, perform DMTet optimization on the preliminary texture, and obtain the optimized texture of the geometric shape.
[0105] In summary, the technical solution of the embodiment of the present application is implemented by utilizing 3D Prior;
[0106] First, initialize NeRF;
[0107] Secondly, the shape of the generated 3D model is controlled by inputting constraints during the NERF generation process;
[0108] Then, use the rendering depth as a condition to guide the training;
[0109] Finally, a 3D model with consistent geometric direction and good effect is quickly generated;
[0110] 3D Prior: 3D prior inspection is to construct a rough 3D model by roughly representing the target to be generated during the 3D model generation process;
[0111] NeRF: Neural Radiance Fields, a neural radiation field model, generates new views of a 3D scene given an image, conditioned on the image and precise pose. It also explicitly defines the 3D shape and appearance of the scene as a continuous function, and can generate a 3D mesh by marching cubes. Before use, the parameters must be initialized and adjusted according to the image to be generated.
[0112] Depth: Depth is a parameter used in rendering models to highlight the sense of space, depth of field and other three-dimensional effects;
[0113] Condition: Guiding conditions are the control conditions that control the generation effect of 3D models during the generation process.
[0114] It should be noted that the drawings in the specification Figure 2 This is a technical schematic diagram of an embodiment of the present application. Figure 2 F in it is the symbol for MLP, which is a multi-layer perceptron. Appearance refers to the texture part, Geometry refers to the geometry part, Initialization refers to the initialization part, and Shap-E is a pre-trained model used to provide rough 3D geometry prior.
[0115] in addition, Figure 2 The operations corresponding to the numeric symbols in are:
[0116] (1) The corresponding operations are: the rough model provided by Shap-E is first used as the geometry initialization;
[0117] (2) The corresponding operations are: when training NeRF, it also serves as a geometric constraint for shape control;
[0118] (3) The corresponding operation is: perform RGB texture optimization based on the obtained NeRF geometry.
[0119] It should be noted that the drawings in the specification Figure 2 This is a technical schematic diagram of an embodiment of the present application.
[0120] The drawings in the specification Figure 3 This is the RGB optimization principle diagram of the embodiment of the present application. Figure 3 RGB refine in it means RGB texture optimization, F represents the symbol of MLP, which is a multi-layer perceptron, Decoder represents decoding the RGB image from the NeRF feature, Encoder represents encoding the RGB image using an encoder of another diffusion model, and LSDS represents the SDS loss function.
[0121] In addition, the drawings of the specification Figure 3 This describes the operation process of the RGB optimization stage:
[0122] First, the RGB image is decoded from the NeRF feature, and then the RGB image is encoded using an encoder of another diffusion model, and finally the SDS loss function is calculated for optimization.
[0123] It should be noted that the drawings in the specification Figure 4 This is the DMTet optimization principle diagram of the embodiment of this application. Figure 4 In which DMTet refine represents the DMTet optimization stage, Render represents rendering, Image represents image, Encoder represents encoder, and LSDS represents SDS loss function.
[0124] In addition, the drawings of the specification Figure 3 Describes the operation process of the DMTet optimization phase:
[0125] First, an RGB image is rendered from the DMTet feature, and then the RGB image is encoded using an encoder of another diffusion model, and finally the SDS loss function is calculated for optimization.
[0126] The overall process of the technical solution of the embodiment of the present application is as follows:
[0127] In the first step, the point cloud generated by Shap-E is used in latent space to initialize the geometric occupancy of NeRF, adaptive shape guidance is used to constrain its training, and the depth of Prior rendering is used as a Condition to guide ControlNet to optimize the NeRF model to obtain better geometry.
[0128] The second step is to perform RGBrefine based on the first step to get the preliminary texture:
[0129] Similar to processes such as DreamFusion, the RGB representation of a view is obtained by the decoder after rendering from (x, y, z, θ), and the image encoder is used to convert the RGB space to the latent space, and then the same diffusion process as the geometry stage is performed to optimize the texture representation of the object in NeRF. The MLP of NeRF in the RGB refinement stage is initialized with the weights of the MLP in the geometry stage.
[0130] The third step is to perform DMTet optimization based on the second step to get a better texture:
[0131] Although RGB space can represent texture well, it lacks the ability to capture finer details. Following Magic3D, DMTet optimization is combined to enhance the quality of texture representation. DMTet training starts with a 3D shape represented by NeRF. A deformable tetrahedral mesh (VT, T) is used for representation, where VT represents a vertex in the mesh T. Each vertex Contains a signed distance field (SDF) value si∈R and a deformation Δvi∈R3 relative to its initial canonical coordinates. DMTet refinement is performed in three steps. First, a surface mesh is extracted from the SDF using a differentiable marching tetrahedron algorithm; second, a differentiable rasterizer is used to render the extracted surface mesh into a high-resolution image; finally, the SDS gradient is used to optimize si and Δvi for each vi.
[0132] Based on the above technical solution, the key points in the technical solution of the embodiment of the present application are: using Prior to initialize NeRF, constrain the production process and guide the training process.
[0133] The technical solution of the embodiment of the present application is used to gradually optimize the geometric shape and texture of a 3D object:
[0134] First, the sparse point cloud generated by Shap-E is used as the prior for initializing the geometric space occupancy of Latent-NeRF;
[0135] In addition, an adaptive shape guidance is proposed to balance the use of 3D priors and the expressiveness of NeRF, and the rendered depth is used as the condition of the controlnet to train NeRF;
[0136] Once valid geometry is obtained, it is given accurate color information through RGB refinement, and the surface is optimized using DMTet to obtain finer details;
[0137] In summary, the FastDream model has achieved significant improvements over existing methods in terms of efficiency, controllability, and geometric stability.
[0138] It should be noted that the 3D task generation device provided in the embodiment of the present application, and its corresponding technical problems, technical means and technical effects are similar to the principles of the 3D task generation method in terms of principle.
[0139] In a third aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method mentioned in the first aspect is implemented.
[0140] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program running on the processor, and the processor implements the method mentioned in the first aspect when executing the computer program.
[0141] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0142] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A 3D task generation method, characterized in that: The method comprises the following steps: Initializing the geometric occupancy of NeRF and optimizing the NeRF by adaptive shape-guided constraint training; Based on the optimized NeRF, the geometric shape is obtained; Performing RGB refinement based on the geometric shape to obtain a preliminary texture of the geometric shape; Performing DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape; Initializing the geometric occupancy of NeRF and adaptively guiding constraint training to optimize the NeRF includes the following steps: Use the point cloud generated by Prior in latent space to initialize the geometric occupancy of NeRF, perform constraint training based on adaptive shape guidance, and use the depth rendered by Prior as a Condition to guide ControlNet to optimize NeRF; Performing DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape comprises the following steps: Using DMTet representation to replace NeRF representation, performing DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape; Initialize the NeRF geometry occupancy, including the following steps: Use Shap-E to generate a rough 3D model and initialize the geometric occupancy of NeRF.
2. A 3D task generation device, characterized in that: The device comprises: A first optimization module is used to initialize the geometric occupancy of NeRF and perform constraint training based on adaptive shape guidance to optimize the NeRF; A geometry acquisition module, which is used to obtain the geometric shape based on the optimized NeRF; A texture generation module, which is used to perform RGB refinement based on the geometric shape to obtain a preliminary texture of the geometric shape; A texture optimization module, which is used to perform DMTet optimization on the preliminary texture to obtain an optimized texture of the geometric shape; The first optimization module is also used to initialize the geometric occupancy of NeRF using the point cloud generated by Prior in latent space, perform constraint training based on adaptive shape guidance, and use the depth rendered by Prior as Condition to guide ControlNet to optimize NeRF; The texture optimization module is further used to use DMTet representation to replace NeRF representation, perform DMTet optimization on the preliminary texture, and obtain the optimized texture of the geometric shape; The first optimization module is also used to generate a rough 3D model using Shap-E to initialize the geometric occupancy of NeRF.
3. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 1 is implemented.
4. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program running on the processor, wherein: When the processor executes the computer program, the method according to claim 1 is implemented.