Three-dimensional generation method based on semantic enhancement hybrid reconstruction

Abstract

The invention discloses a three-dimensional generation method based on semantic enhancement hybrid reconstruction. The three-dimensional generation method comprises the following steps: 1, performing semantic enhancement on an input original text by utilizing a GPT-4 model; 2, generating a two-dimensional image from the text; inputting the semantically enhanced text into a text generation image diffusion model, so that the generated two-dimensional image is beneficial to three-dimensional reconstruction; 3, multiple views are generated from the two-dimensional image, the images generated in the text image diffusion model are input into a multi-view diffusion model, and images at different view angles are generated for three-dimensional reconstruction; 4, performing three-dimensional hybrid reconstruction by using the image and the text, taking the generated multi-view and the original text as input, and generating three-plane representation through a three-dimensional large reconstruction model; 5, generating a three-dimensional grid from the three-plane representation; and taking the three-plane representation as input, and performing contour surface extraction by using FlexiCubes to obtain a final three-dimensional grid. According to the invention, the problems of slow three-dimensional object generation, low quality and misalignment with semantics are solved.

Description

Technical Field

[0001] The present invention realizes an efficient text-to-3D generation method through semantically enhanced sparse view hints and hybrid reconstruction. Background Art

[0002] The generation of three-dimensional (3D) objects has had a transformative impact on multiple fields, including healthcare, architecture, industry, augmented reality (AR), and virtual reality (VR). By providing realistic simulation data, 3D object generation has significantly enhanced our understanding of complex structures and their interactions in real-world environments. However, traditional 3D object generation methods are often very time-consuming and resource-intensive. Recent advances in cross-media technologies, coupled with ongoing research, have accelerated the development of text-to-3D generation. Text-to-3D generation leverages the power of natural language processing (NLP) to generate 3D models from simple text descriptions, which has the potential to democratize 3D content creation by eliminating the need for specialized skills and reducing production time. Despite these advantages, significant challenges remain. The scarcity of high-quality 3D training data and the large computational resources required to render 3D objects pose obstacles to the efficient and accurate generation of 3D assets. These challenges are particularly evident in tasks that require fast and high-quality output, which current methods often fail to meet.

[0003] In summary, the difficulty of text-to-3D object generation lies in how to quickly generate high-quality, semantically aligned 3D objects. Summary of the invention

[0004] The present invention provides a 3D generation method based on semantically enhanced hybrid reconstruction, which can generate high-quality 3D objects from text in only about 25 seconds. The architecture mainly includes: 1. Semantic enhancement through large language models: We first use large language models (such as GPT4) to preprocess the input text, retaining its core semantic content while enriching it with other relevant details. 2. Fine-tuning the text-to-image diffusion model using a low-rank adaptation model (LoRA): In order to mitigate the impact of complex backgrounds in generated images, we fine-tune the diffusion model using a low-rank adaptation model (LoRA) on a subset of the 3D dataset. 3. Multi-view prompt generation: We use a multi-view diffusion model to generate a set of geometrically consistent multi-view visual cues. This process carefully balances the number of views to capture sufficient details while minimizing the risk of structural inconsistencies between views. 4. Sparse visual cues for text injection. Figure 33D reconstruction: We input multi-view features and text features into a large 3D reconstruction model for hybrid reconstruction and generate a three-plane representation. Our method extends its capabilities to multi-modal, multi-view settings, integrating text and sparse view visual cues to improve the accuracy and geometric consistency of the reconstructed 3D model. 5. 3D mesh generation: In order to address the memory and computational challenges associated with traditional volume rendering methods such as NeRF, we integrate FlexiCubes into our pipeline, use FlexiCubes for isosurface extraction, and finally generate a 3D mesh.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step (1) uses the GPT-4 model to semantically enhance the input original text, while retaining its core semantic content and adding relevant details. This preprocessing step enhances the ability of the diffusion model to generate richer and semantically aligned initial images;

[0007] Step (2), text generates a two-dimensional image; in order to reduce the influence of complex background in the generated image, the semantically enhanced text is input into the text diffusion model, so that the generated two-dimensional image is helpful for three-dimensional reconstruction. The text diffusion model has been fine-tuned using a low-rank adaptation model (LoRA).

[0008] Step (3), generating multiple views from the two-dimensional image, inputting the image generated in the Vincent graph diffusion model into the multi-view diffusion model to generate images at different viewing angles for three-dimensional reconstruction;

[0009] Step (4) uses images and texts to perform three-dimensional hybrid reconstruction, taking the generated multi-views and original texts as inputs, and generating a three-plane representation through a three-dimensional large-scale reconstruction model.

[0010] Step (5), generating a 3D mesh from the three-plane representation. The three-plane representation is used as input, and FlexiCubes is used to extract isosurfaces to obtain the final 3D mesh.

[0011] Furthermore, step (1) is specifically as follows:

[0012] Input the original text and instruction prompts into GPT-4 to obtain the semantically enhanced text, as shown in formula (1);

[0013] T s =GPT-4(T o ,T Instruction ) (Formula 1)

[0014] Among them, T o Represents the original input text, T s represents the semantically enhanced text, T InstructionIndicates a command prompt.

[0015] Furthermore, step (2) is specifically as follows:

[0016] 2-1. The semantically enhanced text T s Input to the Wensheng graph diffusion model fine-tuned with LoRA to generate a semantically aligned two-dimensional image, as shown in formula (2);

[0017]

[0018] in, Indicates D LoRA Fine-tuned diffusion model D from Wensheng graph SDXL and L oRA Fine-tune the resulting initial view;

[0019] 2-2. Introduce the trainable low-rank matrix into the Transformer module of the model to achieve efficient fine-tuning without modifying the weights of the original Wensheng graph diffusion model. For the pre-trained weight matrix W0, the LoRA expression updated by low-rank decomposition is as shown in formula (3);

[0020] W0+ΔW=W0+BA (Formula 3)

[0021] in, r<<min(d,k),

[0022] r represents the rank of the LoRA module, d and k represent the dimensions of the matrix, and the corresponding forward propagation formula is modified as follows, as shown in formula (4);

[0023] h=W0x+ΔW x =W0x+BAx (Formula 4)

[0024] In this example, the weight matrix W0 remains frozen and does not receive gradient updates. ΔW represents the intermediate variables. Instead, only the parameters A and B are fine-tuned. A is initialized with a random Gaussian value and B is initialized to zero. x represents the input data.

[0025] 2-3. LoRA introduces an additional downsampling and upsampling auxiliary branch based on the original linear layer, and fine-tunes by training this auxiliary branch. The loss function guiding the fine-tuning process is defined as shown in formula (5):

[0026]

[0027] Among them, ε represents the real noise, ε θ represents the predicted noise, Express expectations. The latent variable at time step t is defined as z0=Enc(x0), where x0 is the input image, Enc is the encoder that maps the image to a low-dimensional latent space, and τ θ (y) represents the encoded text condition, and y represents the text condition.

[0028] Furthermore, step (3) is specifically as follows:

[0029] Take the image generated by the text as input and generate multi-view images; use the multi-view diffusion model to generate a set of geometrically consistent multi-view images. This process carefully balances the number of views to capture sufficient details while minimizing the risk of structural inconsistency across views. As shown in formula (6);

[0030]

[0031] Among them, I vp represents the generated multi-view image, D MV represents the multi-view diffusion model, Represents the initial view.

[0032] Furthermore, step (4) is specifically as follows:

[0033] 4-1. Cross-modal multi-view fusion, generating multi-view images Combined with the original input text T o , a three-plane representation is generated through a large 3D reconstruction model. Based on the large 3D reconstruction model, our method extends its functionality to a multi-modal, multi-view setting, integrating visual cues from text and sparse views to enhance the accuracy and geometric consistency of the reconstructed 3D model, as shown in formulas (7) and (8);

[0034]

[0035] F text =D BERT (To)(Formula 8)

[0036] in, Indicates the kth th image embedding of multi-view images, where Indicates the kth th Global visual embedding of multi-view images, Indicates the kth th The i-th patch encoding of the visual cue of the view, D VIT-DINO represents the image encoder, D BERT represents the semantic encoder. M represents D VIT–DINO The length of the output sequence. F textrepresents text embedding, L stands for D BERT The length of the output sequence.

[0037] 4-2. Apply the cross-modal fusion strategy on each multi-view image, and then adopt the multi-view fusion strategy on all multi-view images, as shown in formulas (9) and (10);

[0038]

[0039] Among them, CFusion represents the cross-modal fusion operation, and VFusion represents the multi-view fusion operation. th The modality fusion of multi-view images and text descriptions adopts a direct fusion method. Replace with Global visual embedding in and F text Global text embedding in The modality fusion feature is constructed by element-wise summation of At the same time, the original local semantic embedding is retained. Use cascade operators to fuse the modal features of multiple views Perform view fusion to obtain a unified feature representation F fus , integrating cross-view and cross-modal information.

[0040] 4-3. Sparse Vision Figure 3 D reconstruction, for fine 3D reconstruction, a three-plane decoder is used to process the fused feature representation F fus The three-plane decoder uses a cross-attention mechanism to combine the three-plane tokens with the fused feature representation F fus are linked and then passed through self-attention and multi-layer perceptron layers to generate a refined three-plane token F triplane , this process is mathematically expressed as shown in formula (11);

[0041] F triplane =TriplaneDecoder(F fus ,τ) (Formula 11)

[0042] Where τ represents the learnable token sequence and F triplane represents a triplane token, and TriplaneDecoder represents a triplane decoder;

[0043] 4-4. Deconvolution and reshaping operations are performed on the three-plane tokens to form the final three-plane representation R triplane , thus achieving more accurate and detailed 3D reconstruction. This process is shown in formula (12);

[0044] Rtriplane =Reshape(DeConv(F triplane )) (Formula 12)

[0045] Among them, Reshape represents the shaping operation, and DeConv represents the deconvolution operation;

[0046] Furthermore, step (5) is specifically as follows:

[0047] To address the memory and computational challenges associated with traditional volume rendering methods such as tri-plane NeRF, we integrate FlexiCubes into our framework. triplane Represents the final 3D mesh generated after processing through FlexiCube. FlexiCube significantly improves the training efficiency and quality of the reconstructed mesh, enabling the generation of smoother and more detailed 3D outputs while maintaining the geometric consistency of the reconstructed object.

[0048] Beneficial effects of the present invention:

[0049] The present invention proposes a method for quickly generating three-dimensional objects, which solves the problems of slow generation, low quality, and semantic misalignment of three-dimensional objects. By optimizing algorithms and data processing processes, the present invention significantly improves the generation speed of three-dimensional objects and effectively shortens the time from design to model completion. This is especially important for scenarios that require rapid iterative design and verification, such as animation production, game development, industrial design and other fields. Due to the comprehensive improvement of the present invention in terms of speed, quality and user interaction, its scope of application is very wide. In addition to traditional film and television animation and game development, it can also be expanded to multiple fields such as online education, telemedicine, urban planning, historical restoration, etc., and has very broad market prospects and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention.

[0051] Figure 2 It is a semantically enhanced multi-view prompt generation module constructed in the method of the present invention.

[0052] Figure 3 is the sparse view of text injection constructed in the method of the present invention Figure 3 D reconstruction module. DETAILED DESCRIPTION

[0053] The detailed parameters of the present invention are further described in detail below.

[0054] like Figure 1As shown, the present invention provides a three-dimensional generation method based on semantically enhanced hybrid reconstruction, specifically a three-dimensional object generation framework that quickly generates high-quality and highly semantically aligned three-dimensional objects.

[0055] Step (1) uses the GPT-4 model to preprocess the input text, adding relevant details while retaining its core semantic content. This preprocessing step enhances the ability of the diffusion model to generate richer and more semantically aligned visual cues. Figure 2 As shown, the details are as follows:

[0056] The original text T o and instruction prompt T Instruction Input to GPT-4, and through GPT-4's powerful natural language processing capabilities, obtain the semantically enhanced text T s , used for subsequent image generation.

[0057] Step (2) generates a two-dimensional image from the text. In order to reduce the impact of the complex background in the generated image, the semantically enhanced text is used as input to the text-to-image diffusion model fine-tuned using LoRA, so that the generated two-dimensional image is helpful for three-dimensional reconstruction, such as Figure 2 As shown, the details are as follows:

[0058] The semantically enhanced text T s Input is given to the low-rank adaptive model D SDXL Fine-tuned text-to-image diffusion model D LoRA , Dx oRA 1,000 high-quality data samples were selected from the Objaverse dataset and their resolution was upscaled from 512×512 to 1024×1024 to match the output size of SDXL generated images. The caption corresponding to each sample was generated using the Cap3D model. These 1,000 image-caption pairs were used to train the LoRA model using the AdamW optimizer with a learning rate of 10-6, a batch size of 2, and a total of 10,000 training steps. The training process was performed on four RTX 4090 GPUs and completed in about 20 minutes. LoRA The number of denoising steps is 50, and it takes about 7 seconds to generate a semantically aligned 2D image from text. The resolution size is 1024×1024.

[0059] Step (3) takes the text-generated images as input to generate multi-view images. We utilize the multi-view generative diffusion model to generate a set of geometrically consistent multi-view visual cues. This process carefully balances the number of views to capture sufficient details while minimizing the risk of structural inconsistency across views, such as Figure 2 As shown, the details are as follows:

[0060] We exploit the multi-view diffusion model to generate a set of geometrically consistent multi-view visual cues I vp This process carefully balances the number of views to capture enough detail while minimizing the risk of structural inconsistencies between views. In our implementation, the generated views are 6, and the model generates images with absolute elevations of 20°, -10°, 20°, -10°, 20°, and -10°, while using the images generated by the text as query views. The corresponding relative azimuths of these images are 30°, 90°, 150°, 210°, 270°, and 330°. Then, the generated 960x640 resolution size view is segmented into six 320x320 resolution views with a white background.

[0061] Step (4) uses images and texts to perform 3D hybrid reconstruction, taking the generated multi-views and original texts as inputs, and generating a three-plane representation through a large 3D reconstruction model, such as Figure 3 As shown, the details are as follows:

[0062] 4-1. Cross-modal multi-view fusion, we will generate multi-view images Combine input text T o , reconstructing 3D objects. Based on the 3D large reconstruction model, our method extends its capabilities to the multimodal, multi-view setting, integrating visual cues from text and sparse views to enhance the accuracy and geometric consistency of the reconstructed 3D model. We use the DINO ViT-B / 16 model as the image encoder for visual cues and BERT as the text encoder. Through the image encoder D VIT-DINO Get image features in Indicates the kth th Global semantic embedding of visual cues from each view, Indicates the kth th The i-th patch encoding of the visual cue of the view. M represents D VIT–DINO The length of the output sequence, for each view M, is 400.

[0063] 4-2. Next, we apply a cross-modal fusion strategy on each view cue and then a multi-view fusion strategy on all view cues. th We adopt a direct fusion method to fuse the visual cues and text descriptions of the views. Replace with Global visual semantics in and F text Global text embedding in The modality fusion feature is constructed by element-wise summation of At the same time, the original local semantic embedding is retained. Next, we use the cascade operator to fuse the features of multiple view modalities. View fusion is performed to obtain a unified feature representation that integrates cross-view and cross-modal information.

[0064] 4-3. Sparse Vision Figure 3 D reconstruction, for fine 3D reconstruction, we use the three-plane decoder TriplaneDecoder to process the fused feature F fus The three-plane decoder uses a cross-attention mechanism to link these three-plane tokens τ with the fused feature tokens, and passes through self-attention and multi-layer perceptron layers to generate refined three-plane tokens F triplane .

[0065] 4-4. These F triplane It is further processed by deconvolution and reshaping operations to form the final three-plane representation R triplane , resulting in more accurate and detailed 3D reconstruction.

[0066] Step (5) convert the three planes into a three-dimensional grid. The three planes represent R triplane The three-dimensional mesh is generated by the isosurface extraction algorithm FlexiCubes, such as Figure 3 As shown, the details are as follows:

[0067] To address the memory and computational challenges associated with traditional volume rendering methods such as tri-planar NeRF, we integrated FlexiCubes into our pipeline. triplane The representation is processed by FlexiCube to generate the final 3D mesh. FlexiCube significantly improves the training efficiency and quality of the reconstructed mesh, enabling the generation of smoother and more detailed 3D outputs while maintaining the geometric consistency of the reconstructed object.

[0068] Table 1: T 3 Quantitative comparison results on Bench

[0069]

[0070] As shown in Table 1 above, 3 The quantitative comparison results are given on Bench. The evaluation of the generation ability of our method spans three categories: single object, single object and environment, and multiple objects. Time(s) indicates the time required to complete the generation of a 3D object, Quality indicates the quality of the generated 3D scene, and Alignment indicates the degree of alignment between the generated 3D scene and the text prompt. Average indicates the average score of the two, and the best result is highlighted in bold.

Claims

1. A 3D generation method based on semantically enhanced hybrid reconstruction, characterized in that: The steps include: Step (1), using the GPT-4 model to semantically enhance the input original text; Step (2), generating a two-dimensional image from the text; in order to reduce the influence of the complex background in the generated image, the semantically enhanced text is input into a text diffusion model so that the generated two-dimensional image is helpful for three-dimensional reconstruction; the text diffusion model has been fine-tuned using a low-rank adaptation model; Step (3), generating multiple views from the two-dimensional image, inputting the image generated in the Vincent graph diffusion model into the multi-view diffusion model to generate images at different viewing angles for three-dimensional reconstruction; Step (4), using the image and text to perform 3D hybrid reconstruction, taking the generated multi-view and the original text as input, and generating a three-plane representation through a 3D large-scale reconstruction model; Step (5), generating a three-dimensional mesh from the three-plane representation; Taking the three-plane representation as input, FlexiCubes is used to perform isosurface extraction to obtain the final 3D mesh.

2. The 3D generation method based on semantic enhanced hybrid reconstruction according to claim 1, characterized in that Step (1) is as follows: Input the original text and instruction prompts into GPT-4 to obtain the semantically enhanced text, as shown in formula (1); T s =GPT-4(T o ,T Instruction ) (Formula 1) Among them, T o Represents the original input text, T s represents the semantically enhanced text, T Instruction Indicates a command prompt.

3. The 3D generation method based on semantic enhanced hybrid reconstruction according to claim 2, characterized in that Step (2) is as follows: 2-1. The semantically enhanced text T s Input to the Wensheng graph diffusion model fine-tuned with LoRA to generate a semantically aligned two-dimensional image, as shown in formula (2); in, Indicated by D LoRA Fine-tuned Wensheng graph diffusion model D SDXL and the initial view produced by LoRA fine-tuning; 2-2. Introduce the trainable low-rank matrix into the Transformer module of the model to achieve efficient fine-tuning without modifying the weights of the original Wensheng graph diffusion model; for the pre-trained weight matrix W0, the LoRA expression updated by low-rank decomposition is as shown in formula (3); W0+ΔW=W0+BA (Formula 3) Among them, W0∈R d×k , B∈R d×r , A∈R r×k , r<<min(d,k), r represents the rank of the LoRA module, d and k represent the dimensions of the matrix, and the corresponding forward propagation formula is modified as follows, as shown in formula (4); h=W0x+ΔWx=W0x+BAx (Formula 4) Among them, the weight matrix W0 remains frozen and does not accept gradient updates. ΔW represents the intermediate variable. Instead, only the parameters A and B are fine-tuned. A is initialized with a random Gaussian value and B is initialized to zero. x represents the input data. 2-3. LoRA introduces an additional downsampling and upsampling auxiliary branch based on the original linear layer, and fine-tunes by training this auxiliary branch; the loss function guiding the fine-tuning process is defined as shown in formula (5): Among them, ε represents the real noise, ε θ represents the predicted noise, express expectations; The latent variable at time step t is defined as z0=Enc(x0), where x0 is the input image, Enc is the encoder that maps the image to a low-dimensional latent space, and τ θ (y) represents the encoded text condition, and y represents the text condition.

4. The 3D generation method based on semantic enhanced hybrid reconstruction according to claim 3, characterized in that Step (3) is as follows: Taking the image generated by the text as input, a multi-view image is generated; a set of geometrically consistent multi-view images is generated using a multi-view diffusion model, as shown in formula (6); Among them, I vp represents the generated multi-view image, D MV represents the multi-view diffusion model, Represents the initial view.

5. A framework for generating three-dimensional objects from text according to claim 4, characterized in that Step (4) is as follows: 4-1. Cross-modal multi-view fusion, generating multi-view images Combined with the original input text T o , a three-dimensional large-scale reconstruction model is used to generate a three-plane representation, as shown in formulas (7) and (8); F text =D BERT (T o ) (Formula 8) in, Indicates the kth th image embedding of multi-view images, where Indicates the kth th Global visual embedding of multi-view images, Indicates the kth th The i-th patch encoding of the visual cue of the view, D VIT-DINO represents the image encoder, D BERT represents the semantic encoder; M represents D VIT-DINO The length of the output sequence; F text represents text embedding, L stands for D BERT The length of the output sequence; 4-2. Apply the cross-modal fusion strategy on each multi-view image, and then adopt the multi-view fusion strategy on all multi-view images, as shown in formulas (9) and (10); Among them, CFusion represents the cross-modal fusion operation, VFusion represents the multi-view fusion operation; for the kth th The modality fusion of multi-view images and text descriptions adopts a direct fusion method; Replace with Global visual embedding in and F text Global text embedding in The modality fusion feature is constructed by element-wise summation of At the same time, the original local semantic embedding is retained; the modal fusion features of multiple views are fused using cascade operators Perform view fusion to obtain a unified feature representation F fus , integrating cross-view and cross-modal information; 4-3. Using the three-plane decoder to process the fused feature representation F fus ; The three-plane decoder uses a cross-attention mechanism to combine the three-plane tokens with the fused feature representation F fus are linked and then passed through self-attention and multi-layer perceptron layers to generate a refined three-plane token F triplane , this process is mathematically expressed as shown in formula (11); F triplane = TriplaneDecoder(F fus , τ) (Equation 11) Among them, τ represents the learnable token sequence, F triplane represents a triplane token, and TriplaneDecoder represents a triplane decoder; 4-4. Deconvolution and reshaping operations are performed on the three-plane tokens to form the final three-plane representation R triplane , this process is shown in formula (12); R triplane = Reshape(DeConv(F triplane )) (Formula 12) Among them, Reshape represents a reshaping operation, and DeConv represents a deconvolution operation.

Images