Three-dimensional model generation method and device

Abstract

The invention discloses a three-dimensional model generation method and device, and relates to the technical field of three-dimensional model generation, and the method comprises the following steps: constructing a pseudo graph diffusion model based on a preset three-dimensional model data set; inputting text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information; and refining the three-plane representation information based on a fractional distillation sampling technology to obtain a three-dimensional model. According to the method, the understanding of text description is improved, so that the limitation of an existing method is overcome while high efficiency and quality are kept, and actual requirements are met.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional model generation, and in particular to a three-dimensional model generation method and device. Background Art

[0002] In existing research, the challenges of generating 3D models from text descriptions mainly focus on two aspects: utilizing 3D diffusion models and improving 2D diffusion models based on 3D representations.

[0003] Current research is often devoted to training 3D diffusion models on popular 3D datasets, but these models have difficulty generating complex objects, especially when dealing with text descriptions. Existing methods, such as Point·E and Shap·E, have limited understanding of complex concepts despite being trained on 3D datasets. Other models use three-plane representations, which achieve efficient mesh generation but still have limitations in understanding some complex cues.

[0004] On the other hand, existing research focuses on extracting 3D knowledge by optimizing pre-trained 2D diffusion models. These methods, such as DreamFusion and SJC, obtain rendering results from 2D diffusion models through fractional distillation sampling to optimize 3D representations. However, these methods usually require a large number of optimization iterations and have certain deficiencies in diversity.

[0005] By reviewing the existing technical solutions, we realize that solving the problem of text-to-3D model conversion requires overcoming the challenges of time efficiency, limitations on text understanding, and generation quality.

[0006] The most similar implementations at present: This type of method focuses on training diffusion models on popular 3D datasets, such as Shap·E and NFD. These models can generate 3D shapes to a certain extent, but due to the lack of 2D image data priors and the low quality of 3D data, their understanding and generation capabilities still need to be improved when processing complex text descriptions.

[0007] Therefore, in order to meet actual needs, a three-dimensional model generation technology is now provided. Summary of the invention

[0008] In view of the defects existing in the prior art, the purpose of this application is to provide a three-dimensional model generation method and device, which can overcome the limitations of the existing methods and meet practical needs while maintaining high efficiency and quality by improving the understanding of text descriptions.

[0009] In order to achieve the above objectives, the technical solution adopted by this application is:

[0010] In a first aspect, the present application provides a method for generating a three-dimensional model, the method comprising the following steps:

[0011] Based on the preset 3D model data set, a pseudo-image diffusion model is constructed;

[0012] Inputting text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information;

[0013] Based on the fractional distillation sampling technology, the three-plane representation information is refined to obtain a three-dimensional model.

[0014] On the basis of the above technical solution, the method further comprises the following steps:

[0015] A lightweight iterative refinement process is performed on the three-dimensional model to obtain a refined three-dimensional model.

[0016] On the basis of the above technical solution, the three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

[0017] Based on the above technical solution, the length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

[0018] Based on the above technical solution, the number of parameters of the three-plane representation information is 3*H*W*C; wherein,

[0019] H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

[0020] In a second aspect, the present application provides a three-dimensional model generation method and device, the device comprising:

[0021] A model building module, which is used to build a pseudo-image diffusion model based on a preset three-dimensional model data set;

[0022] A three-plane information acquisition module, which is used to input text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information;

[0023] The model generation module is used to refine the three-plane representation information based on the fractional distillation sampling technology to obtain a three-dimensional model.

[0024] On the basis of the above technical solution, the device further comprises:

[0025] The model refinement module is used to perform lightweight iterative refinement processing on the three-dimensional model to obtain a refined three-dimensional model.

[0026] On the basis of the above technical solution, the three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

[0027] Based on the above technical solution, the length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

[0028] Based on the above technical solution, the number of parameters of the three-plane representation information is 3*H*W*C; wherein,

[0029] H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

[0030] Compared with the prior art, the advantages of this application are:

[0031] This application overcomes the limitations of existing methods and meets practical needs by improving the understanding of text descriptions while maintaining efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of the steps of the three-dimensional model generation method according to an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the principle of a pseudo-image diffusion model of the three-dimensional model generation method of an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the principle of depth-aware three-plane fitting of the three-dimensional model generation method according to an embodiment of the present application;

[0036] Figure 4 A structural block diagram of a three-dimensional model generating device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0039] The embodiments of the present application provide a three-dimensional model generation method and device, which improve the understanding of text descriptions, thereby overcoming the limitations of existing methods and meeting practical needs while maintaining efficiency and quality.

[0040] In order to achieve the above technical effects, the overall idea of ​​this application is as follows:

[0041] A three-dimensional model generation method, the method comprising the following steps:

[0042] S1. Construct a pseudo-image diffusion model based on a preset three-dimensional model dataset;

[0043] S2, inputting text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information;

[0044] S3. Based on the fractional distillation sampling technology, the three-plane representation information is refined to obtain a three-dimensional model.

[0045] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0046] First, see Figures 1 to 3 The present invention provides a method for generating a three-dimensional model, which comprises the following steps:

[0047] S1. Construct a pseudo-image diffusion model based on a preset three-dimensional model dataset;

[0048] S2, inputting text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information;

[0049] S3. Based on the fractional distillation sampling technology, the three-plane representation information is refined to obtain a three-dimensional model.

[0050] The embodiments of the present application solve the problems of efficiency, generalization and quality in generating three-dimensional models from text descriptions. Existing methods are usually limited in terms of time efficiency and understanding of text, such as the need to perform multiple iterative optimizations on a three-dimensional model, which is time-consuming, or the generated objects are limited to certain specific categories. Therefore, the embodiments of the present application provide a three-dimensional model generation method based on a three-plane diffusion model and fractional distillation sampling. This method utilizes both two-dimensional image data and three-dimensional model data, uses a small amount of text-3D pairs to fine-tune the existing text-image diffusion model, and performs lightweight iterative refinement through fractional distillation sampling, which significantly improves the understanding of the input text description and improves the generation efficiency and quality of the three-dimensional model.

[0051] In the embodiments of the present application, by improving the understanding of text descriptions, the limitations of existing methods are overcome while maintaining efficiency and quality, thereby meeting practical needs.

[0052] Furthermore, the method further comprises the following steps:

[0053] A lightweight iterative refinement process is performed on the three-dimensional model to obtain a refined three-dimensional model.

[0054] Furthermore, the three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

[0055] Furthermore, the length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

[0056] Furthermore, the number of parameters of the three-plane representation information is 3*H*W*C; wherein,

[0057] H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

[0058] It should be noted that the technical advantages of the technical solution of the embodiment of the present application are:

[0059] (1) Innovative construction of a three-plane diffusion model: The key innovation of this application in terms of three-dimensional model generation is the introduction of a three-plane diffusion model. This model is based on a multi-channel pseudo-image and improves the ability to understand the input text description through lightweight iterative refinement, effectively improving the generation efficiency and model quality.

[0060] (2) Superiority of the pseudo-image diffusion model: By using the pseudo-image diffusion model, the present application effectively utilizes the information of two-dimensional image data and three-dimensional model data, and fine-tunes the existing text-image diffusion model through a small amount of text-3D pairs, thereby achieving the understanding of complex text descriptions, thereby improving the generation efficiency and quality of the three-dimensional model.

[0061] (3) Innovative design of lightweight iterative refinement: This application introduces a lightweight iterative refinement process on the generated 3D model. Through the ScoreDistillationSampling strategy, it effectively improves the consistency and quality of the generated model, while achieving high-quality generation at a lower computational cost.

[0062] (4) Integration and optimization of existing technologies: This application integrates the existing text-image diffusion model and applies its advantages to the field of three-dimensional model generation, so that even with a small number of samples, a three-dimensional model with high quality and high consistency can be generated, thereby effectively making up for the shortcomings of the existing technology.

[0063] (5) Overall process of generating a 3D model based on text description: This application provides a complete process from text description to the generation of a 3D model, covering the construction of a three-plane diffusion model, the training of a pseudo-graph diffusion model, and a lightweight iterative refinement process, forming a technical solution with practical application prospects.

[0064] These key innovations constitute the core of the technical solution of this application, and provide strong technical support for solving the efficiency, generalization and quality issues of generating 3D models from text descriptions. In the drafting of the claims, special attention should be paid to protecting these innovations to ensure that the technical contributions of this application are fully covered.

[0065] The advantages of the technical solution of the embodiment of the present application are:

[0066] (1) Significant improvement in generation efficiency and quality: Compared with existing 3D model generation methods, this application introduces a three-plane diffusion model and fractional distillation sampling, which effectively improves the generation efficiency and quality of the generated model. By combining the text-image diffusion model and lightweight iterative refinement in the generation process, this application can generate high-quality 3D models in a short time, greatly reducing the generation time compared to traditional methods.

[0067] (2) It makes up for the problem of 3D data scarcity: In the prior art, the lack of high-quality and large-scale 3D datasets has limited the generalization ability of 3D generative models. However, this application successfully combines the information of 2D images with the generation of 3D models by cleverly utilizing the text-image diffusion model and fractional distillation sampling, thus overcoming the problem of data scarcity and improving the generalization performance of the generative model.

[0068] (3) Realizing the understanding and generation of complex descriptions: This application uses a three-plane diffusion model to better understand text descriptions containing complex concepts. During the training process of the pseudo-graph diffusion model, by introducing fractional distillation sampling, this application successfully achieves efficient understanding and generation of complex descriptions, thereby more accurately expressing the user's intentions in the generated three-dimensional model.

[0069] (4) Improved model consistency and diversity: By introducing lightweight iterative refinement, the present application injects more consistency and diversity into the generated 3D model. Compared with the existing methods that require a large number of optimization iterations, the present application can effectively improve the diversity of the generated model while maintaining consistency, making the generated model richer.

[0070] In summary, this application has successfully solved the problems of low efficiency, poor generalization, and low quality in the existing technology for three-dimensional model generation through technological innovation, and has obvious technical advantages.

[0071] The technical solution of the embodiment of the present application aims to generate a three-dimensional model that conforms to the text description according to the input text description, and the representation of the three-dimensional model is a triangular mesh. The complete operation process of the embodiment of the present application is as follows:

[0072] (1) The pseudo-image diffusion model is trained based on the 3D model dataset Objaverse-LVIS. This process only needs to be performed once, and the trained model can be reused afterwards. The training process of the pseudo-image diffusion model will be described in detail later.

[0073] (2) Use the pre-trained pseudo-graph diffusion model to process the input text description, that is, use the CLIP text encoder to encode it into a text feature vector;

[0074] Six feature maps of size 4*32*32 are sampled from the normal distribution as the initial latent space noise z of the pseudo-image diffusion model. T ;

[0075] The diffusion network of the pre-trained pseudo-graph diffusion model is used to process the text feature vector and the Caintiq space noise to obtain 6 latent space features z of size 4*32*32 after one-step denoising. T-1 , repeat this process T times (usually 50 times) until 6 latent space features z of size 4*32*32 are obtained 0 , which is called the latent space feature sampled by the pseudo-graph diffusion model, and the decoder of the variational encoder-decoder in the pseudo-graph diffusion model converts z 0 Convert to 6 pseudo images x of size 3*256*256 0, stack every two images of these 6 pseudo images into a 6*256*256 feature map, and get a total of three feature maps as the three-plane representation of the sampled object.

[0076] (3) Use fractional distillation sampling to perform lightweight refinement on the three-plane representation of the object: In each iteration, a random camera viewpoint is taken toward the origin, and the parameterized renderer obtained in (1), i.e., the shared multi-layer perceptron, is used to perform volume rendering on the three-plane representation to obtain an RGB rendering of the object at the random camera viewpoint. After adding Gaussian noise to the RGB rendering, a pre-trained image diffusion model (such as DeepFloyd-IF, which is an open source model and can be obtained directly) is input. The image diffusion model also receives text as input. The pre-trained neural network calculates the noise estimate of the noisy RGB rendering. According to the calculation process of the paper "DreamFusion: Text-to-3D using 2D Iffusion", the optimized gradient of the original RGB rendering is obtained, which is then back-propagated to the three-plane feature map and optimized using gradient descent. This process continues for several iterations (usually 2000) to obtain the optimized three-plane feature map. Sample several grid points in space (usually with a resolution of 512*512*512), calculate their spatial features based on the three-plane feature map, and use the parametric renderer obtained in (1) to obtain the R, G, B, and opacity at these positions, and use the marching cube algorithm to obtain the triangular mesh of the object based on the opacity. Use xatlas to perform UV unfolding on the triangular mesh, calculate the three-dimensional coordinates of each point on the 1024*1024 resolution color map, calculate their spatial features on the three-plane feature map, and use the parametric renderer to obtain the R, G, B, and opacity at these positions. Use RGB as the color of these points to obtain the color map of the object. Triangular mesh and color map Figure 1 As the generated three-dimensional model.

[0077] The embodiment of the present application requires the construction of a pseudo-image diffusion model responsible for fast sampling of three-dimensional objects. It is divided into two stages. In the first stage, a three-dimensional representation is fitted for each three-dimensional object, and in the next stage, a text-conditioned pseudo-image diffusion model adjusted from a pre-trained two-dimensional diffusion model is constructed. The second part is a lightweight refinement using fractional distillation sampling (SDS), which converges quickly and further improves the quality of the sampled three-dimensional objects. As shown in the attached figure of the specification Figure 2 As shown,

[0078] Specifically, the drawings in the specification Figure 2It mainly explains how to use the trained diffusion model to generate a 3D model after the trainable diffusion model is established. In the first stage of generation, diffusion model sampling (i.e., Diffusion Sampling), the user enters a text description of the 3D model to be generated (here "askiing penguin wearing a puffy jacket"), and the trained pseudo-image diffusion model (represented by the two blue trapezoids on the left) gradually recovers the three-plane parameter representation of the 3D object from the noise. Its content corresponds to the six gray background images on the left. This process takes about 3 seconds. The three-plane parameter representation is rendered into a multi-view image of the object by the fitted parameterized volume renderer, i.e., the three images shown in the middle of the attached figure. In the second stage of generation, lightweight refinement (i.e., Lightweight Refinement), by using the pre-trained image diffusion model (represented by the two orange trapezoids on the right), the score distillation sampling technology is used to optimize the three-plane parameter representation according to the text description entered by the user to improve the quality of the 3D model. In actual operation, this process takes about 3 minutes.

[0079] The technical solution of the embodiment of the present application performs depth perception three-plane fitting based on the three-plane representation information, and the specific operations are as follows:

[0080] Fit a corresponding three-plane representation to each object in the dataset Objaverse-LVIS, and represent each object as three axis-aligned feature maps F(p) = [F_xy(p_xy), F_xz(p_xz), F_yz(p_yz)]. Then, the color (RGB) and density (opacity) at point p are decoded by a shared multi-layer perceptron D(·; θ). The expected color on the camera ray r is calculated in a volume rendering manner. In order to encourage the correct underlying geometry, the object mask M(r) and the expected depth D(r) along the ray are also supervised. The operation of this process is detailed in the attached figure of the specification. Figure 3 .

[0081] The drawings in the specification Figure 3 , explains the process of obtaining a rendered image from a three-plane parameter representation in three-plane fitting and supervising on multiple signals. Figure 3The left side of the figure is the camera, which is represented by the internal and external parameter matrix of the camera. The three-dimensional object is represented by the three-plane parameters, which are the two-dimensional planes perpendicular to the right side of the camera. These planes are two-dimensional feature maps with length and width H=W=256, and the number of feature channels is C=6. The parameter quantity of the entire three-plane representation is 3HWC. The object is regarded as being in a bounding box of the three planes. Each ray emitted from the camera will pass through a series of points in space. The spatial feature corresponding to the point p on the ray is the feature obtained by aggregating the corresponding features of p projected onto the three planes according to axis alignment (that is, the green, blue and purple blocks in the figure, with a parameter quantity of 3C). The specific expression is as follows:

[0082] F(p)=[F xy (p xy ), F xz (p xz ), F yz (p yz )].

[0083] The points on the ray are sampled to obtain a series of points, which are decoded by a shared multi-layer perceptron to obtain the spatial opacity σ and color c of these points, which are expressed as follows:

[0084] c(p), σ(p)=D(F(p); θ).

[0085] Then the color corresponding to the ray is obtained through the volume rendering process, which is expressed as follows:

[0086]

[0087]

[0088] The object mask and depth corresponding to the ray can also be obtained. The specific expressions of the two are as follows:

[0089]

[0090] The parameters to be optimized in this process are the three-plane parameter representation of each object and the multi-layer perceptron D in the shared volume renderer. The optimization goal is to make the color, mask and depth map of each object rendered from multiple perspectives as close as possible to the actual rendering result, that is, to use the following three losses (color loss, mask loss, depth loss). Among them, C(r), M(r), D(r) are the actual values ​​of the color, mask and depth map, BCE is the cross entropy loss function, represents element-wise multiplication, λ L1 are predefined parameters used to balance different losses.

[0091]

[0092]

[0093]

[0094] In addition, the following loss is applied to the three-plane parameter representation of each object:

[0095] L L2 =‖F‖ 2 ;

[0096] L TV =∑ 1≤i<H ‖F(i,.)-F(i+1,.)‖ 1 +∑ 1≤i<W ‖F(.,i)-F(.,i+1)‖ 1 ;

[0097] L hull =‖(1-O)*F‖ 1 .

[0098] F is the feature plane. For simplicity, the xy, yz, and xz subscripts are omitted here. The actual loss calculation is done on all three feature planes. O is the object mask rendered at three orthogonal viewing angles for each object, i.e., L in the figure. hull The three black and white pictures above. L2 ,L TV ,L hull They are L2 norm loss, total variation loss, and perspective envelope loss. This process uses stochastic gradient descent for parameter optimization. The overall optimization goal is as follows: mask ,λ depth It is a predefined parameter used to balance different losses. Its specific form is as follows:

[0099] L=L color +λ mask L mask +λ depth L depth +λ L2 L L2 +λ TV L TV +λ hull L hull .

[0100] The goal of this step is to obtain the three-plane parameter representation of each 3D model and the shared parameterized renderer data information for volume rendering from a large-scale 3D model dataset. The parameters here are the parameters of the multi-layer perceptron, so that the three-plane parameter representation of each 3D model can be rendered into images of various perspectives by the shared parameterized renderer, and these images are as consistent as possible with the images of the corresponding 3D model at these perspectives. The whole process represents the parameterized 3D model as a three-plane parameter representation.

[0101] The pseudo image diffusion process of the pseudo image diffusion model in the technical solution of the embodiment of the present application is described in detail as follows:

[0102] The Stable Diffusion model (Stable Diffusion 1.5) is used as the base model to build a pseudo-image diffusion model without changing its architecture. Two main challenges are: (1) how to output multiple multi-channel feature maps simultaneously under the architecture, and (2) how to model cross-plane consistency to form a reasonable underlying 3D geometry. Each three-plane representation is regarded as a set of 6 pseudo images and they are stacked along the batch dimension as the model input. A global pseudo-image attention block is designed, which is adapted from the original self-attention block without changing the parameters, and the self-attention calculation on a single image is changed to a global self-attention calculation on 6 pseudo images. In this way, the feature interactions between the three planes are enriched and the 3D knowledge is incorporated into the global attention block. To identify the input image, a learnable embedding is assigned to each pseudo image and added to the time-step embedding in the cross-attention block.

[0103] In detail, this section uses the three-plane parameter representation of the three-dimensional model of each object, the text description of each object in the dataset, the open source image-text description pair dataset LAION5B, and the pre-trained open source image diffusion model - the Stable Diffusion model. The goal is to fine-tune the Stable Diffusion model through a stochastic gradient descent process to obtain a pseudo-image diffusion model so that the pseudo-image diffusion model can generate a three-plane parameter representation of a new object based on the input text. The Stable Diffusion Model mainly consists of three parts, namely 1) a pre-trained CLIP (Text-Image Comparative Pre-training Model) parameter model for processing text. CLIP can process text with indefinite length into a vector embedding parameter representation of a certain length. The parameters of the CLIP model are fixed during the fine-tuning process and are not optimized. 2) A Variational Auto Encoder that encodes and decodes images. VAE can encode a three-channel image of size H×W into a vector of size The four-channel feature map (called latent feature) is decoded from the latent feature as realistically as possible into the original three-channel image. The parameters of this part are fixed during the fine-tuning process. 3) According to the vector embedding of the text and the time step t, the latent diffusion model (latent diffusion) is used to diffuse and denoise the noisy latent feature. The latent diffusion includes a denoising UNet model and a parameter model for embedding t. The UNet model predicts the noise size in the input noisy latent feature based on the embedding representation of t and the vector embedding representation of the text. The parameters of this part of the UNet model will be optimized during the fine-tuning process.

[0104] In each batch of stochastic gradient descent, a three-plane parameter representation-text description pair or a picture-text description pair is randomly selected for fine-tuning training. For batches using three-plane parameter representation-text description pairs, let the batch size be B, then the dimensions of the three-plane parameter representation of a batch are [B, 3, C, H, W], where C = 6. The feature map of each plane is split into two three-channel images to obtain feature images of [6B, 3, H, W] dimensions, called pseudo images. These images are encoded into corresponding latent features through VAE, with dimensions of [6B, 4, H / 8, W / 8]. The text description is processed by CLIP into a vector embedding representation. Randomly sample time step t, add random Gaussian noise of the same dimension to the latent feature, and obtain a noisy latent space feature, where the noise size is related to t and is a pre-defined parameter. The latent space diffusion model embeds the time step t and accepts the text vector embedding representation and the noisy latent space features. In addition, the embedding of time step t is added with the embedding to be optimized to distinguish the 6 pseudo images. In the UNet calculation process, the calculation method of the image self-attention module is changed from self-attention on a single image to self-attention on 6 pseudo images. This change does not change the parameters, but only changes the calculation method, and does not affect the output dimension. The goal is to establish a connection between the 6 pseudo images. The latent space diffusion model predicts random Gaussian noise in the noisy latent feature after calculation. The optimization goal is to make the predicted Gaussian noise as consistent as possible with the Gaussian noise added at the beginning. For batches of image-text description data, the above process is basically the same. The image dimension is [B, 3, H, W], the latent feature dimension is [B, 4, H / 8, W / 8]. The embedding of time step t is not added with the embedding to be optimized during the calculation process. The calculation method of the self-attention module is not changed, and the optimization goal is also the same. In this way, stochastic gradient descent is used to fine-tune the stable diffusion model on the three-plane parameter representation-text description pairs and image-text description pairs to obtain the final pseudo-image diffusion model.

[0105] The process of lightweight refinement of the three-dimensional model in the technical solution of the embodiment of the present application is described in detail as follows:

[0106] After obtaining the pseudo-image diffusion model, it can be diffusely sampled directly from it in seconds, and then further refined using 2D fractional distillation sampling. The three-planes generated by pseudo-image diffusion are optimized using SDS loss along the shared decoder. To prevent over-optimization of 3D shapes, both appearance and geometry are optimized by sampling only small noise levels in SDS. Due to good initialization, it significantly accelerates convergence, thereby achieving efficient 3D content generation.

[0107] In detail, the pseudo-image diffusion model mentioned above, the shared parameterized volume renderer, and the pre-trained open-source image diffusion model are used. According to the text description entered by the user, the goal is to generate a high-quality parameter representation of the 3D model. The pseudo-image diffusion model performs diffusion sampling according to the text description entered by the user, and uses the DDPM sampling algorithm to obtain the three-plane parameter representation of the 3D model. Then, the parameter representation is optimized using the trained open-source image diffusion model, i.e., a lightweight refinement process. During the optimization process, stochastic gradient descent is used to optimize the three-plane parameter representation and the parameterized volume renderer. In each batch, the parameterized volume renderer is used to render several images of random viewpoints for the three-plane representation, and then the image and text description are input into the pre-trained image diffusion model. The optimization target of the image is obtained according to the fractional distillation sampling algorithm, and the parameters of the volume renderer and the three-plane representation are optimized using back propagation. This process is iterated about 2000 times and takes 3 minutes. The principle is to make the distribution of the rendered image and the real image modeled by the image diffusion model according to the text description as close as possible at each viewpoint, thereby improving the generation quality of the 3D model. The fractional distillation sampling algorithm was proposed by the paper "DreamFusion: Text-to-3D using 2D Diffusion". For an image to be optimized, random Gaussian noise is added, and the noisy image and text are used as the output of the pre-trained image diffusion model to predict the Gaussian noise. The optimization direction of the image to be optimized in gradient descent can be calculated by the actual added Gaussian noise and the predicted Gaussian noise.

[0108] In summary, the technical details of the technical solution of the embodiment of the present application are:

[0109] Data level:

[0110] The 20,000 3D objects from the LVIS subset of Objaverse[9] and their corresponding text descriptions in Cap3D

[23] are used as the 3D dataset. In the three-plane fitting stage, 100 random views with a resolution of 512×512 are generated for each object by scaling them to fit a bounding box of unit length and placing them at the origin.

[0111] Three-plane fitting:

[0112] The dimensions of the tri-plane are defined as H = W = 256, C = 6. The shared decoder for volume rendering is a 4-layer MLP with each hidden layer of width 32. It is trained on a set of 500 objects. Subsequently, the shared decoder is frozen and the tri-plane representation is fitted for each object independently, implementing a linearly decaying learning rate. The balancing coefficients of the loss term are as follows: λL1 = 0.2, λmask = 0.1, λdepth = 0.5, λL2 = 0.05, λTV = 0.05, λhull = 1.0. The tri-plane is scaled to [-1, 1] to ensure a similar variance to the image, allowing for easy diffusion training later. The process of fitting one object takes 2 minutes on a single A100 GPU.

[0113] Pseudo-image diffusion:

[0114] We started with Stable Diffusion v2.1 and trained the learnable embeddings using a learning rate of 0.001, and the remaining parameters were trained at a learning rate of 0.0001 for 15,000 iterations. Training used 256×256 real and fake images from LAION-5B with a probability of 50% each. The equivalent batch sizes for tri-plane and real images were set to 384 and 2304, respectively, to match their computational overhead. Text embeddings were discarded with a probability of 5%, and the unconditional diffusion model was jointly trained to achieve bootstrapping without a classifier. During diffusion training, the v-objective and Min-SNR strategies were used to enhance convergence using the noise size defined by DDPM. The entire diffusion training process took about 36 hours using 8 A100 GPUs.

[0115] Text to 3D Generation:

[0116] In pseudo image sampling, 50 sample steps are performed using the DDPM scheduler with a classifier free bootstrapping ratio of 5.0. Sampling a 256×256 resolution three-plane takes 3 seconds. The three-plane based NeRF is optimized with 2000 steps and a batch size of 1, which takes 3 minutes on an A100 GPU. The time step t is sampled in U(0.1,0.5) to introduce small corrections and prevent excessive changes. The classifier free bootstrapping adopted in SDS is 20.

[0117] Of course, it should be noted that when examining the technical solution of this application, it is noted that there may be some alternative solutions in the process of achieving the purpose of the application. However, these alternative solutions cannot completely replace the innovation and uniqueness of this application.

[0118] (1) Existence of alternatives: There are some possible alternatives, such as using different text-image models for fine-tuning optimization, which does not touch the essence of the method but is only an implementation alternative.

[0119] (2) Limitations and irreplaceability: The key innovation of this application is the introduction of a three-plane diffusion model and fractional distillation sampling, which successfully solves the bottleneck problems of efficiency and generalization in the existing technology by combining two-dimensional image and three-dimensional model data in the generative model. This comprehensive utilization method, as well as the lightweight iterative refinement strategy, makes this application unique and advanced in the field of three-dimensional model generation.

[0120] (3) Different implementation methods: Alternative solutions may use different generative models, iterative optimization algorithms or data sets, but it is difficult to achieve the comprehensive effects of high efficiency, high quality and high generalization achieved in this application. In all implementation processes, the three-plane diffusion model and fractional distillation sampling used in this application are essential key structures, and lightweight iterative refinement is a key step to improve the quality of the generative model.

[0121] Overall, although there are some possible alternatives, considering the excellent performance of this application in terms of efficiency, quality and generalization, these alternatives cannot replace the remarkable achievements made by this application in the field of three-dimensional model generation.

[0122] Second, see Figure 4 The present invention provides a three-dimensional model generation device, the device comprising:

[0123] A model building module, which is used to build a pseudo-image diffusion model based on a preset three-dimensional model data set;

[0124] A three-plane information acquisition module, which is used to input text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information;

[0125] The model generation module is used to refine the three-plane representation information based on the fractional distillation sampling technology to obtain a three-dimensional model.

[0126] In the embodiments of the present application, by improving the understanding of text descriptions, the limitations of existing methods are overcome while maintaining efficiency and quality, thereby meeting practical needs.

[0127] Furthermore, the device also includes:

[0128] The model refinement module is used to perform lightweight iterative refinement processing on the three-dimensional model to obtain a refined three-dimensional model.

[0129] Furthermore, the three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

[0130] Furthermore, the length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

[0131] Furthermore, the number of parameters of the three-plane representation information is 3*H*W*C; wherein,

[0132] H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

[0133] It should be noted that the three-dimensional model generation device of the embodiment of the present application and the three-dimensional model generation method mentioned in the first aspect have similar technical principles in terms of technical solutions, technical problems and technical effects, so they will not be elaborated here.

[0134] In the description of the present application, it should be noted that the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0135] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0136] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A three-dimensional model generation method, characterized in that: The method comprises the following steps: Based on the preset 3D model data set, a pseudo-image diffusion model is constructed; Inputting text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information; Based on the fractional distillation sampling technology, the three-plane representation information is refined to obtain a three-dimensional model.

2. The three-dimensional model generation method according to claim 1, characterized in that: The method further comprises the following steps: A lightweight iterative refinement process is performed on the three-dimensional model to obtain a refined three-dimensional model.

3. The three-dimensional model generation method according to claim 1, characterized in that: The three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

4. The three-dimensional model generation method according to claim 3, characterized in that: The length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

5. The three-dimensional model generation method according to claim 3, characterized in that: The number of parameters of the three-plane representation information is 3*H*W*C; wherein, H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

6. A three-dimensional model generation device, characterized in that: The device comprises: A model building module, which is used to build a pseudo-image diffusion model based on a preset three-dimensional model data set; A three-plane information acquisition module, which is used to input text description information of the model to be constructed into the pseudo-graph diffusion model to obtain corresponding three-plane representation information; The model generation module is used to refine the three-plane representation information based on the fractional distillation sampling technology to obtain a three-dimensional model.

7. The three-dimensional model generating device according to claim 1, characterized in that: The device also includes: The model refinement module is used to perform lightweight iterative refinement processing on the three-dimensional model to obtain a refined three-dimensional model.

8. The three-dimensional model generating device according to claim 6, characterized in that: The three-plane representation information is used to construct a two-dimensional feature map of the model to be constructed with three axes aligned.

9. The three-dimensional model generating device according to claim 8, characterized in that: The length and width of the two-dimensional feature map are both 256 pixels, and the number of length feature channels of the two-dimensional feature map is 6.

10. The three-dimensional model generating device according to claim 8, characterized in that: The number of parameters of the three-plane representation information is 3*H*W*C; wherein, H is the number of pixels corresponding to the length of the two-dimensional feature map, W is the number of pixels corresponding to the width of the two-dimensional feature map, and C is the number of length feature channels of the two-dimensional feature map.

Images