Text-to-three-dimensional generation method, device and equipment

Abstract

The invention discloses a method, a device and equipment for generating a three-dimensional object from a text, which can improve the credibility of generating a three-dimensional object from the text. The method comprises the following steps: performing semantic analysis on text description to obtain text embedding; based on text embedding, determining an initial neural radiation field model corresponding to the to-be-modeled object; updating the initial neural radiation field model through the two-dimensional guidance model and the three-dimensional guidance model, and determining a neural radiation field model corresponding to the to-be-modeled object; and performing multi-view rendering on the neural radiation field model, and determining a three-dimensional object corresponding to the to-be-modeled object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is filed based on and claims the priority of U.S. patent application No. 63 / 578,702, filed on August 25, 2023. The entire contents of the patent application are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for generating text into three dimensions. Background Art

[0004] Neural Radiance Fields (NeRF) is a method of representing three-dimensional (3D) scenes using neural networks. Its basic principle is to combine computer graphics and deep learning to learn continuous 3D scene representation from a set of sparse 2D images and generate high-quality new perspective images in arbitrary viewing directions.

[0005] Currently, how to generate accurate and credible 3D content is a long-standing problem in the field of computer vision and graphics. Summary of the invention

[0006] The embodiments of the present application are intended to provide a method, device and apparatus for generating three-dimensional text.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, a method for generating a text to a three-dimensional image is provided, comprising:

[0009] Perform semantic analysis on the text description to obtain text embedding;

[0010] Based on the text embedding, determining an initial neural radiation field model corresponding to the object to be modeled;

[0011] The initial neural radiation field model is updated by using the two-dimensional guidance model and the three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled;

[0012] The neural radiation field model is rendered from multiple perspectives to determine a three-dimensional object corresponding to the object to be modeled.

[0013] In a second aspect, a device for generating a text-to-three-dimensional image is provided, comprising:

[0014] A text parsing model, configured to perform semantic analysis on the text description to obtain text embedding;

[0015] An initialization module, configured to determine an initial neural radiation field model corresponding to the object to be modeled based on the text embedding;

[0016] An updating module is configured to update the initial neural radiation field model through a two-dimensional guidance model and a three-dimensional guidance model to determine a neural radiation field model corresponding to the object to be modeled;

[0017] A rendering module is configured to perform multi-view rendering on the neural radiation field model to determine a three-dimensional object corresponding to the object to be modeled.

[0018] In a third aspect, a text-to-3D generation device is provided, comprising: a memory and a processor; wherein:

[0019] The memory is used to store executable instructions.

[0020] The processor is used to implement the text-to-three-dimensional generation method provided in the present application when executing the executable instructions stored in the memory.

[0021] The present application provides a text-to-three-dimensional generation method, device and equipment, the method comprising: performing semantic analysis on the text description to obtain text embedding; determining the initial neural radiation field model corresponding to the object to be modeled based on the text embedding; updating the initial neural radiation field model through the two-dimensional guidance model and the three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled; performing multi-view rendering on the neural radiation field model to determine the three-dimensional object corresponding to the object to be modeled. In this way, the advantages of the two-dimensional guidance model and the three-dimensional guidance model are combined, and the two-dimensional prior knowledge provided by the two-dimensional guidance model can be used to guide the NeRF model to learn the two-dimensional image characteristics of the object to be modeled, so that the three-dimensional object generated based on its rendering has high-resolution texture details. And the three-dimensional prior knowledge provided by the three-dimensional guidance model is used to guide the NeRF model to learn the three-dimensional model representation of the object to be modeled, so that the three-dimensional object generated based on its rendering has good geometric consistency in multiple perspectives, thereby improving the realism and credibility of the three-dimensional object. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic representation of the current Janus problem. Figure 1 ;

[0023] Figure 2 This is a schematic representation of the current Janus problem. Figure 2 ;

[0024] Figure 3 A flowchart of a method for generating a text-to-3D image provided in an embodiment of the present application Figure 1 ;

[0025] Figure 4 A flowchart of a method for generating a text-to-3D image provided in an embodiment of the present application Figure 2 ;

[0026] Figure 5 A schematic diagram of a process for determining a combined loss using a two-dimensional guidance model and a three-dimensional guidance model provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of a method for generating a text-to-3D image provided in an embodiment of the present application Figure 3 ;

[0028] Figure 7 A schematic diagram of the multi-resolution hash coding process provided by an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a system for generating text into three dimensions provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the composition structure of a text-to-three-dimensional generation device provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the composition structure of a text-to-three-dimensional generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0033] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0034] Part-aware editing: This refers to the ability to edit specific parts or components of a digital object or content with the understanding or recognition of its individual components. In the context of 3D modeling and graphics, part-aware editing means manipulating specific parts or elements of a 3D model without affecting the entire model.

[0035] 2D content: A two-dimensional (width, height) digital model. For example, a two-dimensional image is a flat image that does not contain depth information.

[0036] 3D content: A digital model in three dimensions (width, height, depth) that can be rotated, scaled, and viewed from any angle, unlike a 2D image.

[0037] Text-to-3D Generation Module: This is a system or process that can take a textual description as input and output a 3D model representing the described object or scene.

[0038] Text embedding: Also known as word embedding or sentence embedding, in Natural Language Processing (NLP), this refers to converting text data into a digital form, usually a vector in a high-dimensional space, which can be processed by machine learning algorithms.

[0039] NeRF: Neural Radiance Fields, a method of representing 3D scenes using neural networks. NeRF represents the scene as a continuous function that maps coordinates to color and density, allowing realistic rendering of the scene.

[0040] Geometric consistency: also known as 3D consistency, refers to the 3D model maintaining geometric properties or structural consistency when viewed from all angles.

[0041] Janus problem: Janus problem refers to the situation where the generated 3D objects have wrong views at inappropriate angles. For example, artifacts appear in 3D modeling, repeated objects or patterns appear (such as multiple heads, legs, ears), etc.

[0042] Text-to-3D Generation Module: A module that receives an input text description and generates an initial 3D representation.

[0043] Point cloud: A collection of data points in space, often used for 3D modeling.

[0044] Depth estimation: Techniques for estimating the depth or distance of objects in an image.

[0045] 2D Guidance Model: A model that can be used to synthesize 2D images from text.

[0046] 3D Guidance Model: Can be used to evaluate 3D vision Figure 1 Consistent model.

[0047] Tetrahedron: A polyhedral shape consisting of four triangular faces, six straight edges, and four vertex corners. The tetrahedron is the simplest Platonic solid and is often used as a basic building block for generating meshes in 3D modeling and computer graphics.

[0048] At present, the methods for generating three-dimensional content in related technologies include the following:

[0049] 1. Generate 3D content by randomly sampling in feature space.

[0050] Currently, technologies such as 3D GAN generate 3D shapes by modeling them as a combination of points. Then, new 3D shapes can be generated by predicting the positions of these points by randomly sampling vectors from the feature space. Due to the nature of random sampling, the generated content cannot be controlled by the user.

[0051] 2. Generate 3D content by aligning text embedding vectors in feature space.

[0052] Currently, technologies such as ShapE have improved previous methods by using text embedding instead of randomly sampled feature vectors to generate 3D content, so that users can control the generated results through natural language. However, since point clouds are still used to model 3D content, the visual quality is low and the generated results are not diverse enough.

[0053] 3. Generate 3D content from a single image.

[0054] Currently, technologies such as zero123 can generate 3D content from a single image. These technologies use depth estimation methods and prior knowledge learned from 3D datasets to infer other views from a given 2D view. However, these technologies have the inherent limitations of a single 2D image, that is, the generated model is blurry when it is at other viewing angles.

[0055] 4. Generate 3D content from text using a 2D guidance model.

[0056] Currently, technologies such as Dreamfusion only use text to generate 3D shapes. 3D content is generated by randomly initializing the NeRF model and repeatedly rendering 2D views at random camera positions. Dreamfusion uses a 2D guidance model to give update directions and then updates the NeRF model. However, this approach has a serious Janus problem, that is, repeated objects or patterns such as ears, eyes, and legs appear in the generated 3D content. As shown in Figure 1 In , the rabbit is incorrectly modeled as having 3 ears. Figure 2 The eagle is incorrectly modeled as having 2 heads.

[0057] In summary, the three-dimensional objects or three-dimensional contents generated by the current related art methods may have the problem of inaccurate modeling, which reduces the credibility of the generated three-dimensional objects.

[0058] The embodiments of the present application provide a text-to-three-dimensional generation method, apparatus, and device. The method performs semantic analysis on the description text to extract text embedding, constructs an initial NeRF model based on the text embedding, uses the two-dimensional prior knowledge and the three-dimensional prior knowledge learned by the two-dimensional guidance model and the three-dimensional guidance model as prior references, guides the NeRF model to perform gradient update, and enables the NeRF model to generate a rendered image of the object to be modeled from any perspective, thereby obtaining a three-dimensional object. The three-dimensional object has both high-resolution texture details and good geometric consistency at any perspective, eliminating the Janus problem, thereby improving the realism and credibility of the generated 3D object.

[0059] Figure 3 Schematic diagram of the first process of the method for generating text to three-dimensional in the embodiment of the present application, as shown in FIG. Figure 3 As shown, including:

[0060] S101: Perform semantic analysis on the text description to obtain text embedding.

[0061] In some embodiments, the description text corresponding to the object to be modeled is input into a text parsing model for speech analysis to extract text embedding. The text parsing model refers to a model that uses computer technology to deeply analyze and understand text, which can help computers understand the meaning, context and emotion of human language and extract text embedding from it.

[0062] This process requires converting discrete text inputs into continuous word embeddings. These embeddings encapsulate the semantic essence of the text in a continuous vector space, so that the entire generation process of building three-dimensional objects (3D objects) based on text embeddings can realize the connection between text features and corresponding 3D object properties.

[0063] S102: Determine an initial neural radiation field model corresponding to the object to be modeled based on text embedding.

[0064] In the embodiment of the present application, based on text embedding, the NeRF model is randomly initialized to obtain the initial neural radiation field model corresponding to the object to be modeled. The NeRF model uses a neural network to represent a three-dimensional model. The function of this neural network is that given a viewing angle (the position and direction of the camera, etc.) and the position of a spatial point (a certain point in space), the NeRF model can infer the information of the point (color and density).

[0065] Exemplarily, the initial NeRF model can be composed of a multilayer perceptron (MLP). Generating a three-dimensional object using the initial NeRF model includes two stages: a training stage and a rendering stage. For the training stage, since the network parameters in the initial NeRF model are randomly generated, the network parameters in the initial NeRF model can be updated by performing network training on the initial NeRF model, so that the initial NeRF model continuously learns the three-dimensional representation of the object to be modeled during the training process. In this way, in the rendering stage, the neural radiation field model obtained after training can be used to render a three-dimensional view of the object to be modeled at various perspectives to obtain a three-dimensional object. In this way, the neural radiation field model obtained by training the two-dimensional guidance model and the three-dimensional guidance model can provide a powerful representation of the 3D object, and can dynamically render the 3D object from almost any perspective, thereby ensuring the creation of realistic and coherent 3D content, accurately capturing the subtle differences and details of the geometry and appearance of the object to be modeled.

[0066] S103. The initial neural radiation field model is updated by using the two-dimensional guidance model and the three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled.

[0067] In an embodiment of the present application, during the training phase of the initial neural radiation field model, the initial neural radiation field model can be updated using a pre-trained two-dimensional guidance model and a three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled.

[0068] In some embodiments, a two-dimensional guidance model is used to guide an initial neural radiation field model to perform two-dimensional image restoration based on text embedding and noisy two-dimensional images; the two-dimensional guidance model is obtained by pre-training on a two-dimensional image dataset; the three-dimensional guidance model is used to guide the initial neural radiation field model to evaluate the three-dimensional consistency between two images; the three-dimensional guidance model is obtained by pre-training on a three-dimensional image dataset.

[0069] In some embodiments, two camera positions in a three-dimensional scene corresponding to the initial neural radiation field model are rendered in two dimensions to obtain two two-dimensional images, one of the two-dimensional images is denoised, and a two-dimensional guidance model is used to predict noise based on text embedding and the noisy two-dimensional image to obtain a first predicted noise, and then a first loss for the two-dimensional guidance model to guide the training of the initial neural radiation field model is obtained based on the first predicted noise. Using the three-dimensional guidance model, noise prediction is performed based on the noisy two-dimensional image, another two-dimensional image, and the relative camera pose between the two to obtain a second predicted noise, and then a second loss for the three-dimensional guidance model to guide the training of the initial neural radiation field model is obtained based on the second predicted noise. Combining the first loss with the second loss, a combined loss can be obtained, and the combined loss is used to perform gradient update on the initial neural radiation field model, and the updated gradient is back-propagated to the initial neural radiation field model to update the network parameters of the initial neural radiation field model. The above process is used for iterative training until the preset convergence condition is reached, the training is completed, and the neural radiation field model corresponding to the object to be modeled is obtained.

[0070] In some embodiments, more additional prior knowledge may be combined, such as segmentation, classification and other prior knowledge, to train the initial neural radiation field model. The specific selection is made based on actual conditions and is not limited in the embodiments of the present application.

[0071] S104: Perform multi-view rendering on the neural radiation field model to determine the three-dimensional object corresponding to the object to be modeled.

[0072] In an embodiment of the present application, during the rendering stage, the trained neural radiation field model is rendered from multiple perspectives to determine the three-dimensional object corresponding to the object to be modeled.

[0073] In some embodiments, multi-perspective rendering is performed on the three-dimensional representation corresponding to the neural radiation field model to determine multiple three-dimensional images corresponding to the multiple perspectives; image analysis is performed on the multiple three-dimensional images to determine semantic differences between the multiple three-dimensional images; and based on the semantic differences, the neural radiation field model is updated to determine the three-dimensional object.

[0074] In some embodiments, a minimum editing algorithm may be used to update parameters of the neural radiation field model based on semantic differences, and determine the three-dimensional object based on the updated neural radiation field model. Here, the minimum editing is used to minimize the editing required to achieve consistent 3D reconstruction.

[0075] That is, in order to ensure consistent semantic alignment and coherence from different perspectives, multi-view rendering and analysis can be used to render the generated neural radiation field model, that is, the 3D model representation corresponding to the object to be modeled, from various perspectives and viewpoints. The views obtained from each perspective are thoroughly analyzed to identify any semantic inconsistencies or differences that may occur when viewing the model from different angles. In order to resolve and correct the identified semantic inconsistencies, an optimization algorithm is used. The algorithm uses a minimization strategy to perform precise and minimal edits to the 3D model. By recalibrating the features and elements of the model, the optimization algorithm ensures that the 3D reconstruction remains coherent and consistent across multiple viewpoints, thereby effectively resolving any differences detected during multi-view analysis and improving the credibility of the rendered three-dimensional object.

[0076] It can be understood that in the embodiment of the present application, the advantages of the two-dimensional guidance model and the three-dimensional guidance model are combined, and the initial neural radiation field model generated based on text embedding is guided by the two-dimensional guidance model and the three-dimensional guidance model to learn and train the object to be modeled. In this way, the two-dimensional prior knowledge provided by the two-dimensional guidance model can be used to guide the NeRF model to learn the two-dimensional image characteristics of the object to be modeled, so that the three-dimensional object generated based on its rendering has high-resolution texture details. And the three-dimensional prior knowledge provided by the three-dimensional guidance model is used to guide the NeRF model to learn the three-dimensional model representation of the object to be modeled, so that the three-dimensional object generated based on its rendering has good geometric consistency from multiple perspectives, thereby improving the realism and credibility of the three-dimensional object.

[0077] In some embodiments, the process of updating the initial neural radiation field model by using the two-dimensional guidance model and the three-dimensional guidance model in the above S103 to determine the neural radiation field model corresponding to the object to be modeled can be as follows: Figure 4 As shown, including S1031-S1035, as follows:

[0078] S1031. Perform two-dimensional image rendering on the initial neural radiation field model to obtain a first image and a second image.

[0079] In S1031, a two-dimensional image rendering is performed based on a first camera position and a second camera position in a three-dimensional scene corresponding to the initial neural radiation field model to obtain a first image corresponding to the first camera position and a second image corresponding to the second camera position. Here, the first camera position and the second camera position may include any two camera positions in the three-dimensional scene corresponding to the initial neural radiation field model.

[0080] It should be noted that the two-dimensional image rendering in the training phase is the same as that in the rendering phase, which uses the initial neural radiation field model or the color and opacity of the 3D points predicted by the neural radiation field model, and integrates them along the light to obtain the 2D image pixel value, and then obtains the rendered two-dimensional image.

[0081] S1032: Add sample noise to the first image to obtain a noisy first image.

[0082] S1033. Perform noise prediction based on the noisy first image and text embedding through a two-dimensional guidance model to determine a first loss.

[0083] In some embodiments, a two-dimensional guidance model is used to perform noise prediction based on a noisy first image and text embedding to determine a first predicted noise; and a first loss is determined based on the first predicted noise and sample noise.

[0084] Exemplarily, the two-dimensional guidance model may include a pre-trained diffusion model. The diffusion model is a generative model based on deep learning technology. The diffusion model is based on a simulated physical diffusion process. During the training process, the model simulates the diffusion process by adding noise to the image, so that the image gradually approaches the Gaussian noise distribution. During the generation process, the model reverses the diffusion process by removing noise and gradually restores a clear image. This process is achieved through a trained neural network that learns the ability to predict and remove noise given a noisy image.

[0085] The pre-trained diffusion model is a diffusion model that has been pre-trained on a large-scale image dataset and supports generating high-quality images from text descriptions. Users only need to provide a description text, and the pre-trained diffusion model can generate a first image that matches it.

[0086] Pre-trained diffusion models are trained on a large number of image datasets and usually have the ability to generate images from text descriptions, providing rich image details and texture information. This information can be used as a priori reference to provide guidance in the text-to-3D generation process, improving the quality and detail of the generated 3D model. In text-to-3D generation, this capability can be combined with the 3D generator through appropriate modification and extension to achieve text-to-3D conversion.

[0087] S1034. Determine a second loss by performing noise prediction based on the noisy first image, the second image, and the relative camera pose between the noisy first image and the second image through the three-dimensional guidance model.

[0088] In some embodiments, a three-dimensional guidance module is used to perform noise prediction based on the noisy first image, the second image, and the relative camera pose between the noisy first image and the second image to determine a second predicted noise; and a second loss is determined based on the second predicted noise and the sample noise.

[0089] That is, the update gradient can be predicted by a three-dimensional guidance model (3D guidance model) based on the two rendered 2D images obtained by the initial NeRF model rendering and the relative camera pose between the two rendered 2D images. The 3D guidance model is pre-trained on a large-scale 3D dataset to maximize 3D consistency, thereby facilitating the 3D consistency of the generated 3D content.

[0090] Exemplarily, the three-dimensional guidance model may include a network model of a Zero-1-to-3 structure, or may also include models of other network forms. The specific selection is made according to actual conditions and is not limited in the embodiments of the present application.

[0091] Exemplarily, the first loss and the second loss may include a shape diffusion similarity (SDS) loss, or may include losses calculated using other loss functions. The specific loss is selected based on actual conditions and is not limited in the embodiments of the present application.

[0092] S1035. Combining the first loss and the second loss, the initial neural radiation field model is gradient updated until a preset convergence condition is reached, and the neural radiation field model corresponding to the object to be modeled is determined.

[0093] In some embodiments, the first loss and the second loss are mixed, clipped, and post-processed to determine a combined loss; based on the combined loss, the initial neural radiation field model is gradient updated until a preset convergence condition is reached to determine the neural radiation field model corresponding to the object to be modeled.

[0094] Exemplarily, adaptive mixing, clipping, and post-processing can be performed based on the first loss from the 2D guidance model and the second loss from the 3D guidance model to obtain a combined loss, thereby determining the combined gradient. The combined gradient is back-propagated to the NeRF network to facilitate the learning and alignment of the 3D representation and the rendered image, completing a round of training. Iterative training is performed using the process of S1031-S1035 until the preset convergence condition is reached, and the neural radiation field model corresponding to the object to be modeled is determined.

[0095] In the present application embodiment, model gradient update is an important step in machine learning, which is used to adjust model parameters to minimize prediction error. In the model training process, the error of each sample can be expressed as the distance between the model output and the label (i.e., loss function). The goal of the model is to minimize the mean value of the loss function of all samples (i.e., minimize the overall error). When calculating the gradient of the model, the chain rule and back propagation algorithm can be used to calculate the contribution of each parameter to the loss function, and then the model parameters are updated according to the gradient of the loss function to minimize the error. This process can be iterated many times until the loss function converges or reaches a preset number of iterations.

[0096] Exemplarily, the above process of determining the combined loss can be as follows: Figure 5 As shown. The two-dimensional guidance model performs noise prediction based on the high-resolution image of the object to be modeled, the text embedding, and the noisy first image, and determines the first loss ￡ based on the predicted first predicted noise 2D The three-dimensional guidance model predicts noise based on the noisy first image, the second image rendered by the initial NeRF model, and the camera pose [R, t] between the noisy first image and the second image (where R represents the rotation of the camera coordinate system between the noisy first image and the second image, and t represents the translation of the camera coordinate system between the noisy first image and the second image), and determines the second loss ￡ based on the predicted second predicted noise. 3D Based on the first weight λ 2D and the second weight λ 3D Combined with 2D and £ 3D , and get the combined loss £ g . It can be seen that the embodiment of the present application can use 2D prior knowledge and 3D prior knowledge to optimize the initial NeRF model. Among them, 2D prior knowledge provides high-resolution texture details, while 3D prior knowledge provides good geometric consistency. Compared with the current related technologies that either only use 2D prior knowledge (resulting in the Janus problem) or only use 3D prior knowledge (resulting in low visual quality), the embodiment of the present application can dynamically combine and balance these two prior knowledge methods to combine the advantages of the two methods and improve the realism and credibility of the subsequently generated three-dimensional objects.

[0097] In some embodiments, the method further includes: extracting a three-dimensional mesh from the trained component-aware neural radiation field model; and converting the three-dimensional mesh into a standard three-dimensional file format using a differentiable tetrahedral mesh. The standard three-dimensional file format enables these models to be efficiently rendered using rasterization algorithms on a range of devices including mobile and VR platforms, ensuring wide accessibility and immersive experience.

[0098] Exemplarily, the standard three-dimensional file format may be one of the following: OBJ (Wavefront Object) format, STL (Standard Tessellation Language) format, COLLADA format, etc.

[0099] In some embodiments, Figure 6 As shown, the above process of S102 can be implemented by executing S1021-S1022 as follows:

[0100] S1021. Construct a neural radiation field based on text embedding to determine an initial three-dimensional representation model corresponding to the object to be modeled.

[0101] Traditional NeRF models are good at capturing complex scene details, but they have high computational requirements, especially for highly complex scenes. In the embodiment of the present application, a neural radiation field can be constructed based on text embedding to obtain a neural radiation field model with random parameters as an initial three-dimensional representation model, and this challenge is alleviated by using a multi-resolution hash grid structure for the initial three-dimensional representation model. This structure optimizes computing resources by effectively dividing the scene into manageable blocks, thereby achieving selective and targeted refinement.

[0102] S1022. Perform multi-resolution hash encoding on the initial three-dimensional representation model to determine an initial neural radiation field model.

[0103] In the embodiment of the present application, the initial three-dimensional representation model is subjected to multi-resolution hash coding so that it has a multi-resolution hash grid structure. The multi-resolution hash grid structure uses a hierarchical method to divide the scene into grids of different resolutions. For example, the multi-resolution hash coding process can be as follows: Figure 7 As shown. The input of multi-resolution hash coding includes scenes at various perspectives in the initial three-dimensional representation model: Step 1. For a coordinate point in scene A at a certain perspective in the initial three-dimensional representation model, find the grid where the coordinate point is located at different resolution levels, and assign indexes to the corners (vertices) of the above grids by hash mapping the integer coordinates to obtain the corner indexes corresponding to the grids at each resolution level. For example, the initial three-dimensional representation model includes L resolution levels, and the highest resolution level S among the L resolution levels is max , minimum resolution level S min .like Figure 7 As shown, it can be determined that the coordinate point is in S max The 4 corner indices c of the grid where the level is located f1 to c f4 ; It can be determined that the coordinate point is in S min The 4 corner indices c of the grid where the level is located w1 to cw4 Step 2: For each corner index, find the F-dimensional feature vector corresponding to each corner index from the hash table. min The corner index corresponding to the level is in the hash table c w Search in S max The level corresponds to the corner index in the hash table c f Search in. Step 3. According to the relative position of the coordinate point in the grids of different resolution levels, linearly interpolate the F-dimensional feature vector corresponding to each corner index to obtain the feature vector of the coordinate point at different resolution levels, and the feature vector represents the attributes of the coordinate point at different resolution levels. Step 4. Connect the feature vectors of different resolution levels and the viewing angle through the fully connected deep neural grid, and output the color and density information corresponding to the coordinate point. Through the above steps, the scene A' generated by multi-resolution hash coding of the scene A under the input viewing angle can be obtained, and then the initial neural radiation field model is determined by multi-resolution hash coding of the scenes under each viewing angle of the initial three-dimensional representation model. Among them, steps 1-3 are completed independently at each resolution level. The hierarchical structure of multi-resolution hash coding ensures efficient memory utilization, and accelerates the rendering process by giving priority to calculations in areas where details are required, while applying coarser resolutions where finer details are not required, thereby increasing the speed of generating 3D objects. The current related art solutions take several hours to generate 3D content, while the multi-resolution hash grid method of the embodiment of the present application can reduce the learning time to about 10 minutes.

[0104] It should be noted that in addition to the multi-resolution hash coding network structure, vanilla NeRF, tensor radiation field, etc. can also be used as a network structure to represent a three-dimensional scene. The specific selection is made according to the actual situation, and the embodiments of this application are not limited.

[0105] It should be noted that in scenarios where real-time interactive inspection and analysis of generated 3D content is not required (e.g., generating a video of 3D content), the 3D learning process can be skipped and such a task can be handled by generating a sequence of 2D images that match the desired textual prompts.

[0106] See also Figure 8, for the training process of the initial NeRF model in the embodiment of the present application, for a given coordinate, the corresponding hash embedding is obtained by multi-resolution hash coding, and the two-dimensional image rendering is performed through the MLP network in the initial NeRF model to obtain a 2D RGB image. Sample noise is added to the 2D RGB image to obtain a noisy image. The noisy image and the text prompt are input into the two-dimensional guidance model to obtain a first predicted noise, and a 2D gradient is obtained based on the first predicted noise. The noisy image, the reference image (another two-dimensional rendered image of the initial NeRF model), and the relative camera posture between the noisy image and the reference image are input into the three-dimensional guidance model to obtain a second predicted noise, and a 3D gradient is obtained based on the second predicted noise. The 2D gradient and the 3D gradient are combined to obtain the final gradient value, which is back-propagated to the initial NeRF model.

[0107] The text-to-3D generation method provided in the embodiment of the present application has the following beneficial effects:

[0108] 1) The text-to-3D generation method provided in the embodiments of the present application can generate realistic 3D content, thereby producing impressive visual representations that are very similar to real-world objects and scenes. Users can appreciate the complex details and realistic textures presented in the generated 3D model, thereby enhancing the overall visual experience. The modeled three-dimensional objects are more realistic, providing enhanced immersion and authenticity, making the 3D content more attractive and exciting.

[0109] 2) The embodiments of the present application can effectively solve the Janus problem, ensuring that the generated 3D content remains consistent and coherent from all viewing angles. Users can study the 3D model without encountering problems such as repeated objects or patterns, thereby achieving a more accurate and reliable representation. In addition, since there is no Janus artifact problem, the credibility and usability of the generated content are enhanced, so that the generated content can be effectively applied in a variety of contexts.

[0110] 3) The use of the multi-resolution hash grid method in the embodiment of the present application significantly reduces the time required to generate 3D content, allowing users to quickly obtain 3D content. This accelerated sampling process improves processing efficiency and user experience, making the system more accessible and effectively used in various applications. On this basis, through different design ideas and concepts, the needs of users for rapid iteration can be met, forming a more dynamic and creative workflow.

[0111] 4) The text-to-three-dimensional generation method provided in the embodiment of the present application facilitates real-time research of generated 3D content, allowing users to interactively examine and analyze their creations from various angles and perspectives. Real-time research can enhance the interactivity of 3D content, allowing users to fully understand the design, structure, and details. This capability is particularly valuable for collaborative discussions, design reviews, and decision-making processes that must be dynamically studied.

[0112] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a device for generating text to three-dimensional, such as Fig. 9 As shown, the text to three-dimensional generating device 80 includes:

[0113] A text parsing model 801 is configured to perform semantic analysis on the text description to obtain text embedding;

[0114] An initialization module 802 is configured to determine an initial neural radiation field model corresponding to the object to be modeled based on the text embedding;

[0115] An updating module 803 is configured to update the initial neural radiation field model by using a two-dimensional guidance model and a three-dimensional guidance model to determine a neural radiation field model corresponding to the object to be modeled;

[0116] The rendering module 804 is configured to perform multi-view rendering on the neural radiation field model to determine the three-dimensional object corresponding to the object to be modeled.

[0117] In some embodiments, the update module 803 is further configured to perform two-dimensional image rendering on the initial neural radiation field model to obtain a first image and a second image; add sample noise to the first image to obtain a noisy first image; perform noise prediction based on the noisy first image and the text embedding through the two-dimensional guidance model to determine a first loss; perform noise prediction based on the noisy first image, the second image, and the relative camera pose between the noisy first image and the second image through the three-dimensional guidance model to determine a second loss; and perform gradient update on the initial neural radiation field model in combination with the first loss and the second loss until a preset convergence condition is reached to determine the neural radiation field model corresponding to the object to be modeled.

[0118] In some embodiments, the updating module 803 is further configured to perform noise prediction based on the noisy first image and the text embedding through the two-dimensional guidance model to determine a first predicted noise; and determine the first loss based on the first predicted noise and the sample noise.

[0119] In some embodiments, the update module 803 is further configured to perform noise prediction based on the noisy first image, the second image, and the relative camera pose between the noisy first image and the second image through the three-dimensional guidance module to determine a second predicted noise; and determine the second loss based on the second predicted noise and the sample noise.

[0120] In some embodiments, the update module 803 is further configured to perform at least one of mixing, clipping and post-processing on the first loss and the second loss to determine a combined loss; based on the combined loss, the initial neural radiation field model is gradient updated until the preset convergence condition is reached to determine the neural radiation field model corresponding to the object to be modeled.

[0121] In some embodiments, the two-dimensional guidance model is used to guide the initial neural radiation field model to perform two-dimensional image restoration based on text embedding and noisy two-dimensional images; the two-dimensional guidance model is obtained by pre-training on a two-dimensional image dataset; the three-dimensional guidance model is used to guide the initial neural radiation field model to evaluate the three-dimensional consistency between two images; the three-dimensional guidance model is obtained by pre-training on a three-dimensional image dataset.

[0122] In some embodiments, the initialization module 802 is further configured to construct a neural radiation field based on the text embedding to determine an initial three-dimensional representation model corresponding to the object to be modeled; and perform multi-resolution hash encoding on the initial three-dimensional representation model to determine the initial neural radiation field model.

[0123] In some embodiments, the rendering module 804 is further configured to perform multi-perspective rendering on the three-dimensional representation corresponding to the neural radiation field model to determine multiple three-dimensional images corresponding to multiple perspectives; perform image analysis on the multiple three-dimensional images to determine semantic differences between the multiple three-dimensional images; and update the neural radiation field model based on the semantic differences to determine the three-dimensional object.

[0124] In some embodiments, the rendering module 804 is further configured to adopt a minimal editing algorithm to update parameters of the neural radiation field model based on the semantic difference, and determine the three-dimensional object based on the updated neural radiation field model.

[0125] In some embodiments, the text-to-3D generation device also includes a conversion module, which is configured to extract a three-dimensional mesh from a neural radiation field model corresponding to the object to be modeled; and convert the three-dimensional mesh into a standard three-dimensional file format using a differentiable tetrahedral mesh.

[0126] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0127] In practical applications, the above-mentioned text-to-three-dimensional generation device can be an electronic device that realizes the text-to-three-dimensional generation function, or a chip applied to an electronic device. The electronic device can be independent of the visual back-end system, or it can be part of the visual back-end system. In the present application, the device can realize the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the text-to-three-dimensional generation method provided in any of the above embodiments. And the technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solutions in the text-to-three-dimensional generation method, and this application will not go into details one by one.

[0128] Based on the hardware implementation of each unit in the above-mentioned text-to-3D generation device, the embodiment of the present application also provides a text-to-3D generation device, such as Fig.10 As shown, the text-to-three-dimensional generation device 90 includes: a processor 901 and a memory 902 configured to store a computer program that can be run on the processor;

[0129] The processor 901 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0130] Of course, in practical applications, Fig. 9 As shown, the various components in the text-to-three-dimensional generating device 90 are coupled together via a bus system 903. It can be understood that the bus system 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 903 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 903 in the figure.

[0131] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0132] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0133] Optionally, the text-to-3D generation device 60 may be a chip, and the chip may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0134] Optionally, the chip may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0135] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of a text-to-three-dimensional generation device to complete the steps of the aforementioned method.

[0136] An embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the methods described in the embodiments of the present application are implemented.

[0137] Optionally, the computer program product can be applied to the text-to-three-dimensional generation device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the text-to-three-dimensional generation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0138] The embodiment of the present application also provides a computer program. Optionally, the computer program can be applied to the text-to-3D generation device in the embodiment of the present application. When the computer program is run on a computer, the computer executes the corresponding process implemented by the text-to-3D generation device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0139] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0140] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0141] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0142] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0146] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for generating a text into a three-dimensional image, characterized in that: The method comprises: Perform semantic analysis on the text description to obtain text embedding; Based on the text embedding, determining an initial neural radiation field model corresponding to the object to be modeled; The initial neural radiation field model is updated by using the two-dimensional guidance model and the three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled; The neural radiation field model is rendered from multiple perspectives to determine a three-dimensional object corresponding to the object to be modeled.

2. The method according to claim 1, characterized in that The updating of the initial neural radiation field model by using the two-dimensional guidance model and the three-dimensional guidance model to determine the neural radiation field model corresponding to the object to be modeled includes: Performing two-dimensional image rendering on the initial neural radiation field model to obtain a first image and a second image; adding sample noise to the first image to obtain a noisy first image; performing noise prediction based on the noisy first image and the text embedding by the two-dimensional guidance model to determine a first loss; performing noise prediction based on the noisy first image, the second image, and a relative camera pose between the noisy first image and the second image by the three-dimensional guidance model to determine a second loss; The initial neural radiation field model is gradient updated by combining the first loss and the second loss until a preset convergence condition is reached, thereby determining a neural radiation field model corresponding to the object to be modeled.

3. The method according to claim 2, characterized in that The step of performing noise prediction based on the noisy first image and the text embedding by the two-dimensional guidance model to determine a first loss includes: Performing noise prediction based on the noisy first image and the text embedding by the two-dimensional guidance model to determine a first predicted noise; The first loss is determined based on the first prediction noise and the sample noise.

4. The method according to claim 2, characterized in that: The step of performing noise prediction based on the noisy first image, the second image, and a relative camera pose between the noisy first image and the second image by using the three-dimensional guidance model to determine a second loss includes: performing noise prediction based on the noisy first image, the second image, and a relative camera pose between the noisy first image and the second image by the three-dimensional guidance module to determine a second predicted noise; The second loss is determined based on the second prediction noise and the sample noise.

5. The method according to any one of claims 2 to 4, characterized in that: The step of combining the first loss and the second loss to perform gradient updating on the initial neural radiation field model until a preset convergence condition is reached to determine the neural radiation field model corresponding to the object to be modeled includes: performing at least one of mixing, clipping, and post-processing on the first loss and the second loss to determine a combined loss; Based on the combined loss, the initial neural radiation field model is gradient updated until the preset convergence condition is reached, and the neural radiation field model corresponding to the object to be modeled is determined.

6. The method according to any one of claims 1 to 4, characterized in that: The two-dimensional guidance model is used to guide the initial neural radiation field model to perform two-dimensional image restoration based on text embedding and noisy two-dimensional images; the two-dimensional guidance model is obtained by pre-training on a two-dimensional image dataset; The three-dimensional guidance model is used to guide the initial neural radiation field model to evaluate the three-dimensional consistency between two images; The three-dimensional guidance model is obtained by pre-training on a three-dimensional image dataset.

7. The method according to any one of claims 1 to 4, characterized in that: The determining, based on the text embedding, an initial neural radiation field model corresponding to the object to be modeled comprises: Constructing a neural radiation field based on the text embedding to determine an initial three-dimensional representation model corresponding to the object to be modeled; Multi-resolution hash coding is performed on the initial three-dimensional representation model to determine the initial neural radiation field model.

8. The method according to any one of claims 1 to 4, characterized in that: The performing multi-view rendering on the neural radiation field model to determine the three-dimensional object corresponding to the object to be modeled includes: Performing multi-view rendering on the three-dimensional representation corresponding to the neural radiation field model to determine a plurality of three-dimensional images corresponding to the plurality of viewpoints; performing image analysis on the multiple three-dimensional images to determine semantic differences between the multiple three-dimensional images; Based on the semantic difference, the neural radiation field model is updated to determine the three-dimensional object.

9. The method according to claim 8, characterized in that The updating of the neural radiation field model based on the semantic difference to determine the three-dimensional object includes: A minimization editing algorithm is used to update parameters of the neural radiation field model based on the semantic difference, and the three-dimensional object is determined based on the updated neural radiation field model.

10. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Extracting a three-dimensional mesh from a neural radiation field model corresponding to the object to be modeled; The three-dimensional mesh is converted to a standard three-dimensional file format using a differentiable tetrahedral mesher.

11. A device for generating text into three dimensions, characterized in that: The device comprises: A text parsing model, configured to perform semantic analysis on the text description to obtain text embedding; An initialization module, configured to determine an initial neural radiation field model corresponding to the object to be modeled based on the text embedding; An updating module is configured to update the initial neural radiation field model through a two-dimensional guidance model and a three-dimensional guidance model to determine a neural radiation field model corresponding to the object to be modeled; A rendering module is configured to perform multi-view rendering on the neural radiation field model to determine a three-dimensional object corresponding to the object to be modeled.

12. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images