Text-to-three-dimensional generation method, device and equipment

Abstract

The invention discloses a text-to-three-dimensional generation method, device and equipment, and the method comprises the steps: carrying out the semantic analysis of text description, and obtaining text embedding; embedding and inputting the text into the diffusion model, and outputting a first image matched with the text description; the shape information and the texture information of the modeling object in the first image and the visual angle information are input into a component perception neural radiation field model, second images corresponding to all visual angles are output, and the component perception neural radiation field model supports editing operation on components of the modeling object according to editing information. Thus, the diffusion model provides a 2D image as a priori reference, a guiding component senses the neural radiation field model to generate a rendered image of any view angle of the 3D object, and the fidelity of the generated 3D object is improved. And under the condition that other parts are not influenced, the 3D objects support part-level shape and visual attribute editing operation, so that a text-to-three-dimensional generation result is optimized.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is filed based on and claims the priority of U.S. patent application No. 63 / 578,856, filed on August 25, 2023. The entire contents of the patent application are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for generating text into three dimensions. Background Art

[0004] Neural Radiance Fields (NeRF) is a method of representing three-dimensional (3D) scenes using neural networks. Its basic principle is to combine computer graphics and deep learning to learn continuous 3D scene representation from a set of sparse 2D images and generate high-quality new perspective images in arbitrary viewing directions.

[0005] Generating realistic and editable 3D content is a long-standing problem in computer vision and graphics. Currently, Generative Adversarial Networks (GANs) have become a powerful tool for synthesizing realistic images, but do not allow changing specific parts of the object. Summary of the invention

[0006] The embodiments of the present application are intended to provide a method, device and apparatus for generating three-dimensional text.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, a method for generating a text to a three-dimensional image is provided, comprising:

[0009] Perform semantic analysis on the text description to obtain text embedding;

[0010] Inputting the text embedding into a diffusion model and outputting a first image matching the text description;

[0011] Extracting shape information and texture information of the modeled object in the first image;

[0012] The texture information and shape information of the modeling object and the viewing angle information are input into the component perception neural radiation field model, and a second image corresponding to the viewing angle information is output. The component perception neural radiation field model supports editing operations on the components of the modeling object according to the editing information.

[0013] In a second aspect, a device for generating a text-to-three-dimensional image is provided, comprising:

[0014] Text parsing model, configured to perform semantic analysis on text descriptions to obtain text embeddings;

[0015] A diffusion model configured to determine a first image matching the text description based on the text embedding; extract shape information and texture information of a modeled object in the first image;

[0016] The component-aware neural radiation field model is configured to determine a second image corresponding to the perspective information based on texture information and shape information of the modeling object, as well as perspective information. The component-aware neural radiation field model supports editing operations on the components of the modeling object based on editing information.

[0017] In a third aspect, a device for generating a text-to-three-dimensional image is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0018] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0019] The present application provides a method, device and equipment for generating text to three-dimensional. The method includes: performing semantic analysis on a text description to obtain a text embedding; inputting the text embedding into a diffusion model, and outputting a first image matching the text description; extracting shape information and texture information of a modeled object in the first image; inputting the texture information and shape information of the modeled object, as well as the viewing angle information into a component-perceived neural radiation field model, and outputting a second image corresponding to the viewing angle information. The component-perceived neural radiation field model supports editing operations on the components of the modeled object according to the editing information. In this way, the diffusion model provides a 2D image as a priori reference to guide the component-perceived neural radiation field model to generate a rendered image of a 3D object from any viewing angle, thereby improving the fidelity of the generated 3D object. And without affecting other components, these 3D objects support editing operations on component-level shapes and visual attributes, thereby optimizing the generation results of text to three-dimensional. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of a method for generating a text-to-three-dimensional image in an embodiment of the present application;

[0021] Figure 2 The pipeline diagram of the method for generating three-dimensional text in the embodiment of the present application is shown in FIG. Figure 1 ;

[0022] Figure 3 The pipeline diagram of the method for generating three-dimensional text in the embodiment of the present application is shown in FIG. Figure 2 ;

[0023] Figure 4 A schematic flow chart of a method for implementing a component-perceived neural radiation field model in an embodiment of the present application;

[0024] Figure 5 A pipeline schematic diagram of a component sensing neural radiation field model in an embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of the network architecture of the component perception neural radiation field model in the embodiment of the present application;

[0026] Figure 7 A pipeline diagram of a multi-resolution hash coding network in an embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of the composition structure of a device for generating text to three-dimensional data in an embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of the composition structure of the text-to-three-dimensional generation device in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0030] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0031] Part-aware editing: This refers to the ability to edit specific parts or components of a digital object or content with the understanding or recognition of its individual components. In the context of 3D modeling and graphics, part-aware editing means manipulating specific parts or elements of a 3D model without affecting the entire model.

[0032] 2D content: A two-dimensional (width, height) digital model. For example, a two-dimensional image is a flat image that does not contain depth information.

[0033] 3D content: A digital model in three dimensions (width, height, depth) that can be rotated, scaled, and viewed from any angle, unlike a 2D image.

[0034] Text-to-3D Generation Module: This is a system or process that can take a textual description as input and output a 3D model representing the described object or scene.

[0035] Text embedding: Also known as word embedding or sentence embedding, in Natural Language Processing (NLP), this refers to converting text data into a digital form, usually a vector in a high-dimensional space, which can be processed by machine learning algorithms.

[0036] NeRF: Neural Radiance Fields, a method for representing 3D scenes using neural networks that enables realistic rendering of these scenes.

[0037] The embodiments of the present application provide a text-to-three-dimensional generation method, apparatus, and device. The method performs semantic analysis on a text description to extract text embedding, and inputs the text embedding into a diffusion model to obtain a 2D image as a priori reference, thereby guiding the component perception NeRF model to generate rendered images of 3D objects from any perspective, thereby improving the realism of the generated 3D objects. These 3D objects support component-level editing operations, such as geometric modification and visual enhancement, and will not affect other components.

[0038] Figure 1 Schematic diagram of the process of the method for generating a text-to-three-dimensional image in the embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0039] Step 101: Perform semantic analysis on the text description to obtain text embedding;

[0040] In some embodiments, the text description is input into a text parsing model for speech analysis to extract text embeddings for subsequent diffusion models. The text parsing model refers to a model that uses computer technology to deeply analyze and understand text, which can help computers understand the meaning, context, and emotions of human language and extract text embeddings that can be used in subsequent diffusion models.

[0041] This process requires converting discrete text inputs into continuous word embeddings. These embeddings encapsulate the semantic essence of the text in a continuous vector space, thus enabling the connection of text features with corresponding 3D object properties throughout the generation process.

[0042] Step 102: Input the text embedding into the diffusion model, and output a first image matching the text description;

[0043] The diffusion model is a generative model based on deep learning technology. The diffusion model includes forward diffusion process and reverse diffusion process. The forward diffusion process means that in the diffusion model, starting from a clear text description with clear semantics, the text description is gradually transformed into a fuzzy or disordered state by gradually adding noise or performing other forms of "destruction". In the task of text-to-3D generation, the forward diffusion process can be regarded as transforming the text description into a potential intermediate representation containing rich semantic information. This intermediate representation may be a series of vectors, matrices or more complex data structures that can capture key features in the text description, such as shape, size, color, etc.

[0044] The reverse diffusion process is based on the forward diffusion process, by gradually removing noise or performing other forms of "recovery" to transform the fuzzy or disordered intermediate state into a clear 3D model with a clear structure. Through the reverse diffusion process, the model can gradually remove the noise and redundant information in the intermediate representation, restore the shape, size, color and other features specified in the text description, and thus generate a 3D model that is highly consistent with the text description.

[0045] The diffusion model is a diffusion model pre-trained on a large-scale image dataset. It usually has the ability to generate images from text descriptions and can provide rich image details and texture information. In the task of text-to-3D generation, the pre-trained diffusion model can serve as a powerful prior to provide guidance and constraints for the generation of the component-aware neural radiation field model. The pre-trained diffusion model is used to optimize the 3D model represented by the component-aware neural radiation field model. By back-propagating the gradient, the diffusion model can adjust the parameters of the component-aware neural radiation field model so that the rendered image matches the cognition of the text description. This optimization process ensures that the generated 3D model visually conforms to the text description and has high-quality geometric and texture details, improving the quality and detail performance of the 3D model. In other words, the diffusion model supports the generation of high-quality images from text descriptions. The user only needs to provide a text description, and the diffusion model can generate a first image that matches it.

[0046] Step 103: extracting shape information and texture information of the modeled object in the first image;

[0047] The first image contains the initial shape information and texture information of the modeled object, and the shape information and texture information of the modeled object in the first image are extracted as a priori reference for the subsequent component perception neural radiation field model. The shape information and texture information of the modeled object can be extracted through a diffusion model, through a component perception neural radiation field model, or through a separate neural network model.

[0048] In some embodiments, the modeled object may include a foreground object in a three-dimensional scene, or may include a background object in a three-dimensional scene, and the specific content depends on the content of the text description and other constraints in the generation process.

[0049] Step 104: Input the texture information and shape information of the modeling object and the viewing angle information into the component-perceived neural radiation field model, and output a second image corresponding to the viewing angle information. The component-perceived neural radiation field model supports editing operations on the components of the modeling object according to the editing information.

[0050] In some embodiments, the method further includes: obtaining editing information, inputting the editing information into the component perception neural radiation field model, and performing editing operations on the components of the modeling object.

[0051] The editing operation may include at least one of the following: modification of the geometric shape of the entire modeled object, modification of the visual attributes of the entire modeled object, modification of the geometric shape of a part of the modeled object, and modification of the visual attributes of a part of the modeled object. For example, the geometric shape modification may include rigid transformation and non-rigid transformation, which may include translation, rotation, scaling, distortion, affine, etc. The visual attribute modification may include texture adjustment and color adjustment, etc.

[0052] In some embodiments, the method further includes: deleting the local neural radiation field network of a specific component according to the editing information to delete the corresponding component; or, adding a new local neural radiation field network according to the editing information to add a new component.

[0053] The editing operation may also include: deleting parts, adding parts of other objects as new parts of the current modeling object.

[0054] The component perception neural radiation field model provided in the embodiment of the present application can generate rendering images of 3D objects at any viewing angle, improving the realism of the generated 3D objects. In addition, without affecting other components, these 3D objects support the editing of component-level shapes and visual attributes, thereby optimizing the text-to-three-dimensional generation results and ensuring that the generation results at different viewing angles are consistent with the text description.

[0055] In some embodiments, the method further includes: inputting text embedding, texture information and shape information of the modeled object, and perspective information into the component-perceived neural radiation field model, and outputting a second image corresponding to the perspective information. In other words, text embedding can also be input into the component-perceived neural radiation field model as a priori reference to constrain the text-to-three-dimensional generation process to improve the generation speed and quality of 3D content.

[0056] Figure 2The pipeline diagram of the method for generating three-dimensional text in the embodiment of the present application is shown in FIG. Figure 1 ,like Figure 2 As shown, the text description is input into the text parsing model for semantic analysis to obtain text embedding; the text embedding is input into the diffusion model to generate a first image matching the text description, and the shape information and texture information of the first image are extracted. At the same time, the perspective information is input into the component perception neural radiation field model to obtain the second image at the current perspective. Furthermore, the editing information can also be input into the component perception neural radiation field model to perform component editing operations.

[0057] In some embodiments, the method may also include: adding a first noise to the second image to obtain a noise image; inputting the text embedding and the noise image into a diffusion model to output a second noise; determining a loss function based on the first noise and the second noise; and optimizing the parameters of the component perception neural radiation field model based on the loss function until the loss function converges.

[0058] The diffusion model is a pre-trained diffusion model that can be used as a guide model for the text-to-3D generation process. It guides the model to predict and update the gradient. Based on the calculated gradient, an optimization algorithm (such as gradient descent, Adam, etc.) is used to update the model parameters to ensure that the generation results from different perspectives are consistent with the text description. Model gradient update is an important step in machine learning. It is used to adjust model parameters to minimize prediction errors. This process can be iterated multiple times until the loss function converges or the preset number of iterations is reached.

[0059] The loss function may be a score distillation sampling (SDS) loss function, which optimizes the parameters of the component-aware NeRF model based on SDS so that its rendered images at different viewing angles match the output of the pre-trained diffusion model, thereby generating a 3D scene consistent with the text description.

[0060] Figure 3 The pipeline diagram of the method for generating three-dimensional text in the embodiment of the present application is shown in FIG. Figure 2 ,like Figure 3As shown, the text description is input into the document interpretation model to extract semantic insights and obtain the document embedding. For example, the text description is "a photo of a peacock on a surfboard taken by a SLR camera"; the text embedding is input into the diffusion model (such as a U-Net model) to obtain a first image that matches the text description. As the prior knowledge of the perceptual neural radiation field model, the potential shape code and texture code in the first image are extracted and sent to the component perceptual neural radiation field model to generate a second image x from a different perspective of the first image. Then, imitating the training mode of the diffusion model, noise ε is added to the peacock picture to obtain the noisy peacock image x+ε. Then, the denoising function of the pre-trained diffusion model is used to predict the noise corresponding to the noisy image x+ε. If the noise is predicted If it is close to the real noise ε, then we can think that the image rendered by the component-perceived neural radiation field model has passed the test of denoising and denoising of the diffusion model, and is a picture that is more in line with the prior reference of the diffusion model. Otherwise, it means that the component-perceived neural radiation field model needs to be improved. Through back propagation, we can adjust the weights of the component-perceived neural radiation field model to ensure that its generated results at different viewing angles are consistent with the text description and have high-quality geometric and texture details.

[0061] In some embodiments, the method further includes: extracting a three-dimensional mesh from the component-perceived neural radiation field model; optimizing the three-dimensional mesh using a differentiable tetrahedral mesh to obtain an optimized three-dimensional mesh; and converting the optimized three-dimensional mesh into a standard three-dimensional file format. In this way, the implicit representation in the model is converted into a more intuitive and understandable three-dimensional structure for further analysis, visualization or application. The use of differentiable tetrahedral meshes can flexibly adapt to a variety of complex three-dimensional geometries. By decomposing the three-dimensional space into small tetrahedral units, objects with complex boundaries and internal structures can be accurately represented. Using a standard three-dimensional file format allows these three-dimensional models to be efficiently rendered using rasterization algorithms on a range of devices, including mobile and VR platforms, ensuring wide accessibility and an immersive experience.

[0062] Exemplarily, the standard 3D file format may be an OBJ (Wavefront Object) format, or one of the following: an STL (Standard Tessellation Language) format, an FBX (Filmbox) format, a COLLADA format, or the like.

[0063] The specific implementation of the component perception neural radiation field model is further illustrated by examples. Figure 4 Schematic diagram of the implementation process of the component perception neural radiation field model in the embodiment of the present application, such as Figure 4As shown, the component-aware neural radiation field model is configured to perform the steps of the following method:

[0064] Step 401: decomposing the shape information and texture information of the modeled object into components, and determining component-level shape information and component-level texture information of M components in the modeled object, where M is a positive integer;

[0065] In some embodiments, M may be an integer greater than 1, or may be 1.

[0066] Step 402: assigning a complete ray set corresponding to the view information to the associated component according to the component-level shape information of each component, and determining a ray set for each component, wherein the ray set includes a ray starting point and a ray direction of the associated ray;

[0067] In some embodiments, the viewing angle information may include camera pose information, which may specifically include the position and orientation of the camera. During the camera imaging process, a ray is projected from the projection center of the camera to each pixel, passing through the pixel position in the imaging plane to form a pixel point. This process can be regarded as an object point emitting a ray, passing through the optical center of the camera, and finally projected onto the imaging plane. The complete ray set can be understood as a ray containing all pixel points on the imaging plane, and different viewing angle information corresponds to different complete ray sets. For example, a ray is characterized by the ray starting point and the ray direction, (o, d) is used to represent a ray, o is the coordinate of the ray starting point, and d is the ray direction. Next, the 3D sampling points along each ray are input into the component perception neural radiation field model to obtain 4 scalar values ​​for each sampling point, including volume density and RGB color information.

[0068] The component-aware neural radiation field model provided in the embodiment of the present application can provide rendered images from various perspectives, and uses a strict light association method to assign light (also called rays) to different components, which can ensure that editing one component will not change the appearance of other components. The ability to implement editing at the component level and seamlessly merge components from different objects are the advantages of the embodiment of the present application.

[0069] In some embodiments, each ray is associated with a component, that is, a hard ray assignment method is adopted to ensure that editing one component does not change the appearance of other components. For example, intersection detection is performed with each ray according to the component-level shape information of each component, and by associating the ray with the first component it intersects, a ray set R associated with the mth component can be defined. m , if ray r is assigned to the mth component, then the mth NeRF network is used to render ray r. If a ray is not associated with any component, its color is black. Hard ray component assignment is a basic property,

[0070] In some other embodiments, each ray may be associated with M components, that is, a ray soft assignment method may be used, but more loss functions are required to separate each component. For example, intersection detection is performed with each ray according to component-level shape information of each component, by associating the ray with the M components intersecting with it.

[0071] Step 403: Rendering the components according to the ray set, component-level shape information, and component-level texture information of each component to obtain a rendered image of each component;

[0072] In some embodiments, based on the ray set of each component, one or more sampling points of the associated ray of each component are determined; based on the one or more sampling points of the associated ray of each component and the component-level shape information, the geometric information of the one or more sampling points of the associated ray is determined; based on the one or more sampling points of the associated ray of each component, the ray direction, the component-level shape information and the component-level texture information, the component is rendered to determine the color information of the one or more sampling points of the associated ray; based on the geometric information and color information of the one or more sampling points of the associated ray of each component, the color information of the corresponding pixel of the associated ray of each component is determined.

[0073] It should be noted that one ray corresponds to one pixel point, and each associated ray is sampled to determine one or more sampling points; the geometric information and color information of each sampling point are calculated; and then the color information of all sampling points is fused according to the rendering formula to obtain the color information of the pixel corresponding to the associated ray. Among them, the color information fusion weight of the sampling point is related to the geometric information of the sampling point.

[0074] In some embodiments, the geometric information may be the density of the sampling points, or the geometric information may be the occupancy rate of the sampling points.

[0075] In some embodiments, the method further includes: performing multi-resolution hash encoding on the three-dimensional coordinates of one or more sampling points of the associated ray, performing multi-resolution hash encoding on the ray direction of the associated ray, and using the multi-resolution hash encoding results to predict the geometric information and color information of the sampling points.

[0076] Step 404: Obtain a second image corresponding to the viewing angle information according to the rendered images of each component.

[0077] Figure 5 Schematic diagram of the pipeline of the component perception neural radiation field model in the embodiment of the present application, such as Figure 5 As shown, the component-aware neural radiation field model includes a decomposition network and multiple local neural radiation field networks. The local neural radiation field network is used to reconstruct the corresponding components. The local neural radiation field network includes a structural network and a neural rendering network.

[0078] The decomposition network is configured to perform component decomposition on the shape information and texture information of the modeled object, and determine the component-level shape information and component-level texture information of M components in the modeled object, where M is a positive integer.

[0079] The local neural radiation field network is configured to assign a complete set of rays corresponding to the viewing angle information to the associated component according to the component-level shape information of the first component, and determine the ray set of the first component; and perform component rendering according to the ray set, component-level shape information and component-level texture information of the first component to obtain a rendered image of the first component.

[0080] The component-aware neural radiation field model may also include a merging network to merge the rendered images of each component to obtain a second image.

[0081] In some embodiments, the component-aware neural radiation field model further includes a multi-resolution hash coding network ( Figure 5 ), configured to perform multi-resolution hash coding on the three-dimensional coordinates of one or more sampling points of the associated ray, perform multi-resolution hash coding on the ray direction of the associated ray, and use the multi-resolution hash coding result to predict the geometric information and color information of the sampling point.

[0082] The relevant component-aware NeRF model is good at capturing intricate scene details but is computationally demanding, especially for scenes of high complexity. This challenge is alleviated using a multi-resolution hashing grid structure that optimizes computational resources by effectively partitioning the scene into manageable chunks, enabling selective and targeted refinement.

[0083] The multi-resolution hash grid structure uses a hierarchical approach to divide the scene into grids of different resolutions. This hierarchical structure ensures efficient memory utilization and speeds up the rendering process by prioritizing calculations in areas where details are required while applying a coarser resolution where finer details are not needed.

[0084] The embodiment of the present application provides a set of local neural radiation field (Part-NeRF) networks, including M local neural radiation field networks, which are used to reconstruct corresponding parts. In order to achieve part-level control, hard assignment is enforced between parts and rays to ensure that changing one part does not affect the shape and appearance of other parts. The model does not require any explicit three-dimensional supervision, only assumes supervision from images captured from known cameras and object masks, and is able to perform editing operations on the shape and texture of the generated object.

[0085] In some embodiments, the decomposition network is configured to perform the steps of the following method: mapping the shape information and texture information of the modeled object into shape embeddings and texture embeddings of M components; encoding the shape embeddings and texture embeddings of the M components to obtain component-level shape information and component-level texture information of the M components; and inputting the component-level shape information and component-level texture information of the M components into the local neural radiation field network corresponding to each component, respectively.

[0086] The structural network is configured to perform the steps of the following method: assigning a complete ray set corresponding to the viewing angle information to the associated component according to the component-level shape information of the first component, determining the ray set of the first component, the first component being any component of the M components; mapping the component-level shape information of the first component to a rotation matrix, a translation vector and a scale vector; performing an affine transformation on the sampling points of each associated ray of the first component according to the rotation matrix and the translation vector, obtaining the transformed sampling points of each associated ray of the first component; mapping the ray direction of each associated ray of the first component to the local coordinate system of the first component according to the rotation matrix, obtaining the transformed ray direction of each associated ray of the first component;

[0087] The neural rendering network is configured to perform the steps of the following method: determining a first occupancy rate of the transformed sampling point of each associated ray of the first component according to the transformed sampling point of each associated ray of the first component and the component-level shape information of the first component; determining color information of the transformed sampling point of each associated ray of the first component according to the transformed sampling point of each associated ray of the first component, the transformed ray direction, the component-level shape information of the first component and the component-level texture information; determining color information of a pixel corresponding to each associated ray of the first component according to the first occupancy rate and color information of all the transformed sampling points on each associated ray of the first component.

[0088] In some embodiments, the structure network is further configured to perform the steps of the following method: determining a second occupancy rate of the transformed sampling points of each associated ray of the first component based on the scale vector and the transformed sampling points on each associated ray of the first component;

[0089] The neural rendering network is also configured to perform the steps of the following method: determining a third occupancy based on a first occupancy and a second occupancy of a transformed sampling point of each associated ray of the first component; determining color information of a pixel corresponding to each associated ray of the first component based on the third occupancy and color information of all transformed sampling points on each associated ray of the first component.

[0090] The present application embodiment cleverly coordinates and harmoniously interweaves with affine transformation strategies by deploying a local neural radiation field (Part-NeRF) network. This architectural foundation establishes a potential mature foundation, enabling access to a versatile toolkit to edit various functions across the boundaries of creative capabilities. Among the countless manipulation strategies that can be used, the ability to achieve transformations at the component level and seamlessly merge components from a range of various objects is a prominent feature of this method.

[0091] Figure 6 Schematic diagram of the network architecture of the component perception neural radiation field model in the embodiment of the present application, such as Figure 6 As shown, the component-aware neural radiation field model includes:

[0092] Decomposition network, using two object-specific learnable embeddings to represent the shape code z s and texture code z t , mapped to the embedding of M components and It is fed into two encoders to predict the shape embedding and texture embedding of each part, decomposing the 3D modeled object into shape codes of M parts. and the texture code Allows modification of one component's properties without affecting other components.

[0093] Structural network, the shape code of each component Mapped to the rotation matrix R m , translation vector t m and the scale vector s m , which define the local coordinate system of each component and its spatial extent; if we are interested in specific parts of the modeled object, we can modify the pose, size and appearance of each component independently, which can be enforced by ensuring that each NeRF receives geometric input in its own local coordinate system. m (x) = R m (x+t m ), map the sampling point x to the local coordinate system of the mth component, and obtain the transformed sampling point T m (x), transform the ray direction d to the local coordinate system of the mth component, and obtain the transformed ray direction R m d.

[0094] The neural rendering network, based on the transformed sampling point Tm(x) and the transformed ray direction R m d and shape code and texture code z t , the transformed sampling point is mapped to color c and the first occupancy o, which is specifically expressed as follows:

[0095]

[0096] In some embodiments, the component-perceived neural radiation field model also includes a multi-resolution hash coding network; the multi-resolution hash coding network is configured to perform the steps of the following method: position encoding the three-dimensional coordinates of one or more transformed sampling points of each associated ray of each component to obtain a first position coding result for each component, and inputting the first position coding result into the neural rendering network; position encoding the transformed ray direction of each associated ray of each component to obtain a second position coding result for each component, and inputting the second position coding result into the neural rendering network.

[0097] Figure 7 Schematic diagram of the pipeline of the multi-resolution hash coding network in the embodiment of the present application, such as Figure 7 As shown,

[0098] The core of the multi-resolution hash coding network is: to encode the three-dimensional spatial coordinates x of the sampled discrete points: y = enc(x; θ), θ is the parameter required for encoding, which is the parameter in all hash tables. Assuming there are L resolutions (that is, there are L hash tables), the size of each hash table is T, and each value in the hash table (representing density) is represented by an F-dimensional vector, then the total number of parameters in θ is T×L×F; the encoding result y is sent to the neural rendering network.

[0099] In the embodiment of the present application, the coordinates of the transformed sampling points and the transformed ray directions are multi-resolution hash coded Y(·). Therefore, the formulas of the above color c and the first occupancy rate o can also be expressed as follows:

[0100]

[0101] In order to force each component to capture only a continuous area of ​​the object, the present embodiment calculates the second occupancy of the axis-aligned 3D ellipsoid centered at the origin of the coordinate system. It is expressed as follows:

[0102]

[0103] The second occupancy of the ellipsoid of the mth component Input into the neural rendering model, specifically the transformed point T m The first occupancy rate of (x) With the second occupancy Multiply, the axis length is proportional to the vector s m This gives the final third occupancy of the mth component It is expressed as follows:

[0104]

[0105] Add noise to the rendered second image to obtain the noise image Z T , input into the diffusion model to predict the noise and obtain the noise information Z0. As long as the predicted noise is accurate enough, it means that the second image is accurate enough. The loss function can be simply written as the mean square error loss of the predicted noise and the real noise.

[0106] In some embodiments, the editing information includes at least one of the following: first shape editing information, second shape editing information and texture editing information, and the component-aware neural radiation field model is configured to perform the steps of the following method: inputting the first shape editing information of the first component into the structural network and the neural rendering network to modify the shape of the first component; inputting the second shape editing information of the first component into the structural network to modify one or more of the rotation matrix, the translation vector and the scale vector to achieve scaling, translation and rotation transformations of the first component; inputting the texture editing information of the first component into the neural rendering network to modify the texture of the first component.

[0107] It can be understood that the first shape editing information is used to replace the original component-level shape information with the new component-level shape information. The second shape editing information is used to transform the original shape of the component, such as scaling, translation, and rotation. The texture editing information is used to replace the original component-level texture information with the new component-level texture information, or to adjust the original component-level texture information.

[0108] For example, to apply a rigid transformation to a specific component, simply change the translation vector t of each component m and the rotation matrix R m For example, to translate a component, m Add the appropriate displacement vector. Similarly, to rotate the component, R m To apply a non-rigid transformation to a specific part, similarly, a non-rigid transformation that changes the size of a specific part can be achieved by multiplying the part's rotation matrix R by m Multiply with the appropriate scale matrix.

[0109] The embodiment of the present application provides a component-level editing function, and the posture, size and appearance of each component are independent, so a specific component can be modified separately without affecting other components.

[0110] The text-to-3D generation method provided in the embodiment of the present application has the following beneficial effects:

[0111] a) High-quality 3D content generation

[0112] 1. More realistic 3D content can be generated to better describe real-world objects and scenes.

[0113] 2. Users can immerse themselves in the intricate details and realistic textures seen in the 3D models they create, enhancing the overall visual experience.

[0114] b) Accelerate 3D content generation

[0115] 1. The use of multi-resolution hash coding significantly reduces the time required to generate 3D content, allowing users to obtain results in about 75 minutes.

[0116] 2. This accelerated sampling process improves generation efficiency, making the system more accessible and effective for a variety of applications.

[0117] 3. Generate 3D content based on text description, so that users can quickly create the 3D content they want through simple operations. This user-friendliness lowers the technical threshold, allowing more people to enjoy the fun and convenience brought by text-to-3D technology.

[0118] c) 3D content has component perception and editing capabilities

[0119] 1. Component-aware NeRF enables control at the component level, enabling a range of editing actions to be performed on the component shapes and visual properties of generated objects. These editing operations include rigid and non-rigid transformations applied to object components, blending components from different objects, adding or removing components, and refining the appearance of specific object segments.

[0120] 2. Component-aware NeRF Once the training process is complete, it can mix and modify the shape code and texture code of each component. Each component is independent of each other, which means that modifying one component will not affect the geometry and appearance of other components.

[0121] 3. Pre-trained shape codes and texture codes can also be optimized for second-stage training. Just freeze the pre-trained component-aware NeRF weights and only train the shape code and texture code.

[0122] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a text-to-three-dimensional generation device, such as Figure 8 As shown, the text to three-dimensional generating device 80 includes:

[0123] The text parsing model 801 is configured to perform semantic analysis on the text description to obtain text embedding;

[0124] Diffusion model 802 is configured to determine a first image matching the text description based on the text embedding; extract shape information and texture information of the modeled object in the first image;

[0125] The component-aware neural radiation field model 803 is configured to determine a second image corresponding to the perspective information based on the texture information and shape information of the modeling object, as well as the perspective information. The component-aware neural radiation field model supports editing operations on the components of the modeling object based on the editing information.

[0126] In some embodiments, the component-aware neural radiation field model 803 is configured to execute the steps of the following method: decomposing the shape information and texture information of the modeled object into components to determine the component-level shape information and component-level texture information of M components in the modeled object; assigning a complete set of rays corresponding to the viewing angle information to associated components according to the component-level shape information of each component to determine the ray set of each component, wherein the ray set includes a ray starting point and a ray direction of the associated ray; performing component rendering according to the ray set, component-level shape information and component-level texture information of each component to obtain rendered images of each component; and obtaining a second image corresponding to the viewing angle information according to the rendered images of each component.

[0127] In some embodiments, the component-aware neural radiation field model 803 includes a decomposition network and M local neural radiation field networks, the local neural radiation field network is used to reconstruct the corresponding component, and the local neural radiation field network includes a structure network and a neural rendering network;

[0128] The decomposition network is configured to perform the steps of the following method: mapping the shape information and texture information of the modeled object into the shape embeddings and texture embeddings of M components; encoding the shape embeddings and texture embeddings of the M components to obtain component-level shape information and component-level texture information of the M components; inputting the component-level shape information and component-level texture information of the M components into the local neural radiation field network corresponding to each component;

[0129] The structural network is configured to perform the steps of the following method: assigning a complete ray set corresponding to the viewing angle information to the associated component according to the component-level shape information of the first component, determining the ray set of the first component, the first component being any component of the M components; mapping the component-level shape information of the first component to a rotation matrix, a translation vector and a scale vector; performing an affine transformation on the sampling points of each associated ray of the first component according to the rotation matrix and the translation vector, obtaining the transformed sampling points of each associated ray of the first component; mapping the ray direction of each associated ray of the first component to the local coordinate system of the first component according to the rotation matrix, obtaining the transformed ray direction of each associated ray of the first component;

[0130] The neural rendering network is configured to perform the steps of the following method: determining a first occupancy rate of the transformed sampling point of each associated ray of the first component according to the transformed sampling point of each associated ray of the first component and the component-level shape information of the first component; determining color information of the transformed sampling point of each associated ray of the first component according to the transformed sampling point of each associated ray of the first component, the transformed ray direction, the component-level shape information of the first component and the component-level texture information; determining color information of a pixel corresponding to each associated ray of the first component according to the first occupancy rate and color information of all the transformed sampling points on each associated ray of the first component.

[0131] In some embodiments, the structure network is further configured to perform the steps of the following method: determining a second occupancy rate of the transformed sampling points of each associated ray of the first component based on the scale vector and the transformed sampling points on each associated ray of the first component;

[0132] The neural rendering network is also configured to perform the steps of the following method: determining a third occupancy based on a first occupancy and a second occupancy of a transformed sampling point of each associated ray of the first component; determining color information of a pixel corresponding to each associated ray of the first component based on the third occupancy and color information of all transformed sampling points on each associated ray of the first component.

[0133] In some embodiments, the component-aware neural radiation field model 803 also includes a multi-resolution hash coding network;

[0134] The multi-resolution hash coding network is configured to execute the steps of the following method: multi-resolution hash coding is performed on the three-dimensional coordinates of the transformed sampling point of each associated ray of the first component to obtain a first position coding result of the first component, and the first position coding result is input into the neural rendering network; multi-resolution hash coding is performed on the transformed ray direction of each associated ray of the first component to obtain a second position coding result of the first component, and the second position coding result is input into the neural rendering network.

[0135] In some embodiments, the editing information includes at least one of the following: first shape editing information, second shape editing information, and texture editing information.

[0136] The component-aware neural radiation field model is configured to execute the steps of the following method: inputting the first shape editing information of the first component into the structural network and the neural rendering network to modify the shape of the first component; inputting the second shape editing information of the first component into the structural network to modify one or more of the rotation matrix, the translation vector and the scale vector to achieve the scaling, translation and rotation transformation of the first component; inputting the texture editing information of the first component into the neural rendering network to modify the texture of the first component.

[0137] In some embodiments, the diffusion model 802 is further configured to add a first noise to a second image to obtain a noise image; determine a second noise based on the text embedding and the noise image; determine a loss function based on the first noise and the second noise; and optimize the parameters of the component perception neural radiation field model based on the loss function until the loss function converges.

[0138] In some embodiments, the method further includes: extracting a three-dimensional mesh from a trained component-perceived neural radiation field model; optimizing the three-dimensional mesh using a differentiable tetrahedral mesh to obtain an optimized three-dimensional mesh; and converting the optimized three-dimensional mesh into a standard three-dimensional file format.

[0139] In practical applications, the above-mentioned text-to-three-dimensional generation device can be an electronic device that realizes the text-to-three-dimensional generation function, or a chip applied to an electronic device. The electronic device can be independent of the visual back-end system, or it can be part of the visual back-end system. In the present application, the device can realize the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the text-to-three-dimensional generation method provided in any of the above embodiments. And the technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solutions in the text-to-three-dimensional generation method, and this application will not go into details one by one.

[0140] The above-mentioned device uses a diffusion model to provide a 2D image as a prior reference to guide the component-aware NeRF model to generate rendered images of 3D objects from any perspective, thereby improving the realism of the generated 3D objects without the need for explicit 3D supervision. Unlike related methods that are constrained by detailed annotations of volumetric data, the component-aware NeRF model provided in the embodiment of the present application can skillfully cope with the complexity of generated objects. This is achieved by utilizing guidance from qualitative images and text embedding enclosed in a continuous vector space, making the text-to-three-dimensional generation process more adaptable while avoiding the limitations imposed by traditional 3D supervision styles.

[0141] Furthermore, the part-aware NeRF model supports part-level editing operations of 3D objects, such as geometry modification and visual enhancement, without affecting other parts.

[0142] Based on the hardware implementation of each unit in the above-mentioned text-to-3D generation device, the embodiment of the present application also provides a text-to-3D generation device, such as Fig. 9 As shown, the text-to-three-dimensional generation device 90 includes: a processor 901 and a memory 902 configured to store a computer program that can be run on the processor;

[0143] The processor 901 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0144] Of course, in practical applications, Fig. 9 As shown, the various components in the text-to-three-dimensional generating device 90 are coupled together via a bus system 903. It can be understood that the bus system 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 903 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 903 in the figure.

[0145] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0146] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0147] In some embodiments, the text-to-3D generation device 90 may be a chip, and the chip may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0148] In some embodiments, the chip may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0149] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of a text-to-three-dimensional generation device to complete the steps of the aforementioned method.

[0150] The embodiments of the present application also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the embodiments of the present application.

[0151] Optionally, the computer program product can be applied to the text-to-three-dimensional generation device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the text-to-three-dimensional generation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0152] The embodiment of the present application also provides a computer program.

[0153] Optionally, the computer program can be applied to the text-to-three-dimensional generation device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the text-to-three-dimensional generation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0154] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0155] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "have", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0156] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0157] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0160] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0161] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for generating a text into a three-dimensional image, characterized in that: The method comprises: Perform semantic analysis on the text description to obtain text embedding; Inputting the text embedding into a diffusion model and outputting a first image matching the text description; Extracting shape information and texture information of the modeled object in the first image; The texture information and shape information of the modeling object and the viewing angle information are input into the component perception neural radiation field model, and a second image corresponding to the viewing angle information is output. The component perception neural radiation field model supports editing operations on the components of the modeling object according to the editing information.

2. The method according to claim 1, characterized in that: The component-aware neural radiation field model is configured to perform the following method steps: Decomposing the shape information and texture information of the modeled object into components, and determining component-level shape information and component-level texture information of M components in the modeled object, where M is a positive integer; According to the component-level shape information of each component, a complete ray set corresponding to the viewing angle information is assigned to the associated component, and a ray set of each component is determined, wherein the ray set includes a ray starting point and a ray direction of the associated ray; Rendering the components according to the ray set of each component, the component-level shape information, and the component-level texture information to obtain a rendered image of each component; According to the rendered images of the various components, a second image corresponding to the viewing angle information is obtained.

3. The method according to claim 2, characterized in that The component perception neural radiation field model includes a decomposition network and M local neural radiation field networks, the local neural radiation field network is used to reconstruct the corresponding component, and the local neural radiation field network includes a structural network and a neural rendering network; The decomposition network is configured to perform the steps of the following method: Mapping the shape information and texture information of the modeled object into shape embeddings and texture embeddings of M parts; Encoding the shape embedding and texture embedding of the M components to obtain component-level shape information and component-level texture information of the M components; Inputting the component-level shape information and component-level texture information of the M components into the local neural radiation field network corresponding to each component respectively; The structure network is configured to perform the steps of the following method: Assigning a complete set of rays corresponding to the viewing angle information to an associated component according to component-level shape information of the first component, and determining a set of rays of the first component, where the first component is any one of the M components; Mapping the component-level shape information of the first component into a rotation matrix, a translation vector, and a scale vector; According to the rotation matrix and the translation vector, the sampling points of each associated ray of the first component are affine transformed to obtain transformed sampling points of each associated ray of the first component; Mapping the ray direction of each associated ray of the first component to the local coordinate system of the first component according to the rotation matrix to obtain a transformed ray direction of each associated ray of the first component; The neural rendering network is configured to perform the steps of the following method: Determine a first occupancy rate of the transformed sampling points of each associated ray of the first component according to the transformed sampling points of each associated ray of the first component and the component-level shape information of the first component; Determine color information of the transformed sampling point of each associated ray of the first component according to the transformed sampling point of each associated ray of the first component, the transformed ray direction, component-level shape information and component-level texture information of the first component; Color information of a pixel corresponding to each associated ray of the first component is determined according to first occupancy rates and color information of all transformed sampling points on each associated ray of the first component.

4. The method according to claim 3, characterized in that The fabric network is also configured to perform the following method steps: Determine a second occupancy rate of the transformed sampling points of each associated ray of the first component according to the scale vector and the transformed sampling points of each associated ray of the first component; The neural rendering network is further configured to perform the steps of the following method: determining a third occupancy rate according to the first occupancy rate and the second occupancy rate of the transformed sampling point of each associated ray of the first component; Color information of a pixel corresponding to each associated ray of the first component is determined according to the third occupancy rates and color information of all transformed sampling points on each associated ray of the first component.

5. The method according to claim 3, characterized in that: The component-aware neural radiation field model also includes a multi-resolution hash coding network; The multi-resolution hash coding network is configured to perform the steps of the following method: Performing multi-resolution hash coding on the three-dimensional coordinates of the transformed sampling points of each associated ray of the first component to obtain a first position coding result of the first component, and inputting the first position coding result into the neural rendering network; Multi-resolution hash coding is performed on the transformed ray direction of each associated ray of the first component to obtain a second position coding result of the first component, and the second position coding result is input into the neural rendering network.

6. The method according to claim 3, characterized in that The editing information includes at least one of the following: first shape editing information, second shape editing information and texture editing information, The component-aware neural radiation field model is configured to perform the following method steps: Inputting first shape editing information of the first component into the structural network and the neural rendering network to modify the shape of the first component; Inputting the second shape editing information of the first component into the structural network to modify one or more of the rotation matrix, the translation vector and the scale vector to achieve scaling, translation and rotation transformation of the first component; Inputting the texture editing information of the first component into the neural rendering network to modify the texture of the first component.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: adding a first noise to the second image to obtain a noisy image; Inputting the text embedding and the noise image into the diffusion model, and outputting a second noise; Determining a loss function according to the first noise and the second noise; Based on the loss function, the parameters of the component perception neural radiation field model are optimized until the loss function converges.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: extracting a three-dimensional mesh from a model of the component-sensing neural radiation field; Optimizing the three-dimensional mesh using a differentiable tetrahedral mesh to obtain an optimized three-dimensional mesh; The optimized three-dimensional mesh is converted into a standard three-dimensional file format.

9. A device for generating text into three dimensions, characterized in that: The device comprises: Text parsing model, configured to perform semantic analysis on text descriptions to obtain text embeddings; A diffusion model configured to determine a first image matching the text description based on the text embedding; extract shape information and texture information of a modeled object in the first image; The component-aware neural radiation field model is configured to determine a second image corresponding to the perspective information based on texture information and shape information of the modeling object, as well as perspective information. The component-aware neural radiation field model supports editing operations on the components of the modeling object based on editing information.

10. A device for generating text into three dimensions, characterized in that: include: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 8 when running the computer program.

Images