Text-to-image diffusion model for generalizable mesh generation

Abstract

Certain aspects of the present disclosure provide techniques and apparatuses for improved machine learning. In an example method, a multi-view latent tensor is generated based on processing a text input using a diffusion machine learning model, where the multi-view latent tensor corresponds to a plurality of orthographic projections corresponding to the text input. A tri-plane latent tensor is generated based on the multi-view latent tensor using a transformation machine learning model, and a three-dimensional mesh is generated based on processing the tri-plane latent tensor using a decoder machine learning model.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 537,496, filed December 12, 2023, which is incorporated herein by reference. Background Technology

[0003] This disclosure relates to various aspects of machine learning.

[0004] In recent years, a wide variety of machine learning models have been trained for a large number of similar tasks. For example, generative models (e.g., Generative Adversarial Networks (GANs), diffusion models, etc.) have been trained to generate new output data (e.g., images or text) based on input prompts. For instance, some models are able to receive a text string describing an object (e.g., "sailboat") and use that string to generate a two-dimensional image depicting the requested object (e.g., an image of a sailboat). Summary of the Invention

[0005] Some aspects of this disclosure provide a processor-implemented method comprising: generating a multi-view latent tensor based on processing text input using a diffusion machine learning model, wherein the multi-view latent tensor corresponds to a plurality of orthographic projections corresponding to the text input; generating a three-plane latent tensor based on the multi-view latent tensor using a transformation machine learning model; and generating a three-dimensional mesh based on processing the three-plane latent tensor using a decoder machine learning model.

[0006] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0007] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0008] The accompanying drawings illustrate certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0009] Figure 1 An example workflow for generating 3D meshes using a generative machine learning model, based on some aspects of this disclosure, is described.

[0010] Figure 2 An example workflow is described, based on some aspects of this disclosure, for generating multi-view image latent tensors using a diffusion model to facilitate 3D mesh generation.

[0011] Figure 3 An example workflow for generating tri-plane data based on multi-view image data is described according to some aspects of this disclosure.

[0012] Figure 4 An example workflow for training a machine learning model to generate a 3D mesh, according to some aspects of this disclosure, is described.

[0013] Figure 5 This is a flowchart depicting an example method for generating and texturing 3D meshes using multi-view image data, according to some aspects of this disclosure.

[0014] Figure 6 This is a flowchart depicting an example method for generating a three-dimensional mesh according to some aspects of this disclosure.

[0015] Figure 7 Example processing systems configured to perform various aspects of this disclosure are described.

[0016] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0017] This disclosure provides apparatus, methods, processing systems, and non-transitory computer-readable media for providing improved machine learning. Specifically, aspects relate to improved three-dimensional (3D) mesh generation.

[0018] Despite recent advances in 2D generative modeling (e.g., for image generation), efficient generation of high-quality 3D assets remains a challenging task. A significant challenge in 3D mesh generation is data scarcity: training effective models relies on vast amounts of training data. The most widely available 3D training datasets contain millions of assets, while 2D training datasets typically contain billions of images. Consequently, some conventional methods are insufficient for training generalizable 3D mesh generation models.

[0019] In some aspects of this disclosure, a large and pre-trained 2D diffusion model can be utilized and adapted to generate 3D meshes. More specifically, a pre-trained text-to-image generative model (e.g., a diffusion model that generates images based on text input) can be fine-tuned or refined such that the model collectively predicts or generates multiple orthographic projections of a requested object (e.g., an object described in the text input). In some aspects, this set of orthographic projections is referred to as a "multi-view" image to indicate that the image data contains multiple orthographic projections of an object. In some aspects, a multi-view image may be referred to as a "six-view" image to indicate that the image contains six orthographic projections.

[0020] In some respects, these projections are orthogonal (e.g., each projection is a parallel projection of the object, where the projection lines are orthogonal to the projection plane, such that the view of the object does not include perspective). In some respects, one or more views among a plurality of views are also orthogonal to each other (e.g., perpendicular). For example, in the case of a three-view image (with three orthogonal projections), these views may include a top view (e.g., a parallel projection of the object viewed from above), a side view (e.g., a parallel projection of the object viewed from one side), and a front view (e.g., a parallel projection of the object viewed from directly in front). In some respects, in the case of a six-view image, these views may include a top view, a bottom view, a left view, a right view, a front view, and a rear view.

[0021] In some aspects, transformational machine learning models can be used to process multi-view image data to generate corresponding triplanar latent representations, and these triplanar latent representations can be processed to generate 3D meshes. In other aspects, the disclosed techniques and architectures do not rely on per-sample optimization and can generate high-quality and diverse objects based on text prompts within seconds, which is significantly faster than existing methods. Furthermore, the disclosed architectures demonstrate strong generalization to new objects or components.

[0022] In some aspects of this disclosure, machine learning models can be used to generate 3D meshes for a variety of domains and applications. For example, such 3D mesh generation can be used in video game development (e.g., to generate meshes that can be used as objects in a game), for augmented reality (AR) and / or virtual reality (VR) (e.g., to generate objects for AR / VR displays), for graphic design tasks, for generating assets for films and shows, and so on. Using conventional methods, 3D mesh generation is a very tedious and time-consuming task. However, utilizing aspects of this disclosure, highly detailed and textured meshes can be generated by processing text-based cues using specially trained machine learning models.

[0023] Example workflow for generating 3D meshes using a generative machine learning model.

[0024] Figure 1An example workflow 100 for generating 3D meshes using a generative machine learning model, according to some aspects of this disclosure, is described. In some aspects, workflow 100 is performed by a machine learning system (e.g., a computational system that trains a machine learning model and / or uses a trained machine learning model). In some aspects, models may be trained on one or more computational systems (referred to in some aspects as "training systems"), and these models are then deployed for inference (e.g., mesh generation) on one or more other computational systems (referred to in some aspects as "inference systems").

[0025] In the illustrated workflow 100, text input 105 is accessed for processing using a diffusion model 110 (referred to in some aspects as a diffusion machine learning model) to generate six-view data 115 (or more generally, multi-view data). As used herein, “accessing” data can generally mean receiving, requesting, retrieving, collecting, measuring, generating, obtaining, or otherwise gaining access to data. In the illustrated example, text input 105 typically comprises natural language text describing a desired object or scene. For example, text input 105 may specify an object such as “a green dragon holding a sword” or “a blue humanoid robot with four arms.” In some aspects, text input 105 is provided by a user. In some aspects, text input 105 may be generated using other operations, such as processing audio data using a speech-to-text algorithm. Although not included in the illustrated example, in some aspects, other input data (e.g., image data depicting the desired object) may be used in addition to or in lieu of text input 105 to prompt the model.

[0026] In some aspects, the diffusion model 110 employs an autoencoder architecture (e.g., a variational autoencoder (VAE) model). For example, the diffusion model 110 may use an encoder part to encode the input (e.g., text input 105) into a latent space and a decoder part to decode the latent space into the desired output (e.g., six-view data 115). In some aspects, the diffusion model 110 is a modified or updated version of a pre-trained text-to-image model. For example, a generative model that has been trained to generate images based on text strings (e.g., using a large amount of image data) may be trained or accessed by a machine learning system and may then be refined, fine-tuned, adapted, modified, or otherwise updated based on a relatively small 3D dataset to incorporate 3D knowledge into the architecture, as discussed in more detail below.

[0027] In the illustrated workflow 100, the diffusion model 110 generates six-view data 115. Although depicted as six views for conceptual clarity, in some respects, the six-view data 115 can be any multi-view data (e.g., three-view, four-view, etc.). Generally speaking, the six-view data 115 corresponds to any data that includes or represents multiple orthographic projections depicting an object described by or indicated in the text input 105. For example, the six-view data 115 could be an image of six orthographic projections of an object spatially stitched together (e.g., in two rows, three projections per row).

[0028] In some aspects, the six-view data 115 includes data for a “front” view (e.g., a parallel projection from the front of the object), a “rear” view (e.g., a parallel projection from behind the object), a “left” view (e.g., a parallel projection from the left side of the object), a “right” view (e.g., a parallel projection from the right side of the object), a “top” view (e.g., a parallel projection from the top of the object), and a “bottom” view (e.g., a parallel projection from the bottom of the object). In some aspects, the six-view data 115 has three pairs of corresponding or parallel orthographic projections. That is, the “top” view and the “bottom” view may be referred to as a “parallel pair” or “a pair of parallel orthographic projections” to indicate that these two views were generated using projection lines parallel to each other (which, compared to, for example, the “left” view, the “right” view, the “front” view, and the “rear” view, each use projection lines orthogonal to or perpendicular to the “top” view and the “bottom” view). Similarly, the “left” view and the “right” view may be referred to as a parallel pair, and the “front” view and the “rear” view may be referred to as a parallel pair.

[0029] In some respects, as discussed in more detail below, the six-view data 115 is image data. For example, the six-view data 115 could be an image (e.g., six orthographic projections stitched together in space) with four channels (e.g., where each pixel has four values, such as red, green, blue, and α values ​​to define the pixel's opacity). In some respects, the six-view data 115 is instead a set of intermediate features generated by the inner layers (e.g., the penultimate layer) of the diffusion model 110. For example, instead of using such image data to generate a 3D mesh, the diffusion model 110 could be trained to generate six-view image data, and the six-view data 115 could be an intermediate feature tensor (e.g., with hundreds of channels) generated by the penultimate layer. In some respects, the final layer of the diffusion model 110 can be used to process the intermediate features to generate the six-view image (as discussed below).

[0030] In some aspects, as discussed in more detail below, six-view data 115 is generated by processing the data through multiple iterations of a diffusion model 110. For example, the diffusion model 110 may include a denoising backbone that processes random (e.g., Gaussian) noise along with text input 105 to generate a relatively denoised latent tensor. This denoised tensor can then be iteratively processed using this backbone to generate data with progressively reduced noise until a final iteration produces a latent tensor that can be decoded to generate a high-quality six-view image. In some aspects, instead of using this final latent representation, the diffusion model 110 may perform an additional iteration using added conditional tokens to imbue the latent representation with 3D knowledge. The features of the penultimate layer of the model are then available as the six-view data 115, as discussed in more detail below.

[0031] In the illustrated workflow 100, six-view data 115 is processed by transformation module 120 (referred to in some aspects as a transformation machine learning model) to generate tri-plane data 125. In some aspects, transformation module 120 may include one or more non-parametric operations (e.g., spatial slicing, rotating and / or flipping slices of the six-view data 115, and stitching slices corresponding to parallel views, as discussed in more detail below). The preprocessed data can then be processed using one or more parametric components (e.g., one or more convolutional layers) to generate tri-plane data 125.

[0032] In some respects, the triplanar data 125 (also referred to in some respects as the triplanar latent tensor) is a triplanar representation of a 3D object (e.g., a feature plane aligned using three axes). In some respects, the triplanar data 125 can be volumetrically rendered or projected to create a 3D mesh.

[0033] In the illustrated example, the three-plane data 125 is processed by a decoder model 130 (referred to in some aspects as a decoder machine learning model) to generate an output 3D mesh 135. In the illustrated example, mesh 135 is a three-dimensional uppercase letter "F". For example, text input 105 could include "uppercase letter F".

[0034] In some respects, decoder model 130 may correspond to a pre-trained autoencoder model or a portion thereof. For example, as discussed in more detail below, a VAE-based model may be trained to generate a 3D mesh based on input data (e.g., an input point cloud), where the latent space (between the encoder and decoder) corresponds to a three-plane representation of that input. After training this mesh generation model, the encoder and decoder portions can be used to facilitate fine-tuning of the diffusion model 110, as discussed in more detail below. Furthermore, the decoder portion of the mesh generation model may be used as decoder model 130 during inference (e.g., to generate a 3D mesh during runtime).

[0035] In some respects, mesh 135 can generally be any format and type of mesh, depending on the specific implementation. For example, in some respects, mesh 135 is a surface mesh (e.g., a set of polygonal faces, such as triangles, quadrilaterals, etc.). As another example, in some respects, mesh 135 can be a solid mesh or a volumetric mesh (e.g., a set of three-dimensional elements, such as tetrahedrons, hexahedrons, polyhedra, etc.).

[0036] In some respects, as discussed in more detail below, the mesh 135 may be textured. For example, in some respects, the tri-plane data 125 itself encodes texture information from (e.g., generated by diffusion model 110) the six-view data 115. Thus, in some respects, the mesh 135 may already include texture material. In some respects, as discussed below, the six-view data 115 may additionally be used directly to add further detail to the texture of the mesh 135. For example, in some respects, a UV map can be generated by processing the initial mesh 135 using a UV unwrapping algorithm. UV mapping typically involves projecting a three-dimensional mesh onto a two-dimensional plane to facilitate texturing or coloring the mesh using a two-dimensional image. Generally, the letters “U” and “V” are conventionally used to represent the axes of a two-dimensional image. In some respects, machine learning systems do not use UV mapping but instead use a UVW mapping method, which allows the use of a third dimension in the texture (via the “W” axis). The UV texture can be initialized based on the original colors predicted from the tri-plane data 125 and the UV map. In some aspects, one or more final layers of the diffusion model 110 (e.g., using a pre-trained decoder) can be used to process the six-view data 115 to generate a six-view image. Pixel values ​​from the six-view image (e.g., red, green, blue, and α (RGBA)) can be mapped onto an initialized UV texture. This refined texture can then be used to texture the mesh 135 based on the UV mapping, resulting in significantly improved detail.

[0037] Additionally, in some aspects, additional texture information can be encoded into the three-plane data 125 and / or the six-view data 115. In some aspects, the additional texture can correspond to different texture modes. For example, a set of six-view data 115 can be generated to produce a texture including the diffuse or primary color of the object, a second set of six-view data 115 can be generated to produce a texture including the surface normals of the object, and a third set of six-view data 115 can be generated to produce a texture including metallic properties and roughness properties. Generally speaking, a set of six-view data 115 can be generated for any texture mode or material property of the object. By using such additional information, the resulting mesh 135 texture can be significantly improved.

[0038] As discussed above, Mesh 135 can be used for a variety of purposes. For example, Mesh 135 can be used directly as a 3D asset in video games, movies, shows, etc. In some aspects, Mesh 135 can be used as an initial or starting mesh, allowing users to quickly refine the initial mesh to generate the final mesh. In other aspects, a machine learning system (or another system) can render one or more images depicting Mesh 135 from one or more viewpoints or angles and output the rendered images via one or more displays. This allows users to easily visually observe, evaluate, and / or modify Mesh 135.

[0039] Example workflow for generating multi-view image latent tensors using a diffusion model to facilitate 3D mesh generation

[0040] Figure 2 An example workflow 200 for generating multi-view image latent tensors using a diffusion model to facilitate 3D mesh generation, according to some aspects of this disclosure, is described. In some aspects, workflow 200 is powered by a machine learning system (such as those referenced above). Figure 1 The machine learning system under discussion is used to execute this. In some respects, workflow 200 is designed for... Figure 1 Additional details are provided for the operation of diffusion model 110.

[0041] In the illustrated workflow, text input 205 is processed (which may correspond to...) Figure 1 The text input 105) is used to generate an intermediate tensor 225 (which can correspond to the text input 105) ... Figure 1 The six-view data 115). Specifically, as illustrated, diffusion model 210 is used (which may correspond to...). Figure 1 The diffusion model 210 is used to process the text input 205 to generate the six-view latent tensor 215. In some respects, as discussed above and illustrated by arrow 217, the diffusion model 210 can be used to iteratively process the data over multiple iterations in order to progressively denoise the six-view latent tensor 215.

[0042] For example, in some aspects, the first iteration of processing the data may include processing a noisy latent tensor (e.g., with random Gaussian noise) in conjunction with text input 205 (or its encoded version) to generate a first six-view latent tensor 215 for the first iteration. As indicated by arrow 217, the six-view latent tensor 215 can then be used as input latent representation (along with text encoding) during the second iteration to generate a new six-view latent tensor 215. This process may be repeated any number of iterations until a six-view latent tensor 215 is generated during the final iteration. As illustrated, text input 205 (e.g., embeddings generated based on text) is used to modulate the denoising process. Although not depicted in the illustrated example, in some aspects, additional data may be used to modulate the denoising. For example, in some aspects, denoising may be further modulated based on time-step embeddings (e.g., indicating which iteration is being performed), positional embeddings, etc.

[0043] In the illustrated example, a diffusion model 210 is used to process the six-view latent tensor 215 for one additional iteration or the final iteration. During this final iteration, an additional condition token 220 can be used to conditionalize the model. For example, in some aspects, the condition token 220 may correspond to a "make it 3D" token, which 3D conditions the diffusion model 210 to ensure that the final output adequately incorporates the relevant 3D properties. As illustrated, the machine learning system does not output the final six-view latent tensor (e.g., as...). Figure 1 Instead of using the six-view data 115 in the model, intermediate tensors 225 (e.g., features generated by the penultimate layer of the diffusion model 210) are used as the output representation. As discussed above, these intermediate tensors 225 can be used to generate a three-plane representation of the object.

[0044] Although the illustrated example depicts the use of the same diffusion model 210 for generating the six-view latent tensor 215 to generate the intermediate tensor 225, other components may be used in some aspects. For example, in some aspects, the diffusion model 210 may be used to generate the six-view latent tensor 215 over multiple iterations. Once the iterations are complete, different machine learning models (e.g., including one or more convolutional operations) may be used to process the final six-view latent tensor 215 to generate the intermediate tensor 225.

[0045] Example workflow for generating triplane data based on multi-view image data

[0046] Figure 3 An example workflow 300 for generating tri-plane data based on multi-view image data is described according to some aspects of this disclosure. In some aspects, workflow 300 is powered by a machine learning system (such as the one referenced above). Figures 1 to 2 The machine learning system under discussion is used to execute this. In some respects, workflow 300 is designed for... Figure 1Additional details are provided regarding the operation of the conversion module 120.

[0047] In the illustrated workflow 300, six-view data 305 (which may correspond to...) are processed. Figure 1 Six-view data 115 and / or Figure 2 The intermediate tensor 225) is used to generate three-plane data 335 (which can correspond to the intermediate tensor 225) ... Figure 1 The three-plane data 125). In the illustrated example, the six-view data 305 includes or corresponds to a plurality of orthographic projections 310A to 310F. Specifically, as illustrated, the six-view data 305 depicts the capital letter "F" (e.g., the six-view data 305 may have been generated based on text such as "capital F"), and includes a front orthographic projection 310A, a rear orthographic projection 310D, a top orthographic projection 310B, a bottom orthographic projection 310E, a left orthographic projection 310C, and a right orthographic projection 310F. As illustrated, each orthographic projection 310 is spatially stitched together to form the six-view data 305.

[0048] In the illustrated example, for clarity and visualization, the six-view data 305 is depicted as image data. However, as discussed above, the six-view data 305 could be a latent tensor (e.g., Figure 2 The six-view latent tensor 215) and / or intermediate feature tensors (e.g., Figure 2 The intermediate tensor 225). As discussed above, such latent tensors and / or intermediate features can still include or correspond to six (or other numbers) orthographic views or projections. For example, the front view orthographic projection 310A may include latent representations and / or feature data of the orthographic projection of the letter "F" from the front. In some aspects, the six-view data 305 is represented as having dimensions The tensor, in which W is the height of each orthographic projection 310 (in pixels), and W is the width of each orthographic projection 310 (in pixels). It is the channel depth of the six-view data 305.

[0049] As illustrated, transformation operation 315 is used to process the six-view data 305 to generate stitched data 320. Transformation operation 315 typically corresponds to one or more non-parametric transformations or operations used to align the parallel views of the six-view data 305 to match the orientation of the projections. As discussed above, in some aspects, each orthographic projection 310 in the six-view data 305 may have corresponding or parallel orthographic projections 310, where parallel views correspond to projections from opposite sides of the object. For example, front orthographic projection 310A and rear orthographic projection 310D may form a first parallel pair, top orthographic projection 310B and bottom orthographic projection 310E may be a second parallel pair, and left orthographic projection 310C and right orthographic projection 310F may be a third parallel pair.

[0050] In the illustrated workflow 300, transformation operation 315 typically includes slicing the six-view data 305 to separate the orthographic projections 310 (e.g., splitting them into six separate tensors). Transformation operation 315 may then include flipping (also referred to in some respects as mirroring) and / or rotating one or more of the sliced ​​orthographic projections 310 to align each orthographic projection 310 with its parallel view (e.g., horizontally mirroring the rear orthographic projection 310D such that the features of the rear orthographic projection 310D are aligned with the features of the front orthographic projection 310A). In the illustrated example, transformation operation 315 also includes stitching the parallel views or corresponding views along the channel dimension.

[0051] Specifically, as illustrated, the realigned front view orthographic projection 310A and rear view orthographic projection 310D are stitched together to form the stitched orthographic projection 325A. Furthermore, the realigned top view orthographic projection 310B and bottom view orthographic projection 310E are stitched together to form the stitched orthographic projection 325B, and the realigned left view orthographic projection 310C and right view orthographic projection 310F are stitched together to form the stitched orthographic projection 325C. In some aspects, the stitched data 320 is represented as having dimensions. The tensor.

[0052] In the illustrated workflow 300, the concatenated data 320 is then processed using a transformation model 330 to generate a set of triplane data 335. In some respects, the transformation model 330 (also referred to in some respects as a transformational machine learning model) typically includes one or more parameterized operations (e.g., convolution using learned weights, with values ​​learned during training).

[0053] Example workflow for training a machine learning model to generate a 3D mesh

[0054] Figure 4An example workflow 400 for training a machine learning model to generate a 3D mesh, according to some aspects of this disclosure, is described. In some aspects, workflow 400 is performed by a machine learning system (such as those referenced above). Figures 1 to 3 The workflow 400 is used to execute the training of the various machine learning components discussed above, such as the machine learning system in question. Figure 1 diffusion model 110 and / or Figure 1 The conversion module 120 (e.g., Figure 3 Transformation model 330).

[0055] In the depicted workflow 400, a baseline ground truth or target 3D mesh 430 is evaluated to generate a six-view image 435. That is, a six-view image (e.g., six orthographic projections or views) can be generated based on rendering the mesh 430 from each of six angles (without perspective). In the illustrated example, the six-view image 435 is processed using a 2D encoder 440 to generate a baseline ground truth six-view latent tensor 445. In some respects, the 2D encoder 440 is a machine learning model or component trained (or pre-trained) to generate latent embeddings.

[0056] In the illustrated example, text input 405 (which may include natural language text, as referenced above) Figure 1 Text input 105 and Figure 2 The text input 205 (discussed) is processed by the diffusion model 110 to generate a six-view latent tensor 415. In some respects, as discussed above, the diffusion model 110 may be a pre-trained (e.g., text- and image-based trained) text-to-image model, and the machine learning system may use workflow 400 to refine or update the parameters of the model to produce an improved diffusion model 110 that can generate a 3D-aware six-view.

[0057] In some aspects, as discussed above, generating the six-view latent tensor 415 may include using a diffusion model 110 to process data (such as text input 405 and intermediate latent tensors) for one or more iterations. In some aspects, during the first iteration, a random noise tensor may be provided as input to initiate denoising. In some aspects, a baseline truth six-view latent representation (such as the six-view latent tensor 445) may be used alternatively.

[0058] For example, in the illustrated workflow 400, a noise operation 448 can be used to process the baseline truth six-view latent tensor 445 to add a certain amount of noise (e.g., Gaussian noise) to create a noisy truth latent representation. This noisy baseline truth latent representation, along with the embedding of text input 405, can then be fed as input to the diffusion model 110 to generate a (denoised) six-view latent tensor 415. This facilitates the training of the diffusion model 110.

[0059] As illustrated, the baseline truth six-view latent tensor 445 and the generated six-view latent tensor 415 are used to generate the diffusion loss 450. That is, the diffusion loss 450 can be generated based on the difference between the six-view latent tensor 445 and the six-view latent tensor 415. The diffusion loss 450 can typically be formulated using various loss formulas, such as mean squared error. In some respects, the diffusion loss 450 can be used to update one or more parameters of the diffusion model 110 during training.

[0060] In the illustrated example, intermediate operation 452 is also used to process the baseline truth six-view latent tensor 445 to generate an intermediate feature tensor, which is used as input to transformation module 120. For example, as discussed above, intermediate operation 452 may correspond to processing the six-view latent tensor 445 using diffusion model 110 for a final single iteration (along with the 3D token), and the features of the penultimate layer of the diffusion model may be used as input to transformation module 120. In other respects, as discussed above, the input to transformation module 120 may be the six-view latent tensor 445 itself.

[0061] As illustrated, the transformation module 120 generates a three-plane latent tensor 425 based on the module's input (e.g., directly based on the six-view latent tensor 445, or based on intermediate features generated using the six-view latent tensor 445).

[0062] Additionally, in the illustrated example, a 3D encoder 455 is used to generate a benchmark ground truth triplane latent tensor 460 based on a target mesh 430. In some aspects, the 3D encoder 455 may be an encoder from a VAE-based model trained to generate a 3D mesh based on input data. For example, the model may be trained to receive a point cloud and (using the 3D encoder 455) generate a latent spatial representation of the input (e.g., a triplane representation). Furthermore, during training, a decoder component may be trained to generate a 3D mesh. This mesh may be compared to a benchmark ground truth mesh to train the encoder and decoder components. In some aspects, after training the model, the encoder may be used as a 3D encoder 455 to train the diffusion model 110 and / or the transformation module 120. Furthermore, as discussed above, the decoder portion of the model may be used during runtime as... Figure 1 Decoder model 130.

[0063] In some respects, instead of providing the mesh 430 directly to the 3D encoder 455, the mesh 430 can be processed through one or more other operations, such as generating a point cloud based on the mesh 430. The point cloud can then be used as input to the 3D encoder 455.

[0064] In the illustrated example, the generated three-plane latent tensor 425 and the baseline ground truth three-plane latent tensor 460 are then used to generate the reconstruction loss 465. That is, the reconstruction loss 465 can be generated based on the difference between the three-plane latent tensor 425 and the three-plane latent tensor 460. The reconstruction loss 465 can typically be formulated using various loss formulas, such as mean squared error. In some aspects, the reconstruction loss 465 can be used to update the transformation module 120 during training (e.g., Figure 3 One or more parameters of the transformation model 330.

[0065] In some aspects, any number of training grids 430 can be used to execute workflow 400 any number of times to generate diffusion loss 450 and reconstruction loss 465 to train the model. In some aspects, backpropagation can be used to train the model to update the weights or other parameters of each layer of each model. In some aspects, the machine learning system can update parameters based on each training paradigm (e.g., using stochastic gradient descent) and / or based on batches of training paradigms (e.g., using batch gradient descent).

[0066] In some respects, the diffusion model 110 and the transformation module 120 can be trained independently (as illustrated in workflow 400). That is, the parameters of the diffusion model 110 can be updated based on the grid 430 without considering the output of the transformation module 120, and the transformation module 120 can be updated based on the grid 430 without considering the output of the diffusion model 110. In this way, the diffusion model 110 and the transformation module 120 can be trained at different times, on different systems, and / or for different time lengths or number of iterations.

[0067] In some aspects, the diffusion model 110 and the transformation module 120 can be trained jointly (e.g., end-to-end). For example, instead of using the baseline ground truth six-view latent tensor 445 to generate the triplane latent tensor 425, the machine learning system can use the generated six-view latent tensor 415 to generate the triplane latent tensor 425. In some aspects, the diffusion model 110 can be trained by performing a single denoising iteration. That is, a first diffusion loss can be generated based on a first version of the noisy baseline ground truth six-view latent representation processed as input (e.g., a six-view latent tensor 445 with a first amount of added noise) to generate a corresponding denoised output. A second diffusion loss can also be generated based on a second version of the noisy baseline ground truth six-view latent representation processed as input (e.g., a six-view latent tensor 445 with a second different amount of added noise) to generate a corresponding denoised output.

[0068] Therefore, if an end-to-end joint training operation is used, the transformation module 120 can be trained to predict the three-plane latent tensor 425 based on the "partially denoised" six-view latent tensor 415. In some respects, this can serve as a regularization technique that makes the transformation module 120 robust to noisy inputs.

[0069] In some respects, after training, the diffusion model 110 and the transformation module 120 (together with the decoder model discussed above) can be deployed for inference. Generally speaking, deploying the model may include any operations used to prepare or provide the model for inference (by the machine learning system or by another system).

[0070] Example methods for generating and texturing 3D meshes using multi-view image data

[0071] Figure 5 This is a flowchart depicting an example method 500 for generating and texturing 3D meshes using multi-view image data, according to some aspects of this disclosure. In some aspects, method 500 is powered by a machine learning system (such as the one referenced above). Figures 1 to 4 The machine learning system discussed is used to execute this. In some respects, method 500 is executed by a dedicated inference system.

[0072] At box 505, the machine learning system accesses the text prompt (e.g., Figure 1 Text input 105 and / or Figure 2 (Text input 205). For example, as discussed above, text prompts may include natural language text describing one or more objects, scenes, environments, or other entities that the 3D mesh is expected to target.

[0073] At box 510, the machine learning system uses a diffusion machine learning model (e.g., Figure 1 and / or Figure 4 The diffusion model 110, and / or Figure 2 The diffusion model 210 generates a six-view latent tensor. For example, as discussed above, a machine learning system can use the diffusion model to process data in multiple iterations to iteratively denoise the six-view latent tensor. Although the illustrated example depicts a six-view latent tensor for conceptual clarity, as discussed above, a machine learning system can alternatively generate any multi-view latent tensor.

[0074] In some aspects, machine learning systems do not generate a single six-view latent tensor, but can additionally generate one or more auxiliary six-view latent tensors. For example, in some aspects, each auxiliary six-view latent tensor may correspond to a rotated version of the original six-view latent tensor (e.g., depicting six orthographic projections of an object from different angles, such as diagonally from left-front, right-front, etc., compared to the front, back, left, right, top, and bottom views of the original latent representation). In some aspects, using such auxiliary six-view latent tensors can help the model generate higher-quality 3D meshes. That is, a three-plane latent and / or final mesh can be generated based on both the six-view latent tensor and the auxiliary six-view latent tensors.

[0075] As another example, in some aspects, each auxiliary six-view latent tensor in the auxiliary six-view latent tensor may correspond to a different resolution (e.g., initially generating the six-view latent tensor at a relatively low resolution, and gradually increasing the resolution across diffusion iterations). In some aspects, using such auxiliary six-view latent tensors can help improve model robustness and / or reduce the computational load of processing data using a diffusion model. That is, a three-plane latent and / or final mesh can be generated based on both the six-view latent tensor and the auxiliary six-view latent tensors.

[0076] As another example, in some aspects, each of the auxiliary six-view latent tensors in the auxiliary six-view latent tensor may correspond to a different material property (e.g., a first latent tensor for diffuse or primary colors, a second latent tensor for metallic features, a third latent tensor for surface normal features, etc.). In some aspects, using such auxiliary six-view latent tensors can improve the texturing of the output mesh. That is, the texture of the final mesh can be generated based on both the six-view latent tensor and the auxiliary six-view latent tensors.

[0077] At box 515, the machine learning system uses tokens with 3D tokens (e.g., Figure 2 The diffusion model of condition token 220 is used to generate intermediate feature sets (e.g., Figure 2 The intermediate tensor 225). For example, as discussed above, the six-view latent tensor (generated during box 510) can be processed for a single or final iteration of the diffusion model, along with a special token indicating the embedding of 3D features into the output. The features generated by the penultimate layer of the model can then be used as intermediate features generated during box 515.

[0078] At box 520, the machine learning system is based on using a transformation module (e.g., Figure 1 The transformation module 120) processes intermediate features to generate a three-plane latent tensor (e.g., Figure 1 Three-plane data 125 and / or Figure 3(335) Three-plane data. For example, as discussed above, machine learning systems can slice, flip, rotate, and / or stitch together individual orthographic projections (e.g., using...). Figure 3 Transformation operation 315) to produce the concatenated tensor (e.g., Figure 3 The concatenated data is 320). Then a transformation model can be used (e.g., Figure 3 The concatenated tensor is processed by a transformation model (330), which can be a small convolutional neural network.

[0079] At box 525, the machine learning system generates a mesh based on a three-plane latent tensor (e.g., Figure 1 (Grid 135). For example, as discussed above, machine learning systems can use decoder models (e.g., Figure 1 The decoder model 130 is used to process the three-plane latent tensor.

[0080] At box 530, the machine learning system may optionally texturize the mesh, at least in part, based on the six-view image (generated based on the six-view latent tensor generated at box 510). For example, as discussed above, a UV texture baking method can be utilized, which leverages orthogonal views predicted during the six-view diffusion process. In some aspects, these views may contain fine texture details missing in the decoded mesh. Therefore, in some aspects, a post-processing procedure can be used to "bake" the intermediate six views onto the decoded mesh. In some aspects, this texturizing process comprises three steps.

[0081] First, UV mapping is performed. In some aspects, the machine learning system employs an automatic UV unwrapping algorithm to create a UV map that can be used to project a 2D texture onto a 3D mesh. Next, texture initialization is performed. In some aspects, the machine learning system then initializes the UV texture, such as by mapping each texel in the texture to its corresponding 3D coordinates, sampling the corresponding tri-plane features at that point, and converting the tri-plane features to RGB using a machine learning model (e.g., a multilayer perceptron (MLP) in the decoder used to create the mesh based on the tri-planes). Third, the machine learning system can perform six-view baking. During this process, the machine learning system can project pixels from (decoded) orthogonal views (e.g., from a six-view image generated based on a six-view latent representation) onto the UV texture. In some aspects, where multiple six-view pixels correspond to the same texel, the machine learning system can average the pixel values. In some aspects, this texture baking process can significantly enhance the visual quality of the mesh.

[0082] In these ways, using Method 500, machine learning systems can quickly and efficiently generate high-quality 3D meshes with high-quality visual textures based solely on simple text input. These meshes can be used for various purposes and objectives, as discussed above.

[0083] Example methods for generating 3D meshes

[0084] Figure 6 This is a flowchart depicting an example method 600 for generating a 3D mesh according to some aspects of this disclosure. In some aspects, method 600 is performed by a machine learning system (such as the one mentioned above). Figures 1 to 5 The machine learning system in question is used to execute this.

[0085] At box 605, a multi-view latent tensor is generated based on processing the text input using a diffusion machine learning model, where the multi-view latent tensor corresponds to multiple orthographic projections corresponding to the text input.

[0086] In some respects, the text input specifies the object, and multiple orthographic projections correspond to six orthographic views of the object.

[0087] At box 610, a transformational machine learning model is used to generate a tri-plane latent tensor based on the multi-view latent tensor; and

[0088] In some respects, generating triplane tensors involves using a diffusion machine learning model to process multi-view latent tensors and three-dimensional conditional tokens to generate intermediate tensors.

[0089] In some respects, generating multi-view potential tensors involves using a diffusion machine learning model to perform multiple iterations of processing data without a 3D condition token, and generating intermediate tensors involves using a diffusion machine learning model to perform a single iteration of processing data.

[0090] In some aspects, generating a triplane latent tensor includes: for each corresponding pair of parallel orthographic projections from a plurality of orthographic projections, the corresponding pair corresponding to a relative view of an object specified in the text input; realigning at least one orthographic projection in the corresponding pair of parallel orthographic projections to match the orientation; stitching the realigned pair of parallel orthographic projections together; and processing the stitched orthographic projections using a transformational machine learning model.

[0091] At box 615, a 3D mesh is generated based on processing the three-plane latent tensor using a decoder machine learning model.

[0092] In some aspects, method 600 also includes: generating a multi-view image based on a multi-view latent tensor; generating a two-dimensional texture based on the multi-view image; and texturing a three-dimensional mesh based on the two-dimensional texture.

[0093] In some respects, generating a 2D texture includes: generating a UV map based on a 3D mesh; initializing the 2D texture based on the UV map and based on mapping each texel in the 2D texture to a corresponding point on the 3D mesh; and projecting a multi-view image onto the initialized 2D texture.

[0094] In some respects, method 600 also includes: generating multiple textures using a diffusion machine learning model, each of the multiple textures corresponding to a corresponding texture modality; and further texturing the 3D mesh based on the multiple textures.

[0095] In some respects, method 600 also includes: generating an auxiliary multi-view latent tensor using a diffusion machine learning model; and generating a triplane latent tensor based on both the multi-view latent tensor and the auxiliary multi-view latent tensor.

[0096] In some respects, the auxiliary multiview latent tensor has a relatively low resolution compared to the multiview latent tensor.

[0097] In some respects, method 600 also includes: rendering an image depicting a three-dimensional mesh, and outputting the rendered image via a display.

[0098] Example processing system for machine learning

[0099] Figure 7 Various aspects configured to perform this disclosure are described (including, for example, references to...). Figures 1 to 6 Example processing system 700 (the described techniques and methods). In some aspects, processing system 700 may correspond to a machine learning system. For example, processing system 700 may correspond to a system for training machine learning models (e.g., diffusion models and / or transformation modules) and / or for image generation using trained machine learning models. Although depicted as a single system for conceptual clarity, in some aspects, as discussed above, the operations described below with respect to processing system 700 may be distributed across any number of devices or systems.

[0100] Processing system 700 includes a central processing unit (CPU) 702, which in some examples may be a multi-core CPU. Instructions that can be executed at CPU 702 may be loaded, for example, from program memory associated with CPU 702 or from a memory partition (e.g., a partition of memory 724).

[0101] The processing system 700 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, a multimedia component 710 (e.g., a multimedia processing unit), and a wireless connectivity component 712.

[0102] NPUs (such as the NPU 708) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0103] NPUs (such as the NPU 708) are configured to accelerate the performance of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0104] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0105] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0106] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0107] In some implementations, the NPU 708 is part of one or more of the CPU 702, GPU 704, and / or DSP 706.

[0108] In some examples, the wireless connectivity component 712 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 712 is also coupled to one or more antennas 714.

[0109] The processing system 700 may also include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISPs) 718 associated with any type of image sensor, and / or a navigation component 720, which may include a satellite-based positioning system component (e.g., GPS or GLONASS) and an inertial positioning system component.

[0110] The processing system 700 may also include one or more input and / or output devices 722, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0111] In some examples, one or more processors in the processing system 700 may be based on the ARM or RISC-V instruction set.

[0112] The processing system 700 also includes a memory 724, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 724 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 700.

[0113] Specifically, in this example, memory 724 includes a diffusion component 724A, a transformation component 724B, a decoder component 724C, and a texture component 724D. Although not depicted in the illustrated example, memory 724 may also include other components, such as a training component for training a machine learning model, a generation component for managing image generation using the trained machine learning model, etc. (Although not depicted for clarity of concept...) Figure 7 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) may be implemented together or separately.

[0114] As illustrated, memory 724 also includes a set of model parameters 724E (e.g., parameters of one or more machine learning models or components thereof). For example, model parameters 724E may include parameters for, for example... Figure 1 The memory 724 contains parameters for each of the components, such as the diffusion model 110, the conversion module 120, and / or the decoder model 130. Although not depicted in the illustrated example, the memory 724 may also include other data, such as training data (e.g., training examples including text prompts and target grids).

[0115] The processing system 700 also includes a diffusion circuit 726, a conversion circuit 727, a decoder circuit 728, and a texture circuit 729. The depicted circuits and other circuits not depicted (such as inference circuits) can be configured to perform various aspects of the techniques described herein.

[0116] As discussed above, diffusion component 724A and / or diffusion circuit 726 (which may correspond to Figure 1 , Figure 3 and / or Figure 4 The diffusion model 110, and / or Figure 2 The diffusion model 210 can be used to generate multi-view latent representations and images based on text prompts. For example, the diffusion component 724A and / or the diffusion circuit 726 may include or use machine learning models to iteratively denoise the multi-view latent tensor to generate multi-view images with multiple orthographic projections from different views.

[0117] Conversion component 724B and / or conversion circuit 727 (which may correspond to) Figure 1 and / or Figure 4 The transformation module 120 can be used to convert multi-view data into three-plane data, as discussed above. For example, transformation component 724B and / or transformation circuit 727 may include or use non-parametric transformations to align corresponding parallel views from the multi-view data, and may include or use one or more machine learning models to generate a three-plane latent representation based on the transformed multi-view data.

[0118] Decoder component 724C and / or decoder circuit 728 (which may correspond to...) Figure 1 The decoder component 130 can be used to generate a 3D mesh based on the input triplane data, as discussed above. For example, the decoder component 724C and / or the decoder circuit 728 may include or use a machine learning model to decode the triplane data in order to generate a mesh.

[0119] Texture component 724D and / or texture circuit 729 can be used to generate textures and apply them to 3D meshes, as discussed above. For example, texture component 724D and / or texture circuit 729 can generate UV maps and textures based on multi-view images.

[0120] Although for clarity Figure 7 While depicted as separate components and circuits, the diffusion circuit 726, conversion circuit 727, decoder circuit 728, and texture circuit 729 may be implemented jointly or separately in other processing devices of the processing system 700, such as within the CPU 702, GPU 704, DSP 706, NPU 708, etc.

[0121] Generally speaking, the processing system 700 and / or its components can be configured to perform the methods described herein.

[0122] It is worth noting that, in other aspects, such as when the processing system 700 is a server computer, various aspects of the processing system 700 can be omitted. For example, the multimedia component 710, wireless connectivity component 712, sensor processing unit 716, ISP 718, and / or navigation processor 720 can be omitted in other aspects. Furthermore, the various aspects of the processing system 700 can be distributed among multiple devices.

[0123] Example Terms

[0124] Specific implementation examples are described in the following numbered clauses: Clause 1: A method comprising: generating a multi-view latent tensor based on processing a text input using a diffusion machine learning model, wherein the multi-view latent tensor corresponds to a plurality of orthographic projections corresponding to the text input; generating a three-plane latent tensor based on the multi-view latent tensor using a transformation machine learning model; and generating a three-dimensional mesh based on processing the three-plane latent tensor using a decoder machine learning model.

[0125] Clause 2: The method according to Clause 1, wherein: the text input specifies an object, and the plurality of orthographic projections correspond to six orthographic views of the object.

[0126] Clause 3: The method according to any one of Clauses 1 to 2, wherein generating the three-plane tensor comprises: using the diffusion machine learning model to process the multi-view latent tensor and the three-dimensional condition token to generate an intermediate tensor.

[0127] Clause 4: The method according to Clause 3, wherein: generating the multi-view potential tensor includes: using the diffusion machine learning model to perform multiple iterations of processing data without the 3D condition token, and generating the intermediate tensor includes: using the diffusion machine learning model to perform a single iteration of processing data.

[0128] Clause 5: The method according to any one of Clauses 1 to 4, wherein generating the three-plane latent tensor comprises: for each corresponding pair of parallel orthographic projections from the plurality of orthographic projections, the corresponding pair corresponding to a relative view of an object specified in the text input: realigning at least one orthographic projection in the corresponding pair of parallel orthographic projections to match orientation; and stitching the realigned pair of parallel orthographic projections; and using the transformation machine learning model to process the stitched orthographic projections.

[0129] Clause 6: The method according to any one of Clauses 1 to 5 further comprises: generating a multi-view image based on the multi-view latent tensor; generating a two-dimensional texture based on the multi-view image; and texturing the three-dimensional mesh based on the two-dimensional texture.

[0130] Clause 7: The method according to Clause 6, wherein generating the two-dimensional texture comprises: generating a UV map based on the three-dimensional mesh; initializing the two-dimensional texture based on the UV map and based on corresponding points on the three-dimensional mesh by mapping each texel in the two-dimensional texture to a point on the three-dimensional mesh; and projecting the multi-view image onto the initialized two-dimensional texture.

[0131] Clause 8: The method according to any one of Clauses 6 to 7 further comprises: generating a plurality of textures using the diffusion machine learning model, each of the plurality of textures corresponding to a corresponding texture modality; and further texturing the three-dimensional mesh based on the plurality of textures.

[0132] Clause 9: The method according to any one of Clauses 1 to 8 further comprises: generating an auxiliary multi-view latent tensor using the diffusion machine learning model; and generating the triplane latent tensor based on both the multi-view latent tensor and the auxiliary multi-view latent tensor.

[0133] Clause 10: The method according to Clause 9, wherein the auxiliary multi-view latent tensor has a relatively low resolution compared to the multi-view latent tensor.

[0134] Clause 11: The method according to any one of Clauses 1 to 10 further includes: rendering an image depicting the three-dimensional mesh; and outputting the rendered image via a display.

[0135] Clause 12: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 11.

[0136] Clause 13: A processing system comprising components for performing the method according to any one of Clauses 1 to 11.

[0137] Clause 14: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 11.

[0138] Clause 15: A computer program product embodied on a computer-readable storage medium, the computer-readable storage medium including code for performing the method according to any one of Clauses 1 to 11.

[0139] Additional Notes

[0140] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0141] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or superior to other aspects.

[0142] As used in this article, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0143] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0144] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0145] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless otherwise specifically stated, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC §112(f) unless it is expressly referred to herein by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person skilled in the art or will later be known are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: One or more memories, the one or more memories including processor-executable instructions; and One or more processors, the one or more processors being configured to execute processor-executable instructions and cause the processing system to: A multi-view latent tensor is generated by processing text input using a diffusion machine learning model, wherein the multi-view latent tensor corresponds to multiple orthographic projections corresponding to the text input; A transformational machine learning model is used to generate a three-plane latent tensor based on the multi-view latent tensor; as well as A three-dimensional mesh is generated by processing the three-plane latent tensor using a decoder machine learning model.

2. The processing system according to claim 1, wherein: The text input specifies an object, and The plurality of orthographic projections correspond to the six orthographic views of the object.

3. The processing system according to claim 1, wherein, In order to generate the three-plane latent tensor, the one or more processors are configured to execute processor-executable instructions and cause the processing system to use the diffusion machine learning model to process the multi-view latent tensor and the three-dimensional condition token to generate intermediate tensors.

4. The processing system according to claim 3, wherein: To generate the multi-view potential tensor, the one or more processors are configured to execute processor-executable instructions and cause the processing system to use the diffusion machine learning model to perform multiple iterations of processing data without the 3D condition token. In order to generate the intermediate tensor, the one or more processors are configured to execute processor-executable instructions and cause the processing system to perform a single iteration of processing data using the diffusion machine learning model.

5. The processing system of claim 1, wherein, in order to generate the three-plane latent tensor, the one or more processors are configured to execute processor-executable instructions and cause the processing system to: For each corresponding pair of parallel orthographic projections from the plurality of orthographic projections, the corresponding pair corresponds to a relative view of the object specified in the text input: Realign at least one orthogonal projection in the corresponding parallel orthogonal projection pair to match the orientation; as well as The corresponding parallel orthographic projection pairs are spliced ​​and realigned. as well as The transformation machine learning model is used to process the stitched orthographic projection.

6. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions and cause the processing system to: Generate multi-view images based on the multi-view latent tensor; Generate a two-dimensional texture based on the multi-view image; and The three-dimensional mesh is textured based on the two-dimensional texture.

7. The processing system according to claim 6, wherein, In order to generate the two-dimensional texture, the one or more processors are configured to execute processor-executable instructions and enable the processing system to: Generate UV mapping based on the 3D mesh; Based on the UV mapping, the two-dimensional texture is initialized by mapping each texel in the two-dimensional texture to a corresponding point on the three-dimensional mesh; as well as The multi-view image is projected onto the initialized two-dimensional texture.

8. The processing system of claim 6, wherein the one or more processors are configured to further execute processor-executable instructions and cause the processing system to: Multiple textures are generated using the diffusion machine learning model, each texture corresponding to a corresponding texture modality; and The three-dimensional mesh is further textured based on the multiple textures.

9. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions and cause the processing system to: The diffusion machine learning model is used to generate auxiliary multi-view latent tensors; and The three-plane latent tensor is generated based on both the multi-view latent tensor and the auxiliary multi-view latent tensor.

10. The processing system of claim 9, wherein the auxiliary multi-view latent tensor has a relatively low resolution compared to the multi-view latent tensor.

11. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions and cause the processing system to: Render an image depicting the three-dimensional mesh; and The rendered image is output to the monitor.

12. A processor-implemented method for machine learning, the processor-implemented method comprising: A multi-view latent tensor is generated by processing text input using a diffusion machine learning model, wherein the multi-view latent tensor corresponds to multiple orthographic projections corresponding to the text input; A transformational machine learning model is used to generate a three-plane latent tensor based on the multi-view latent tensor; as well as A three-dimensional mesh is generated by processing the three-plane latent tensor using a decoder machine learning model.

13. The processor-implemented method according to claim 12, wherein: The text input specifies an object, and The plurality of orthographic projections correspond to the six orthographic views of the object.

14. The processor-implemented method of claim 12, wherein generating the three-plane latent tensor comprises: The diffusion machine learning model is used to process the multi-view latent tensor and the 3D conditional token to generate intermediate tensors.

15. The processor-implemented method according to claim 14, wherein: Generating the multi-view latent tensor includes: using the diffusion machine learning model to perform multiple iterations of processing the data without the 3D condition token, and Generating the intermediate tensor involves performing a single iteration of the data processing using the diffusion machine learning model.

16. The processor-implemented method of claim 12, wherein generating the three-plane latent tensor comprises: For each corresponding pair of parallel orthographic projections from the plurality of orthographic projections, the corresponding pair corresponds to a relative view of the object specified in the text input: Realign at least one orthogonal projection in the corresponding parallel orthogonal projection pair to match the orientation; as well as The corresponding parallel orthographic projection pairs are spliced ​​and realigned. as well as The transformation machine learning model is used to process the stitched orthographic projection.

17. The processor-implemented method according to claim 12, further comprising: Generate multi-view images based on the multi-view latent tensor; Generate a two-dimensional texture based on the multi-view image; as well as The three-dimensional mesh is textured based on the two-dimensional texture.

18. The processor-implemented method of claim 17, wherein generating the two-dimensional texture comprises: Generate UV mapping based on the 3D mesh; as well as Based on the UV mapping, the two-dimensional texture is initialized by mapping each texel in the two-dimensional texture to a corresponding point on the three-dimensional mesh; as well as The multi-view image is projected onto the initialized two-dimensional texture.

19. The processor-implemented method according to claim 12, further comprising: The diffusion machine learning model is used to generate auxiliary multi-view latent tensors; as well as The three-plane latent tensor is generated based on both the multi-view latent tensor and the auxiliary multi-view latent tensor.

20. One or more non-transitory computer-readable media, the non-transitory computer-readable media comprising processor-executable instructions, which, when executed by one or more processors of a processing system, cause the processing system to: A multi-view latent tensor is generated by processing text input using a diffusion machine learning model, wherein the multi-view latent tensor corresponds to multiple orthographic projections corresponding to the text input; A transformational machine learning model is used to generate a three-plane latent tensor based on the multi-view latent tensor; and A three-dimensional mesh is generated by processing the three-plane latent tensor using a decoder machine learning model.

Images