Four-dimensional object and scene model synthesis using generative models

By using processor systems and potential diffusion models, high-quality 4D video content is generated from text input, which solves the problem of 4D content that is difficult to generate authenticity and consistency in the prior art, and achieves the effect of high diversity and avoiding saturation and blur.

CN120107454APending Publication Date: 2025-06-06NVIDIA CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411769546.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-01-25
Filing Date
2024-12-04
Publication Date
2025-06-06

AI Technical Summary

Technical Problem

The prior art is difficult to generate high-quality four-dimensional (4D) content from text input, such as objects or scenes represented by 3D data over time, especially under resource constraints, and it is difficult to ensure authenticity and consistency across space, orientation and time.

Method used

Using a processor system, the content model is initialized by receiving text input, and the content model is updated using the potential diffusion model until the convergence condition is met. The system includes circuitry and processing units that are able to render frame sequences, determine metrics, and modify content models based on metrics to generate realistic 4D video content.

Benefits of technology

It realizes the generation of 4D video content with high diversity and avoids saturation and fuzzy output from text input, while ensuring global geometric consistency, solving the problem of generating high-quality 4D content under resource limitations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120107454A_ABST
    Figure CN120107454A_ABST
Patent Text Reader

Abstract

The invention discloses four-dimensional object and scene model synthesis using generative models. In various examples, systems and methods related to generating a four-dimensional (4D) content model (e.g., a 4D content model for rendering a realistic sequence of 3D data frames) are disclosed. The system may initialize a 3D component (e.g., a 3D Gaussian splash representation) of the 4D content model based at least on the cues of the 4D content. The system may configure motion and / or dynamics of a sequence of frames by evaluating frames rendered from a 4D content model using one or more potential diffusion models (LDMs), including video LDMs. The system may perform operations such as autoregressive generation of frames to create a long sequence of content, motion amplification to facilitate realistic dynamic motion generation, and regularization to facilitate generation of complex dynamics.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 606,193, filed on December 5, 2023, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Machine learning models such as neural networks can be used to represent environments or scenes. For example, images and / or video data of certain portions of an environment can be used to interpolate or otherwise determine a more complete representation of the environment. However, without a large amount of input direction and / or computational resources, it is difficult to generate more complex scene representations in a way that provides realism and consistency across space, orientation, and / or time. Summary of the invention

[0004] Embodiments of the present disclosure relate to synthetic four-dimensional object and scene models based on inputs including text inputs. Compared to conventional systems such as those described above, systems and methods according to the present disclosure may allow realistic four-dimensional (4D) content to be generated from text inputs, such as representing at least one of an object or scene over time using three-dimensional (3D) data (e.g., having three spatial dimensions). For example, systems and methods according to the present disclosure may generate 4D video content that may have diversity (e.g., more information and / or variation relative to the input), avoid saturation, blurry output, and / or have global geometric consistency.

[0005] At least one aspect relates to a processor. The processor may include one or more circuits that can be used to receive input indicating one or more features of content, the content including at least one of an object or a scene. The one or more circuits can initialize a content model based on the input to represent the input in three spatial dimensions and a time dimension. The one or more circuits can update the content model by rendering one or more frame sequences from the content model, determining metrics for the one or more sequences using a potential diffusion model, and modifying the content model based on the metrics until a convergence condition is met. The one or more circuits can cause at least one of the following: (i) performing a simulation using the updated content model or (ii) presenting the updated content model using a display.

[0006] In some embodiments, the latent diffusion model includes one or more layers configured for the time dimension. The latent diffusion model may include or be coupled with an optimizer configured to determine a metric based at least on a gradient associated with a given frame in one or more frame sequences. In some embodiments, the content model is conditioned on camera movement associated with three spatial dimensions and a time value associated with the time dimension, and one or more circuits may render a given frame sequence in the one or more frame sequences according to a given camera pose for the given frame sequence and provide the given frame sequence as input to the latent diffusion model.

[0007] In some embodiments, one or more circuits may update the content model based on a predetermined input that identifies a camera pose for a given frame sequence and a time point for one or more frames in the given frame sequence. One or more circuits may determine a metric based on a given frame sequence rendered according to the predetermined input.

[0008] In some embodiments, the content model includes a deformation field and may include at least one of a Gaussiansplatting representation, a neural radiation field (NeRF), a mesh representation, or a point cloud. In some embodiments, one or more circuits are configured to update the content model according to a physical model configured to measure (e.g., parameterize) the physics-based realism of the motion represented in one or more frame sequences. In some embodiments, one or more circuits may render a first frame for a first time point and a second frame for a second time point after the first time point from the updated content model, modify the updated content model according to the second frame, and render a third frame for a third time point after the second time point from the modified content model according to the second frame.

[0009] In some embodiments, one or more circuits are used to update the content model based on a physics model configured to measure physics-based realism of motion represented in one or more frame sequences. In some embodiments, one or more circuits are used to identify from the updated content model at least one of a joint of an object, a movement property of the object, or a deformation property of the object represented by the updated content model.

[0010] At least one aspect relates to a system. The system may include one or more processing units for performing operations including: receiving an input indicating one or more features of content, the content including at least one of an object or a scene. The one or more processing units may initialize a content model based on the input to represent the input in three spatial dimensions and a temporal dimension. The one or more processing units may update the content model by rendering one or more frame sequences from the content model, determining a metric for the one or more sequences using a potential diffusion model, and modifying the content model based on the metric until a convergence condition is met. The one or more processing units may cause at least one of the following: (i) performing a simulation using the updated content model or (ii) presenting the updated content model using a display.

[0011] In some embodiments, the latent diffusion model includes one or more layers configured for the time dimension. The latent diffusion model may include or be coupled with an optimizer configured to determine a metric based at least on a gradient associated with a given frame in one or more frame sequences. In some embodiments, the content model is conditioned on camera movements associated with three spatial dimensions and a time point associated with the time dimension, and one or more circuits may render a given frame sequence in the one or more frame sequences according to a given camera pose for the given frame sequence, and provide the given frame sequence as input to the latent diffusion model.

[0012] In some embodiments, one or more processing units may update the content model based on a predetermined input that identifies a camera pose for a given frame sequence and a time point for one or more frames in the given frame sequence. One or more processing units may determine a metric based on a given frame sequence rendered according to the predetermined input.

[0013] In some embodiments, the content model includes a deformation field and may include at least one of a Gaussian splat (e.g., Gaussian Splat) representation, a neural radiance field (NeRF), a mesh representation, or a point cloud. In some embodiments, one or more processing units may update the content model based on a physical model configured to measure the physics-based realism of the motion represented in one or more frame sequences. In some embodiments, one or more processing units may render a first frame for a first time point and a second frame for a second time point after the first time point based on the updated content model, modify the updated content model based on the second frame, and render a third frame for a third time point after the second time point from the modified content model based on the second frame.

[0014] In some embodiments, the one or more processing units update the content model based on a physics model configured to measure physics-based realism of motion represented in the one or more frame sequences. In some embodiments, the one or more processing units identify from the updated content model at least one of a joint of an object, a motion property of the object, or a deformation property of the object represented by the updated content model.

[0015] At least one aspect relates to a method. The method may include receiving, by one or more processors, an input indicating at least one of an object or a scene. The method may include initializing, by the one or more processors, multiple spatial dimensions of a content model of at least one of the object or the scene based at least on the input. The method may include updating, by the one or more processors, the content model to have a temporal dimension in response to evaluating multiple frames rendered from the content model at multiple points in time using a latent diffusion model having one or more temporal layers to generate an updated content model. The method may include outputting, by the one or more processors, one or more frames from the updated content model.

[0016] In some embodiments, the content model includes a 3D Gaussian splash representation corresponding to multiple spatial dimensions, the 3D Gaussian splash representation coupled to a multi-layer perceptron (MLP) corresponding to the time dimension. In some embodiments, the latent diffusion model includes one or more layers configured for the time dimension. The latent diffusion model may include or be coupled with an optimizer configured to determine a metric based at least on a gradient associated with a given frame in one or more frame sequences. In some embodiments, the latent diffusion model is conditioned on camera movement associated with three spatial dimensions, and one or more circuits may render a given frame sequence in one or more frame sequences according to a given camera pose for the given frame sequence.

[0017] In some embodiments, the method includes rendering a given frame sequence of the one or more frame sequences according to a predetermined input identifying a camera pose for the given frame sequence. The method may include determining a metric according to the given frame sequence rendered according to the predetermined input.

[0018] In some embodiments, the method includes updating the content model based on the physics model. The physics model can measure the physics-based realism of the motion represented in the one or more frame sequences. In some embodiments, the method includes rendering a first frame for a first time point and a second frame for a second time point after the first time point from the updated content model, modifying the updated content model based on the second frame, and rendering a third frame for a third time point after the second time point from the modified content model based on the second frame.

[0019] The processors, systems and / or methods described herein may be implemented by or included in at least one of the following: a system for generating synthetic data; a system for performing simulation operations; a system for performing conversational AI operations; a system for performing collaborative content creation of 3D assets; a system including one or more language models (e.g., a large language model (LLM)); a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, and / or mixed reality (MR) content; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system associated with an autonomous or semi-autonomous machine (e.g., an in-vehicle infotainment system); a system including one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present system and method for machine learning models for synthetic image generation that complement neural field representations are described in detail below with reference to the accompanying drawings, wherein:

[0021] Figure 1 is a block diagram of an example system for generating 4D content according to some embodiments of the present disclosure;

[0022] Figure 2 is a block diagram of an example system for generating dynamic 4D content according to some embodiments of the present disclosure;

[0023] Figure 3 is a flowchart of an example of a method for generating 4D content according to some embodiments of the present disclosure;

[0024] Figure 4 is a block diagram of an example content streaming system suitable for implementing some embodiments of the present disclosure;

[0025] Figure 5 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0026] Figure 6 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Disclosed are systems and methods related to synthetic four-dimensional (4D) object and / or scene models that can be generated / established / configured based on text input, for example, synthesizing 4D dynamic image data structures (e.g., Gaussian splash representations or neural radiation fields) from high-resolution text to video diffusion models. For example, systems and methods according to the present disclosure can allow language-based input (e.g., text, speech, and / or audio indicating one or more features of an object or scene) to be processed by a model to generate a three-dimensional (3D) video, for example, to provide 4D content representing one or more features. In some embodiments, systems and methods according to the present disclosure can provide 4D content in the form of a 3D representation (e.g., 3D Gaussian splash) coupled with a deformation field (e.g., a deformation field based on a multilayer perceptron (MLP)).

[0028] Various machine learning models (e.g., diffusion models) can be used to generate image data based on text input. For example, a diffusion model configured (e.g., pre-trained) using training data that includes text mapped to images can be used to generate two-dimensional (2D) images from text input. While some 4D systems have been developed, these systems tend to generate data with little diversity and data that lacks image quality (e.g., oversaturated, blurry outputs); this may be due to the use of large classifier-free guidance weights in the diffusion models underlying these systems. Such systems may also have geometrically inconsistent results, such as due to the lack of direct 3D knowledge or awareness (e.g., by independently rendering different views of a 3D object). Some systems may also operate in pixel space, limiting the resolution available for operation and / or requiring large computational resources.

[0029] Systems and methods according to the present disclosure may implement a 4D content generation system (e.g., three image dimensions plus time) to generate video content from text input. The system may include a content data structure (e.g., a content model) and a latent diffusion model (LDM). The content model may be constructed to represent 4D content, such as a 4D representation of an object and / or scene. For example, the content model may have dimensions [x, y, z, t] corresponding to three spatial dimensions and one temporal dimension. The content data structure may be, for example, an object model or a scene model, such as a neural radiance field (NeRF), an implicit function (e.g., a multilayer perceptron (MLP) neural network), a grid-based representation, a point-based representation (e.g., a point cloud), a 3D Gaussian splash representation, a hexahedron, a grid representation, a trihedron extended with a time component or a deformation field, or various combinations thereof.

[0030] The LDM can perform functions including at least one of text-to-image, text-to-video (e.g., two spatial dimensions plus time), or text-to-3D data generation. For example, the LDM can include or be coupled with one or more machine learning models, including large-scale models, such as a 2D text-to-image model; large-scale text-to-video (video LDM); a 3D diffusion model (which can be used to ensure that every frame of a 4D sequence is 3D consistent), or various combinations thereof. The LDM can be configured by fine-tuning and / or transfer learning one or more temporal layers added to the text-to-image LDM. By using the LDM, a high-resolution rendering (e.g., from a content model) can be provided to the LDM for processing, which can avoid image quality issues and / or avoid the need for a separate super-resolution model.

[0031] The LDM may be used to evaluate (e.g., score) one or more rendered frame sequences from a content data structure in order to update and / or optimize a content model, thereby optimizing a 4D sequence that may be generated from the content model. For example, the LDM may be used to generate gradients based on the evaluation, which may be back-propagated to render one or more rendered frame sequences. The evaluation and back-propagation may be performed, for example, using an optimizer / scoring technique such as fractional distillation sampling (SDS) or variational fractional distillation (VSD); using VSD may be used to increase the diversity of the output and may reduce the likelihood of generating blurry or oversaturated output. Video data (e.g., one or more frames of a video) may be used to initialize the content model.

[0032] LDMs (e.g., text-to-video LDMs) can be fine-tuned to adjust for camera pose and / or motion. This can mitigate geometric inconsistencies due to the way 3D features are added to content. Optimization (e.g., using VSD) can use adjustments related to camera pose (e.g., informing the model of a pose such as "side view"), and scoring function determinations with and without the adjustments can be compared to account for view adjustments. This can allow for the reduction of non-classifier guidance weights, thereby overcoming issues with blurry images, low diversity, and / or oversaturation.

[0033] The system can be used to generate long 4D sequences, e.g. with many frames (long relative to the time dimension). For example, the ending frame of a first sequence can be used for input conditioning of a second sequence. These operations can be performed in an autoregressive manner to generate 4D sequences of arbitrary length.

[0034] Customization or personalization can be performed by fine-tuning the LDM using one or more images, such as images of objects and / or scenes to be represented using the content model. Physics models can be used in the system to facilitate the generation of realistic motion. 3D assets such as joints, motion profiles, or other information about how a 3D object in a 4D representation moves can be extracted from the content model.

[0035] The systems and methods described herein may be used for a variety of purposes, such as, but not limited to, for synthetic data generation, machine control, machine motion, machine driving, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0036] The disclosed embodiments may be included in a variety of different systems, such as systems for performing synthetic data generation operations, automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems implementing one or more language models (e.g., LLMs), systems implemented at least in part in a data center, systems for performing conversational AI operations, systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0037] refer to Figure 1 , Figure 11 is an example computing environment including system 100 according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are presented as examples only. In addition to the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may also be used, and some elements may be omitted completely. In addition, many elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and position. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in a memory. System 100 may include any function, model (e.g., machine learning model), operation, routine, logic, or instruction to perform functions such as configuring a diffusion model 112, a scene representation 124, and / or a motion generator 128, as described herein, thereby configuring a machine learning model to operate as a diffusion model.

[0038] The system 100 may include or be coupled to one or more data sources 104. The data source 104 may include, for example, any of a variety of databases, data sets, or data repositories. The data source 104 may include data for configuring any of a variety of machine learning models (e.g., model 112; scene representation 124). The one or more data sources 104 may be maintained by one or more entities, which may be entities that maintain the system 100 or may be separate from the entities that maintain the system 100. In some embodiments, the system 100 uses data from different data sets, such as by performing at least a first configuration (e.g., updating or training) of the model 112 using data from a first data source 104, and performing at least a second configuration of the model 112 using training data elements from a second data source 104. For example, the first data source 104 may include publicly available data, while the second data source 104 may include domain-specific data (access to which may be restricted compared to the data of the first data source 104). The data 108 may include data from any suitable image data set, including labeled and / or unlabeled image data. In some examples, data source 104 includes data from a large-scale image dataset (eg, ImageNet) available from a variety of sources and services.

[0039] The data source 104 may include, but is not limited to, data 108, such as one or more of text, speech, audio, image, and / or video data. The system 100 may perform various pre-processing operations on the data, such as filtering, normalizing, compressing, decompressing, enlarging or reducing, cropping, and / or converting to grayscale (e.g., from image and / or video data). The images (including videos) of the data 108 may correspond to one or more views of a scene captured by an image capture device (e.g., a camera), or computationally generated images, such as simulated or virtual images or videos (including by modifying an image from an image capture device). Each image may include a plurality of pixels, such as pixels arranged in rows and columns. An image may include image data assigned to one or more pixels of the image, such as color, brightness, contrast, intensity, depth (e.g., for a three-dimensional (3D) image), or various combinations thereof. Data 108 may include video and / or video data structured into multiple frames (e.g., image frames, video frames), such as in the form of a sequence of frames, where each frame is assigned a time index (e.g., time step, time point) and has image data assigned to one or more pixels of the image.

[0040] In some embodiments, the image data and / or video data of the data 108 includes camera pose information. The camera pose information may indicate a viewpoint from which the data 108 is represented. For example, the camera pose information may indicate at least one of a position or an orientation of a camera (e.g., a real or virtual camera) according to which the data 108 is captured or represented.

[0041] The system 100 may train, update, or configure one or more models 112 (e.g., machine learning models). The machine learning models 112 may include machine learning models or other models that may generate target outputs based on various types of inputs. The machine learning models 112 may include one or more neural networks. The neural network may include an input layer, an output layer, and / or one or more intermediate layers (e.g., hidden layers), each of which may have respective nodes. The system 100 may train / update the neural network by modifying or updating one or more parameters (e.g., weights and / or biases) of respective nodes of the neural network in response to an evaluation of a candidate output of the neural network.

[0042] Model 112 can be or include various neural network models, including models that can effectively operate on or generate data, including but not limited to image data, video data, text data, voice data, audio data, or various combinations thereof. Machine learning model 112 may include one or more transformers, recurrent neural networks (RNNs), long short-term memory (LSTM) models, other network types, or various combinations thereof. Machine learning model 112 may include generative models, such as generative adversarial networks (GANs), Markov decision processes, variational autoencoders (VAEs), Bayesian networks, autoregressive models, autoregressive encoder models (e.g., including encoders that generate latent representations (e.g., in an embedding space) of model inputs (e.g., representations of different dimensions than the inputs) and / or models that generate decoders that represent outputs from latent representations), or various combinations thereof.

[0043] Models 112 may include at least one diffusion model 112. The diffusion model may include a network, such as a denoising network 116. For example, in brief overview, the diffusion model may include a denoising network 116 that is configured (e.g., pre-trained, trained, updated, fine-tuned, and / or has transfer learning applied) using training data of data 108 including data elements to which noise is applied, and the denoising network 116 is configured to modify the noise-enhanced data elements to restore the (noise-free) data elements.

[0044] The model 112 may include (e.g., the denoising network 116 may be implemented as) a latent diffusion model (LDM). The LDM may include or be coupled to an encoder 114. The encoder 114 may include a neural network for encoding (e.g., compressing) data into a compressed latent space (e.g., a latent tensor, a latent representation, a latent coding) of lower dimensionality, thereby allowing operations to be performed more efficiently in the latent space. For example, this may allow the model 112 to receive high-resolution image and / or video data to configure the model 112 while maintaining a target performance level. For example, the encoder 114 may allow the model 112 to improve computational and memory efficiency relative to a pixel space diffusion model (DM) by first training the encoder 114 to transform an input image (e.g., an input image of the data 108) into a spatially low-dimensional latent space of reduced complexity, from which the original data can be reconstructed with high fidelity. The method may be implemented with a regularized autoencoder that reconstructs the input image and a decoder (e.g., a decoder neural network). Compared to a corresponding pixel space DM with similar performance, the latent space may be smaller in terms of parameter count and / or memory consumption of the model 112 operating in the latent space.

[0045] The model 112 may include a denoising network 116 as part of the LDM, and the denoising network 116 may be coupled to the encoder 114 to perform operations on the data that the encoder 114 maps to the latent space. The system 100 may configure the denoising network 116 by having the denoising network 116 reproduce the example data to which noise has been applied. In some embodiments, the system 100 configures the denoising network 116 by conditioning the denoising network 116 based on a conditional input (e.g., a text input), thereby allowing the denoising network 116 to generate an output in response to receiving the input (e.g., at runtime / inference).

[0046] For example, the system 100 may perform diffusion on one or more images x of the data 108 0 (and / or image frames of a video). The system 100 may perform diffusion by applying noise to (e.g., diffusing) the data 108 to determine training data points (e.g., diffused or noisy data, such as a noisy image x T , e.g., a diffused frame as described with reference to Figure 2 . For example, the system 100 may add noise to the data 108 (e.g., add numerical values representing noise to the data 108 in the same data format as the data 108) to determine training data points. The system 100 may use one or more noise distributions to determine the noise to add to the data 108, and these noise distributions may indicate the noise level according to the time t, where 0 < t < T, such that applying the noise corresponding to time T may result in a training data point x T representing Gaussian noise. For example, the noise may be a sample from a distribution such as a Gaussian distribution. The system 100 may apply the noise according to or relative to the duration t. The duration t may be a value within the time interval, e.g., a value between zero and the maximum value T of the time interval. The duration t may be a multiple of a number of discrete time steps between zero and T. The maximum value T may correspond to a certain amount of time such that the result of applying the noise for duration T may be indistinguishable or nearly indistinguishable from Gaussian noise. For example, the system 100 may apply diffusion to the image x 0 within the duration T to determine a training data point (e.g., a noisy image) x T .

[0047] Further referring to Figure 1 , for example but not limited to, the denoising network 116 may be implemented using a U-Net (e.g., a convolutional neural network including a downscaling path and an upscaling path). The denoising network 116 may receive the training data point x T and determine an estimated output in response to receiving the training data point x T . The estimated output may have the same format as the training data point x T , e.g., be an image having the same number of channels as the training data point x TAn image of the same number of pixel rows and pixel columns (and / or the same format as data 108 compressed by encoder 114, e.g., where denoising network 116 generates an estimated output and provides the estimated output to a decoder network to decode into data 108).

[0048] In some embodiments, the system 100 can enable the model 112 (e.g., an LDM implemented by a denoising network 116) to learn to model a data distribution x by iterative denoising using the denoising network 116, and can be trained (e.g., updated) using denoising score matching. The noise schedule can be parameterized by a diffusion time, during which the log signal-to-noise ratio is monotonically decreasing. The denoiser model can receive a diffusion input parameterized by a learnable parameter, and can optimize a denoising score matching target based on conditional information (e.g., a text prompt), a target vector (e.g., random noise), a forward diffusion process, a reverse generative process, etc. The input image x can be perturbed to Gaussian random noise over a maximum diffusion time (e.g., time T). The iterative generative denoising process using the learned denoiser (e.g., the denoising neural network 116) can be initialized from Gaussian noise to synthesize new data.

[0049] Further references Figure 1 , the system 100 can configure the model 112 as a video diffusion model or include a video diffusion model, such as a video LDM. For example, the system 100 can configure the model 112 (e.g., the denoising network 116) to include or be coupled with at least one temporal layer 118. This can allow the denoising network 116 (e.g., together with the temporal layer 118) to generate an output having image data (e.g., in two or three spatial dimensions) over time (e.g., based on the operation of the at least one temporal layer 118). The temporal layer 118 can be, for example, one or more neural network layers, such as an attention neural network layer.

[0050] The system 100 can use the video data of the data 108 (e.g., one or more sequences of video frames retrieved from the data 108) to configure the temporal layer 118. For example, the system 100 can update / train the temporal layer 118 (e.g., independently of the denoising network 116 or together with the denoising network 116) to align the multiple images generated by the denoising network 116 into frames of the video, for example, by aligning the multiple images generated by the denoising network 116 into consecutive frames of the video, referred to as the first video. The first video can be an initial low temporal resolution (or low frame rate, low FPS) video and a low spatial resolution video (referred to as the first video) upsampled in the manner described herein.

[0051] The system 100 can train or update at least one temporal layer 118 by modifying or updating one or more parameters (e.g., weights and / or biases) of various nodes of the neural network in response to evaluating the candidate outputs of the at least one temporal layer 118. The output of the at least one temporal layer 118 can be used to evaluate whether the at least one temporal layer 118 has been sufficiently trained / updated to meet a target performance indicator, such as an indicator indicating the accuracy of the at least one temporal layer 118 in generating outputs. Such evaluation can be performed based on various types of losses. For example, the system 100 can use a function such as a loss function to evaluate a condition for determining whether the at least one temporal layer 118 is (sufficiently) configured to meet the target performance indicator. The condition can be a convergence condition, such as a condition that is satisfied in response to factors such as the output of the function satisfying the target performance indicator or threshold, the number of training iterations, the training convergence of the at least one temporal attention neural network layer 118, or various combinations thereof.

[0052] As described above, the encoder 114 can map an input image (e.g., image data 106) from an image or pixel space to a lower dimensional compressed latent space (e.g., a latent tensor, a latent representation, a latent coding, etc.), where the denoising network 116 can be updated or trained more efficiently in terms of power consumption, memory consumption, and / or time. The decoder (not shown) can map the latent coding back to the image space. Compared to operating directly in the image or pixel space, the model 112 fine-tuned from the image diffusion model (e.g., a video LDM including the denoising network 116 and the temporal layer 118) can simultaneously model / generate / exploit an increased number of video frames given a fixed memory budget, thereby facilitating long-term video generation.

[0053] In some embodiments, temporal layer 118 can be updated or trained using video data 108 in a manner described herein. In some examples, model 112 is fine-tuned or updated using video data 108 to allow model 112 to avoid introducing temporal incoherence when decoding a sequence of frames generated from a latent space (e.g., multiple latent tensors, latent representations, latent encodings, etc. corresponding to multiple input images). In some examples, image data 106 and video data 108 can be in the same domain. In some examples, image data 106 and video data 108 can be in different domains, or image data 106 and video data 108 can have a domain gap.

[0054] For example, the model 112 (e.g., the video LDM) may include one or more layers configured to process image data and / or spatial data, and one or more temporal layers 118 configured for the temporal dimension, such as by fine-tuning a neural network including a denoising network 116 (already updated / trained on image data) and one or more temporal layers 118 included with the denoising network 116. The system 100 may include an optimizer (e.g., included in or coupled to the updater 140) to configure the model 112 to update one or more parameters (e.g., weights, biases) of the model 112 based at least on a gradient generated for the model 112.

[0055] System 100 may include at least one scene representation 124. Scene representation 124 may include any data structure, image representation, function, model (e.g., a neural network or other machine learning model), operation, routine, logic, or instruction to represent a scene or a portion of a scene, such as one or more objects in the scene (e.g., scene representation 124 may include object representations, such as 3D object models, and may or may not include additional objects or environmental features relative to the objects represented by the 3D object models). For example, scene representation 124 may include two-dimensional (2D) and / or three-dimensional (3D) image data. The image data may represent a 3D environment, such as a real-world, physical, simulated, or virtual environment. Scene representation 124 may be a data structure that can be queried based on a direction (e.g., a vector representing a direction) and output an image based on the query, thereby outputting an image including image data representing a view of the scene perceived along the direction. In some embodiments, scene representation 124 includes at least one machine learning model (e.g., a neural network, such as the neural radiance field described herein) that can be used to generate an image data structure as a representation of a scene, object, and / or environment. For example, the image data structure may be a data structure that may be queried to retrieve one or more 2D or 3D portions of a representation, thereby retrieving pixel outputs of the representation. The scene representation 124 may be generated and / or updated using at least one of real or synthetic image data, such as image data captured in a physical / real world environment using an image capture device, or synthetic image data generated to represent a virtual or simulated environment.

[0056] For example, the scene representation 124 may include at least one 3D Gaussian volume, such as one or more 3D Gaussian splashes, to represent 3D image data (e.g., point or particle data representing at least three spatial dimensions of an object and / or scene, such as including depth information in addition to 2D image data). For example, the scene representation 124 may include a plurality of N 3D Gaussian volumes, each of the N 3D Gaussian volumes corresponding to a time point, each 3D Gaussian volume having parameters θ, such as a position μ i , covariance Σi (and / or standard deviation σ i ), opacity η i and / or color i The system 100 can render the view (e.g., rendering 132) onto the image plane of the 2D camera based on the given 3D Gaussian volume to produce a 2D Gaussian volume with a projected mean and a projected covariance. The system 100 can determine the color C(p) of a given image pixel p of the rendered view based on the point-based volume rendering, for example:

[0057]

[0058] in is the projected mean, is the projection covariance, and j iterates the Gaussian volume from pixel p to Gaussian volume i along the ray through the scene. In some embodiments, to speed up rendering, system 100 divides the image plane into tiles and performs rendering on at least a subset of the tiles in parallel. In some embodiments, the covariance is isotropic, which can facilitate more expressive motion of the 3D representation (caused by motion generator 128).

[0059] In some embodiments, the scene representation 124 includes at least one neural radiance field (NeRF). The NeRF can be a neural network that is configured to determine (e.g., infer) any of a plurality of views of a scene, such as by being trained, configured, and / or updated using one or more images (2D or 3D) and / or video data of the scene, such as by configuring / updating the neural network so as to be able to output a view consistent with a provided image and / or video (e.g., image data 106, video data 108, or other image and / or video data). For example, the NeRF can render a view based on at least one of an origin or an orientation of the view. The NeRF can render at least one view having an orientation that is different from the respective orientations of the image and / or video data used to configure the NeRF. In some embodiments, the NeRF includes a neural network that is configured to generate values ​​of an image data structure. In some embodiments, the NeRF includes at least one of a neural network or an image data structure configured to generate an image data structure. The NeRF can be defined as a function f θ :(p, d)→(c, σ), which converts the 3D position of the scene and viewing direction Mapping to volume density σ∈[0,∞) and color (e.g., RGB color) c∈[0,1] 3. System 100 can implement NeRF as at least one neural network that includes one or more multi-layer perceptrons (MLPs), such as a single global MLP, or multiple local MLPs, each local MLP corresponding to a local portion of the scene (e.g., based on local features arranged in a grid). System 100 can implement NeRF as a hash table, such as a multi-resolution hash position encoding. System 100 can configure parameters of NeRF (e.g., parameter θ) using a loss function. In some embodiments, scene representation 124 includes HexPlanes.

[0060] Further references Figure 1 , the system 100 can retrieve one or more renderings 132 (e.g., renderers) from the scene representation 124 (or an image data structure corresponding to the scene representation 124). The rendering 132 can be an image frame, such as a 2D image frame, of a given view of the 3D object and / or scene represented by the scene representation 124. The system 100 can receive a query to retrieve one or more renderings 132 from the scene representation 124. For example, the system 100 can receive a query indicating at least one of a camera pose, an origin, an orientation, or a point in time of the rendering 132, and can retrieve the rendering 132 from the scene representation 124 according to the query. The system 100 can retrieve the rendering 132 by causing the scene representation 124 to output the rendering 132 according to at least one of the camera pose, the origin, the orientation, or the point in time. In some embodiments, as described above, for example, with respect to a function f of a 3D Gaussian volume and / or NeRF, the scene representation 124 is or includes a rendering function g with a parameter θ, which may be a differentiable function, and the system 100 may cause the rendering function to render the rendering 132.

[0061] The scene representation 124 may provide a rendering 132 of multiple views with different viewpoints (e.g., perspectives corresponding to at least one of a camera pose, origin, or orientation) and / or corresponding to different points in time. For example, the scene representation 124 may provide or output at least a first view of a 3D scene and a second view of the 3D scene, the first view of the 3D scene and the second view of the 3D scene possibly having different viewpoints and / or representing poses of objects in the 3D scene at different points in time.

[0062] Further references Figure 1In some embodiments, the model 112 includes at least one multi-view diffusion model. The multi-view diffusion model can be configured to generate images (e.g., image frames) of 3D objects (e.g., 3D objects represented by the scene representation 124). For example, the multi-view diffusion model can be a 3D-aware text-conditional model that can generate views of one or more objects of the scene representation 124 in response to receiving an indication of a given view (e.g., camera pose, origin, or orientation) of the views to be generated. In some embodiments, the multi-view diffusion model is configured based on at least the diffusion model (e.g., image model (e.g., text-to-image model) 112), for example by fine-tuning a potential text-to-image model using 3D object data (e.g., of the image data 106).

[0063] like Figure 1 As shown, the model 112 can receive input 120 and can generate output in response to the input 120. The input 120 can include any one or more text, voice, audio, image and / or video input data, and the model 112 can generate output based on at least these, such as generating a 2D image, a 3D image and / or a video output. For example, the input 120 can represent text information such as "the dog is running", and in response to the text information, the text-to-image model 112 can generate an image frame of the dog running, the multi-view diffusion model 112 can generate multiple image frames from multiple camera views of the dog running, and the text-to-video model 112 can generate a sequence of image frames representing the dog running (e.g., showing the movement of the dog in the frame sequence).

[0064] System 100 may allow for personalized 4D generation, such as for personalization of scene representation 124. For example, system 100 may receive an input including one or more images representing a subject (e.g., a person, a user), and may apply the one or more images as part of input 120 to model 112 and / or scene representation 124, such as initializing scene representation 124 using the one or more images.

[0065] In some embodiments, the system 100 performs view guidance (e.g., adjustments based on camera poses) to generate a scene representation 124 using the multi-view diffusion model 112. For example, the system 100 can augment the input 120 with text representing the views of one or more camera poses, such as, but not limited to, for example, "top view", "front view", "side view", "back view". This can be used to amplify the effect of the directional text hint augmentation.

[0066] Further references Figure 1, scene representation 124 may include or be coupled to motion generator 128. Scene representation 124 and motion generator 128 may together form a 4D representation (e.g., a sequence of 3D frames represented by scene representation 124 with motion between frames defined by motion generator 128). Motion generator 128 may include a function that may determine a state (e.g., position, pose) of one or more objects of scene representation 124 at a first point in time based at least on a state of one or more objects at a second point in time prior to the first point in time. Motion generator 128 may include a deformation field Δ Φ The motion generator 128 may have parameters x, y, z, τ (eg, three spatial dimension parameters and one temporal dimension parameter). For example, the deformation field may be Δ Φ (x, t, y, τ), and can output (e.g., predict) a displacement (Δx, Δy, Δz) given a 3D position (x, y, z) and a time τ. In some embodiments, the deformation field is or includes a neural network, such as a multi-layer perceptron (MLP) Φ. As further described herein with reference to the operation of updater 140, the deformation field Φ can be updated by updater 140 based on metrics 136 from evaluation of rendering 132 by model 112.

[0067] In some embodiments, the system 100 regularizes the deformation field, which can allow nearby Gaussians of the scene representation 124 to deform in a similar manner. For example, the system 100 can perform regularization by determining the 3D mean and diagonal covariance matrix of the 3D Gaussians of the scene representation at multiple time points τ of the 4D sequence, and can modify at least one of the 3D Gaussians to reduce the difference relative to one or more other 3D Gaussians in the 3D Gaussians, such as using Jenson-Shannon divergence for regularization. This can allow the distribution moments of the Gaussians to remain approximately constant over time, and can allow the system 100 to generate meaningful complex dynamics.

[0068] In some embodiments, the system 100 combines multiple scene representations 124 and / or motion generators 128. For example, the system 100 can combine (e.g., add) a first scene representation 124 and corresponding motion generator 128 with a second scene representation 124 and corresponding motion generator 128.

[0069] In some embodiments, the system 100 adjusts the video LDM of the model 112 according to the frame rate of the scene representation 124. For example, the system 100 can configure the video LDM according to the selected frame rate (e.g., from 4 frames per second, 8 frames per second, 12 frames per second), and can generate the rendering 132 according to the selected frame rate to provide the rendering 132 to the video LDM. This can allow the system 100 to generate a sufficiently long 4D animation and a temporally smooth 4D animation.

[0070] The system 100 may include at least one updater 140. The updater 140 may configure (e.g., train, modify, update, etc.) the scene representation 124 and / or the motion generator 128, or one or more components thereof, based at least on one or more metrics 136 generated by the model 112 or based on outputs (e.g., estimated outputs) from the model 112. In some embodiments, the model 112 is frozen (e.g., its parameters, such as weights and / or biases, are not changed) during operation of the updater 140 to update at least one of the scene representation 124 or the motion generator 128. In some embodiments, the updater 140 may configure the model 112 (e.g., perform various diffusion model training operations, potential diffusion model training operations, adjustments of the model 112, classifier / non-classifier guided training of the model 112, etc.) based on the data 106, 108 input to the given model 112 and the estimated output generated by the given model 112 in response to the input data.

[0071] For example, the updater 140 may use various objective functions, such as cost functions, scoring functions, and / or gradient functions, to evaluate the estimated (e.g., candidate) output determined (e.g., generated, produced) by the model 112 in response to receiving the rendering 132 as input. The updater 140 may update the model 112 in response to the output of the objective function, such as in response to whether a comparison between the estimated output and the corresponding data meets various convergence criteria (e.g., the output of the objective function is less than a threshold output or does not vary by more than a predetermined value over a number of iterations; training for a threshold number of iterations has been completed; the model 112 meets performance criteria (e.g., regarding output quality, accuracy of a downstream classifier operating on the output of the model 112, etc.). The objective function may include, for example, but not limited to, a least squares function, an L1 norm, or an L2 norm.

[0072] Further references Figure 1 In some embodiments, one or more models 112 may generate a metric 136 in response to one or more renderings 132 (processing of one or more renderings 132). Model 112 may perform a fractional distillation process (e.g., but not limited to fractional distillation sampling (SDS), variational fractional distillation) to determine metric 136. Model 112 may receive a given rendering 132 as input, diffuse the input (e.g., apply noise to the input according to parameters of model 112) to generate a diffused rendering z, and determine metric 136 based on at least the diffused rendering z, such as determining a gradient based on at least the diffused rendering z. System 100 may determine that metric 136 includes a gradient of a scoring function (e.g., Kullback-Leibler divergence (KLD)) of one or more models 112.

[0073] For example, to determine that the metric 136 includes a gradient for updating the scene representation 124 (e.g., updating parameters θ of the function g(θ)), the system 100 may use a corresponding scoring function of at least one of the text-to-image model 112 or the multi-view diffusion model 112 (each of which may score the diffuse rendering 132 input to the corresponding model 112 relative to the output of the corresponding model 112 adjusted according to the input 120) as a function for determining the gradient of the metric 136. The use of the text-to-image model 112 and the multi-view diffusion model 112 may allow for enforcing the quality of the rendering 132. To determine that the metric 136 includes a gradient for updating the motion predicted by the motion generator 128 (e.g., updating parameters of the MLP of the deformation field Φ), the system 100 may use a corresponding scoring function of at least one of the text-to-image model 112 or the text-to-video model 112 (each of which may score the diffuse rendering 132 input to the corresponding model 112 relative to the output of the corresponding model 112 adjusted according to the input 120).

[0074] For example, the system 100 may determine the gradient of the metric 136 as at least one of a fractional distillation sampling (SDS) loss or a variational fractional distillation (VSD) loss. Thus, the system 100 may use the metric 136 to facilitate ensuring that the rendering 132 from the scene representation 124 has a high likelihood under the priors defined by any one or more of the models 112 used for scoring. The metric 136 may include or apply various terms to incorporate specific functionality into how the updater 140 updates at least one of the scene representation 124 or the motion generator 128, such as weights or scaling factors corresponding to functionality such as motion amplification, view guidance, negative cues, and / or physical model-based verification; the values ​​of these terms may be configurable to control the effects of these functions.

[0075] The updater 140 may update at least one of the scene representation 124 or the motion generator 128 based at least on the corresponding metric 136 (e.g., updating the scene representation 124 based at least on the gradient of the scoring function of the text-to-image model 112 and the multi-view diffusion model 112; updating the motion generator 128 based at least on the gradient of the scoring function of the text-to-image model 112 and the text-to-video model 112). For example, the updater 140 may provide an update 144 to back-propagate the corresponding gradients into the scene representation 124 and / or the motion generator 128, such as updating the parameters θ and / or updating the deformation field Φ. For example, the updater 140 may modify the parameters θ and / or the deformation field Φ based on the corresponding gradients using various optimization algorithms (including, but not limited to, gradient descent).

[0076] Further references Figure 1, the system 100 can perform autoregressive generation of 4D content to extend the length (e.g., number of frames) of the 4D content. For example, the system 100 can select a first frame at a second time point (e.g., after a first initial time point) of a first sequence of multiple frames of a scene representation 124 and a motion generator 128, such as an intermediate frame of the first sequence. The system 100 can use the first frame as the initial frame of the scene representation 124 of the second frame sequence, and update the second motion generator 128 (e.g., a second deformation field) based on at least the first frame to allow the second motion generator 128 to be used to render a third frame at a third time point after the second time point. In some embodiments, the system 100 interpolates between overlapping portions of the frames of the first sequence and the second sequence (e.g., using a weighting function). The system 100 can iteratively perform such autoregressive operations to link multiple frame sequences together.

[0077] In some embodiments, the system 100 performs amplification of motion of an object represented by the 4D representation. For example, the system 100 may apply a scaling factor to the metric 136 (or a parameter corresponding to the metric, such as a gradient determined by the video diffusion model 112 for determining a given frame i or a component for determining the metric 136) to amplify the motion. The system 100 may control the value of the scaling factor; for example, a value of 1 may correspond to no scaling, while a value greater than 1 may increase the motion of the given frame i. In some embodiments, the system 100 determines the scaling metric 136 for the given frame i based at least on the metric 136, the scaling factor, and an average of the metrics 136 for the frames of the 4D representation.

[0078] The system 100 may implement negative hint guidance for the configuration of at least one of the scene representation 124 or the motion generator 128. For example, the system 100 may apply one or more negative hints (e.g., "low motion," "static statue," "no movement," "no motion") indicating motion restrictions as input to one or more models 112 (e.g., during configuration of the motion generator 128 (and the scene representation 124)). This may be performed in a manner similar to the absence of classifier guidance to allow the 4D representation to have a more dynamic and vivid scene.

[0079] In some embodiments, the system 100 includes or is coupled with a physical model, such as a model indicating expected motion of an object. The system 100 can use the physical model to evaluate the pose and / or motion of an object represented by the scene representation 124. For example, the system 100 can provide one or more renderings 132 from the scene representation 124 as input to the physical model to generate expected motion from the physical model, and compare the expected motion with the actual motion between the one or more renderings 132 and the second rendering 132 to perform the evaluation. The system 100 can update the scene representation 124 based at least on the evaluation, such as modifying the scene representation 124 to reduce the difference between the expected motion and the actual motion.

[0080] In some embodiments, the system 100 uses the scene representation 124 to extract 3D assets from the scene representation 124. For example, the system 100 can perform any of a variety of image processing operations (e.g., segmentation, feature detection, etc.) on one or more renderings 132 to retrieve assets, such as joints, motion, and / or deformation.

[0081] Reference now Figure 2 , Figure 2 Depicted is an example computing environment including system 200 according to some embodiments of the present disclosure. System 200 can be used to perform text to 4D operations with high-quality output. It should be understood that this and other arrangements described herein are presented only as examples. In addition to or in place of the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) can also be used, and some elements can be completely omitted. In addition, many elements described herein are functional entities that can be implemented as discrete or distributed components or with other components, and can be implemented in any suitable combination and position. The various functions performed by the entity described herein can be performed by hardware, firmware and / or software. For example, various functions can be implemented by a processor executing instructions stored in a memory. System 200 can include any function, model (e.g., machine learning model), operation, routine, logic or instruction for performing functions such as configuring, deploying, updating and / or generating the output of a machine learning model (including text to video model 216 and text to image model 220), as described herein. System 200 may incorporate features of system 100, such as facilitating periodic updating or modification of content model 204 using updater 140. System 200 may be implemented, at least in part, by the same or different entities or devices that implement system 100.

[0082] System 200 may include at least one content model 204. Content model 204 may be combined with reference Figure 1The content model 204 may include features of at least one of the scene representation 124 or the motion generator 128 described herein. For example, the content model 204 may include a 3D object and / or scene representation, such as one or more 3D Gaussian volumes and / or NeRFs, coupled with a deformation field (e.g., represented by an MLP). Figure 2 As described, the system 200 may be used to update the deformation field after configuring the scene representation; the system 200 may similarly be used to perform configuration of the scene representation (e.g., using the multi-view diffusion model 112 to generate gradients for scoring renderings 132 of multiple different views of a 3D Gaussian volume and / or NeRF representation of a 3D object and / or 3D scene, thereby allowing the system 200 to generate a static 3D Gaussian volume representing the hint 210).

[0083] The system 200 may receive one or more prompts 210. The prompts 210 may indicate one or more features of a 4D representation generated by the system 200, such as one or more features of one or more views of the content model 204. The prompts 210 may be received from one or more user input devices that may be coupled to the system 200. The prompts 210 may include any of a variety of data formats, including but not limited to text, voice, audio, image, or video data, indicating instructions corresponding to features of the 4D representation generated by the system 200. The prompts 210 may indicate, for example but not limited to, information about the category and / or characteristics of an object to be represented by the 4D representation. In some embodiments, the system 200 presents a prompt requesting one or more features through a user interface and receives the prompts 210 from the user interface. The prompts 210 may be received as semantic information (e.g., text, voice, speech, etc.) and / or image information (e.g., input of pixels indicating an area in a scene).

[0084] As described above, the system 200 may initialize a 4D representation of the content model 204 using a static 3D scene first generated based at least on the prompt 210. Subsequently, the system 200 may render a plurality of frames 208 from the content model 204. The frames 208 may be 4D images with corresponding camera poses c i (For example, c 1 、c 2 、c 3 and c 4 ) and the corresponding time step τ i (For example, τ 1 , τ 2 , τ 3 , τ 4 ). In this way, each frame 208 may correspond to a given camera pose and time step of the 4D representation of the content model 204.

[0085] In addition to rendering frame 208, system 200 may also render at least one frame 212 from content model 204. Frame 212 may have at least one camera pose or time step different from one or more frames 208. For example, Figure 2 As shown, frame 212 is for a time step τ 4 And according to the random camera pose In some embodiments, a subset of frames 212 are rendered from the same camera pose as frame 208 , and at least one other frame 212 is rendered from a different camera pose than frame 208 .

[0086] The system 200 may diffuse the frames 208 and 212 (e.g., add noise to the frames 208 and 212 using a forward diffusion process of the model 112). In response to diffusing the frames 208 and / or 212, the system 200 may provide the (diffused) frames 208 as input to a text-to-video model 216, such as to a denoising network 116 of the text-to-video model 216. The text-to-video model 216 may include, for example, a denoising network 116 of the video diffusion model 112 and one or more temporal layers 118. In some embodiments, the system 200 may input the diffused frames 208 to the denoising network 116 via an encoder 114 so as to perform a score / gradient determination operation in the latent space of the denoising network 116 and the temporal layers 118 (and may similarly input the diffused frames 212 to a corresponding denoising network 116 of a text-to-image diffusion model via a corresponding encoder 114 of the text-to-image diffusion model).

[0087] In response to receiving the input frame 208, the text-to-video model 216 may be used to generate a gradient (e.g., as for Figure 1 136). The updater 140 may update the content model 204 according to the gradients, for example, by performing an optimization (e.g., but not limited to, gradient descent) to modify the parameters of the content model 204 until a convergence condition is achieved. The system 200 may iteratively perform rendering of the frames 204, 208, generate gradients using the models 216, 220, and update the content model 204 (e.g., by back-propagating the gradients into the content model 204) until a convergence condition is achieved.

[0088] In response to the diffused frame 212, the system 200 may provide the (diffused) frame 212 as input to a text-to-image model 220, such as to a denoising network 116 of the text-to-image model 220. The text-to-image model 220 may include, for example, a reference Figure 1In response to receiving the input frame 212, the text-to-image model 220 may be used to generate a gradient (e.g., as for the score of the input frame 212 with respect to the parameter (e.g., parameter Φ) of the content model 204. Figure 1 In some embodiments, the system 200 determines the combined gradient in response to the processing of the frame 208 by the text-to-video model 216 and the processing of the frame 212 by the text-to-image model 220, and the updater 140 can update the content model 204 according to the combined gradient. Thus, the system 200 can use the text-to-video model 216 to allow the content model 204 to have realistic dynamics (e.g., motion) for 4D content, and can use the text-to-image model 220 to allow the content model 204 to render the frame 204 with a realistic, high-quality representation of the 3D scene at each time step relative to the prompt 210.

[0089] Reference now Figure 3 Each block of the method 300 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in a memory. The method can also be embodied as computer usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, Figure 1 and Figure 2 The method 300 is described using a system as an example. However, the method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0090] Figure 3 is a flow chart illustrating a method 300 for generating 4D content according to some embodiments of the present disclosure. Various operations of method 300 may be implemented by the same or different devices or entities at different points in time. For example, one or more first devices may implement operations related to configuring a diffusion machine learning model, one or more second devices may implement operations related to configuring a 4D content model, and one or more third devices may implement operations related to receiving user input requesting that content be generated by the diffusion machine learning model and / or the 4D content model and presented or otherwise provided. One or more third devices may maintain a neural network model, or may access the neural network model using, for example, but not limited to, an API provided by one or more first devices and / or one or more second devices.

[0091] The method 300 includes, at block B302, receiving input indicating 4D content to be generated. The 4D content may be a sequence of frames of 3D content data that may be arranged to have realistic motion of an object represented by the 3D content data across space and time. The input may include one or more of text, audio, speech, image, or video data, such as received as a prompt, such as through a conversational interface. The input may include one or more images of a subject to be represented by the 4D content, such as a user or other person, to allow personalization of the 4D content.

[0092] The method 300 may include, at block B304, initializing a 3D content model based at least on the input. The 3D content model may include at least one of a 3D Gaussian model (e.g., 3D Gaussian Splatting), NeRF, a point cloud, or HexPlanes. The 3D content model may be initialized by iteratively rendering frames from the 3D content model for a plurality of different camera poses, evaluating the rendered frames using a machine learning model, and updating the 3D content model based on the evaluation until a convergence condition is met. The machine learning model may include at least one of a multi-view diffusion model or a text-to-image diffusion model, such that the evaluation may provide a metric (e.g., an output of a scoring function; a gradient) of the rendering relative to the parameters of the 3D content model. The 3D content model may be updated based on the metric (e.g., using, but not limited to, gradient descent). In some embodiments, the metric includes a component for camera pose adjustment and / or trajectory adjustment.

[0093] The method 300 may include updating the 4D content model at block B306 based at least on the input. The 4D content model may include a 3D content model and a motion generator, such as a dynamic model. For example, the motion generator may include a machine learning model, such as an MLP, which may be configured to generate motion of features of an object represented by the 3D content model between frames rendered based on the 3D content model. For example, multiple frames may be rendered from a 3D content model (e.g., from a 3D Gaussian, etc.). The rendered frames may include one or more first frames having a predetermined camera pose, and one or more second frames having the same second pose and / or a different pose (e.g., a randomly selected pose) as the first frame. The rendered frames may be provided as input to at least one of a text-to-image model (which may be the same model as the model used to initialize the 3D content model) or a text-to-video model. The text-to-video model may be a diffusion model, such as a potential diffusion model that has been fine-tuned and / or transfer learning performed to incorporate temporal layers into the diffusion model. At least one of the text-to-image model or the text-to-video model may evaluate the rendered frames, such as the gradient of the data used to generate the rendered frames relative to the parameters of the motion generator. The motion generator may be updated based on the gradients, for example by back-propagating the gradients through the configuration of the motion generator until convergence / threshold criteria are met. In some embodiments, a physical model is used to evaluate the rendered frames. The motion generator may be updated based on one or more metrics such as view guidance, regularization, motion amplification, and / or negative cues.

[0094] The method 300 includes, at block B308, outputting a rendered frame from the updated 4D content model. For example, this may include causing at least one of: (i) performing a simulation using the rendered frame from the updated 4D content model or (ii) presenting the rendered frame using a display. In some embodiments, the rendered frame is processed to detect 3D assets, such as joints, motion, or deformation of an object represented by the 4D content model. In some embodiments, the 4D content is autoregressively generated, such as by using a first frame from a first instance of a 4D content model configuration as an input to a second instance configuration of the 4D content model.

[0095] Example content streaming system

[0096] See now Figure 4 , Figure 4 is an example system diagram for a content streaming system 400 according to some embodiments of the present disclosure. Figure 4 Includes (one or more) application servers 402 (which may include Figure 5 ), client device(s) 404 (which may include components, features, and / or functionality similar to the example computing device 500 of Figure 5) and (one or more) networks 406 (which may be similar to the (one or more) networks described herein). In some embodiments of the present disclosure, system 400 can be implemented to perform diffusion model and NeRF training and runtime operations. The application session can correspond to a game streaming application (e.g., NVIDIA GeFORCE NOW), a remote desktop application, a simulation application (e.g., an autonomous or semi-autonomous vehicle simulation), a computer-aided design (CAD) application, a virtual reality (VR) and / or augmented reality (AR) streaming application, a deep learning application, and / or other application types. For example, system 400 can be implemented to receive input indicating one or more features of output generated using a neural network model, provide input to the model so that the model generates output, and use the output for various operations including display or simulation operations.

[0097] In system 400, for an application session, client device(s) 404 may receive only input data in response to input to input device(s), send the input data to application server(s) 402, receive encoded display data from application server(s) 402, and display the display data on display 424. Thus, computationally more intensive calculations and processing are offloaded to application server(s) 402 (e.g., rendering for graphical output of the application session—particularly ray or path tracing—is performed by the GPU(s) of game server(s) 402). In other words, the application session is streamed from application server(s) 402 to client device(s) 404, thereby reducing the requirements for graphics processing and rendering by client device(s) 404.

[0098] For example, with respect to instantiation of an application session, client device 404 may display a frame of the application session on display 424 based on receiving display data from one or more application servers 402. Client device 404 may receive input from one of the input devices(s) and generate input data in response, such as providing a prompt as input, for generating 4D content. Client device 404 may send the input data to application server(s) 402 via communication interface 420 and over network 406 (e.g., the Internet), and application server(s) 402 may receive the input data via communication interface 418. CPU(s) may receive the input data, process the input data, and send the data to GPU(s), which causes GPU(s) to generate a rendering of the application session. For example, the input data may represent movement of a user's character in a game session of a game application, firing a weapon, reloading, passing a ball, steering a vehicle, etc. Rendering component 412 may render an application session (e.g., a result representing input data), and rendering capture component 414 may capture the rendering of the application session as display data (e.g., as image data capturing a rendered frame of the application session). The rendering of the application session may include lighting and / or shadow effects of ray or path tracing computed using one or more parallel processing units (such as GPUs) of application server(s) 402, which may further employ the use of one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques. In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components such as vGPUs, vCPUs, etc.—may be used by application server(s) 402 to support the application session. Encoder 416 may then encode the display data to generate encoded display data, and the encoded display data may be sent to client device 404 over network 406 via communication interface 418. The client device 404 may receive the encoded display data via the communication interface 420 , and the decoder 422 may decode the encoded display data to generate the display data. The client device 404 may then display the display data via the display 424 .

[0099] Example computing device

[0100] Figure 55 is a block diagram of an example computing device 500 suitable for implementing some embodiments of the present disclosure. The computing device 500 may include an interconnect system 502 that directly or indirectly couples the following devices: a memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, an input / output (I / O) port 512, an input / output component 514, a power supply 516, one or more presentation components 518 (e.g., one or more displays), and one or more logic units 520. In at least one embodiment, one or more computing devices 500 may include one or more virtual machines (VMs), and / or any of their components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 508 may include one or more vGPUs, one or more of the CPUs 506 may include one or more vCPUs, and / or one or more of the logic units 520 may include one or more virtual logic units. As such, computing device(s) 500 may include discrete components (eg, a full GPU dedicated to computing device 500 ), virtual components (eg, a portion of a GPU dedicated to computing device 500 ), or a combination thereof.

[0101] although Figure 5 The various blocks of are shown as being connected with wires via interconnect system 502, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 518 (such as a display device) may be considered to be I / O component 514 (e.g., if the display is a touch screen). As another example, CPU 506 and / or GPU 508 may include memory (e.g., memory 504 may represent a storage device in addition to the memory of GPU 508, CPU 506, and / or other components). In other words, Figure 5 The computing devices described herein are merely illustrative. No distinction is made between categories such as "workstations," "servers," "laptops," "desktop computers," "tablet computers," "client devices," "mobile devices," "handheld devices," "game consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are contemplated herein. Figure 5 within the range of computing devices.

[0102] The interconnection system 502 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 502 may be arranged in various topologies including, but not limited to, a bus, a star, a ring, a mesh, a tree, or a hybrid topology. The interconnection system 502 may include one or more bus or link types, such as an industrial standard architecture (ISA) bus, an extended industrial standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a fast peripheral component interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, the CPU 506 may be directly connected to the memory 504. Further, the CPU 506 may be directly connected to the GPU 508. In the case of a direct connection or a point-to-point connection between components, the interconnection system 502 may include a PCIe link to perform the connection. In these examples, the computing device 500 does not need to include a PCI bus.

[0103] Memory 504 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 500. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0104] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 504 may store computer readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 500. As used herein, computer storage media does not include the signals themselves.

[0105] Computer storage media may embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal (such as a carrier wave or other transmission mechanism), and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. As an example and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer readable media.

[0106] The CPU 506 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. Each of the (one or more) CPUs 506 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of handling multiple software threads simultaneously. The (one or more) CPUs 506 may include any type of processor and may include different types of processors depending on the type of computing device 500 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 500, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 500 may include one or more CPUs 506 in addition to one or more microprocessors or supplementary coprocessors such as math coprocessors.

[0107] In addition to or in place of the CPU 506, one or more GPUs 508 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 508 may be an integrated GPU (e.g., with one or more of the CPUs 506) and / or one or more of the GPUs 508 may be a discrete GPU. In an embodiment, one or more of the (one or more) GPUs 508 may be a coprocessor of one or more of the (one or more) CPUs 506. The GPU 508 may be used by the computing device 500 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the GPU 508 may be used for general-purpose computing on a GPU (GPGPU). The GPU 508 may include hundreds of cores capable of handling hundreds of software threads simultaneously. The GPU 508 may generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command received from the CPU 506 via a host interface). GPU 508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 504. GPU 508 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 508 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

[0108] In addition to or in lieu of the CPU 506 and / or GPU 508, one or more logic units 520 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 500 to perform one or more of the methods and / or processes described herein. In an embodiment, the (one or more) CPUs 506, (one or more) GPUs 508, and / or (one or more) logic units 520 may perform any combination of methods, processes, and / or portions thereof, separately or jointly. One or more of the logic units 520 may be a part of and / or integrated into one or more of the CPUs 506 and / or GPUs 508, and / or one or more of the logic units 520 may be discrete components or otherwise external to the CPUs 506 and / or GPUs 508. In an embodiment, one or more of the logic units 520 may be a coprocessor of one or more of the one or more CPUs 506 and / or one or more of the one or more GPUs 508.

[0109] Examples of logic unit 520 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0110] The communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 500 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 510 may include components and functions that enable communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, the logic unit 520 and / or the communication interface 510 may include one or more data processing units (DPUs) to transmit data received over a network and / or over the interconnect system 502 directly to one or more GPUs 508 (e.g., memory of one or more GPUs 508). In some embodiments, multiple computing devices 500 or components thereof (which may be similar or different in various aspects to one another) may be communicatively coupled to send and receive data to perform the various operations described herein, for example, to facilitate reduction in latency.

[0111] I / O ports 512 may enable computing device 500 to be logically coupled to other devices including I / O components 514, presentation components 518, and / or other components, some of which may be built into (e.g., integrated into) computing device 500. Illustrative I / O components 514 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 514 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by a user, such as generating prompts, image data 106, and / or video data 108. In some instances, the input may be sent to an appropriate network element for further processing, such as modifying and registering an image. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with a display of computing device 500 (as described in more detail below). The computing device 500 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 500 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables detection of motion. In some examples, the computing device 500 may use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0112] The power supply 516 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 516 may provide power to the computing device 500 to enable the components of the computing device 500 to operate.

[0113] The presentation component(s) 518 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 518 may receive data from other components (e.g., GPU 508, CPU 506, DPU, etc.) and output data (e.g., as images, video, sound, etc.).

[0114] Sample Data Center

[0115] Figure 6 An example data center 800 is shown that can be used in at least one embodiment of the present disclosure, such as system 100 and / or system 200 implemented in one or more examples of data center 600. Data center 600 can include a data center infrastructure layer 610, a framework layer 620, a software layer 630, and / or an application layer 640.

[0116] like Figure 6 As shown, the data center infrastructure layer 610 may include a resource coordinator 612, grouped computing resources 614, and node computing resources ("node CRs") 616(1)-616(N), where "N" represents any integer, positive integer. In at least one embodiment, the node CRs 616(1)-616(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output ("NW In some embodiments, one or more of the node CRs 616 (1) - 616 (N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node CRs 616 (1) - 616 (N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of the node CRs 616 (1) - 616 (N) may correspond to a virtual machine (VM).

[0117] In at least one embodiment, the computing resources 614 of the grouping may include a separate grouping of node CR616 in many racks contained in one or more racks (not shown) or contained in a data center (also not shown) at different geographical locations. The separate grouping of node CR616 in the computing resources 614 of the grouping may include computing resources, network resources, memory resources or storage resources of the grouping, which may be configured or allocated to support one or more workloads. In at least one embodiment, several node CR616 including CPU, GPU, DPU and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules and / or network switches in any combination.

[0118] Resource coordinator 612 may configure or otherwise control one or more nodes CR 616(1)-616(N) and / or grouped computing resources 614. In at least one embodiment, resource coordinator 612 may comprise a software design infrastructure ("SDI") management entity for data center 600. Resource coordinator 612 may comprise hardware, software, or some combination thereof.

[0119] In at least one embodiment, Figure 6 As shown, the framework layer 620 may include a job scheduler 628, a configuration manager 634, a resource manager 636, and / or a distributed file system 638. The framework layer 620 may include a framework that supports software 632 of the software layer 630 and / or one or more applications 642 of the application layer 640. The software 632 or the application 642 may include network-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 620 may be, but is not limited to, a type of free open source software web application framework that can utilize the distributed file system 638 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 628 may include a Spark driver to facilitate the scheduling of workloads supported by the various layers of the data center 600. The configuration manager 634 may be able to configure different layers, such as the software layer 630 and the framework layer 620 including Spark and a distributed file system 638 for supporting large-scale data processing. The resource manager 636 may be able to manage the clustered or grouped computing resources mapped to or allocated to support the distributed file system 638 and the job scheduler 628. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 614 at the data center infrastructure layer 610. The resource manager 636 may coordinate with the resource coordinator 612 to manage these mapped or allocated computing resources.

[0120] In at least one embodiment, the software 632 included in the software layer 630 may include software used by at least portions of the node CRs 616(1)-616(N), the grouped computing resources 614, and / or the distributed file system 638 of the framework layer 620. The one or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0121] In at least one embodiment, the applications 642 included in the application layer 640 may include one or more types of applications used by the node CRs 616(1)-616(N), the grouped computing resources 614, and / or at least multiple portions of the distributed file system 638 of the framework layer 620. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments, such as training, configuring, updating, and / or executing the machine learning model 112.

[0122] In at least one embodiment, any of configuration manager 634, resource manager 636, and resource coordinator 612 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. The self-modification actions can save a data center operator of data center 600 from making potentially poor configuration decisions and potentially avoiding underutilized and / or underperforming portions of a data center.

[0123] According to one or more embodiments described herein, data center 600 may include tools, services, software, or other resources to train one or more machine learning models (e.g., train machine learning model 112) or use one or more machine learning models to predict or infer information (e.g., generate scene representation 124, motion generator 128, and / or content model 204). For example, one or more machine learning models may be trained by computing weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 600. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 600 by using weight parameters computed by one or more training techniques (such as, but not limited to, those described herein).

[0124] In at least one embodiment, the data center 600 may use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or reasoning using the above resources. In addition, the above one or more software and / or hardware resources may be configured to allow users to train or perform information reasoning services, such as image recognition, speech recognition, or other artificial intelligence services.

[0125] Example network environment

[0126] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to communicate with the client device, server, or other device types. Figure 5 The computing device 500 of the embodiment of the present invention may be implemented on one or more instances of the computing device 500, for example, each device may include similar components, features and / or functions of the computing device 500. In addition, in the case of implementing a backend device (e.g., a server, NAS, etc.), the backend device may be included as part of the data center 600, and the example of the data center 600 is referred to in this article with respect to Figure 6 Describe in more detail.

[0127] The components of the network environment can communicate with each other via one or more networks that can be wired, wireless, or both. The network can include multiple networks or networks of networks. As an example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or a public switched telephone network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0128] Compatible network environments may include one or more peer-to-peer network environments, in which case the network environment may not include a server, and one or more client-server network environments, in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0129] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for one or more applications of the software supporting the software layer and / or the application layer. The software or application may include web-based service software or application programs, respectively. In an embodiment, one or more of the client devices may use web-based service software or application programs (e.g., by accessing the service software and / or application programs via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework such as a distributed file system that can perform large-scale data processing (e.g., "big data")

[0130] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed in multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server can assign at least a portion of the function to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0131] One or more client devices may include herein Figure 5 At least some of the components, features, and functionality of one or more of the described example computing devices 500. By way of example and not limitation, the client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smart phone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0132] The present disclosure may be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) executed by a computer or other machine, such as a personal data assistant or other handheld device. In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs a particular task or implements a particular abstract data type. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0133] As used herein, the description of "and / or" about two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. In addition, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0134] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in conjunction with other current or future technologies to include different steps or combinations of steps similar to the steps described in this document. In addition, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, unless and in addition to the order of the individual steps being explicitly described, the terms should not be interpreted as implying any particular order among or between the various steps disclosed herein.

Claims

1. A processor, comprising: One or more circuits for: receiving input indicating one or more characteristics of content, the content comprising at least one of an object or a scene; Initializing a content model based on the input to represent the input in three spatial dimensions and a temporal dimension; updating the content model by rendering one or more sequences of frames from the content model, determining a metric for the one or more sequences using a latent diffusion model, and modifying the content model based on the metric until a convergence condition is satisfied; as well as At least one of: (i) causing a simulation to be performed using the updated content model or (ii) causing presentation of the updated content model using a display.

2. The processor of claim 1 , wherein the latent diffusion model comprises one or more layers configured for the time dimension and comprises or is coupled to an optimizer to determine the metric based at least on a gradient associated with a given frame in the one or more sequences of frames.

3. The processor of claim 1 , wherein the content model is conditioned on camera movements associated with the three spatial dimensions and time values ​​associated with the time dimension, and the one or more circuits are configured to render a given frame sequence in the one or more frame sequences according to a given camera pose for the given frame sequence and provide the given frame sequence as an input to the latent diffusion model.

4. The processor of claim 1 , wherein the one or more circuits are configured to: updating the content model according to a predetermined input, the predetermined input identifying a camera pose for the given frame sequence and a time point of one or more frames in the given frame sequence; and The metric is determined based on the given sequence of frames rendered according to the predetermined input.

5. The processor of claim 1 , wherein the content model comprises: a deformation field for representing motion in the one or more sequences of frames; as well as At least one of a Gaussian splatter representation, a neural radiation field (NeRF), a mesh representation, or a point cloud.

6. The processor of claim 1 , wherein the one or more circuits are configured to: Rendering a first frame for a first time point and a second frame for a second time point after the first time point according to the updated content model; modifying the updated content model according to the second frame; and A third frame for a third time point after the second time point is rendered from the modified content model according to the second frame.

7. The processor of claim 1, wherein the input comprises natural language data and one or more images, and the one or more circuits are configured to update the content model based on the one or more images.

8. The processor of claim 1, wherein the one or more circuits are configured to update the content model based on a physics model to measure the physics-based realism of motion represented in the one or more frame sequences.

9. The processor of claim 1, wherein the one or more circuits are to identify from the updated content model at least one of a joint of an object represented by the updated content model, a motion property of the object, or a deformation property of the object.

10. The processor of claim 1, wherein the processor is included in at least one of the following: Systems for generating synthetic data; A system for performing simulation operations; Systems for performing conversational AI operations; A system for performing collaborative content creation of 3D assets; A system including one or more large language models (LLMs); Systems for performing digital twin operations; A system for performing light transport simulations; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

11. A system comprising: One or more processing units, configured to perform operations, the operations comprising: receiving input indicating one or more characteristics of content, the content comprising at least one of an object or a scene; Initializing a content model based on the input to represent the input in three spatial dimensions and a temporal dimension; updating the content model by rendering one or more sequences of frames from the content model, determining a metric for the one or more sequences using a latent diffusion model, and modifying the content model based on the metric until a convergence condition is satisfied; and At least one of: (i) causing a simulation to be performed using the updated content model or (ii) causing presentation of the updated content model using a display.

12. The system of claim 11, wherein the latent diffusion model comprises one or more layers configured for the time dimension and comprises or is coupled to an optimizer to determine the metric based at least on a gradient associated with a given frame in the one or more frame sequences.

13. The system of claim 11, wherein the content model is conditioned on camera movement associated with the three spatial dimensions and a time value associated with the time dimension, and the one or more processing units are configured to perform operations comprising: A given frame sequence of the one or more frame sequences is rendered according to a given camera pose for the given frame sequence, and the given frame sequence is provided as an input to the latent diffusion model.

14. The system of claim 11, wherein the one or more processing units are configured to perform operations comprising: updating the content model according to a predetermined input, the predetermined input identifying a camera pose for the given frame sequence and a time point of one or more frames in the given frame sequence; as well as The metric is determined based on the given sequence of frames rendered according to the predetermined input.

15. The system of claim 11, wherein the content model comprises: deformation field; as well as At least one of a Gaussian splatter representation, a neural radiation field (NeRF), a mesh representation, or a point cloud.

16. The system of claim 11, wherein the one or more processing units are configured to perform operations comprising: Rendering a first frame for a first time point and a second frame for a second time point after the first time point according to the updated content model; modifying the updated content model according to the second frame; and A third frame for a third time point after the second time point is rendered from the modified content model according to the second frame.

17. The system of claim 11, wherein the input comprises natural language data and one or more images, and the one or more circuits are configured to perform operations including updating the content model based on the one or more images.

18. The system of claim 11, wherein the system is included in at least one of the following: Systems for generating synthetic data; A system for performing simulation operations; Systems for performing conversational AI operations; A system for performing collaborative content creation of 3D assets; A system including one or more large language models (LLMs); Systems for performing digital twin operations; A system for performing light transport simulations; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A method comprising: receiving, by one or more processors, input indicative of at least one of an object or a scene; initializing, by the one or more processors, a plurality of spatial dimensions of a content model of the at least one of the object or the scene based at least on the input; updating, by the one or more processors, the content model having a temporal dimension in response to evaluating a plurality of frames rendered from the content model at a plurality of points in time using a latent diffusion model having one or more temporal layers to generate an updated content model; and One or more frames from the updated content model are output by the one or more processors.

20. The method of claim 19, wherein the content model comprises a 3D Gaussian splatter representation corresponding to the plurality of spatial dimensions, the 3D Gaussian splatter representation coupled with a multi-layer perceptron (MLP) corresponding to the temporal dimension.