Feature cache based generative video editing for dynamic frame generation

Abstract

The invention relates to feature cache based generative video editing for dynamic frame generation. Various examples, systems, and methods relating to feature cache-based generative video editing for dynamic frame generation are disclosed. The system may apply the first frame as an input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model. The system may store the first embedding in a cache, where the cache includes a second embedding of a second frame. The system may generate a third frame using a machine learning model based at least on the cache, where the third frame is associated with the first frame. The system may output the third frame to a video stream that includes the fourth frame.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 690,571, filed September 4, 2024, the disclosure of which is incorporated by reference herein in its entirety. BACKGROUND

[0003] Some systems can use diffusion-based models to edit videos. These systems can generate edited images, but lack the temporal consistency required for real-time video streaming scenarios, resulting in flickering and / or other unattractive visual effects. Other systems can produce temporally consistent video frames based on pre-computed inter-frame correspondences, but these systems have high processing latency and thus are unable to perform real-time video editing tasks. Some systems generate edited video streams by relying on marker-merging techniques, which discard fine-grained or motion-specific details, resulting in a lack of temporal consistency across frames. SUMMARY

[0004] Systems and methods related to feature cache-based generative video editing for dynamic frame generation are disclosed. Systems and methods according to the present disclosure can implement AI video generation models, such as diffusion models, that use a feature cache to store useful video frame feature data to facilitate the video generation process. The feature cache can be used to selectively maintain information related to the scene of a generated video that is relevant to temporal considerations while maintaining the ability to process in real-time or near real-time.

[0005] Conventional diffusion models, while effective, are slow due to their large size and the large number of iterations (e.g., 30 to 50 iterations) required to generate a single image and thus are not suitable for real-time applications. Furthermore, these models lack temporal consistency as they are designed to generate individual images (e.g., based on text prompts) rather than a series of related images over time.

[0006] In contrast to conventional systems such as those described above, systems and methods according to the present disclosure can enable real-time edited video generation with shorter frame generation times and / or more frames per second by conditioning diffusion models to generate one or more temporally consistent edited frames from video input by querying relevant features stored in a feature cache. For example, systems and methods herein can leverage a caching mechanism and temporal embeddings to facilitate the video generation process in real-time applications.

[0007] Real-time applications can include, but are not limited to, style transfer (e.g., converting an existing video style to a claymation, a pixel art, a pencil sketch, a hard resin, and / or other style), image enhancement (e.g., including receiving a low resolution video (e.g., a video lacking in detail) and adding details back to the video), and object replacement tasks (e.g., detecting an object (e.g., a dog) in a video and replacing the object with another object (e.g., a cat)). In certain examples, the systems and methods herein can edit visual details in a video that indicate one weather (e.g., rain) to instead indicate a different weather (e.g., sunny).

[0008] In some embodiments, the technology described herein involves one or more processors including processing circuitry to apply a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model. The processing circuitry can store the first embedding in a cache, where the cache contains a second embedding of a second frame. The processing circuitry can generate a third frame using the machine learning model based at least on the cache, where the third frame is associated with the first frame. The processing circuitry can output the third frame into a video stream containing a fourth frame.

[0009] In some embodiments, the processing circuitry is to store the first embedding in a first slot of the cache. The processing circuitry can store the second embedding in a second slot of the cache. In some embodiments, the first frame and the second frame are separated by an interval.

[0010] In some embodiments, the processing circuitry is to interpolate a fifth frame based at least on the fourth frame and the third frame, where the fourth frame is previously generated. The processing circuitry can output the fifth frame into the video stream between the fourth frame and the third frame.

[0011] In some embodiments, the processing circuitry is to remove the third embedding from the cache according to a respective weight in response to determining that a number of stored embeddings exceeds a cache capacity.

[0012] In some embodiments, the processing circuitry is to predict a fifth frame based at least on the third frame. The processing circuitry can store a third embedding of the fifth frame in the cache.

[0013] In some embodiments, the processing circuitry is to assign a weight for at least one of the first embedding or the second embedding determined according to a duration of the respective embedding in the cache.

[0014] In some embodiments, the fourth frame is associated with the second frame.

[0015] In some embodiments, the fifth frame is predicted using optical flow.

[0016] In some embodiments, the technology described herein relates to a method. The method can include applying a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model. The method can include storing the first embedding in a first slot of a cache, where the cache contains a second embedding of a second frame stored in a second slot of the cache. The method can include generating a third frame using the machine learning model based at least on the cache. The method can include outputting the generated third frame to a video stream containing a generated fourth frame.

[0017] In some embodiments, the method can include interpolating a fifth frame based at least on the generated fourth frame and the generated third frame, and outputting the interpolated fifth frame to the video stream between the generated fourth frame and the generated third frame.

[0018] In some embodiments, the method can include determining an optical flow between the third frame and a fifth frame, where the fifth frame includes raw image data, predicting a sixth frame based at least on the optical flow, and storing a third embedding of the sixth frame in the cache.

[0019] In some embodiments, the method can include assigning a first weight to the first embedding, the first weight determined according to a respective duration of the first embedding in the cache.

[0020] In some embodiments, a first embedding of the generated fourth frame is associated with a third embedding of the second frame.

[0021] In some embodiments, the method can include expanding a self-attention layer of the machine learning model based at least on the cache.

[0022] In some embodiments, the method can include linearly decreasing a second weight of the second embedding based at least on storing the first embedding in the cache.

[0023] In some embodiments, the method can include removing the second embedding from the cache based at least on the second weight in response to determining that a number of stored embeddings exceeds a cache capacity.

[0024] In certain aspects, the technology described herein relates to a system comprising one or more processors to apply a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model. The one or more processors can store the first embedding in a first slot of a cache, wherein the cache contains a second embedding of a second frame located in a second slot of the cache. The one or more processors can generate a third frame using the machine learning model based at least on the cache, wherein the third frame is associated with the first frame. The one or more processors can output the third frame to a video stream containing a fourth frame generated. BRIEF DESCRIPTION OF DRAWINGS

[0025] The present system and method for feature cache based generative video editing for dynamic frame generation is described in detail below with reference to the attached drawing figures, wherein:

[0026] Figure 1 is a block diagram of an example generative model system in accordance with implementation of some embodiments of the present disclosure;

[0027] Figure 2 is a block diagram of an example cache based generative video editing system in accordance with implementation of some embodiments of the present disclosure;

[0028] Figure 3 is a flow diagram of an example of a method for generating frames using a cache in accordance with implementation of some embodiments of the present disclosure;

[0029] Figure 4 is a block diagram of an example content streaming system suitable for implementing some embodiments of the present disclosure;

[0030] Figure 5 is a block diagram of an example computing device suitable for implementing at least some embodiments of the present disclosure;

[0031] Figure 6 is a block diagram of an example data center suitable for implementing at least some embodiments of the present disclosure;

[0032] Figure 7 is a block diagram of an example generative model comprising a transformer encoder-decoder suitable for implementing at least some embodiments of the present disclosure; and

[0033] Figure 8 is a block diagram of an example generative model comprising a decoder-only transformer architecture suitable for implementing at least some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] The disclosed systems and methods relate to feature cache-based generative video editing for dynamic frame generation. For example, the systems and methods herein can enable real-time edited video generation with shorter frame generation times and / or more frames per second.

[0035] Some systems can use diffusion-based models to edit videos. These systems can generate edited images, but lack the temporal consistency required for real-time video streaming scenarios such as gaming, resulting in flickering and / or other unattractive visual effects. Other systems are able to produce temporally consistent video frames based on pre-computed inter-frame correspondences, but these systems have high processing latency and thus are unable to perform real-time video editing tasks.

[0036] Some systems generate edited video streams by relying on marker merging techniques, such as merging archived features of past frames with incoming frame features to create a compact feature bank. However, this merging process often discards fine-grained or motion-specific details, resulting in a lack of temporal consistency between frames.

[0037] In contrast to traditional systems such as those described above, systems and methods according to the present disclosure can enable real-time edited video generation with shorter frame generation times and / or more frames per second by conditioning a diffusion model to generate one or more temporally consistent edited frames from a video input by querying relevant stored features in a feature cache. In some embodiments, the system can include a feature cache containing features of previous frames of a video input stream. The system can extract features from a new input frame. The system can generate one or more temporal embeddings based on at least the extracted features. The system can store the temporal embeddings in the feature cache. By querying relevant stored features in the feature cache, the system can condition a diffusion model to generate one or more temporally consistent edited frames, and can output these frames to or as a video stream.

[0038] In some embodiments, the system can maintain global consistency (e.g., coherence between frames in a multi-frame output) by storing features from multiple frames. The system can maintain temporal consistency for long sequences by storing frame features at intervals, such as every fourth frame, and removing past features from the feature cache in response to a cache size exceeding a capacity and / or a time threshold. The past feature for removal can be selected according to a longest duration of past features in the feature cache. The system can reduce processing time by skipping editing every other input frame (or every few input frames). The system can generate one or more frames to replace skipped frames by performing frame interpolation between edited frames (e.g., using a real-time intermediate flow estimation (RIFE) technique). The system can generate predicted frames based at least on edited and / or interpolated frames. The system can generate predicted frames using an optical flow technique. The system can store one or more embeddings of predicted frames in the feature cache to maintain consistency between a current frame and a next input frame to be edited.

[0039] As used herein, the term “feature” refers to a numerical representation or descriptive attribute that captures key characteristics of an input, such as an image, video frame, or portion thereof, extracted from the input. In the field of computer vision, features can include low-level visual patterns (e.g., edges, corner points, textures), mid-level representations (e.g., object parts, semantic regions), or high-level abstract information (e.g., class-related representations). Features can be extracted using handcrafted techniques (e.g., SIFT, HOG) or learned representations generated by one or more neural network layers, such as convolutional layers in a convolutional neural network (CNN).

[0040] As used herein, the term “embedding” refers to a vectorized representation that encodes semantic or contextual information of an input or portion thereof (key, query, and / or value). Embeddings can be derived from features by projecting the extracted features into a continuous vector space (typically with lower dimensionality) using learned transformations. Embeddings can be used to facilitate comparison, classification, retrieval, or other downstream processing tasks. In certain embodiments, embeddings can be generated using fully connected layers, pooling operations, and / or attention mechanism-based encoding mechanisms applied to extracted features.

[0041] As used herein, “features” and “embeddings” can be used interchangeably where context permits, particularly when referring to intermediate representations output by neural networks. As used herein, the term “feature” generally refers to a representation directly extracted from input data (e.g., a convolutional feature map), while an “embedding” can refer to a transformed or encoded version of such features, which are typically used for downstream tasks or inter-component communication.

[0042] As used herein, the term“self-attention” refers to a mechanism by which each element in an input sequence (e.g., a sequence of features or embeddings) is processed with respect to every other element in the sequence to compute a context-aware representation. In computer vision applications, self-attention allows a model to assign weights to different spatial locations in an image or different frames in a video, enabling the model to selectively focus on relevant parts of the input.

[0043] In various embodiments, self-attention can act on features, embeddings, or both. For example, a visual transformer can apply self-attention to patch-wise embeddings derived from image features to compute a refined representation that captures global dependencies. Thus, features, embeddings, and self-attention can be used in combination in a processing pipeline to implement tasks such as classification, detection, segmentation, and / or generation.

[0044] Reference Figure 1 , Figure 1 is a block diagram of an example generative model system 100. In Figure 1 In the example shown, the generative model system 100 includes an input processor 105, an embedding component 120, a retrieval component 192, a plug-in / API 195, and a generative model (LM) 130 (which can include a GAN, a VLM, a multi-modal LM, etc.).

[0045] The processor 105 can receive input data 101, which can include image data, video data, and / or other types of visual or multi-modal input data (e.g., sensor data, 3D models, CAD design drawings, USD scene graphs). The input processor 105 can resize data to a standard size compatible with the format of the respective input channel, and / or can normalize pixel values to a common range (e.g., 0 to 1) to ensure consistent representation. The embedding component 120 can encode image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In certain embodiments where input 101 includes audio data, the input processor 105 can resample audio files to a consistent sampling rate for uniform processing, and the embedding component 120 can extract and encode audio features using any known technique— such as in the form of a spectrogram (e.g., a mel-spectrogram). In certain embodiments where input 101 includes video data, the input processor 105 can extract frames and / or apply resizing to the extracted frames. The embedding component 120 can extract features from input 101, such as optical flow embeddings or video embeddings, and / or can encode temporal information or sequences of frames. In certain embodiments where input 101 includes multi-modal data, the embedding component 120 can fuse representations of different types of data (e.g., text, images, audio, USD, video, designs, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc. In sequential frame implementations (e.g., video), the embedding component 120 can also use optical flow, recurrent layers, and / or spatio-temporal convolutions to extract temporal features. For multi-modal input, fusion techniques (e.g., concatenation, attention-based alignment, or joint embedding space) can be used to unify visual and contextual input.

[0046] In some embodiments, the retrieval component 192 can be used to retrieve base information as part of the conditional input to the generative model 130. For example, the retrieval component 192 can acquire text, visual, or structured information from external data sources or knowledge bases (e.g., images, image-text pairs, visual labels, or descriptive labels) based on the context or intended transformation of the input image. The retrieved content can be embedded and provided to the generative model 130 to inform or refine the denoising process. The retrieval component 192 can also access image libraries, prior embeddings, or external visual knowledge graphs to support the frame generation task.

[0047] In some embodiments, a plugin or API-based architecture 195 can be implemented, allowing the system to interact with external services, such as CAD tools, asset databases, cloud-based visual processing APIs, or knowledge retrieval engines. For example, if the input 101 involves transforming a 3D design into a stylized rendering, the plugin / API 195 can provide ambient lighting data or texture references from a material database.

[0048] The generative model 130 and / or other components of the generative model system 100 can use different types of neural network architectures depending on the particular implementation. For example, a transformer-based architecture, such as the architecture used in models like GPT, can be implemented, and the transformer-based architecture can include a self-attention mechanism that weighs the importance of different features in the input and / or a feed-forward network that processes the output of the self-attention layer, applies a non-linear transformation to the input representation, and / or extracts higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multi-modal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectures, such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, and so on. Thus, the embedding component 120 can apply the encoded representation of the input 101 to the generative model 130. The generative model 130 can process the encoded representation of the input 101 to generate the output 190, which can include one or more final output images.

[0049] As described herein, in some embodiments, the generative model 130 can be configured to access or use (or be able to access or use) plugins / APIs 195 (which can include one or more plugins, application programming interfaces (APIs), databases, data repositories, repositories, and so on). For example, for certain tasks or operations that are not ideally suited for the generative model 130, the model can have instructions for accessing one or more plugins / APIs 195 (e.g., third-party plugins) to assist in processing the current input (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using the retrieval component 192). Thus, the model can rely not only on its own knowledge obtained from training on large datasets and / or data retrieved using the retrieval component 192, but also on the specialized knowledge or optimized properties of one or more external resources, such as the plugins / APIs 195.

[0050] Reference Figure 2 , Figure 2An example system for feature cache based generative video editing is shown in accordance with some implementations of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted and / or others added in various places, as is Figure 1 , 7 and 8), one or more computing devices or components thereof (e.g., as shown in Figure 4 and 5 ), and / or one or more data centers or components thereof (e.g., as shown in Figure 6 ).

[0051] The system 200 can process an input video 202. The input video 202 can contain one or more input frames 210, 220, 230, 240, 250, and 260. The input video 202 can be a video stream. The input video 202 can be a video file (e.g.,.mp4,.mov). The input video 202 can contain real-time input frames.

[0052] The system 200 can contain at least one frame processor 204 to process the input video 202. The frame processor 204 can comprise or be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or any combination thereof. The frame processor 204 can process one or more input frames of the input video 202. The frame processor 204 can process frames in a streaming manner, thereby supporting an unlimited number of input frames. The frame processor 204 can select frames to edit. For example, the frame processor 204 can edit every other frame, such that editing of the remaining frames is skipped (e.g., frames 210, 230, and / or 250 are selected for editing). The interval at which the frame processor 204 selects frames can be predetermined, e.g., based at least on a target frame generation speed and / or a target frame resolution. The frame processor 204 can provide the selected frames as input to the frame editor 208.

[0053] The system 200 can include at least one feature cache 206. The feature cache 206 can store (e.g., maintain, archive) features from one or more input frames 202. The feature cache 206 can condition a diffusion-based model (e.g., frame editor 208) to generate one or more temporally consistent edited frames. The feature cache 206 can store one or more features (e.g., embeddings 214, 254, 264) of the input video 202. The feature cache 206 can include one or more slots 248 for storing one or more features, such that features (e.g., temporal and / or spatial embeddings) are stored together in the feature cache 206 (e.g., for quick access by the system 200) while maintaining a distinction between features corresponding to individual frames in the one or more input frames 202 through each individual slot 248. The capacity of the feature cache 206, which represents the number of slots 248 in the feature cache 206 that can be used to store features, can be predetermined based at least on available computer resources, target frame generation speed, and / or target frame resolution. The system 200 can assign each slot 248 a respective index. The capacity of the feature cache 206 can correspond to a maximum number of features that can be stored, e.g., a maximum number of input frames whose features can be stored. The feature cache 206 can store a capacity of 5, 8, 10, or a different number of frames of features. The feature cache 206 can not be at capacity (e.g., one or more slots 248 are empty), such as when the system 200 receives the first few frames as input.

[0054] The frame processor 204 can extract features from one or more input frames. The frame processor 204 can include an embedding component 120 or be associated with an embedding component 120 in Figure 1 Figure 1 ​The embedding components 120 in the frame buffer 202 are coupled to the frame processor 204. The frame processor 204 can process one or more input frames by extracting visual features using one or more neural network layers, such as convolutional layers, fully connected layers, residual blocks, and / or attention-based mechanisms. The one or more neural network layers can be the same as or different from the one or more neural network layers of the diffusion-based model of the frame editor 208. The extracted features can correspond to localized spatial patterns within the input frames and can be represented as multi-dimensional tensors (e.g., with a shape corresponding to a number of channels, height, and width). The frame processor 204 can generate one or more embeddings 214, 254, and / or 264 by applying dimensionality reduction operations, such as global average pooling, flattening, and / or projection through a fully connected layer, thereby converting the extracted features (e.g., feature maps) to embeddings (e.g., dense, fixed-length vector representations). The embeddings can include keys, queries, and / or values for the input frames. The frame processor 204 can store the embeddings 214, 254, and / or 264 in the feature cache 206 at intervals. For example, the feature cache can store an embedding for one frame every N frames (e.g., N = 2, N = 4, N = 6, or any interval).

[0055] The feature cache 206 can assign one or more weights to the cached embeddings to prioritize features from newer frames. The feature cache 206 can linearly decrease the weights of older frame features in the feature cache 206 (e.g., dial down the weights of older frame features in the feature cache 206). Based at least on the frame processor 204 adding information to the feature cache 206, the feature cache 206 can apply temporal embedding weights, denoted as TE j , where j corresponds to an index of a temporal embedding (e.g., a feature for a frame) in the feature cache 206. For example, the feature cache 206 can determine the temporal embedding weight for a feature for a frame j in the feature cache 206 as: j

[0056]

[0057] where A is the “age” of the frame feature, starting from an “age” of 0 for the current frame, and S is a scaling factor that adjusts the impact of the temporal embedding. The feature cache 206 can apply the temporal embedding weights to the embeddings, for example, by summing and / or concatenating the temporally weighted vector representations (e.g., temporal encodings). The feature cache 206 can apply the temporal embedding weights to the embeddings by multiplying the embeddings by the temporal embedding weights. In this example, the maximum value of the temporal embedding weights is 1.0 (e.g., high importance) for the features for the current frame in the feature cache 206. By applying the temporal embedding weights to the features in the feature cache, the system 200 can decrease the importance of features for older frames in the self-attention, thereby enhancing local frame consistency and preserving global frame consistency.

[0058] ​Feature cache 206 can reach its capacity, for example, such that each slot 248 is filled. Feature cache 206 can remove one or more embeddings to prevent exceeding the cache's capacity. For example, once the cache size exceeds the capacity of M frames (e.g., M=8), feature cache 206 can remove frames from the cache, so feature cache 206 can store embeddings of up to 8 frames, with each frame's embedding stored in its corresponding slot. At least based on the receipt of a new input frame, feature cache 206 can remove one or more embeddings of a frame according to the association weights assigned to one or more embeddings. For example, feature cache 206 can remove embeddings of a frame according to the lowest association weight (e.g., TE). j =0.01) Remove features. This results in empty slots (e.g., leaving 7 slots still full). Feature cache 206 can linearly reduce the allocation weight of one or more stored embeddings, thereby indicating that the duration of each of the remaining one or more embeddings in the cache increases. Feature cache 206 can store one or more embeddings of a new input frame in the emptied slots.

[0059] System 200 may include at least one frame editor 208 for receiving one or more frames as input and generating one or more edited frames. Frame editor 208 may include one or more generative models (e.g., diffusion-based models). Generative models may include encoder and / or decoder blocks. These blocks may include residual convolutional units and / or transformer modules. Transformer modules may include self-attention layers, cross-attention layers, and / or feedforward networks. Unlike systems that apply batch-based methods (e.g., denoising multiple frames simultaneously, extending self-attention layers to cross-frame attention, and / or employing label merging), system 200 may extend self-attention to feature cache 206, thereby incorporating information from past frames and / or the predicted next frame, as further described herein. For example, frame editor 208 may receive (e.g., a real object, person, animal, and / or environment) input frame 250 and apply a diffusion-based generative model to convert frame 250, for example, into a claymation-style rendering. This model may iteratively denoise the latent representation of the frame using encoder and / or decoder blocks. The encoder block can extract hierarchical features using residual convolutional units and / or transformer modules containing self-attention and / or cross-attention layers. Frame editor 208 can extend the self-attention layer using one or more embeddings (e.g., embeddings 214 and / or 254) from feature cache 206 by manipulating (e.g., concatenating, summing, and / or multiplying) one or more embeddings 254 from input frame 250 with one or more stored embeddings 214 from feature cache 206 (e.g., keys, queries, and / or values). This enables frame editor 208 to generate a temporally consistent output frame 252. One or more stored embeddings 214 can be temporally weighted embeddings (e.g., temporal embeddings) such that newer temporal embeddings with higher weights from feature cache 206 have a greater influence on self-attention compared to older temporal embeddings with lower weights from feature cache 206. Conditional data (e.g., style cues) can be merged via cross-attention layers. The decoder block can reconstruct edited (e.g., claymation-styled) frames as noise is gradually eliminated. The frame editor 208 can store one or more edited frames in the past frame buffer 216. The frame editor 208 can provide edited frames as input to the frame interpolator 218. The frame editor 208 can output edited frames 252 to the output video 224.

[0060] System 200 may include at least one past frame buffer 216 for storing one or more previously edited frames. Past frame buffer 216 may allow system 200 to skip frames (e.g., every other frame) when performing diffusion-based editing. System 200 may use past frame buffer 216 to generate one or more interpolated frames between edited frames, thereby reducing the time required to generate video output and / or reducing flicker in output video 224. Interpolated frames (e.g., interpolated frames 222 and 242) may correspond to skipped frames (e.g., frames 220 and 240), respectively. Past frame buffer 216 may include a predetermined number of slots (e.g., one available slot). Past frame buffer 216 may store, for example, the most recently edited frame 232. Frame interpolator 218 may retrieve the most recently edited frame 232. Frame interpolator 218 may perform interpolation of a new frame 242 between frame 232 and the currently edited frame 252.

[0061] System 200 may include at least one frame interpolator 218 for generating one or more frames (e.g., interpolated frames 222 and / or 242) using interpolation techniques. Frame interpolator 218 may contain one or more neural networks, such as one or more neural network architectures, such as convolutional neural networks (CNNs), residual networks (ResNet), U-Net architectures, transformer-based models, and / or recurrent neural networks (RNNs). Frame interpolator 218 may apply one or more interpolation techniques (e.g., RIFE, Recurrent Full-Field Transform (RAFT), FlowNet, and / or accelerated frame interpolation methods, such as those used in NVIDIA Deep Learning Frame Generation (DLFG) and / or NVIDIA Deep Learning Performance Presets (DLPP)) to generate one or more frames based at least on edited frames from past frame buffers 216. Compared to systems that edit each frame individually, the systems and methods of this paper can interpolate frames between edited frames while preserving spatial, temporal, and / or contextual frame information, thereby increasing frames per second (fps) while reducing computational resources (e.g., memory).

[0062] For example, frame interpolator 218 can retrieve the currently edited frame 252 and the previously edited frame 232. Frame interpolator 218 can retrieve frames 252 and 232 from frame editor 208 and past frame buffer 216, respectively. Frame interpolator 218 can apply at least one neural network (e.g., IFNet) to estimate an intermediate stream corresponding to the midpoint between the two frames. This intermediate stream can be used to warp the first and second frames toward the intermediate timestamp (e.g., moving corresponding pixels in the input frames to the same position in the potential intermediate frame). The fusion module can combine the warped frames to synthesize an interpolated frame (e.g., combining pixels from the two input frames).

[0063] System 200 can generate output video 224. Output video 224 may contain one or more edited frames. Output video 224 may contain one or more interpolated frames. In some examples, every other frame of the output video is an edited frame, such as edited frames 212, 232, and 252. In these examples, the remaining frames are interpolated frames, such as interpolated frames 222 and 242.

[0064] In some examples, the interpolated frames of the output video 224 correspond to the positions of frames skipped by the frame processor 204 for editing. For example, with three input frames 210, 220, and 230, the frame processor 204 can provide the first frame 210 and the third frame 230 as input to the frame editor 208, skipping the second frame. In this example, although the second frame 220 is skipped for editing, the frame processor 204 can extract one or more embeddings of the second frame and store them in the feature buffer 206. The edited first frame 212 can correspond to the first frame 210. The edited third frame 232 can correspond to the third frame 230. The intermediate timestamp associated with the interpolated frame 222 can be associated with the timestamp corresponding to the skipped frame 220.

[0065] System 200 may include at least one frame predictor 226 for predicting one or more frames. The frame predictor may contain one or more generative models. Frame predictor 226 may apply optical flow techniques to predict optical flow based at least on one or more frames of the output video 224 and the next unprocessed frame 260. Frame predictor 226 may generate predicted frames 262 based at least on predicted motion. Frame predictor 226 may store one or more embeddings of predicted frames 262 in feature cache 206, allowing system 200 to apply information from past, present, and future frames from feature cache 206 to increase temporal coherence in the frame generation task.

[0066] For example, in order to determine the representation of the current frame C i (For example, edited frame 252) and the next frame N to be processed. i (e.g., motion (e.g., optical flow) F between two reference frames of the unprocessed input frame 260) i Frame predictor 226 can predict the current frame C i and the next frame N to be processed i Convert to grayscale frame GC i and GN i ,in:

[0067] GC i =RGB2Gray(C i )

[0068] GN i =RGB2Gray(Ni )

[0069] To determine the optical flow F i Frame predictor 226 can calculate GC i and GN i Pixel-level motion (e.g., displacement vector field) between frames. Frame predictor 226 can use optical flow F i Twisted GC i To the predicted next frame NP i (For example, predict frame 262). For example:

[0070] NP i =GC i (x+F i (x))

[0071] Where x represents frame GC i The coordinates, F i (x) represents frame GC i The optical flow vector at the coordinates of . The frame predictor 226 can predict the frame NP i Features are attached to feature cache 206 and combined with information from predicting future frames to ensure temporal consistency of frame generation (e.g., generated by frame editor 208).

[0072] Now for reference Figure 3 , Figure 3 This is a flowchart illustrating an example of method 300 for generating frames using a cache. Each block of method 300 described herein contains a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors executing instructions stored in one or more memories. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services), provided as a microservice via an application programming interface (API) or a plug-in to another product, and so on. Furthermore, regarding... Figure 2 The system described herein illustrates method 300 by way of example. However, this method may be performed additionally or alternatively by any system or combination of systems, including but not limited to the system described herein.

[0073] Method 300 at block 302 may include receiving one or more input frames as input, such as from a live video source that supports an unlimited number of input frames. These one or more input frames may be received via computing device 500. Figure 5 The I / O component 514 receives the input frames. Method 300 may include real-time processing of the one or more input frames.

[0074] Method 300 at box 304 may include: applying a first frame from one or more input frames as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model. The machine learning model may include one or more neural networks (e.g., CNN). The one or more neural networks may include one or more neural network layers, such as convolutional layers, residual blocks, and / or attention-based mechanisms. Method 300 at box 304 may include preprocessing the first frame (e.g., raw input data), such as resizing the first frame to a target resolution, converting the color space (e.g., RGB to YUV), and / or normalizing pixel values ​​to a predefined range (e.g., 0 to 1). Method 300 at box 304 may include: using one or more convolutional layers to extract low-level features of the first frame, such as edges, corners, and / or textures. Method 300 at box 304 may include: representing the low-level features as one or more feature maps. Method 300 at box 304 may include: applying a non-linear activation function (e.g., ReLU) and / or normalization operations (e.g., batch normalization, layer normalization) to the feature maps to stabilize the feature distribution of the feature maps. Method 300 at box 304 may include using one or more residual blocks to refine the feature maps to extract intermediate and / or high-level features. Method 300 at box 304 may include: applying one or more attention-based mechanisms (such as self-attention mechanisms) to refine the feature maps by capturing long-range spatial dependencies. Method 300 at box 304 may include: performing downsampling operations (e.g., max pooling and / or stride convolution) on one or more feature maps to reduce the resolution of one or more feature maps, for example, to produce one or more embeddings containing a first embedding (e.g., a fixed-length vector embedding). In some embodiments, method 300 at box 304 may include: dividing one or more feature maps into spatial patches and projecting each patch onto a low-dimensional vector space to generate one or more embeddings. One or more embeddings can encode semantic, spatial, temporal, and / or contextual information. Method 300 at box 304 may include: generating a temporal code indicating the time step of the first frame in one or more input frames. Method 300 at box 304 may also include: adding the temporal code to the first embedding, such as by summing and / or concatenating the first embedding with the temporal code. The resulting embedding can serve as a compact, high-level representation of the input frames for use in downstream processing tasks.

[0075] Method 300 at block 306 may include: storing a first embedding in a cache, wherein the cache contains a second embedding of the second frame. Method 300 at block 306 may include: storing frames at intervals. For example, method 300 at block 306 may include: storing the first embedding in the cache at least based on an interval between the second frame and the first frame. This interval can be any number of frame intervals (e.g., 2 frames, 4 frames, 5 frames). For example, for a 4-frame interval, where the second frame occurs at time step 1, where time step 1 corresponds to the first frame in the frame sequence, and the second embedding of the second frame is stored in the cache, and the first frame occurs at timestamp 5, where timestamp 5 corresponds to the fifth frame in the frame sequence, method 300 at block 306 may include: storing the first embedding of the first frame in the cache. Method 300 at block 306 may include: assigning a first weight to the first embedding and assigning a second weight to the second embedding, the first weight and the second weight being determined based on their respective durations of embedding in the cache.

[0076] Method 300 at box 308 may include: generating a third frame using a machine learning model, at least based on a cache, where the third frame is associated with the first frame. The model may include a neural network architecture (e.g., U-Net). Method 300 at box 308 may include: processing the current frame (e.g., the first frame) using the machine learning model. The machine learning model may include encoder blocks and decoder blocks. Each encoder or decoder block may include residual convolutional units and / or transformer modules. Transformer modules may include self-attention layers, cross-attention layers, and / or feedforward networks. Method 300 at box 308 may include: extending the self-attention layer of each transformer module to incorporate one or more embeddings from the cache. For example, method 300 at box 308 may include: concatenating one or more current embeddings of the current frame (e.g., the first embedding of the first frame) with embeddings stored in the cache (e.g., the second embedding of the second frame), thereby enabling the model to generate a third frame (e.g., the output frame). The generated third frame can therefore exhibit temporal and stylistic consistency with previous frames without batch processing all frames simultaneously.

[0077] Method 300 at box 310 may include: outputting a third frame to a video stream containing a fourth frame. The fourth frame may be a frame previously generated by a machine learning model. Method 300 at box 310 may include: generating a fifth frame based at least on the fourth and third frames. Method 300 at box 310 may include: performing interpolation techniques (e.g., RIFE, RAFT, FlowNet, DLFG, and / or DLPP) to generate the fifth frame. Method 300 at box 310 may include: outputting the fifth frame between the fourth and third frames to make the output video stream temporally coherent.

[0078] Method 300 at box 312 may include: providing output video stream data for display on a display device. Method 300 at box 312 may include: displaying the output video to a user. For example, method 300 at box 312 may include: for a user wearing an augmented reality device, pointing the device's lens at the sky in cloudy weather to present a processed video stream of sunshine and clear skies, maintaining temporal coherence and / or other contextual coherence.

[0079] Example content streaming system

[0080] Now for reference Figure 4 , Figure 4 This is an example system diagram of a content streaming system 400 according to some embodiments of this disclosure. Figure 4 Including application server 402 (which may include with Figure 5 Example computing device 500 (similar components, features and / or functions), client device 404 (which may include similar components, features and / or functions to...) Figure 5 The example computing device 500 may have similar components, features, and / or functions, as well as a network 406 (which may be similar to the network described herein). In some embodiments of this disclosure, system 400 may be implemented. Application sessions may correspond to game streaming applications (e.g., NVIDIA GeForce), remote desktop applications, simulation applications (e.g., automated or semi-automatic vehicle simulations), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other types of applications.

[0081] In system 400, for an application session, one or more client devices 404 may, in response to input from one or more input devices, simply receive input data, send the input data to one or more application servers 402, receive encoded display data from one or more application servers 402, and display the display data on display 424. Therefore, computationally intensive computation and processing are offloaded to one or more application servers 402 (e.g., rendering of the graphics output for the application session—specifically ray or path tracing—is performed by one or more GPUs of one or more game servers 402). In other words, the application session is streamed from one or more application servers 402 to one or more client devices 404, thereby reducing the graphics processing and rendering requirements of one or more client devices 404.

[0082] For example, for an instance of an application session, client device 404 can display frames of the application session on display 424 based on display data received from application server 402. Client device 404 can receive input from one of the input devices and generate input data in response. Client device 404 can transmit input data to application server 402 via communication interface 420 and via network 406 (e.g., the Internet), and application server 402 can receive input data via communication interface 418. CPU can receive input data, process input data, and transmit data to GPU, causing GPU to generate a rendering of the application session. For example, input data can represent user character movement, weapon firing, reloading, passing, vehicle turning, etc., in a game session of a game application. Rendering component 412 can render the application session (e.g., the result representing the input data), while rendering capture component 414 can capture the rendering of the application session as display data (e.g., image data as captured rendering frames of the application session). Rendering of the application session may include lighting and / or shadow effects using ray or path tracing, which are computed using one or more parallel processing units (such as GPUs) of the application server 402. These units may also use one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques. In some implementations, the application server 402 may use one or more virtual machines (VMs) (e.g., including one or more virtual components such as vGPUs, vCPUs, etc.) to support the application session. The encoder 416 may then encode the display data to generate encoded display data, which may be transmitted to the client device 404 via the network 406 through the communication interface 418. The client device 404 may receive the encoded display data via the communication interface 420, and the decoder 422 may decode the encoded display data to generate display data. The client device 404 may then display the display data via the display 424.

[0083] The systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing and / or any other suitable application.

[0084] The disclosed implementations can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0085] Example language model

[0086] In at least some implementations, language models such as Large Language Models (LLM), Small Language Models (SLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or similar content based on context provided in input prompts or queries. In implementations, these language models can be considered “large” based on the fact that the model is trained on massive datasets and has an architecture containing a large number of learnable network parameters (weights and biases) (e.g., millions or billions of parameters). LLMs / VLMs / MMLMs / etc. can be implemented to aggregate textual data, analyze and extract insights from data (e.g., text, images, videos, etc.), and generate new text / images / videos, etc., in user-specified styles, tones, and / or formats. The LLM / SLM / VLM / MMLM / etc. disclosed herein may be specifically designed for text processing in some embodiments, while in other embodiments, a multimodal LLM may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format) and / or video. For example, a visual language model (VLM), or more generally a multimodal language model (MMLM), may be implemented to accept images, video, audio, text, 3D designs (e.g., CAD) and / or other input data types, and / or generate or output images, video, audio, text, 3D designs and / or other output data types.

[0087] Various types of LLM / SLM / VLM / MMLM / etc. architectures can be implemented in a variety of embodiments. For example, different architectures can be implemented that use different techniques to understand and generate outputs such as text, audio, video, images, 2D and / or 3D design or asset data. In some embodiments, LLM / SLM / VLM / MMLM / etc. architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can be used; while in other embodiments, transformer architectures—such as those relying on self-attention and / or cross-attention mechanisms (e.g., between contextual data and textual data)—can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines incorporating LLM / SLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction units). The LLM / SLM / VLM / MMLM / etc. disclosed herein may include encoder and / or decoder blocks. For example, discriminative models or encoder-only models, such as BERT (Bidirectional Encoder Representation from Transformer), can be implemented for tasks involving language understanding (such as classification, sentiment analysis, question answering, and named entity recognition). As another example, generative models or decoder-only models, such as GPT (Generative Pre-trained Transformer), can be implemented for tasks involving language and content generation (such as text completion, story generation, and dialogue generation). LLM / SLM / VLM / MMLM / etc., which include both encoder and decoder components, can be implemented, such as T5 (Text-to-Text Transformer), to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting; any architecture type (including but not limited to those described herein) can be implemented depending on the specific implementation and the task performed using LLM / SLM / VLM / MMLM / etc.

[0088] In various implementations, LLM / SLM / VLM / MMLM / etc. can be trained using unsupervised learning, where LLM / SLM / VLM / MMLM / etc. learn patterns from large amounts of unlabeled text / audio / video / image / design / USD / etc. data. Due to the large amount of training, in some implementations, the model may not need to be trained for a specific task or domain. LLM / SLM / VLM / MMLM / etc. that have already been extensively pre-trained on large amounts of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / SLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., fine-tuning or adjusting cue or tagging to bias the language model towards a specific task or domain), and / or using other fine-tuning or customization techniques to optimize the model for a specific task and / or domain.

[0089] In some implementations, the LLM / SLM / VLM / MMLM / etc. of this disclosure can be implemented using various model alignment techniques. For example, in some implementations, guardrails can be implemented to identify inappropriate or unintended inputs (e.g., prompts) and / or outputs of the model. To this end, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unintended inputs using LLM / SLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated using LLM / SLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some implementations, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "safety" models can be trained to identify inputs and / or outputs that are "safe," acceptable, or desirable for a particular application / implementation, and / or "unsafe" or undesirable inputs and / or outputs. Therefore, the LLM / SLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope and / or undesirable for a particular application / implementation.

[0090] In some implementations, LLM / SLM / VLM / etc. can be configured or enabled to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, if at least part of the prompt is related to a restaurant or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. As another example, if at least part of the response requires mathematical calculations, the model can access one or more mathematical plugins or APIs to help solve the problem, and then use the response from the plugins and / or APIs in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each input prompt for each question / question / request / process / operation, etc., is generated. Therefore, a model can rely not only on the knowledge it gains from training on a large dataset, but also on the expertise or optimized features of one or more external resources (such as APIs, plugins, etc.).

[0091] In some implementations, multiple language models (e.g., LLM / SLM / VLM / MMLM, etc., multiple instances of the same language model, and / or multiple prompts provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data repositories, stores, etc.) to provide output in response to a unified query or in response to different parts of a query. In at least one implementation, the same input query and prompts (e.g., constraint sets, conditioners, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) corpora). In one or more implementations, the language models can be different versions of the same base model. In one or more implementations, at least one language model can be instantiated as multiple agents—e.g., more than one prompt can be provided to constrain, guide, or otherwise influence the style, content, or characters of the provided output. In one or more example non-limiting implementations, the same language model can be required to provide output corresponding to different roles, perspectives, personalities, or different knowledge bases—as defined by the provided prompts.

[0092] In any such implementation, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instance proxies of at least one language model, and / or the outputs provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more implementations, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more implementations, the language model can be required to generate or otherwise acquire output associated with the input source material. This association can include, for example, generating a title or portion of text embedded (e.g., as metadata) into the input source text or image. In one or more implementations, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence of a target word in a portion of text or an object in an image, and to annotate the text or image to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in a selected dataset, for example, but not limited to this.

[0093] Example computing device

[0094] Figure 5 This is a block diagram of an example computing device 500 suitable for implementing some embodiments of the present disclosure. The computing device 500 may include an interconnect system 502 directly or indirectly coupled to the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, input / output (I / O) ports 512, input / output components 514, a power supply 516, one or more presentation components 518 (e.g., one or more displays), and one or more logic units 520. In at least one embodiment, one or more computing devices 500 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 508 may include one or more vGPUs, one or more CPUs 506 may include one or more vCPUs, and / or one or more logic units 520 may include one or more virtual logic units. Accordingly, one or more computing devices 500 may include discrete components (e.g., a full GPU dedicated to computing device 500), virtual components (e.g., a portion of the GPU dedicated to computing device 500), or a combination thereof.

[0095] althoughFigure 5 The various boxes are shown as being connected to lines via interconnect system 502, but this is not intended to be limiting and is merely for clarity. For example, in some implementations, presentation component 518 (such as a display device) may be considered I / O component 514 (e.g., if the display is a touchscreen). As another example, CPU 506 and / or GPU 508 may include memory (e.g., memory 504 may represent a storage device in addition to the memory of GPU 508, CPU 506, and / or other components). In other words, Figure 5 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 5 Within the scope of computing devices.

[0096] Interconnect system 502 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 502 may include one or more bus or link types, such as Industry Standard Architecture (ISA) buses, Extended Industry Standard Architecture (EISA) buses, Video Electronics Standards Association (VESA) buses, Peripheral Component Interconnect (PCI) buses, Fast Peripheral Component Interconnect (PCIe) buses, and / or another type of bus or link. In some implementations, there is a direct connection between components. For example, CPU 506 may be directly connected to memory 504. Further, CPU 506 may be directly connected to GPU 508. In cases where there is a direct connection or point-to-point connection between components, interconnect system 502 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 500.

[0097] The memory 504 may include any of a variety of computer-readable media. The computer-readable media may be any available medium accessible by the computing device 500. The computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0098] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 504 may store computer-readable instructions (e.g., representing programs and / or program elements, such as operating systems). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 500. As used herein, computer storage media does not include the signal itself.

[0099] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0100] CPU 506 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. Each of the one or more CPUs 506 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of handling multiple software threads simultaneously. The one or more CPUs 506 may include any type of processor and may include different types of processors depending on the type of computing device 500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 500, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 500 may also include one or more CPUs 506.

[0101] In addition to or replacing CPU 506, one or more GPUs 508 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. One or more GPUs 508 may be integrated GPUs (e.g., having one or more CPUs 506) and / or one or more GPUs 508 may be discrete GPUs. In embodiments, one or more GPUs 508 may be a coprocessor of one or more CPUs 506. GPUs 508 may be used by computing device 500 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 508 may be used for general-purpose computing on a GPU (GPGPU). GPUs 508 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 508 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 506 via a host interface). GPU 508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 504. GPU 508 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 508 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0102] In addition to or replacing CPU 506 and / or GPU 508, one or more logic units 520 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 506, one or more GPUs 508, and / or one or more logic units 520 may perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 520 may be part of and / or integrated into one or more CPUs 506 and / or GPUs 508, and / or one or more logic units 520 may be discrete components or otherwise external to CPUs 506 and / or GPUs 508. In embodiments, one or more logic units 520 may be coprocessors of one or more CPUs 506 and / or one or more GPUs 508.

[0103] Examples of logic unit 520 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0104] The communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 500 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 510 may include components and functions that enable communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.) and / or the Internet). In one or more embodiments, the logic unit 520 and / or the communication interface 510 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 502 to one or more GPUs 508 (e.g., memory of one or more GPUs 508).

[0105] I / O port 512 enables computing device 500 to be logically coupled to other devices including I / O component 514, presentation component 518, and / or other components, some of which may be built into (e.g., integrated into) computing device 500. Illustrative I / O component 514 includes microphone, mouse, keyboard, joystick, gamepad, game controller, disc satellite dish, scanner, printer, wireless device, etc. I / O component 514 provides a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some instances, the input may be sent to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 500 (as described in more detail below). The computing device 500 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 500 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 500 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0106] Power supply 516 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 516 may provide power to computing device 500 so that the components of computing device 500 can operate.

[0107] One or more presentation components 518 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 518 may receive data from other components (e.g., GPU 508, CPU 506, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0108] Example Data Center

[0109] Figure 6 An example data center 800 that can be used in at least one embodiment of this disclosure is shown. The data center 800 may include a data center infrastructure layer 610, a framework layer 620, a software layer 630, and / or an application layer 640.

[0110] like Figure 6As shown, the data center infrastructure layer 610 may include a resource coordinator 612, grouped computing resources 614, and node computing resources (“nodes CR”) 616(1)-616(N), where “N” represents any integer, a positive integer. In at least one embodiment, the nodes CR 616(1)-616(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW”) devices, and network network interfaces (“NW”). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some implementations, one or more nodes CR616(1)-616(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some implementations, nodes CR616(1)-616(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR616(1)-616(N) may correspond to virtual machines (VMs).

[0111] In at least one embodiment, the grouped computing resources 614 may include separate groups of nodes CR616 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The separate groups of nodes CR616 within the grouped computing resources 614 may include grouped computing resources, network resources, memory resources, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0112] Resource coordinator 612 can be configured or otherwise controlled to control one or more nodes CR616(1)-616(N) and / or grouped computing resources 614. In at least one embodiment, resource coordinator 612 may include a Software Design Infrastructure (“SDI”) management entity for data center 600. Resource coordinator 612 may include hardware, software, or some combination thereof.

[0113] In at least one implementation, such as Figure 6As shown, framework layer 620 may include a job scheduler 628, a configuration manager 634, a resource manager 636, and / or a distributed file system 638. Framework layer 620 may include a framework for software 632 supporting software layer 630 and / or one or more applications 642 of application layer 640. Software 632 or application 642 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 620 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 638 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM (Hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 628 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of data center 600. Configuration manager 634 may be able to configure different layers, such as software layer 630 and framework layer 620 including Spark and distributed file system 638 for supporting large-scale data processing. Resource manager 636 may be able to manage cluster or grouped computing resources mapped to or allocated for supporting distributed file system 638 and job scheduler 628. In at least one embodiment, cluster or grouped computing resources may include grouped computing resources 614 at data center infrastructure layer 610. Resource manager 636 may coordinate with resource coordinator 612 to manage these mapped or allocated computing resources.

[0114] In at least one embodiment, the software 632 included in software layer 630 may include software used by at least a plurality of portions of nodes CR616(1)-616(N), grouped computing resources 614, and / or the distributed file system 638 of framework layer 620. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0115] In at least one embodiment, the application 642 included in the application layer 640 may include one or more types of applications used by at least a plurality of portions of nodes CR616(1)-616(N), grouped computing resources 614, and / or the distributed file system 638 of the framework layer 620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments.

[0116] In at least one implementation, any of the configuration manager 634, resource manager 636, and resource coordinator 612 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free the data center operator of data center 600 from making potentially undesirable configuration decisions and potentially avoid underutilized and / or poorly performing portions of the data center.

[0117] According to one or more embodiments described herein, data center 600 may include tools, services, software, or other resources for training / updating one or more machine learning models or for using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 600. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 600 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0118] In at least one implementation, the data center 600 may use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to train or execute information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0119] Example network environment

[0120] A suitable network environment for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5 The implementation is carried out on one or more instances of computing device 500, for example, each device may include similar components, features and / or functions of computing device 500. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 600, the examples of which in this document are relative to... Figure 6 To describe in more detail.

[0121] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. A network can include multiple networks or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0122] A compatible network environment may include one or more peer-to-peer network environments—in which case the network environment cannot include a server—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0123] In at least one implementation, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework of software and / or application layers supporting the software layer and / or application layer. The software or application may include web-based service software or applications, respectively. In an implementation, one or more client devices may use the web-based service software or application (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, free and open-source software web application frameworks that can use distributed file systems for large-scale data processing (e.g., "big data").

[0124] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions may be distributed across multiple locations from a central or core server (e.g., across one or more data centers distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0125] One or more client devices may be included in this discussion. Figure 5 At least some of the components, features, and functions of one or more example computing devices 500 described. By way of example and not limitation, a client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0126] Figure 7 This is a block diagram of an example implementation where the generative model 730 includes a transformer-based encoder-decoder architecture for the diffusion process. For example, suppose the input image frame is encoded (e.g., by...) Figure 1 The embedded component 120 in the model is encoded into a corresponding embedding (e.g., of size 512). The techniques described herein can be used to add temporal encoding and / or time-step encoding to each embedding to represent the temporal relationships between frames and the progress of the overall denoising scheme. Thus, the generated embeddings can be applied as part of the iterative denoising process of the diffusion model to one or more encoders 735 of the generative model 730.

[0127] In an example implementation, encoder 735 forms an encoder stack, where each encoder contains a self-attention layer and a feedforward network. In the example transformer architecture for image processing, each token embedding flows through a separate path. Thus, each encoder can accept a sequence of embedding vectors, pass each vector through the self-attention layer, then through the feedforward network, and then to the next encoder in the stack. Any suitable self-attention technique can be used. For example, to compute self-attention for each token, a query vector, a key vector, and a value vector can be generated from the token embeddings; the attention score can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying it by the value vector, and summing the weighted value vectors. The encoder can apply multi-head self-attention, where attention operations are performed in parallel on multiple learned projections. Any number of encoders can be stacked to generate context vectors for encoding the input. Attention projection layer 740 can transform the context vectors into attention vectors (e.g., keys and values) for decoder 745.

[0128] In the example implementation, decoder 745 constitutes a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer (which uses attention vectors (keys and values) from encoder 735 to focus on relevant parts of the input), and a feedforward network. Similar to encoder 735, in the example transformer architecture, each token embedding flows through a separate path in decoder 145. At each step of the iterative denoising process, decoder 745 may receive a noisy latent representation corresponding to an input frame at a specific time step of the diffusion schedule. Decoder 745 may use self-attention to model intra-frame dependencies and encoder-decoder attention to integrate contextual information from encoder 735. Decoder 745 may then output a denoised latent representation. This denoised representation may be passed to generation mechanism 755, which may update the latent representation based on predicted noise or directly predict clean data samples, depending on the diffusion model formulation. The process may be repeated a predetermined number of steps (or until a convergence threshold is reached), iteratively refining the latent representation at each step. Decoders may optionally incorporate temporal step embeddings and / or spatial location encodings during each denoising step.

[0129] Therefore, decoder 745 can output an updated latent representation of the image being processed at each denoising step. Generation mechanism 755 can apply this output to compute the next state in the denoised sequence, progressively reducing noise at each time step until a final clean output image is generated. Classifier 750 can optionally be used to perform auxiliary tasks (e.g., image content classification or time step prediction) and can contain one or more neural network layers to project the decoded representation onto the target dimension. In some implementations, generation mechanism 755 can implement a diffusion sampling procedure to traverse the inverse process in the latent space from the initial noisy input to the final denoised output.

[0130] Figure 8 This is a block diagram of an example implementation, where generative model 130 includes a decoder-only transformer architecture for a diffusion-based image processing system. For example, Figure 8 The operation method of the decoder 160 and Figure 7 The decoder in the 745 is similar, but the difference is... Figure 8 Each decoder 860 in the architecture omits the encoder-decoder self-attention layer because no separate encoder is used in this architecture. Therefore, the decoders 860 can form a decoder stack, where each decoder contains a self-attention layer and a feedforward network. Instead of processing discretely labeled input sequences, the decoders 860 can receive latent representations of noisy image frames at a given time step of a diffusion schedule. Temporal step encoding and / or spatial location encoding can be applied to the latent representation before it is input to the decoder 860.

[0131] andFigure 7 Similar to decoder 745, each embedding (e.g., corresponding to an image patch or spatial region) can flow through a separate path in decoder 860. Decoder 860 can be combined with classifier 865 and generation mechanism 870 to participate in the iterative denoising process, progressively refining the latent image representation in each diffusion step. In each step, decoder 860 can output an updated latent representation or predicted noise components, which generation mechanism 870 can apply to compute the latent state for the next time step. This process can be repeated for a predetermined number of time steps, gradually transforming the initial noise into a fully denoised image.

[0132] The operation of classifier 865 and generator 870 can be compared with... Figure 7 The classifier 850 and generative mechanism 855 are similar. For example, the classifier 865 may optionally project the decoded latent representation into a low-dimensional space to aid predictions such as class labels or denoising confidence scores. The generative mechanism 870 may apply the output of the decoder 860 to estimate the clean image for the current step or predict the noise to be removed, depending on the model configuration. The architectures described herein, and others, are illustrative only, and other suitable transformer-based or hybrid architectures may be implemented within the scope of this disclosure.

[0133] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, a program module (including routines, programs, objects, components, data structures, etc.) refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0134] As used herein, statements relating to two or more elements “and / or” should be understood as referring to only one element or a combination of elements. For example, “element A, element B and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B and C. Furthermore, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0135] This document provides a detailed description of the subject matter of this disclosure to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to those described herein, in conjunction with other existing or future techniques. Furthermore, although the terms “step” and / or “box” may be used herein to denote different elements of the method employed, these terms should not be construed as implying any particular order between the various steps disclosed herein, unless the order of the various steps is explicitly described.

Claims

1. One or more processors comprising processing circuitry to: apply a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model; store the first embedding in a cache, wherein the cache comprises a second embedding of a second frame; generate a third frame using the machine learning model based at least on the cache, wherein the third frame is associated with the first frame; and output the third frame to a video stream containing a fourth frame.

2. The one or more processors of claim 1, wherein the processing circuitry is to: store the first embedding in a first slot in the cache; and store the second embedding in a second slot in the cache.

3. The one or more processors of claim 1, wherein the first frame and the second frame are separated by an interval.

4. The one or more processors of claim 1, wherein the processing circuitry is to: interpolate a fifth frame based at least on the fourth frame and the third frame, wherein the fourth frame is previously generated; and output the fifth frame to the video stream, between the fourth frame and the third frame.

5. The one or more processors of claim 1, wherein, the processing circuitry is to remove a third embedding from the cache in accordance with a corresponding weight in response to determining that a number of stored embeddings exceeds a cache capacity.

6. The one or more processors of claim 1, wherein the processing circuitry is to: predict a fifth frame based at least on the third frame; and store a third embedding of the fifth frame in the cache.

7. The one or more processors of claim 1, wherein the processing circuitry is to assign a weight to at least one of the first embedding or the second embedding determined from a duration of a corresponding embedding in the cache.

8. The one or more processors of claim 1, wherein the fourth frame is associated with the second frame.

9. The one or more processors of claim 6, wherein the fifth frame is predicted using optical flow.

10. The one or more processors of claim 1, wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more small language models (SLMs); a system for performing operations using one or more visual language models (VLMs); Systems for performing operations using a multi-modal language model (MMLM); Systems for generating synthetic data; Systems for generating synthetic data using AI; Systems comprising one or more virtual machines (VMs); Systems using or deploying one or more inference microservices; Systems comprising one or more machine learning models deployed in a service or microservice and an operating system-level virtualization package; Systems implemented at least in part in a data center; or Systems implemented at least in part using cloud computing resources.

11. A method comprising: applying a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model; storing the first embedding in a first slot in a cache, wherein the cache includes a second embedding of a second frame in a second slot in the cache; generating a third frame using the machine learning model based at least on the cache, wherein the third frame is associated with the first frame; and outputting the generated third frame to a video stream comprising a generated fourth frame.

12. The method of claim 11, further comprising: interpolating a fifth frame based at least on the generated fourth frame and the generated third frame; and outputting the interpolated fifth frame to the video stream, between the generated fourth frame and the generated third frame.

13. The method of claim 11, further comprising: determining an optical flow between the third frame and a fifth frame, wherein the fifth frame comprises raw image data; predicting a sixth frame based at least on the optical flow; and storing a third embedding of the sixth frame in the cache. assigning a first weight to the first embedding, the first weight determined according to a respective duration of the first embedding in the cache. the first frame and the second frame are separated by an interval.

14. The method of claim 11, further comprising: the first embedding of the generated fourth frame is associated with a third embedding of the second frame.

15. The method of claim 11, wherein, extending a self-attention layer of the machine learning model based at least on the cache.

16. The method of claim 11, wherein, 18. The method of claim 11, further comprising:

17. The method of claim 11, further comprising: linearly decreasing a second weight of the second embedding based at least on storing the first embedding in the cache. removing the second embedding from the cache based at least on the second weight in response to determining that a number of stored embeddings exceeds a cache capacity.

20. A system comprising one or more processors to:

19. The method of claim 18, further comprising: apply a first frame as input to a machine learning model to retrieve a first embedding of the first frame from the machine learning model; store the first embedding in a first slot in a cache, wherein the cache includes a second embedding of a second frame in a second slot in the cache; generate a third frame using the machine learning model based at least on the cache, wherein the third frame is associated with the first frame; and output the third frame to a video stream comprising a generated fourth frame. ​ ​

Images