Efficient diffusion machine learning model

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. During a first iteration of processing data using a first de-noised backbone of a teacher diffusion machine learning model, a first potential tensor is generated using a lower resolution block of the first de-noised backbone. During a first iteration of processing data using a second de-noised backbone of a student diffusion machine learning model, a second potential tensor is generated using an adapter block of the second de-noised backbone. A loss is generated based on the first potential tensor and the second potential tensor, and one or more parameters of the adapter block are updated based on the loss.

Description

Highly efficient diffusion machine learning model

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 488,786, filed October 17, 2023, which is incorporated herein by reference. Background Technology

[0003] This disclosure relates to various aspects of machine learning.

[0004] Recently, a wide variety of machine learning architectures have been used to perform numerous tasks with high accuracy and reliability. For example, generative models have recently been used to generate image and / or video outputs based on text and other inputs. For instance, models have been trained to provide text-based image and / or video content generation, text-based image and / or video content editing, image and / or video enhancement (e.g., super-resolution and colorization), and image and / or video compression, among other things.

[0005] Various generative model architectures have been used. However, generative models, such as diffusion-based models, are generally computationally expensive. In addition to the high training costs, many generative models, such as diffusion models, rely on iterative inference or generation processes (e.g., reprocessing feature maps multiple times) to generate the output. For example, diffusion models (also known in some respects as backdiffusion models) typically use a backdiffusion step involving the execution of a computationally expensive denoising function. This step is often performed dozens of times to generate a single output image. In these ways, some conventional generative models consume significant computational resources while resulting in substantial latency in generating a single prediction. Summary of the Invention

[0006] Some aspects of this disclosure provide a processor-implemented method comprising: during a first iteration of processing data using a denoised backbone of a diffusion machine learning model, generating a first latent tensor using lower-resolution blocks of the denoised backbone; during the first iteration, generating a first feature tensor based on processing the first latent tensor using higher-resolution blocks of the denoised backbone, the higher-resolution blocks using a higher resolution than the lower-resolution blocks; generating a second latent tensor based on processing the first latent tensor using adapter blocks of the denoised backbone; and during a second iteration of processing the data using the denoised backbone, generating a second feature tensor based on processing the second latent tensor using the higher-resolution blocks.

[0007] Some aspects of this disclosure provide a processor-implemented method comprising: during a first iteration of processing data using a first denoised backbone of a teacher diffusion machine learning model, generating a first latent tensor using a lower-resolution block of the first denoised backbone; during a first iteration of processing data using a second denoised backbone of a student diffusion machine learning model, generating a second latent tensor using an adapter block of the second denoised backbone; generating a loss based on the first latent tensor and the second latent tensor; and updating one or more parameters of the adapter block based on the loss.

[0008] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0009] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0010] The accompanying drawings illustrate certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0011] Figure 1 illustrates an example workflow for an efficient generative machine learning model based on some aspects of this disclosure.

[0012] Figures 2A, 2B, and 2C depict example high-efficiency model architectures for generative machine learning according to some aspects of this disclosure.

[0013] Figure 3 illustrates an example architecture of an encoder-decoder adapter block for generative machine learning in a high-efficiency model architecture, according to some aspects of this disclosure.

[0014] Figure 4 illustrates an example architecture for training a highly efficient generative machine learning model according to some aspects of this disclosure.

[0015] Figure 5 is a flowchart depicting an example method for training a generative machine learning model according to some aspects of this disclosure.

[0016] Figure 6 is a flowchart depicting an example method for inference using a generative machine learning model according to some aspects of this disclosure.

[0017] Figure 7 is a flowchart depicting an example method for processing data using a diffusion machine learning model according to some aspects of this disclosure.

[0018] Figure 8 is a flowchart depicting an example method for training a diffusion machine learning model according to some aspects of this disclosure.

[0019] Figure 9 illustrates an example processing system configured to perform various aspects of this disclosure.

[0020] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0021] Various aspects of this disclosure provide apparatus, methods, processing systems, and nontransitory computer-readable media for providing improved generative machine learning-based methods.

[0022] Using diffusion machine learning models to process data typically involves iteratively applying a noise prediction function to denoise noisy samples into denoised samples, starting from the noise (e.g., at time or in iterations). (white Gaussian noise) and in time or iteration Moving towards the final output (e.g., the output image). As used herein, "time" or "time step" often refers to the iterations used to process data using the model. For example, processing... The data from each iteration can be referred to as processed data. Data at time steps. In some respects, the noise prediction function can be decomposed into low-resolution (or lower-resolution) and high-resolution (or higher-resolution) denoising functions, as discussed in more detail below.

[0023] Low-resolution denoisers typically receive a low-resolution latent representation and predict the denoised latent tensor. Generally, denoising (in some respects, referred to as backdiffusion) is performed in the latent space to reduce computational complexity, as the higher-resolution image space may be too large to be reasonably manipulated within. In some aspects, the generation process begins with a white-noise image and iteratively removes noise to generate an output image. At each iteration, given the denoised latent tensor and noisy input features, the high-resolution denoiser predicts the denoised output. This process can then be repeated for a desired number of iterations.

[0024] In some conventional architectures, from to For each iteration (also known as a sampling step), both lower and higher resolution operations are performed. It should be noted that, in some respects, by convention, The process decrements (rather than increments) in each iteration. This is because the denoising process reverses the diffusion process, and the decrementing method allows the use of the same notation to identify corresponding stages or iterations in both the forward and reverse paths. However, in some aspects, the relative stability of the low-resolution latent tensor across sampling steps can be leveraged to reduce the computational cost and latency resulting from latent tensor generation. More specifically, in some aspects, lower-resolution blocks of the model can be used only for a subset of iterations or sampling steps. During other iterations, efficient approximations can be used to generate latent tensors.

[0025] In some aspects, the denoised latent tensor in at least some iterations is approximated using an adapter function or block, discussed in more detail below. In some aspects, the adapter is implemented as a shallow convolutional network without computationally expensive operations such as self-attention or cross-attention. In some aspects, during the sampling process (e.g., during inference), the latent tensor can be generated (also known as denoised) by switching between lower-resolution computation blocks and relatively more efficient adapters based on various criteria, such as defined clock scheduling (also known as intermittent or periodic scheduling). For example, a highly efficient adapter can be used every other iteration or over several iterations before the lower-resolution block is used again.

[0026] In this way, the adapter (which typically consumes fewer computational resources and / or results in reduced latency compared to conventional lower-resolution operations) can be used to substantially reduce the latency and computational cost of generating model outputs during inference. Furthermore, various additional inputs and / or skip connections can be combined with the adapter block compared to some conventional systems to produce more accurate or desired model results with relatively fewer iterations.

[0027] Example workflow for high-efficiency generative machine learning models

[0028] Figure 1 illustrates an example workflow 100 for an efficient generative machine learning model according to some aspects of this disclosure. In some aspects, workflow 100 is performed by a machine learning system, such as a computational system that trains a diffusion model and / or uses the diffusion model during inference.

[0029] The illustrated example depicts two iterations 101A and 101B (collectively referred to as iteration 101) of processing data using a diffusion machine learning model. Specifically, iterations 101A and 101B depict processing data during two consecutive iterations of a denoising backbone of the diffusion machine learning model. As used herein, "denoising backbone" generally refers to a component of a generative model that performs iterative denoising operations to generate an output image. As discussed above, in some aspects, each iteration 101 includes using a lower resolution (which may also be referred to as a first resolution) operation to generate the latent tensor of iteration 101, and using a higher resolution (which may also be referred to as a second resolution) operation to generate a set of output features for that iteration based on the latent tensor. In some conventional methods, as discussed above, each iteration includes using a higher (second) resolution block 102 and a lower (first) resolution block 112. In some aspects of this disclosure, some iterations 101 may use an adapter block 115 instead of the lower resolution block 112 to generate the latent tensor.

[0030] In the illustrated example, as discussed in more detail below, the feature tensor generated by block 105B is used as input to block 105A of higher resolution block 102 during subsequent iterations 101B. Although depicted as discrete components for conceptual clarity, in some respects, iteration 101 can be implemented by processing all or a subset of the generated data using the same hardware and / or software components. For example, the same higher resolution block 102 can be used in each iteration 101, processing a new input feature tensor each time (e.g., during a given iteration, higher resolution block 102 can process feature tensors generated during the immediately preceding previous iteration to generate new feature tensors for the immediately following iteration).

[0031] Specifically, as illustrated, iteration 101A uses higher resolution block 102 and lower resolution block 112 to generate output features, and iteration 101B uses (the same) higher resolution block 102 with adapter block 115 to generate output features. As illustrated, each iteration 101 typically processes data at multiple different scales through one or more downsampling and upsampling operations. In some respects, this may be referred to as the U-Net architecture.

[0032] Specifically, block 105A of the higher resolution block 102 receives the iterative input tensor (e.g., features from previous iterations or input data to the model). In some respects, the input tensor may be referred to as... In some respects, the higher-resolution block 102 operates at full resolution (also known as the second resolution). That is, the higher-resolution block 102 can operate on the input data at the same or original size or dimensions as the data, while the lower-resolution block 112 operates on data at a smaller or lower resolution (first resolution). The higher-resolution block 102 is typically used for computation. In some respects, the higher resolution block 102 is based on Operate the next tensor, where Indicates the application of higher resolution block 102, It is iteration During the input to the higher resolution block 102, and It is generated by lower-resolution blocks (or adapters) for iteration. The denoised latent tensor is discussed in more detail below.

[0033] Block 105A performs some operations or transformations (e.g., convolution and self-attention) and feeds the resulting tensor to downsampling operation 107A. Downsampling operation 107A typically uses any suitable downsampling technique to reduce the size or dimensionality of the tensor (e.g., reduce the spatial size of the features). As illustrated, the output of block 105A is also provided to block 105B of the higher-resolution block 102 via skip connection 106A. This skip connection 106A can be implemented using various operations, such as identity mapping and convolution operations. In some respects, skip connection 106A can improve model stability.

[0034] As illustrated, the output of downsampling operation 107A is provided to the first block 110A in the lower resolution block 112. In some respects, the lower resolution block 112 is used in iterations. During this period, the latent tensors are based on those generated by higher resolution block 102 (e.g., output by block 105A and / or downsampling operation 107A). To perform iteration The denoised latent tensor In some respects, lower resolution block 112 is based on The denoised latent tensor is computed, where Indicates the application of lower resolution block 112, and It is iteration The output comes from block 105A during this period.

[0035] Lower-resolution block 112 typically includes various operations (indicated by exemplary blocks 110A to 110G) capable of performing various operations such as convolution and attention. In the illustrated example, lower-resolution block 112 also includes evaluations at multiple scales. Specifically, the input embedding (from block 105A) is processed by block 110A, which produces the output of block 110B. The output of block 110B is then downsampled by downsampling operation 107B and provided to block 110C. In some aspects, lower-resolution block 112 may also include a skip connection 106B between block 110B and block 110F. The output of block 110C is processed by block 110D, which produces the data input for block 110E.

[0036] Next, the output of block 110E is upsampled via upsampling operation 109A and provided as input to block 110F. The output of block 110F is used as input to block 110G, and the output of block 110G is then upsampled via upsampling operation 109B and used as input to block 105B of the higher resolution block 102. Block 105B then processes this input (in some respects, and the output of block 105A via skip connection 106A) to generate an output feature tensor, which serves as the output of iteration 101A.

[0037] In some respects, the data generated by block 105A and / or by downsampling operation 107A may be referred to as an embedding (generated by the higher resolution block 102) and / or as a noisy latent tensor. That is, the output provided from the higher resolution block 102 to the lower resolution block 112 may be referred to as an embedding or a noisy latent tensor. The output generated by block 110G may similarly be referred to as a denoised latent tensor. That is, the lower resolution block 112 generates an incrementally denoised latent tensor based on the received noisy tensor.

[0038] In the illustrated example, the second iteration 101B does not use the lower resolution block 112. Instead, the adapter block 115 is used to generate the denoised latent tensor for iteration 101B. Specifically, as illustrated, the feature tensor (generated during iteration 101A) is used as input to the higher resolution block 102 (e.g., block 105A) to generate an output, which is downsampled via downsampling operation 107A and provided as input to the adapter block 115. In the illustrated example, the adapter block 115 also receives the denoised latent tensor generated by block 110G during the first iteration 101A as input. Based on these inputs, the adapter block 115 generates a new denoised latent tensor for the second iteration 101B, which is then upsampled via upsampling operation 109B and provided as input to block 105B of the higher resolution block 102.

[0039] As discussed above, compared to the lower resolution block 112, the adapter block 115 can typically be implemented using fewer computational resources and / or in a way that results in reduced latency. In this way, iteration 101B can be executed substantially faster and with reduced computational cost compared to iteration 101A.

[0040] Generally, the specific operation or configuration of adapter block 115 may vary depending on the specific implementation. For example, in some aspects, adapter block 115 implements or includes an identity mapping that copies the potential tensors generated during iteration 101A (e.g., the denoised tensors from block 110G) to the next iteration 101B. In some aspects, this computationally efficient use of the identity mapping can produce acceptable outputs in some domains, especially when the number of sampling steps (e.g., the number of iterations) is sufficiently high.

[0041] In some respects, adapter block 115 is parameterized. That is, adapter block 115 can use a set of learned parameters (with values ​​learned during training) to generate the latent tensor for iteration 101B. In some respects, this can lead to a more efficient approximation, resulting in improved output with reduced cost (and in some cases, a reduced number of iterations).

[0042] In some respects, adapter block 115 is defined as a convolutional U-Net using two-scale representations. In some respects, to ensure its computational efficiency, adapter block 115 may exclude self-attention and cross-attention operations. In some respects, in addition to the U-Net architecture, adapter block 115 may also include other operations, such as isotropic convolution stacking. In some respects, as discussed in more detail below, adapter block 115 may use an encoder-decoder architecture.

[0043] Although two iterations 101 are depicted for conceptual clarity, in some respects any number of iterations can be used to process the data using the diffusion model. Furthermore, while the illustrated example depicts evaluating the data at three resolutions in iteration 101A, any number of resolutions (e.g., any number of downsampling and upsampling operations) can be used. Similarly, the specific arrangement and configuration of block 105 in the higher resolution block 102 and block 110 in the lower resolution block 112 are presented only for conceptual clarity. The actual arrangement and configuration of blocks 105 and 110 may vary depending on a particular specific implementation.

[0044] Additionally, although not depicted in the illustrated example, in some respects, some or all of the blocks 105 in the higher resolution block 102 and / or some or all of the blocks 110 in the lower resolution block 112 may also receive additional data as input, such as embedding the original cue into the model (e.g., text embedding of a string of words that is provided to the model as input or cue to generate an image).

[0045] Similarly, adapter block 115 may or may not receive input from upsampling operation 107A. Adapter block 115 may or may not receive additional input, such as literal embeddings of the input string and temporal embeddings indicating which iteration is being performed. For example, to enable adapter block 115 to perform based on diffusion steps or iterations... The adapter block 115 may receive a time step or iteration embedding indicating which iteration is being performed (e.g., which iteration is generating the denoised potential tensor) as input.

[0046] Example high-efficiency model architecture for generative machine learning

[0047] Figures 2A, 2B, and 2C respectively depict example high-efficiency model architectures 200A, 200B, and 200C for generative machine learning according to some aspects of this disclosure. In some aspects, Figures 2A, 2B, and 2C depict the operation of example denoising backbones of diffusion models. In some aspects, architectures 200A, 200B, and 200C are used by machine learning systems, such as computational systems that train diffusion models and / or use diffusion models during inference (e.g., the machine learning system discussed above with reference to Figure 1)).

[0048] Specifically, architecture 200A depicts a feedforward adapter architecture, architecture 200B depicts a recurrent adapter architecture, and architecture 200C uses a multi-input feedforward adapter architecture. Each of architectures 200A, 200B, and 200C includes four iterations using a diffusion machine learning model to process the data, but any number of iterations can be used. As discussed above, adapter blocks can be used in at least some of these iterations, rather than in all iterations using lower-resolution blocks (which can be relatively expensive and slow).

[0049] As depicted in Figure 2A, the feedforward adapter architecture 200A typically involves using adapter blocks to map a latent tensor from one iteration (e.g., the first iteration) to one or more subsequent iterations. Specifically, in the illustrated example, the input feature tensor 205A (e.g., feature tensors from previous layers in the diffusion model and / or embeddings of actual text input into the model) is processed by at least a portion of a higher-resolution block 102A (which may correspond to higher-resolution block 102 in Figure 1), which generates a latent tensor 210A (also referred to in some respects as a noisy latent tensor), as discussed above. For example, latent tensor 210A may be generated by block 105A of Figure 1.

[0050] As illustrated, latent tensor 210A is processed by lower-resolution block 112A (which may correspond to lower-resolution block 112 in FIG. 1) to generate latent tensor 210B (also referred to in some respects as the denoised latent tensor, as discussed above). For example, latent tensor 210B may be generated by block 110G in FIG. 1. As illustrated, latent tensor 210B is then processed by higher-resolution block 102A to generate feature tensor 205B. For example, feature tensor 205B may be generated by block 105B in FIG. 1. As discussed above, feature tensor 205B is used as the output tensor of the iteration and as the input feature tensor for subsequent iterations.

[0051] Specifically, as illustrated, feature tensor 205B is used as input to higher resolution block 102B (which may correspond to iteration 101B of Figure 1). Although depicted as discrete higher resolution blocks 102A and 102B for clarity of concept, in some respects, higher resolution blocks 102A and 102B can be implemented by processing data using a single higher resolution block 102 at different times (e.g., using higher resolution block 102 to process feature tensor 205A at a first time, and then using the same higher resolution block 102 to process the resulting output feature tensor 205B at subsequent times). In other respects, higher resolution block 102A may differ from higher resolution block 102B.

[0052] In the illustrated example, the latent tensor 210B generated during the first iteration is also provided as input to adapter block 115A (e.g., 115 in Figure 1). During the second iteration, adapter block 115A generates latent tensor 210C (also referred to as the denoised latent tensor, as discussed above). Then, latent tensor 210C is provided as input to higher resolution block 102B during the second iteration. It should be noted that in the illustrated architecture 200A, adapter block 115A does not receive input from higher resolution block 102B. That is, feature tensor 205B generated during the previous iteration can be used by higher resolution block 102B to generate feature tensor 205C, but feature tensor 205B is not accessed or used by adapter block 115A to generate the new latent tensor 210C. Therefore, for example, returning to Figure 1, the connection between block 105A and adapter block 115 may be severed or may not exist.

[0053] As illustrated, the higher resolution block 102B processes the input feature tensor 205B and the latent tensor 210C to generate a new feature tensor 205C during the second iteration.

[0054] In architecture 200A, feature tensor 205C is then used as input to higher resolution block 102C during a subsequent (third) iteration. Although depicted as discrete higher resolution block 102C for conceptual clarity, as discussed above, higher resolution block 102C can be identical to higher resolution blocks 102A and 102B, such as by processing data using a single higher resolution block 102 at a later time (e.g., after feature tensor 205B is processed to generate feature tensor 205C). In other respects, higher resolution block 102C can differ from one or more of higher resolution blocks 102A and 102B.

[0055] In the illustrated example, the latent tensor 210B generated during the first iteration is also provided as input to adapter block 115B. During the third iteration, adapter block 115B generates latent tensor 210D (also referred to as the denoised latent tensor, as discussed above). Then, latent tensor 210D is provided as input to the higher resolution block 102C during the third iteration.

[0056] Although depicted as discrete adapter blocks 115A and 115B for clarity of concept, in some respects, adapter blocks 115A and 115B can be implemented by using a single adapter block 115 to process data at different times (e.g., using adapter block 115 to process latent tensor 210B to generate latent tensor 210C at a first time, and then using the same adapter block 115 to process latent tensor 210B to generate latent tensor 210D at a subsequent time). In other respects, adapter blocks 115A and 115B can be different from each other.

[0057] As illustrated, the higher resolution block 102C processes the input feature tensor 205C and the latent tensor 210D to generate a new feature tensor 205D during the third iteration.

[0058] In architecture 200A, feature tensor 205D is then used as input to higher resolution block 102D during a subsequent (fourth) iteration. Although depicted as discrete higher resolution block 102D for conceptual clarity, as discussed above, higher resolution block 102D can be identical to higher resolution blocks 102A, 102B, and 102C, such as by processing data using a single higher resolution block 102 in subsequent times. In other respects, higher resolution block 102D can differ from one or more of higher resolution blocks 102A, 102B, and 102C.

[0059] In the illustrated example, lower-resolution block 112B is used to process latent tensor 210E (generated by higher-resolution block 102D) to generate latent tensor 210F, instead of using the adapter block in the fourth iteration. Latent tensor 210F is then used by higher-resolution block 102D (in combination with feature tensor 205D) to generate feature tensor 205E. Although depicted as discrete lower-resolution blocks 112A and 112B for conceptual clarity, in some respects, lower-resolution blocks 112A and 112B can be implemented by processing data using a single lower-resolution block 112 at different times. In other respects, lower-resolution blocks 112A and 112B can be different blocks.

[0060] In the illustrated example, architecture 200A uses lower-resolution block 112 for one iteration, then uses adapter block 115 for two iterations, and then again uses lower-resolution block 112 for a fourth iteration. Therefore, in this example, the first and last iterations use lower-resolution blocks. In other examples, any combination or order of lower-resolution blocks and adapters can be used.

[0061] In some respects, the number of iterations or steps that can be performed using adapter block 115 (instead of lower resolution block 112) can be a configurable hyperparameter or a learnable parameter. For example, the user can configure the model to use lower resolution block 112 for the first iteration, then for every subsequent iteration, every second iteration thereafter, and every third iteration thereafter, and so on. Although four iterations using a denoised backbone to process the data are depicted for conceptual clarity, in this respect, the architecture can use any number of iterations to generate the output.

[0062] Although not depicted in the illustrated example, in some respects, adapter block 115 may receive further input (in addition to the latent tensor 210 from previous iterations). For example, in some respects, adapter block 115 may receive a temporal embedding indicating which iteration is being performed. As another example, adapter block 115 may receive a textual embedding (e.g., a contrastive language-image pre-training (CLIP) embedding representing a string that is provided as input to the diffusion model.

[0063] Although not depicted in the illustrated example, in some aspects, one or more components of the diffusion model may exist before and / or after the depicted denoising backbone. For example, the input text may undergo various processing steps before being provided to the higher-resolution block 102 during the first iteration. In some aspects, the input to the initial iteration of the denoising backbone is a random tensor (e.g., a white noise image), and textual cues may be used as additional input to guide the denoising. Similarly, one or more downstream components (e.g., a decoder) may be used to process the output feature tensor 205 of the final iteration of the backbone to generate the final output of the model (e.g., the generated image).

[0064] Advantageously, the feedforward architecture 200A (where the latent tensor 210B from the first iteration is reused at multiple future iterations) prevents error accumulation. That is, because adapter block 115 approximates the latent tensor in a given iteration, repeatedly applying the adapter to the previously approximated latent tensor could allow any introduced error to accumulate through iterations (which could lead to more frequent application of lower-resolution block 112 instead of adapter block 115 to reduce such errors). However, by using the depicted feedforward architecture, such errors do not accumulate.

[0065] Turning to Figure 2B, architecture 200B depicts a recurrent adapter architecture. As illustrated in Figure 2B, the recurrent adapter architecture 200B largely mirrors the feedforward adapter architecture 200A of Figure 2A and typically involves using adapter blocks to map latent tensors from one iteration (e.g., the first iteration) to immediately following subsequent iterations. Specifically, in the illustrated example, the input feature tensor 205A is processed by at least a portion of the higher resolution block 102A to generate latent tensor 210A, and latent tensor 210A is processed by the lower resolution block 112A to generate latent tensor 210B.

[0066] As illustrated, the latent tensor 210B is then processed by the higher-resolution block 102A to generate a feature tensor 205B, which is used as input to the higher-resolution block 102B. Although depicted as discrete higher-resolution blocks 102A and 102B for clarity of concept, in some respects, the higher-resolution blocks 102A and 102B can be implemented by processing the data using a single higher-resolution block 102 at different times (e.g., using the higher-resolution block 102 to process the feature tensor 205A at a first time, and then using the same higher-resolution block 102 to process the resulting output feature tensor 205B at subsequent times). In other respects, the higher-resolution block 102A may differ from the higher-resolution block 102B.

[0067] In the illustrated example, the latent tensor 210B generated during the first iteration is also provided as input to adapter block 115A, as discussed above. During the second iteration, adapter block 115A generates latent tensor 210C, which is then provided as input to higher resolution block 102B during the second iteration. As illustrated, higher resolution block 102B processes the input feature tensor 205B and latent tensor 210C to generate a new feature tensor 205C during the second iteration. In architecture 200B, feature tensor 205C is then used as input to higher resolution block 102C during a subsequent (third) iteration.

[0068] In the illustrated example, the latent tensor 210C (generated during the second iteration) is used by adapter block 115B in the third iteration, instead of providing the latent tensor 210B (generated during the first iteration) as input to adapter block 115B. In some respects, if the fourth iteration also uses the adapter (instead of the lower resolution block), the latent tensor 210D generated during the third iteration will be used in the fourth iteration to generate new latent tensors.

[0069] During the third iteration, adapter block 115B generates latent tensor 210D based on latent tensor 210C. Then, latent tensor 210D is provided as input to higher resolution block 102C during the third iteration. As illustrated, higher resolution block 102C processes the input feature tensor 205C and latent tensor 210D to generate a new feature tensor 205D during the third iteration. Feature tensor 205D is used as input to higher resolution block 102D during a subsequent (fourth) iteration to generate latent tensor 210E, and lower resolution block 112B processes latent tensor 210E to generate latent tensor 210F. Then, latent tensor 210F is used by higher resolution block 102D (in combination with feature tensor 205D) to generate feature tensor 205E.

[0070] In the illustrated example, architecture 200B uses lower-resolution block 112 for one iteration, followed by adapter block 115 for two iterations, and then again uses lower-resolution block 112 for a fourth iteration. In some respects, the number of iterations or steps performed using adapter block 115 (instead of lower-resolution block 112) can be a configurable hyperparameter or a learnable parameter. For example, a data scientist could configure the model to use lower-resolution block 112 for the first iteration, then for every subsequent iteration, every second iteration thereafter, and every third iteration thereafter, and so on. Although four iterations using a denoised backbone to process the data are depicted for conceptual clarity, in reality, the architecture can use any number of iterations to generate the output. Therefore, in this example, the first and last iterations use lower-resolution blocks. In other examples, any combination or order of lower-resolution blocks and adapters can be used.

[0071] Although not depicted in the illustrated example, in some respects, adapter block 115 may receive further input (in addition to the latent tensor 210 from previous iterations), as discussed above. For example, in some respects, adapter block 115 may receive a temporal embedding indicating which iteration is being performed. As another example, adapter block 115 may receive a literal embedding (e.g., a CLIP embedding) representing a cue string provided as input to the diffusion model.

[0072] Although not depicted in the illustrated example, in some respects, one or more components of the diffusion model may exist before and / or after the depicted denoising backbone. For example, the input text (and white noise input) may undergo various processing before being provided to the higher resolution block 102 during the first iteration. Similarly, one or more downstream components may be used to process the output feature tensor 205 of the backbone in the final iteration to generate the final output of the model (e.g., the generated image).

[0073] Advantageously, the recurrent architecture 200B (where the latent tensor 210B from a given iteration can be reused in immediately following iterations, but not in further iterations beyond those subsequent iterations) improves the flexibility of the model. For example, while the feedforward architecture 200A discussed above can be restricted to consistently using the same adapter interleaving settings (e.g., always skipping lower-resolution blocks for the same number of iterations), the recurrent architecture 200B allows the model (or the user) to dynamically determine how many steps or iterations to perform before switching back to lower-resolution blocks.

[0074] Turning to Figure 2C, architecture 200C depicts a multi-input feedforward adapter architecture. As illustrated in Figure 2C, the multi-input feedforward adapter architecture 200C largely mirrors the feedforward adapter architecture 200A of Figure 2A and typically involves using adapter blocks to map latent tensors from one iteration (e.g., the first iteration) to one or more subsequent iterations. Specifically, in the illustrated example, the input feature tensor 205A is processed by at least a portion of the higher resolution block 102A, which generates a latent tensor 210A, as discussed above. As illustrated, latent tensor 210A is processed by the lower resolution block 112A to generate a latent tensor 210B, which is then processed by the higher resolution block 102A to generate the feature tensor 205B.

[0075] Feature tensor 205B is used as input to higher resolution block 102B (e.g., the same higher resolution block during subsequent iterations). In the illustrated example, latent tensor 210B generated during the first iteration is also provided as input to adapter block 115A. During the second iteration, adapter block 115A generates latent tensor 210D. In the illustrated architecture 200B, adapter block 115A also receives and evaluates the embeddings generated by higher resolution block 102B during the second iteration, rather than only evaluating latent tensor 210B as input. That is, latent tensor 210G (referred to in some aspects as embeddings) is also received as input by adapter block 115A.

[0076] Using latent tensors 210B and 210G, adapter block 115A generates latent tensor 210C (e.g., a denoised latent tensor), which is then provided as input to higher resolution block 102B during the second iteration. As illustrated, higher resolution block 102B processes the input feature tensor 205B and latent tensor 210C to generate a new feature tensor 205C during the second iteration.

[0077] In architecture 200C, feature tensor 205C is then used as input to higher resolution block 102C during a subsequent (third) iteration. Using feature tensor 205C, higher resolution block 102C generates latent tensor 210H, which is provided as input to adapter block 115B. In the illustrated example, latent tensor 210B generated during the first iteration is also provided as input to adapter block 115B.

[0078] During the third iteration, adapter block 115B generates latent tensor 210D based on latent tensor 210B and latent tensor 210H, and provides latent tensor 210D to higher resolution block 102C during the third iteration. Higher resolution block 102C processes input feature tensor 205C and latent tensor 210D to generate a new feature tensor 205D during the third iteration. Feature tensor 205D is then used as input to higher resolution block 102D during a subsequent (fourth) iteration.

[0079] In the illustrated example, the lower-resolution block 112B is used to process the latent tensor 210E (generated by the higher-resolution block 102D) to generate the latent tensor 210F, instead of using the adapter block in the fourth iteration. The latent tensor 210F is then used by the higher-resolution block 102D (in combination with the feature tensor 205D) to generate the feature tensor 205E.

[0080] In the illustrated example, architecture 200C uses lower-resolution block 112 for one iteration, followed by adapter block 115 for two iterations, and then again uses lower-resolution block 112 for a fourth iteration. In some respects, the number of iterations or steps performed using adapter block 115 (instead of lower-resolution block 112) can be a configurable hyperparameter or a learnable parameter. For example, a data scientist could configure the model to use lower-resolution block 112 for the first iteration, then for every subsequent iteration, every second iteration thereafter, and every third iteration thereafter, and so on. Although four iterations using a denoised backbone to process the data are depicted for conceptual clarity, in reality, the architecture can use any number of iterations to generate the output. Therefore, in this example, the first and last iterations use lower-resolution blocks. In other examples, any combination or order of lower-resolution blocks and adapters can be used.

[0081] Although not depicted in the illustrated example, in some respects, adapter block 115 may receive further input (in addition to the potential tensor 210 from previous iterations and the potential tensor 210 from the higher-resolution block 102 in the same iteration). For example, in some respects, adapter block 115 receives a temporal embedding indicating which iteration is being performed. As another example, adapter block 115 may receive a literal embedding (e.g., a CLIP embedding) representing a string that is provided as input to the diffusion model.

[0082] Although not depicted in the illustrated example, in some respects, one or more components of the diffusion model may exist before and / or after the depicted denoising backbone. For example, the input text (and the same white noise image) may undergo various processing steps before being provided to the higher resolution block 102 during the first iteration. Similarly, one or more downstream components may be used to process the output feature tensor 205 of the backbone in the final iteration to generate the final output of the model (e.g., the generated image).

[0083] Advantageously, the multi-input feedforward architecture 200C (where the latent tensor 210B from the first iteration is reused at multiple future iterations) prevents error accumulation, as discussed above. Furthermore, in some aspects, the addition of feedback from higher-resolution paths (e.g., latent tensors 210C and 210E from higher-resolution block 102) can provide error correction to the generated denoised latent quantities. In some aspects, this error correction can improve the accuracy of the resulting feature tensors. In this way, in some aspects, architecture 200C may be able to generate improved and / or equivalent output images using fewer iterations (and therefore fewer computational resources and reduced latency).

[0084] Example architecture of encoder-decoder adapter blocks for generative machine learning models

[0085] Figure 3 illustrates an example architecture of an encoder-decoder adapter block for a generative machine learning model according to some aspects of this disclosure. In some aspects, the depicted architecture is used by a machine learning system, such as a computational system that trains a diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B and / or 2C)).

[0086] As discussed above, by reusing the denoised latent tensors generated in one iteration for one or more subsequent iterations, the diffusion model can use more efficient adapter blocks for at least a subset of the iterations of the denoised backbone (using more expensive lower-resolution blocks for one or more of these iterations). However, while this practice reduces the computational complexity and latency of the denoised backbone, storing these latent tensors (e.g., storing them in memory) for future use imposes a certain amount of increased memory burden on the inference process in some respects.

[0087] In some respects, to reduce this burden, the illustrated encoder-decoder adapter block 115 architecture can be used. In the illustrated example, the operation of adapter block 115 can be partitioned across iterations, rather than being used during a given iteration (e.g., to generate denoised latent tensors for a given iteration). Specifically, in the illustrated example, adapter block 115 includes encoder block 305 that can be used during one iteration, and decoder block 310 that can be used during subsequent iterations, as discussed in more detail below.

[0088] In some respects, encoder block 305 roughly corresponds to a parameterized component (e.g., one or more layers of a neural network) trained to generate a compressed or reduced version of the input tensor, while decoder block 310 similarly corresponds to a parameterized component (e.g., one or more layers of a neural network) trained to reconstruct the original input based on compressed data or to create an illusion of it. In some respects, the encoder-decoder architecture can be referred to as a bottleneck.

[0089] In the illustrated example, the input latent tensor (such as latent tensor 210B (e.g., a denoised latent tensor generated by the adapter or by the lower-resolution block during a given iteration)) is provided as input to encoder block 305, in addition to being used by the higher-resolution block to generate output features. Encoder block 305 processes or transforms latent tensor 210B to generate compressed tensor 315. As discussed above, compressed tensor 315 typically corresponds to a compressed version of latent tensor 210B. For example, compressed tensor 315 may have a smaller size or memory footprint (e.g., due to dimensionality reduction of latent tensor 210B).

[0090] In some respects, instead of transferring the latent tensor 210B itself (e.g., instead of storing latent tensor 210B in memory until the next iteration), the compressed tensor 315 is stored. This reduces the memory footprint of the operation. During the next iteration, the compressed tensor 315 can be retrieved from memory and processed using decoder block 310 to generate latent tensors, such as latent tensor 210C.

[0091] In some respects, latent tensor 210C approximates latent tensor 210B. That is, encoder block 305 and / or decoder block 310 can be trained to attempt to align the output of decoder block 310 (latent tensor 210C) with the input to encoder block 305 (latent tensor 210B). Therefore, in some respects, the adapter can use or add one or more additional components to adapt the original denoised latent tensor from the first iteration to a new denoised latent tensor in the second iteration. For example, the output of decoder block 310 can be processed by another adapter component (e.g., adapter block 115A of FIG. 2A, discussed above) to generate a new denoised latent tensor.

[0092] In some respects, encoder block 305 and / or decoder block 310 may be trained to perform this adaptation internally, rather than training encoder block 305 and decoder block 310 to preserve latent tensor 210B. For example, the output of decoder block 310 itself may be a new denoised latent tensor 210C for the current iteration (e.g., an adapted version of denoised latent tensor 210B from an earlier iteration).

[0093] In this way, by maintaining the compressed tensor 315 between iterations (rather than the entire potential tensor 210B), the memory footprint of the adapter block 115 between iterations can be significantly reduced, further improving the computational efficiency of the diffusion model (especially in memory-constrained environments).

[0094] Example architecture for training highly efficient generative machine learning models

[0095] Figure 4 illustrates an example architecture for training a highly efficient generative machine learning model according to some aspects of this disclosure. In some aspects, the depicted workflow can be used to train a denoised backbone for a diffusion model, such as the higher resolution block 102, lower resolution block 112, and / or adapter block 115 discussed above with reference to Figures 1, 2A, 2B, 2C, and / or Figure 3. In some aspects, the depicted training workflow is performed by a machine learning system, such as a computational system that trains the diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B, 2C, and / or Figure 3)).

[0096] In the illustrated example, the student model 440, corresponding to the high-efficiency denoising backbone, is trained on top of the teacher model 400, which uses regular lower-resolution blocks 412, to generate denoised latent tensors at each iteration (while the student model 440 uses a more efficient adapter block 115 for at least some iterations, as discussed above). Although the illustrated example depicts a sequence of blocks (e.g., multiple higher-resolution blocks 402A and 402B, etc.), in some respects, as discussed above, this sequence of blocks can be achieved by performing operations on the same blocks at different times. For example, higher-resolution block 402A may correspond to processing a first set of data (e.g., input 405A) using a given set of weights at a first time, and higher-resolution block 402B may correspond to processing a second set of data (e.g., feature tensor 405B) using the same given set of weights at different times.

[0097] As illustrated, input 405A is provided to higher resolution block 402A of teacher model 400 and to higher resolution block 102A of student model 440. Higher resolution block 402A processes input 405A to generate latent tensor 410A, and higher resolution block 102A processes input 405A to generate latent tensor 210A. In the illustrated example, latent tensors 410A and 210A are then processed by lower resolution blocks 412A and 112A, respectively, to generate denoised latent tensors 410B and 210B, respectively.

[0098] Next, latent tensors 410B and 210B are processed by at least a portion of higher resolution blocks 402A and 102A, respectively, to generate feature tensors 405B and 205B. As illustrated, feature tensors 405B and 205B are used as inputs to higher resolution blocks 402B and 102B, respectively. In teacher model 400, higher resolution block 402B generates latent tensor 410C based on feature tensor 405B. This latent tensor 410C is then provided as input to lower resolution block 412B, which generates a denoised latent tensor 410D. In student model 440, latent tensor 210B (generated by lower resolution block 112A during previous iterations) is provided to adapter block 115A, which generates a denoised latent tensor 210C.

[0099] As illustrated, the denoised latent tensors 410D and 210C are then processed by higher resolution blocks 402B and 102B, respectively, to generate feature tensors 405C and 205C. Feature tensors 405C and 205C are used as inputs to the higher resolution blocks 402C and 102C, respectively. In teacher model 400, higher resolution block 402C generates latent tensor 410E based on feature tensor 405C. This latent tensor 410E is then provided as input to lower resolution block 412C, which generates the denoised latent tensor 410F. In student model 440, latent tensor 210B (generated by lower resolution block 112A during previous iterations) is provided to adapter block 115B, which generates the denoised latent tensor 210D.

[0100] Next, the denoised latent tensors 410F and 210D are processed by higher-resolution blocks 402C and 102C, respectively, to generate feature tensors 405D and 205D. Feature tensors 405D and 205D are used as inputs to higher-resolution blocks 402D and 102D, respectively. In the illustrated example, higher-resolution blocks 402D and 102D output latent tensors 410G and 210E, respectively. Then, latent tensors 410G and 210E are processed by lower-resolution blocks 412D and 112B, respectively, to generate denoised latent tensors 410H and 210F, respectively.

[0101] Next, the latent tensors 410H and 210F are processed by at least a portion of the higher resolution blocks 402D and 102D, respectively, to generate the feature tensors 405E and 205E, respectively. As discussed above, this process can be repeated any number of iterations to compute the backbone of the diffusion model.

[0102] In the illustrated example, to train student model 440, the latent tensors 210 (e.g., 210A to 210F) generated by student model 440 are compared with the corresponding latent tensors 410 generated by teacher model 400. Specifically, in the illustrated example, the latent tensor 210A generated by higher resolution block 102A (e.g., a latent tensor generated based on input 405A during the first iteration of data processing) is compared with the latent tensor 410A generated by higher resolution block 402A of teacher model 400 in the same iteration (e.g., generated based on input 405A). This is depicted by loss component 450A.

[0103] Similarly, latent tensor 210B (generated in the first iteration by lower-resolution block 112A) is compared with latent tensor 410B (generated in the first iteration by lower-resolution block 412A) to generate loss component 450B. Additionally, latent tensor 210C (generated in the second iteration by adapter block 115A) is compared with latent tensor 410D (generated in the second iteration by lower-resolution block 412B) to generate loss component 450C. Latent tensor 210D (generated in the third iteration by adapter block 115B) is compared with latent tensor 410F (generated in the third iteration by lower-resolution block 412C) to generate loss component 450D. Furthermore, the latent tensor 210E (generated in the fourth iteration from the higher resolution block 102D) is compared with the latent tensor 410G (generated in the fourth iteration from the higher resolution block 402D) to generate the loss component 450E, and the latent tensor 210F (generated in the fourth iteration from the lower resolution block 112B) is compared with the latent tensor 410H (generated in the fourth iteration from the lower resolution block 412D) to generate the loss component 450F.

[0104] In some respects, the parameters of student model 440 are updated to minimize or at least reduce the cumulative loss component 450A to 450F. For example, the loss of student model 440 can be defined as the reconstruction error, such as by using Equation 1 below, where It's a loss. It is the number of iterations (e.g., the number of times the data is processed using a denoised backbone). It is in iteration During this period, the latent tensor 410 was generated by the teacher model 400, and It is in iteration The latent tensor 210 is generated by student model 440 during this period.

[0105] (1)

[0106] In some respects, the student model 440 can be trained via backdistillation (e.g., using only text input instead of the corresponding target image) and / or via forward distillation (e.g., using both text input and the corresponding target image).

[0107] Generally, the depicted architecture can be used to refine the student model 440 based on any number of inputs 405 (e.g., the process can be repeated any number of times). Furthermore, in some aspects, the student model 440 can be updated based on each individual input 405 (e.g., using stochastic gradient descent) or based on batch inputs 405 (e.g., using batch gradient descent). Using the depicted architecture, the teacher model 400 (which uses a computationally expensive lower-resolution block 412 in each iteration) can be used to efficiently train the student model 440 (which uses a computationally more efficient adapter block 115 in at least some iterations). In this way, the student model 440 learns to efficiently generate accurate outputs.

[0108] Example methods for training generative machine learning models

[0109] Figure 5 is a flowchart depicting an example method 500 for training a generative machine learning model according to some aspects of this disclosure. In some aspects, method 500 is performed by a machine learning system (such as a computational system that trains a diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B, 2C, 3 and / or 4)).

[0110] At box 505, the machine learning system accesses the teacher model. As used herein, “accessing” data can generally include receiving, retrieving, requesting, collecting, generating, measuring, acquiring, or otherwise obtaining access to data. For example, the machine learning system may access a pre-trained diffusion model, such as teacher model 400 in Figure 4. In some respects, as discussed above, the teacher model generally corresponds to a generative model using a denoised backbone (e.g., a set of denoising operations applied iteratively and / or sequentially to the data to generate the model output), where the denoising backbone includes the use of lower-resolution blocks (e.g., lower-resolution block 412 in Figure 4) during each such iteration.

[0111] At box 510, the machine learning system accesses the training data to be used to train the efficient diffusion model. In some respects, as discussed above, such as when forward distillation is being used, the training data may include text prompts (as input) and corresponding target images (as target output). In other respects, as discussed above, such as when back distillation is being used, the training data may consist only of text prompts (without image data). Generally, the machine learning system can access the training data in any order (including random or pseudo-random).

[0112] At box 515, the machine learning system generates one or more latent tensors by processing the training data using a teacher model. For example, as discussed above with reference to Figure 4, the machine learning system can generate latent tensor 410 using higher resolution block 402 and / or lower resolution block 412. In some aspects, at box 515, the machine learning system generates latent tensors during a single iteration of the denoised backbone of the teacher model.

[0113] At box 520, the machine learning system generates one or more latent tensors by processing the training data using a student model (e.g., student model 440 of Figure 4). For example, as discussed above with reference to Figure 4, the machine learning system may use higher resolution block 102, lower resolution block 112, and / or adapter block 115 to generate latent tensor 210. In some aspects, at box 520, the machine learning system generates latent tensors during a single iteration of the denoised backbone of the student model.

[0114] At box 525, the machine learning system computes or otherwise determines one or more latent tensor losses (e.g., using reconstruction error) between the latent tensors generated by the teacher model and the latent tensors generated by the student model. For example, the machine learning system may use Equation 1 above to generate the loss. In some aspects, the number of loss components used to generate the backbone for processing the data in a given iteration may vary depending on the specific implementation and architecture. For example, in some aspects, the machine learning system may generate two losses (e.g., one for the latent tensors generated from higher resolution blocks and one for the latent tensors generated from lower resolution blocks and / or adapter blocks) for each iteration.

[0115] At box 530, the machine learning system determines whether the denoising backbone has at least one more iteration remaining. That is, the machine learning system determines whether the feature tensors should be processed at least one more time using the backbone. In some respects, as discussed above, the number of iterations used can be user-defined. If at least one iteration remains, method 500 returns to box 515, where the machine learning system generates a new set of latent tensors by processing the output of previous iterations (e.g., the feature tensors generated during the last iteration) using the teacher model. The machine learning system similarly processes the previous outputs of the student model at box 520 to generate a new set of latent tensors.

[0116] Returning to box 530, if the machine learning system determines that no further iterations are needed, method 500 continues to box 535. At box 535, the machine learning system updates one or more parameters of the student model based on the generated loss, as discussed above. In this way, compared to using more complex, lower-resolution blocks, the student model learns to generate efficient and accurate outputs using adapter blocks at least for some iterations.

[0117] At box 540, the machine learning system determines whether one or more termination criteria are met. Generally, termination criteria can correspond to a wide variety of factors, depending on the specific implementation. For example, in some aspects, the machine learning system may determine whether additional training data is available, whether the student model has reached a good or desired accuracy, whether training is still in progress or has stopped, whether a defined number of iterations, computational resources, and / or time have been used to train the model, etc.

[0118] If, at box 540, the machine learning system determines that the termination criteria are not met, method 500 returns to box 510. If the termination criteria are met, method 500 continues to box 545, where the machine learning system deploys the student model for inference. Generally, deploying the student model can include a wide variety of operations and typically corresponds to any steps taken to prepare or provide a model for runtime use. For example, the machine learning system may instantiate or use the model locally, or it may send a trained student model to one or more inference systems, etc.

[0119] Although the illustrated example depicts the training of a denoised backbone for a student model, in some respects, one or more components of the diffusion model may exist before and / or after the denoised backbone. For example, input text may undergo various processing before being provided to the backbone during the first iteration. Similarly, one or more downstream components may be used to process the output feature tensors of the backbone in the final iteration to generate the final output of the model (e.g., the generated image). In some respects, such other components may also be trained based on a teacher model, and / or the student model may use pre-trained parameters of these other components (e.g., from the teacher model).

[0120] Example methods for inference using generative machine learning models

[0121] Figure 6 is a flowchart depicting an example method 600 for inference using a generative machine learning model according to some aspects of this disclosure. In some aspects, method 600 is performed by a machine learning system, such as a computational system that trains a diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B, 2C, 3, 4 and / or 5)).

[0122] At box 605, the machine learning system accesses input data during runtime. For example, as discussed above, the input data may include textual data (e.g., natural language text) to be used to generate the image. In some aspects, the input may also include other elements, such as image data (e.g., where the textual input indicates how to modify or edit the provided image). In some aspects, the input to the denoising backbone may be a white noise image in the first iteration (and a progressively denoised image in subsequent iterations) and cue text (or an embedding of the cue).

[0123] At box 610, the machine learning system generates a first denoised latent tensor based on the input data using higher-resolution blocks of a denoised backbone of a diffusion machine learning model. Although not depicted in the illustrated example, in some respects, the model may include additional processing prior to the denoised backbone (e.g., generating embeddings based on text input). This processed data is then used as input to the higher-resolution blocks during the first iteration of processing the data with the denoised backbone.

[0124] At box 615, the machine learning system determines whether one or more adapter criteria are met. In some aspects, adapter criteria typically indicate whether computationally expensive lower-resolution blocks should be used during the current iteration of processing data using the backbone, or whether more efficient adapter blocks should be used. In some aspects, the criteria include evaluating predefined architectures or configurations (e.g., specifying the adapter block for each...). The first iteration uses a lower resolution block, and the remaining iterations use an adapter block.

[0125] If, at box 615, the machine learning system determines that the adapter criterion is not met, method 600 continues to box 620, whereby the machine learning system uses a lower-resolution block of the backbone (e.g., by processing the latent tensor generated at box 610 using a lower-resolution block) to generate an iterative second latent tensor. In some aspects, as discussed above, the machine learning system may also process other data to generate the second latent tensor. For example, data such as the embeddings of the input data may also be used as input to the lower-resolution block. Method 600 then continues to box 630.

[0126] Returning to box 615, if the machine learning system determines that the adapter criterion is met, method 600 continues to box 625, where the machine learning system uses the adapter block to generate a second latent tensor. For example, in some aspects, the machine learning system processes previous latent tensors (generated during previous iterations by lower-resolution blocks or adapter blocks) to generate new latent tensors for the current iteration. In some aspects, as discussed above, the machine learning system may also process other data to generate the second latent tensor. For example, data such as embeddings of input data, embeddings indicating which iteration or time step is currently being processed, and the latent tensors generated at box 610, etc., may also be used as input to the adapter block. Method 600 then continues to box 630.

[0127] At box 630, the machine learning system generates the feature tensor for the current iteration by processing the second latent tensor (generated by the lower resolution block at box 620 or by the adapter block at box 625) using a higher resolution block, as discussed above.

[0128] At box 635, the machine learning system determines whether the denoising backbone has at least one more iteration remaining. That is, the machine learning system determines whether the backbone should be used to process the feature tensors at least one more time. In some aspects, as discussed above, the number of iterations used can be defined by the model's architecture or configuration (e.g., instructing eight iterations to be performed). In other aspects, the machine learning system determines whether to use another iteration based on the quality of the generated feature tensors (e.g., by evaluating the newly generated feature tensors using one or more quality techniques or metrics, and exiting the backbone if the quality is high enough).

[0129] If at least one iteration remains, method 600 returns to box 610, where the machine learning system generates a new latent tensor by processing the output of the previous iteration (e.g., the feature tensor generated at box 630) using a higher resolution block of the backbone.

[0130] Returning to box 635, if the machine learning system determines that no further iterations are required, method 600 continues to box 640. At box 640, the machine learning system generates and outputs an image as the output from the diffusion machine learning model based on the feature tensors generated at box 630. For example, the feature tensors generated during the final iteration of the backbone can be provided to one or more additional layers or components of the diffusion model (e.g., decoders, one or more fully connected layers, attention layers, and nonlinear layers, etc.) to generate the image. In some respects, since the final feature tensor itself is generated in part based on previous feature tensors, the output image can be said to be generated (at least in part) based on each feature tensor in the feature tensors generated by the denoised backbone.

[0131] Example methods for processing data using diffusion machine learning models

[0132] Figure 7 is a flowchart depicting an example method 700 for processing data using a diffusion machine learning model according to some aspects of this disclosure. In some aspects, method 700 is performed by a machine learning system (such as a computational system that trains the diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B, 2C, 3, 4, 5 and / or 6)).

[0133] At box 705, during the first iteration of processing the data using a denoised backbone of a diffusion machine learning model, a lower-resolution block of the denoised backbone is used to generate the first latent tensor.

[0134] At box 710, during the first iteration, the first feature tensor is generated based on processing the first latent tensor using a higher resolution block with a denoised backbone, the higher resolution block using a higher resolution than the lower resolution block.

[0135] At box 715, the second potential tensor is generated based on the adapter block that uses the denoised backbone to process the first potential tensor.

[0136] In some aspects, the generation of the second latent tensor is further performed based on processing the embeddings corresponding to the second iteration using an adapter block. In some aspects, the generation of the second latent tensor is further performed based on processing the embeddings corresponding to the input of the diffusion machine learning model using an adapter block. In some aspects, the generation of the second latent tensor is further performed based on processing the embeddings generated by the higher-resolution block using an adapter block. In some aspects, the adapter block performs an identity mapping.

[0137] In some aspects, the adapter block uses a set of learned parameters to generate a second latent tensor based on a first latent tensor. In some aspects, the adapter block performs one or more convolutional operations to generate the second latent tensor. In some aspects, the adapter block includes an encoder and a decoder, and generating the second latent tensor includes: generating a compressed tensor based on processing the first latent tensor using the encoder, and generating the second latent tensor based on processing the compressed tensor using the decoder.

[0138] In some respects, a lower resolution block is used to generate a first latent tensor to generate a first delay, and an adapter block is used to generate a second latent tensor to generate a second delay, wherein the second delay is less than the first delay.

[0139] In some respects, the second potential tensor is not generated based on the first feature tensor.

[0140] At box 720, during the second iteration of processing the data using a denoised backbone, a second feature tensor is generated based on processing the second latent tensor using a higher resolution block.

[0141] In some aspects, method 700 also includes generating a third latent tensor based on processing the first latent tensor using an adapter block. In some aspects, method 700 also includes generating a third feature tensor based on processing the third latent tensor using a higher resolution block during a third iteration of processing the data using a diffusion machine learning model.

[0142] In some aspects, method 700 also includes generating a third latent tensor based on processing the second latent tensor using an adapter block. In some aspects, method 700 also includes generating a third feature tensor based on processing the third latent tensor using a higher resolution block during a third iteration of processing the data using a diffusion machine learning model.

[0143] In some aspects, method 700 also includes generating a third latent tensor using lower-resolution blocks during a third iteration of processing the data using a diffusion machine learning model. In some aspects, method 700 also includes generating a third feature tensor based on processing the third latent tensor using higher-resolution blocks during the third iteration.

[0144] In some respects, the diffusion machine learning model is trained using distillation from the teacher machine learning model, and the teacher machine learning model uses multiple higher resolution blocks and corresponding multiple lower resolution blocks.

[0145] In some respects, method 700 also includes: generating an image based at least in part on a first feature tensor and a second feature tensor; and outputting the image as the output from a diffusion machine learning model.

[0146] Example methods for training diffusion machine learning models

[0147] Figure 8 is a flowchart depicting an example method 800 for training a diffusion machine learning model according to some aspects of this disclosure. In some aspects, method 800 is performed by a machine learning system, such as a computational system that trains the diffusion model and / or uses the diffusion model during inference (e.g., the machine learning system discussed above with reference to Figures 1, 2A, 2B, 2C, 3, 4, 5, 6 and / or 7)).

[0148] At box 805, during the first iteration of processing the data using the first denoised backbone of the teacher diffusion machine learning model, the first latent tensor is generated using a lower-resolution block of the first denoised backbone.

[0149] At box 810, during the first iteration of processing the data using the second denoised backbone of the student diffusion machine learning model, the adapter block of the second denoised backbone is used to generate the second latent tensor.

[0150] In some respects, the generation of the second potential tensor is further performed based on using an adapter block to process the embedding corresponding to the first iteration.

[0151] In some respects, the generation of the second latent tensor is further based on using an adapter block to process the embeddings corresponding to the inputs of the student diffusion machine learning model.

[0152] In some respects, the generation of the second potential tensor is further performed based on using adapter blocks to process the embeddings generated by the higher resolution blocks of the second denoised backbone.

[0153] In some respects, the adapter block performs one or more convolution operations to generate a second potential tensor.

[0154] In some respects, the adapter block includes an encoder and a decoder, and generating the second potential tensor includes: generating a compressed tensor based on processing the third potential tensor using the encoder, and generating the second potential tensor based on processing the compressed tensor using the decoder.

[0155] At box 815, the loss is generated based on the first and second potential tensors.

[0156] At box 820, update one or more parameters of the adapter block based on the loss.

[0157] In some aspects, method 800 further includes: updating one or more parameters of a higher resolution block of the second denoised backbone based on a loss; and updating one or more parameters of a lower resolution block of the second denoised backbone based on the loss.

[0158] In some aspects, method 800 also includes: generating a third latent tensor based on processing the second latent tensor using an adapter block; and generating a feature tensor based on processing the third latent tensor using a higher resolution block of a second denoised backbone during a second iteration of processing the data using a Student diffusion machine learning model.

[0159] In some aspects, method 800 also includes: during a second iteration of processing the data using a student diffusion machine learning model, generating a third latent tensor using lower-resolution blocks of a second denoised backbone; and during the second iteration, generating a feature tensor based on processing the third latent tensor using higher-resolution blocks of the second denoised backbone.

[0160] Example processing system for machine learning

[0161] In some aspects, the architectures, workflows, techniques, and methods described with reference to Figures 1 through 8 can be implemented on one or more devices or systems. Figure 9 depicts an example processing system 900 configured to perform various aspects of this disclosure, including, for example, the techniques and methods described with reference to Figures 1 through 8. In some aspects, processing system 900 may correspond to a machine learning system, such as a computational system that trains a diffusion model and / or uses the diffusion model during inference (e.g., the machine learning systems discussed above with reference to Figures 1, 2A, 2B, 2C, 3, 4, 5, 6, 7, and / or 8). Although depicted as a single system for clarity of concept, in some aspects, as discussed above, the operations described below with reference to processing system 900 can be distributed across any number of devices or systems.

[0162] The processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed at the CPU 902 may be loaded, for example, from program memory associated with the CPU 902 or from a memory partition (e.g., a partition of memory 924).

[0163] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, a multimedia component 910 (e.g., a multimedia processing unit), and a wireless connectivity component 912.

[0164] NPUs (such as the NPU 908) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0165] NPUs (such as the NPU 908) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0166] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0167] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0168] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0169] In some implementations, the NPU 908 is part of one or more of the CPU 902, GPU 904, and / or DSP 906.

[0170] In some examples, the wireless connectivity component 912 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 912 is also coupled to one or more antennas 914.

[0171] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0172] The processing system 900 may also include one or more input and / or output devices 922, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, and microphones.

[0173] In some examples, one or more processors in the processing system 900 may be based on the ARM or RISC-V instruction set.

[0174] The processing system 900 also includes a memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, the memory 924 includes a computer-executable component that can be executed by one or more of the aforementioned processors of the processing system 900.

[0175] Specifically, in this example, memory 924 includes a higher resolution component 924A, a lower resolution component 924B, an adapter component 924C, and a processing component 924D. Memory 924 also includes a set of model parameters 924E for one or more models (e.g., for a teacher model used to train a high-efficiency diffusion model, such as teacher model 400 of Figure 4, and / or for a high-efficiency diffusion model, such as student model 440 of Figure 4). Although not depicted in the illustrated example, memory 924 may also include other data, such as training data. Although depicted as discrete components in Figure 9 for conceptual clarity, the illustrated components (and other undepicted components) may be implemented jointly or separately in various respects.

[0176] The processing system 900 also includes higher resolution circuitry 926, lower resolution circuitry 927, adapter circuitry 928, and processing circuitry 929. The depicted circuitry and other circuitry not depicted can be configured to perform various aspects of the techniques described herein.

[0177] For example, higher resolution component 924A and / or higher resolution circuit 926 (which may correspond to higher resolution block 102 of FIG. 1, FIG. 2A, FIG. 2B, FIG. 2C and / or higher resolution block 402 of FIG. 4) may correspond to a higher (e.g., full) resolution portion of the denoised backbone of the diffusion model. For example, higher resolution component 924A and / or higher resolution circuit 926 may be used to process input features to generate (noisy) latent quantities as part of the denoised backbone of the diffusion model, and to process (denoised) latent quantities to generate output feature data, as discussed above.

[0178] Lower resolution component 924B and / or lower resolution circuit 927 (which may correspond to lower resolution block 112 of FIG. 1, FIG. 2A, FIG. 2B, FIG. 2C and / or FIG. 4, and / or lower resolution block 412 of FIG. 4) may correspond to the lower (e.g., reduced) resolution portion of the denoised backbone of the diffusion model. For example, lower resolution component 924B and / or lower resolution circuit 927 may process the (noisy) latent tensors generated by higher resolution component 924A and / or higher resolution circuit 926 to generate (denoised) latent tensors as discussed above.

[0179] Adapter component 924C and / or adapter circuit 928 (which may correspond to adapter block 115 of FIG. 1, FIG. 2A, FIG. 2B, FIG. 2C, FIG. 3 and / or FIG. 4, and / or lower resolution block 412 of FIG. 4) may correspond to an adapter that can be used to replace a lower (e.g., reduced) resolution portion of the denoised backbone for one or more iterations. For example, adapter component 924C and / or adapter circuit 928 may process the (denoised) latent tensors generated during previous iterations of the backbone, the (noisy) latent tensors generated during the current iteration by higher resolution component 924A and / or higher resolution circuit 927, temporal embeddings, and / or input embeddings to generate the (denoised) latent quantity for the current iteration, as discussed above.

[0180] Processing component 924D and / or processing circuitry 929 are typically used to perform training and / or other processing (or preprocessing) involved in the diffusion model. For example, in some aspects, processing component 924D and / or processing circuitry 929 may generate input embeddings (e.g., CLIP embeddings) based on input data and provide these embeddings as input to a denoising backbone. As another example, in some aspects, processing component 924D and / or processing circuitry 929 may perform downstream processing on features generated by the denoising backbone to generate model outputs (e.g., synthetic images). As yet another example, processing component 924D and / or processing circuitry 929 may generate loss components and / or update the parameters of the diffusion model during the training phase.

[0181] Although depicted as separate components and circuits in Figure 9 for clarity, the higher resolution circuit 926, the lower resolution circuit 927, the adapter circuit 928, and / or the processing circuit 929 may be implemented jointly or separately in other processing devices of the processing system 900, such as within the CPU 902, GPU 904, DSP 906, and NPU 908.

[0182] Generally speaking, the processing system 900 and / or its components can be configured to perform the methods described herein.

[0183] It is worth noting that, in other aspects, such as when the processing system 900 is a server computer, components of the processing system 900 may be omitted. For example, in other aspects, the multimedia component 910, the wireless connectivity component 912, the sensor processing unit 916, the ISP 918, and / or the navigation processor 920 may be omitted. Furthermore, the components of the processing system 900 may be distributed among multiple devices.

[0184] Example Terms

[0185] Specific implementation examples are described in the following numbered clauses: Clause 1: A method comprising: during a first iteration of processing data using a denoised backbone of a diffusion machine learning model, generating a first latent tensor using lower-resolution blocks of the denoised backbone; during the first iteration, generating a first feature tensor based on processing the first latent tensor using higher-resolution blocks of the denoised backbone, the higher-resolution blocks using a higher resolution than the lower-resolution blocks; generating a second latent tensor based on processing the first latent tensor using adapter blocks of the denoised backbone; and during a second iteration of processing the data using the denoised backbone, generating a second feature tensor based on processing the second latent tensor using the higher-resolution blocks.

[0186] Clause 2: The method according to Clause 1, wherein the generation of the second potential tensor is further performed based on using the adapter block to process the embedding corresponding to the second iteration.

[0187] Clause 3: The method according to any one of Clauses 1 to 2, wherein the generation of the second potential tensor is further performed based on using the adapter block to process the embedding corresponding to the input of the diffusion machine learning model.

[0188] Clause 4: The method according to any one of Clauses 1 to 3, wherein the generation of the second potential tensor is further performed based on processing the embedding generated by the higher resolution block using the adapter block.

[0189] Clause 5: The method according to any one of Clauses 1 to 4, wherein the adapter block includes an identity mapping.

[0190] Clause 6: The method according to any one of Clauses 1 to 5, wherein the adapter block uses a set of learned parameters to generate the second potential tensor based on the first potential tensor.

[0191] Clause 7: The method according to Clause 6, wherein the adapter block performs one or more convolution operations to generate the second potential tensor.

[0192] Clause 8: The method according to any one of Clauses 6 to 7, wherein: the adapter block includes an encoder and a decoder, and generating the second potential tensor includes: generating a compressed tensor based on processing the first potential tensor using the encoder, and generating the second potential tensor based on processing the compressed tensor using the decoder.

[0193] Clause 9: The method according to any one of Clauses 1 to 8, the method further comprising: generating a third latent tensor based on processing the first latent tensor using the adapter block; and generating a third feature tensor based on processing the third latent tensor using the higher resolution block during a third iteration of processing the data using the diffusion machine learning model.

[0194] Clause 10: The method according to any one of Clauses 1 to 9, the method further comprising: generating a third latent tensor based on processing the second latent tensor using the adapter block; and generating a third feature tensor based on processing the third latent tensor using the higher resolution block during a third iteration of processing the data using the diffusion machine learning model.

[0195] Clause 11: The method according to any one of Clauses 1 to 10, the method further comprising: during a third iteration of processing the data using the diffusion machine learning model, generating a third latent tensor using the lower resolution block; and during the third iteration, generating a third feature tensor based on processing the third latent tensor using the higher resolution block.

[0196] Clause 12: The method according to any one of Clauses 1 to 11, wherein: the diffusion machine learning model is trained using distillation from a teacher machine learning model, and the teacher machine learning model uses a plurality of higher resolution blocks and corresponding plurality of lower resolution blocks.

[0197] Clause 13: The method according to any one of Clauses 1 to 12, wherein: the lower resolution block is used to generate the first latent tensor to generate a first delay, the adapter block is used to generate the second latent tensor to generate a second delay, and the second delay is less than the first delay.

[0198] Clause 14: The method according to any one of Clauses 1 to 13, wherein the second potential tensor is not generated based on the first feature tensor.

[0199] Clause 15: The method according to any one of Clauses 1 to 14, the method further comprising: generating an image based at least in part on the first feature tensor and the second feature tensor; and outputting the image as output from the diffusion machine learning model.

[0200] Item 16: A method comprising: during a first iteration of processing data using a first denoised backbone of a teacher diffusion machine learning model, generating a first latent tensor using lower-resolution blocks of the first denoised backbone; during a first iteration of processing data using a second denoised backbone of a student diffusion machine learning model, generating a second latent tensor using adapter blocks of the second denoised backbone; generating a loss based on the first latent tensor and the second latent tensor; and updating one or more parameters of the adapter blocks based on the loss.

[0201] Clause 17: The method according to Clause 16 further comprises: updating one or more parameters of a higher resolution block of the second denoising backbone based on the loss; and updating one or more parameters of a lower resolution block of the second denoising backbone based on the loss.

[0202] Clause 18: The method according to any one of Clauses 16 to 17, wherein the generation of the second potential tensor is further performed based on using the adapter block to process the embedding corresponding to the first iteration.

[0203] Clause 19: The method according to any one of Clauses 16 to 18, wherein the generation of the second latent tensor is further performed based on using the adapter block to process the embedding corresponding to the input to the student diffusion machine learning model.

[0204] Clause 20: The method according to any one of Clauses 16 to 19, wherein the generation of the second potential tensor is further performed based on processing the embedding generated by the higher resolution block of the second denoising backbone using the adapter block.

[0205] Clause 21: The method according to any one of Clauses 16 to 20, wherein the adapter block performs one or more convolution operations to generate the second potential tensor.

[0206] Clause 22: The method according to any one of Clauses 16 to 21, wherein: the adapter block includes an encoder and a decoder, and generating the second potential tensor includes: generating a compressed tensor based on processing a third potential tensor using the encoder, and generating the second potential tensor based on processing the compressed tensor using the decoder.

[0207] Clause 23: The method according to any one of Clauses 16 to 22, the method further comprising: generating a third latent tensor based on processing the second latent tensor using the adapter block; and generating a feature tensor based on processing the third latent tensor using a higher resolution block of the second denoised backbone during a second iteration of processing the data using the Student Diffusion Machine Learning Model.

[0208] Clause 24: The method according to any one of Clauses 16 to 23, the method further comprising: during a second iteration of processing the data using the Student Diffusion Machine Learning Model, generating a third latent tensor using lower-resolution blocks of the second denoised backbone; and during the second iteration, generating a feature tensor based on processing the third latent tensor using higher-resolution blocks of the second denoised backbone.

[0209] Clause 25: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 24.

[0210] Clause 26: A processing system comprising components for performing the method according to any one of Clauses 1 to 24.

[0211] Clause 27: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 24.

[0212] Clause 28: A computer program product embodied on a computer-readable storage medium, said computer-readable storage medium including code for performing a method according to any one of Clauses 1 to 24.

[0213] Additional Notes

[0214] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Additionally, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0215] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0216] As used in this article, the phrase “at least one of the items” refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0217] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0218] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0219] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be construed in accordance with 35 USC §112(f) unless it is expressly referred to herein by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known thereafter, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: One or more memories, the one or more memories including processor-executable instructions; and one or more processors configured to execute processor-executable instructions and cause the processing system to: generate a first latent tensor using lower-resolution blocks of the first denoised backbone during a first iteration of processing data using a first denoised backbone of a teacher diffusion machine learning model; During the first iteration of processing the data using the second denoising backbone of the student diffusion machine learning model, an adapter block of the second denoising backbone is used to generate a second latent tensor; a loss is generated based on the first latent tensor and the second latent tensor. And update one or more parameters of the adapter block based on the loss.

2. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: update one or more parameters of the higher resolution blocks of the second denoising backbone based on the loss; and update one or more parameters of the lower resolution blocks of the second denoising backbone based on the loss.

3. The processing system according to claim 1, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute the processor executable instructions to cause the processing system to use the adapter block to process the embedding corresponding to the first iteration.

4. The processing system according to claim 1, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute the processor executable instructions to cause the processing system to use the adapter block to process the embeddings corresponding to the inputs to the student diffusion machine learning model.

5. The processing system according to claim 1, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute processor-executable instructions to enable the processing system to use the adapter block to process the embedding generated by the higher resolution block of the second denoising backbone.

6. The processing system of claim 1, wherein the adapter block performs one or more convolution operations to generate the second potential tensor.

7. The processing system according to claim 1, wherein: The adapter block includes an encoder and a decoder, and in order to generate the second potential tensor, the one or more processors are configured to further execute processor-executable instructions to cause the processing system to generate a compressed tensor based on processing the third potential tensor using the encoder; And the second potential tensor is generated based on processing the compressed tensor using the decoder.

8. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: generate a third latent tensor based on processing the second latent tensor using the adapter block; and generate a feature tensor based on processing the third latent tensor using a higher resolution block of the second denoising backbone during a second iteration of processing the data using the Student diffusion machine learning model.

9. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: generate a third latent tensor using lower-resolution blocks of the second denoised backbone during a second iteration of processing the data using the Student diffusion machine learning model; and generate a feature tensor based on processing the third latent tensor using higher-resolution blocks of the second denoised backbone during the second iteration.

10. A processor-implemented method, the processor-implemented method comprising: During the first iteration of processing the data using the first denoised backbone of the teacher diffusion machine learning model, the first latent tensor is generated using the lower resolution blocks of the first denoised backbone. During the first iteration of processing the data using the second denoising backbone of the student diffusion machine learning model, an adapter block of the second denoising backbone is used to generate a second latent tensor; a loss is generated based on the first latent tensor and the second latent tensor. And update one or more parameters of the adapter block based on the loss.

11. The processor-implemented method according to claim 10, further comprising: The loss is used to update one or more parameters of the higher resolution block of the second denoising backbone; And based on the loss, update one or more parameters of the lower resolution block of the second denoising backbone.

12. The processor-implemented method of claim 10, wherein the generation of the second potential tensor is further performed based on processing the embedding corresponding to the first iteration using the adapter block.

13. The processor-implemented method of claim 10, wherein the generation of the second latent tensor is further performed based on using the adapter block to process the embedding corresponding to the input of the student diffusion machine learning model.

14. The processor-implemented method of claim 10, wherein the generation of the second potential tensor is further performed based on processing the embedding generated by the higher resolution block of the second denoising backbone using the adapter block.

15. The processor-implemented method of claim 10, wherein the adapter block performs one or more convolution operations to generate the second potential tensor.

16. The processor-implemented method according to claim 10, wherein: The adapter block includes an encoder and a decoder, and generating the second potential tensor includes: generating a compressed tensor based on processing a third potential tensor using the encoder; and generating the second potential tensor based on processing the compressed tensor using the decoder.

17. The processor-implemented method according to claim 10, further comprising: A third potential tensor is generated based on the use of the adapter block to process the second potential tensor; And during the second iteration of processing the data using the student diffusion machine learning model, a feature tensor is generated based on processing the third latent tensor using a higher resolution block of the second denoised backbone.

18. The processor-implemented method according to claim 10, further comprising: During the second iteration of processing the data using the student diffusion machine learning model, a third latent tensor is generated using lower-resolution blocks of the second denoised backbone. And during the second iteration, a feature tensor is generated based on processing the third latent tensor using a higher resolution block of the second denoised backbone.

19. One or more non-transitory computer-readable media, the non-transitory computer-readable media comprising processor-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to: generate a first latent tensor using lower-resolution blocks of the first denoised backbone during a first iteration of processing data using a first denoised backbone of a teacher diffusion machine learning model; During the first iteration of processing the data using the second denoising backbone of the student diffusion machine learning model, an adapter block of the second denoising backbone is used to generate a second latent tensor; a loss is generated based on the first latent tensor and the second latent tensor. And update one or more parameters of the adapter block based on the loss.

20. The non-transitory computer-readable medium of claim 19, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: update one or more parameters of a higher resolution block of the second denoising backbone based on the loss; and update one or more parameters of a lower resolution block of the second denoising backbone based on the loss.

21. The non-transitory computer-readable medium according to claim 19, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute the processor executable instructions to cause the processing system to use the adapter block to process the embedding corresponding to the first iteration.

22. The non-transitory computer-readable medium according to claim 19, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute the processor executable instructions to cause the processing system to use the adapter block to process the embeddings corresponding to the inputs to the student diffusion machine learning model.

23. The non-transitory computer-readable medium according to claim 19, wherein, In order to generate the second potential tensor, the one or more processors are configured to further execute processor-executable instructions to enable the processing system to use the adapter block to process the embedding generated by the higher resolution block of the second denoising backbone.

24. The non-transitory computer-readable medium of claim 19, wherein the adapter block performs one or more convolution operations to generate the second potential tensor.

25. The non-transitory computer-readable medium according to claim 19, wherein: The adapter block includes an encoder and a decoder, and in order to generate the second potential tensor, the one or more processors are configured to further execute processor-executable instructions to cause the processing system to generate a compressed tensor based on processing the third potential tensor using the encoder; And the second potential tensor is generated based on processing the compressed tensor using the decoder.

26. The non-transitory computer-readable medium of claim 19, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: generate a third latent tensor based on processing the second latent tensor using the adapter block; and generate a feature tensor based on processing the third latent tensor using a higher resolution block of the second denoising backbone during a second iteration of processing the data using the Student diffusion machine learning model.

27. The non-transitory computer-readable medium of claim 19, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to: generate a third latent tensor using lower-resolution blocks of the second denoised backbone during a second iteration of processing the data using the Student diffusion machine learning model; and generate a feature tensor based on processing the third latent tensor using higher-resolution blocks of the second denoised backbone during the second iteration.

28. A processing system, the processing system comprising: A component for generating a first latent tensor using lower-resolution blocks of the first denoised backbone during a first iteration of processing data using a first denoised backbone of a teacher diffusion machine learning model; A component for generating a second latent tensor using an adapter block with a second denoised backbone during a first iteration of processing data using a second denoised backbone of a student diffusion machine learning model; a component for generating a loss based on the first latent tensor and the second latent tensor; and a component for updating one or more parameters of the adapter block based on the loss.

29. The processing system according to claim 28, further comprising: A component for updating one or more parameters of the higher resolution block of the second denoising backbone based on the loss; And components for updating one or more parameters of the lower-resolution block of the second denoising backbone based on the loss.

30. The processing system of claim 28, wherein the generation of the second latent tensor is further performed based on: using the adapter block to process the embedding corresponding to the first iteration; using the adapter block to process the embedding corresponding to the input to the student diffusion machine learning model; and using the adapter block to process the embedding generated by the higher resolution block of the second denoising backbone.

Images