Training goals for distillation-guided diffusion models

Abstract

A method for training a diffusion model includes compressing the diffusion model by removing at least one of: one or more model parameters or one or more gigabit multiply-add operations (GMACs). The method also includes performing guidance conditionalization to train the compressed diffusion model, the guidance conditionalization combining conditional and unconditional outputs from the respective teacher model. The method further includes performing a step-number distillation on the compressed diffusion model after the directed conditionalization.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 492,492, filed October 23, 2023, entitled “TRAINING OBJECTIVES FORDISTILLING GUIDED DIFFUSION MODELS”, the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] All aspects of this disclosure relate to improved training objectives for distillation-guided diffusion models. Background Technology

[0004] Artificial neural networks can comprise interconnected groups of artificial neurons (e.g., neuron models). An artificial neural network (ANN) can be a computing device or represented as a method to be performed by a computing device. A convolutional neural network (CNN) is a type of feedforward ANN. A CNN can comprise an ensemble of neurons, where each neuron has a receptive field and collectively constructs the input space. CNNs such as deep convolutional neural networks (DCNs) have numerous applications. Specifically, these neural network architectures are used in a variety of technologies, such as image recognition, speech recognition, acoustic scene classification, keyword retrieval, autonomous driving, and other classification tasks.

[0005] In machine learning and data generation, diffusion refers to the method by which generative models transform data through a sequence of reversible transformations. These generative models are called diffusion models. During the diffusion process, a diffusion model starts with a distribution (usually Gaussian) and gradually transforms the data into the desired distribution, thus facilitating tasks such as image synthesis and denoising. Diffusion models require significant computational resources, such as power, memory, and / or processor load, leading to a trade-off between training time and the quality of the generated data. Summary of the Invention

[0006] Some aspects of this disclosure relate to a method for training a diffusion model, which includes compressing the diffusion model by removing one or more model parameters and / or one or more gigabit multiply-accumulate (GMAC) operations. The method also includes performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining conditional and unconditional outputs from a corresponding teacher model. The method also includes performing step distillation on the compressed diffusion model after guided conditionalization.

[0007] Some other aspects of this disclosure relate to an apparatus including means for compressing a diffusion model by removing one or more model parameters and / or one or more GMACs. The apparatus also includes means for performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining conditional and unconditional outputs from a corresponding teacher model. The apparatus also includes means for performing step distillation on the compressed diffusion model after guided conditionalization.

[0008] In some other aspects of this disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for compressing a diffusion model by removing one or more model parameters and / or one or more GMACs. The program code also includes program code for performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining conditional and unconditional outputs from the corresponding teacher model. The program code also includes program code for performing step distillation on the compressed diffusion model after guided conditionalization.

[0009] Additionally, some other aspects of this disclosure relate to an apparatus having one or more processors and one or more memories coupled to the processors and storing instructions that, when executed by the processors, are operable to cause the apparatus to compress a diffusion model by removing one or more model parameters and / or one or more GMACs. Execution of these instructions also causes the apparatus to perform bootstrapping conditionation to train the compressed diffusion model, the bootstrapping conditionation combining conditional and unconditional outputs from the corresponding teacher model. Execution of these instructions further causes the apparatus to perform step distillation on the compressed diffusion model after bootstrapping conditionation.

[0010] In some aspects of this disclosure, a method for training a diffusion model includes randomly selecting a teacher model from a set of teacher models for each iteration of a step-distillation training process. The method also includes applying a clipped input space within the step-distillation of the randomly selected teacher model at each iteration. The method further includes updating the parameters of the diffusion model at each iteration based on guidance from the randomly selected teacher model.

[0011] Some other aspects of this disclosure relate to an apparatus including components for randomly selecting a set of teacher models from a set of teacher models for each iteration of a step-distillation training process. The apparatus also includes components for applying a clipped input space within the step-distillation of the randomly selected teacher models at each iteration. The apparatus further includes components for updating the parameters of the diffusion model at each iteration based on guidance from the randomly selected teacher models.

[0012] In some other aspects of this disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for randomly selecting a teacher model from a set of teacher models for each iteration of a step-distillation training process. The program code also includes program code for applying a pruned input space within the step-distillation of the randomly selected teacher model at each iteration. The program code also includes program code for updating the parameters of the diffusion model at each iteration based on guidance from the randomly selected teacher model.

[0013] Additionally, some other aspects of this disclosure relate to an apparatus having one or more processors and one or more memories coupled to the processors and storing instructions that, when executed by the processors, are operable to cause the apparatus to randomly select a teacher model from a set of teacher models for each iteration of a step-distillation training process. Execution of these instructions also causes the apparatus to apply a pruned input space within the step-distillation of the randomly selected teacher model at each iteration. Furthermore, execution of these instructions causes the apparatus to update the parameters of the diffusion model at each iteration based on guidance from the randomly selected teacher model.

[0014] In some aspects of this disclosure, a method for training a control neural network includes initializing a baseline diffusion model for training the control neural network, each stage of the control neural network training pipeline corresponding to an element of the baseline diffusion model. The method also includes training the control neural network in a step-by-step manner, each stage of the control neural network training pipeline receiving input from the previous stage of the control neural network training pipeline and a corresponding element of the diffusion model.

[0015] Some other aspects of this disclosure relate to an apparatus including components for initializing a baseline diffusion model for training a control neural network, each stage of the control neural network training pipeline corresponding to an element of the baseline diffusion model. The apparatus also includes components for training the control neural network in a step-by-step manner, each stage of the control neural network training pipeline receiving input from the previous stage of the control neural network training pipeline and a corresponding element of the diffusion model.

[0016] In some other aspects of this disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for initializing a baseline diffusion model for training a control neural network, each stage of the control neural network training pipeline corresponding to an element of the baseline diffusion model. The program code also includes program code for training the control neural network in a step-by-step manner, each stage of the control neural network training pipeline receiving input from the previous stage of the control neural network training pipeline and corresponding elements of the diffusion model.

[0017] Additionally, some other aspects of this disclosure relate to an apparatus having one or more processors and one or more memories coupled to the processors and storing instructions that, when executed by the processors, are operable to initialize a baseline diffusion model for training a control neural network, each stage of the control neural network training pipeline corresponding to an element of the baseline diffusion model. Execution of these instructions also causes the apparatus to train the control neural network in a stepwise manner, each stage of the control neural network training pipeline receiving input from the previous stage of the control neural network training pipeline and corresponding elements of the diffusion model.

[0018] Additional features and advantages of this disclosure will be described below. Those skilled in the art will understand that this disclosure can be readily used as the basis for modifying or designing other structures for implementing the same purposes as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of this disclosure as set forth in the appended claims. Novel features considered characteristic of this disclosure, in both their organization and manner of operation, along with further objects and advantages, will be better understood when considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to be a definition of a limitation of this disclosure. Attached Figure Description

[0019] The features, substance, and advantages of this disclosure will become more apparent when understood in conjunction with the accompanying drawings, in which the same reference numerals are consistently used for identification.

[0020] Figure 1 Example implementations of neural networks using a system-on-a-chip (SoC) (including a general-purpose processor) according to certain aspects of this disclosure are illustrated.

[0021] Figure 2A , Figure 2B and Figure 2C This is a diagram illustrating various aspects of a neural network according to this disclosure.

[0022] Figure 2D This is a diagram illustrating an exemplary deep convolutional network (DCN) according to various aspects of this disclosure.

[0023] Figure 3 This is a block diagram illustrating an exemplary deep convolutional network (DCN) according to various aspects of this disclosure.

[0024] Figure 4 This is a block diagram illustrating an exemplary software architecture that enables modularization of artificial intelligence (AI) functions according to various aspects of this disclosure.

[0025] Figure 5This is a block diagram illustrating examples of training pipelines for training diffusion models according to various aspects of this disclosure.

[0026] Figure 6A This is a block diagram illustrating a first variant of an efficient architecture for a diffusion-based generative model according to various aspects of this disclosure.

[0027] Figure 6B This is a block diagram illustrating a second variant of an efficient architecture for a diffusion-based generative model according to various aspects of this disclosure.

[0028] Figure 7 This is a block diagram illustrating examples of guided conditionalization according to various aspects of this disclosure.

[0029] Figure 8 This is a block diagram illustrating examples of step distillation according to various aspects of this disclosure.

[0030] Figure 9 This is a block diagram illustrating examples of step distillation processes according to various aspects of this disclosure.

[0031] Figure 10 It is a block diagram of the ControlNet architecture based on various aspects of this disclosure.

[0032] Figure 11 This is a block diagram illustrating examples of ControlNet training pipelines according to various aspects of this disclosure.

[0033] Figure 12 This is a flowchart illustrating examples of processes for training a stable diffusion model according to various aspects of this disclosure.

[0034] Figure 13 This is a flowchart illustrating an example of a classifier-free guided (CFG) sensing step distillation process according to various aspects of this disclosure.

[0035] Figure 14 This is a flowchart illustrating examples of processes for training control neural network models according to various aspects of this disclosure. Detailed Implementation

[0036] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only configurations in which the described concepts can be practiced. To provide a comprehensive understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0037] Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects described. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the various aspects of this disclosure described. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0038] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” need not be interpreted as superior to or better than other aspects.

[0039] While specific aspects are described, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects are mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or purpose. Rather, aspects of this disclosure are intended to be widely applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0040] In machine learning and data generation, diffusion refers to a generative model's approach of morphologically transforming data through a reversible chain of transformations. These generative models (often called diffusion models) initiate a distribution (typically Gaussian) and progressively transform the data into the desired distribution. This stepwise process facilitates tasks associated with a variety of applications, such as image synthesis and denoising, including but not limited to digital media enhancement, medical imaging, and autonomous systems. However, diffusion models can consume significant computational resources, such as, but not limited to, power, memory, and / or processor load, leading to a trade-off between training time and the quality of the generated data. In some cases, training time can depend on the exact sequence of training levels, the loss function, and / or model initialization.

[0041] It may be desirable to reduce the amount of computational resources used by the diffusion model. Various aspects of this disclosure relate to reducing the inference time of the diffusion model. Inference time refers to the time from receiving input to generating the final output from the diffusion model. Some aspects also relate to reducing the inference time of the ControlNet model.

[0042] In some examples, a training pipeline for a diffusion model (e.g., stable diffusion) is specified to reduce inference time. The training pipeline may include a novel UNet architecture with reduced gigabyte multiply-accumulate operations (GMAC) and model parameters, thereby reducing memory usage and other computational resources. This UNet architecture acts as a more streamlined engine, providing similar or better performance to conventional convolutional network architectures while consuming fewer resources.

[0043] Furthermore, the training pipeline includes a bootstrap conditionalization block that trains the student model using both conditional and unconditional generation, effectively doubling the speed of data generation. This bootstrap conditionalization not only accelerates the generation process but also preserves the integrity and quality of the generated data. In some examples, bootstrap conditionalization using unconditional training methods, along with step distillation with dropout, ensures that the student model remains aligned with the teacher model across all training levels. This alignment prevents pattern collapse.

[0044] Finally, the training pipeline includes step distillation blocks to reduce the number of forward passes in the UNet. For example, the number of forward passes can be reduced from approximately forty to six, which reduces training time. In some examples, step distillation can trim the input space.

[0045] In some examples, one or more training blocks of a stable diffusion training pipeline can be used to train a control neural network (such as ControlNet). In some examples, the UNet architecture can be used to train a control neural network. Furthermore, guided conditionalization and step distillation can be applied to control neural networks.

[0046] These strategic enhancements during the training process of diffusion models, inspired by the efficient UNet architecture, guided conditionation, and step distillation, reduce computational requirements while maintaining (if not enhancing) the quality of the generated data. Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. In some examples, training pipelines including the UNet architecture, guided conditionation, and step distillation can reduce the amount of computational resources used by the diffusion model and / or the control neural network model, thereby reducing the corresponding inference time of the diffusion model and / or the control neural network model.

[0047] Figure 1An example implementation of a System-on-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU configured to train a stable diffusion model, implement a stable diffusion model, train a ControlNet model, and / or implement a ControlNet model. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a weighted neural network), latency, frequency window (bin) information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or may be loaded from memory block 118.

[0048] SOC 100 may also include additional processing blocks tailored for specific functions, such as GPU 104, DSP 106, connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one specific implementation, NPU 108 is implemented within CPU 102, DSP 106, and / or GPU 104. SOC 100 may also include sensor processor 114, image signal processor (ISP) 116, and / or navigation module 120, which may include a global positioning system.

[0049] The SOC 100 may be based on the ARM instruction set. In various aspects of this disclosure, the instructions loaded into the general-purpose processor 102 may include code for performing operations, such as referring to... Figure 12 , Figure 13 and Figure 14 The described process includes one or more operations of 1200, 1300, and / or 1400.

[0050] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0051] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0052] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0053] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0054] The connections between layers of a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In the fully connected neural network 202, neurons in the first layer can transmit their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in the first layer can connect to a finite number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0055] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. Convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0056] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as an in-vehicle camera) is illustrated. The DCN 200 in this example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0057] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and forward passes can then be computed to produce output 222. The DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates 28x28 feature maps. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

[0058] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14×14) is smaller than the size of the first set of feature maps 218 (e.g., 28×28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0059] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 can be the probability that image 226 includes one or more features.

[0060] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0061] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0062] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, a new image (e.g., a speed limit sign in image 226) can be presented to DCN 200, and output 222 can be generated through the forward pass of DCN 200. This output can be considered as an inference or prediction of DCN 200.

[0063] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0064] DCN is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0065] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0066] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The output of the convolutional connections can be thought of as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as correction, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0067] Figure 3 This is a block diagram illustrating a DCN 350. A DCN 350 can include multiple layers of different types based on connectivity and weight sharing. For example... Figure 3 As shown, DCN 350 includes convolutional blocks 354A and 354B. Each convolutional block in convolutional blocks 354A and 354B can be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0068] Although only two convolutional blocks 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks 354A and 354B may be included in the DCN 350 according to design preferences.

[0069] Convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate feature maps. Normalization layer 358 can normalize the output of the convolutional filters. For example, normalization layer 358 can provide whitening or lateral suppression. Max pooling layer 360 can provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0070] Parallel filter banks of deep convolutional networks can be loaded onto an SOC 100 (e.g., Figure 1 The CPU 102 or GPU 104 of the SOC 100 can be used to achieve high performance and low power consumption. In an alternative implementation, a parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN 350 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0071] The DCN 350 may also include one or more fully connected layers 362 (FC1 and FC2). The DCN 350 may also include logistic regression (LR) layers 364. Weights (not shown) to be updated are located between each of the layers 356, 358, 360, 362, and 364 of the DCN 350. The output of each layer (e.g., 356, 358, 360, 362, and 364) can be used as input to the next layer in the DCN 350 (e.g., 356, 358, 360, 362, and 364) to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in convolutional block 354A. The output of the DCN 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data includes a feature from the feature set.

[0072] Figure 4 This is a block diagram illustrating an exemplary software architecture 400 with modular artificial intelligence (AI) functionality. According to various aspects of this disclosure, using architecture 400, applications can be designed that enable SOC 420 (which may be similar to...) Figure 1 The various processing blocks of the SOC 100 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) support one or more operations, such as reference... Figure 14 One or more operations of one or more of the processes described in process 1200, 1300, or 1400 and / or one or more operations of AI application 402. Architecture 400 may be included, for example, in a computing device such as a smartphone.

[0073] AI application 402 can be configured to invoke functions defined in user space 404, which may, for example, provide the detection and recognition of a scene indicating the current location of the computing device (including architecture 400). For example, AI application 402 may configure microphones and cameras differently depending on whether the recognized scene is an office, lecture hall, restaurant, or an outdoor environment such as a lake. AI application 402 may make requests to compiled program code associated with libraries defined in the AI ​​Function Application Programming Interface (API) 406. This request may ultimately rely on the output of a deep neural network configured to provide inferred responses based on, for example, video and location data.

[0074] Runtime engine 408 (which may be compiled code of a runtime framework) may be further accessible to AI application 402. AI application 402 may cause runtime engine 408 to request inference, for example, at specific time intervals or when triggered by events detected by the user interface of AI application 402. Upon causing runtime engine 408 to provide an inference response, the runtime engine may then signal to the operating system (OS) space 410 running on SOC 420, such as kernel 412. In some examples, kernel 412 may be a LINUX kernel. The operating system may then enable sequential relaxation of quantization to be performed on CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. CPU 422 may be directly accessible by the operating system, while other processing blocks may be accessed via drivers, such as drivers 414, 416, or 418 for DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to run on a combination of processing blocks such as CPU 422, DSP 424 and GPU 426, or on NPU 428.

[0075] In machine learning and data generation, diffusion refers to a generative model's approach of morphologically transforming data through a reversible chain of transformations. These generative models (often called diffusion models) initiate a distribution (typically Gaussian) and progressively transform the data into a desired distribution. This stepwise process facilitates tasks associated with a variety of applications, such as image synthesis and denoising, including but not limited to digital media enhancement, medical imaging, and autonomous systems. The transformations are designed to be reversible, meaning they can be reversed. This feature allows the diffusion model to learn how to generate new data by running the transformation process in reverse. During the forward process (from simple to complex distributions), noise is added at each step, further diffusing the data. During the reverse process (from complex distributions to the generated data), noise is progressively removed at each step, gradually refining the generated data to resemble real data. The diffusion model can be trained by tuning its parameters to minimize the difference between the generated and real data. This training process can use a set of real data as a reference. Once trained, the diffusion model can generate new data by starting with samples from an initial simple distribution and applying the learned transformations in reverse, thereby gradually removing noise to produce realistic data.

[0076] Examples of tasks performed by diffusion models include text-to-image generation and image or video editing. Solutions from diffusion models can be used with cameras (e.g., camcorders) as well as for personal computation for image or video generation and editing. Diffusion models can also efficiently generate synthetic image data for training deep learning models, which can be used for autonomous driving, as well as for extended reality (XR), augmented reality (AR), and virtual reality (VR) applications.

[0077] Diffusion-based generative models, such as Stable Diffusion developed by STABILITY AI, ImageGen developed by Google, VideoLDM developed by NVIDIA, and FireFly developed by Adobe, are computationally expensive. For example, iterative denoising requires multiple forward passes of the neural network for each generation. The neural network can be a UNet architecture, named so because the network is U-shaped, including downsampling and upsampling paths.

[0078] Diffusion models can consume significant computational resources, such as, but not limited to, power, memory, and / or processor load, leading to a trade-off between training time and the quality of the generated data. In most cases, training a diffusion model requires substantial computational resources in terms of both data and training time. In some cases, training time can depend on the exact sequence of training levels, the loss function, and / or model initialization.

[0079] Various aspects of this disclosure relate to reducing the inference time of diffusion models. Inference time refers to the time from receiving input to generating the final output from the diffusion model. Some aspects also relate to reducing the inference time of ControlNet models. In some examples, a diffusion model (e.g., stable diffusion) training pipeline is specified to address various aspects of the overall loss function and improves efficiency in terms of training and inference time.

[0080] Some conventional training systems aim to simplify the structure of the model or reduce the number of training steps. Various aspects of this disclosure relate to simplifying the diffusion model while also reducing the number of training steps. Such aspects can reduce training time without sacrificing inference accuracy (e.g., quality). This dual approach of simplifying the model and reducing the number of steps in training distinguishes the various aspects of this disclosure from conventional training processes.

[0081] Figure 5 This is a block diagram illustrating examples of a training pipeline 500 for training a diffusion model according to various aspects of this disclosure. The training pipeline 500 may also be referred to as a stable diffusion training pipeline 500. In various aspects of this disclosure, such as Figure 5 As shown in the example, a sequence of training pipeline 500 is specified to reduce training and inference time. Figure 5 As shown in the example, training pipeline 500 at box 502... arrive The box begins. In most cases of stable diffusion, diffusion models inherently predict epsilon (…). ), which represents incremental noise, while the parameter The velocity is represented. In some other models, the prediction may be directly the data itself for each stage of the diffusion process. In such examples, the diffusion model is transformed from ε-space to velocity space because the velocity space provides a larger signal-to-noise ratio, which may be beneficial for step distillation. This transformation helps improve the performance of the diffusion model. The e-to-v box 502 is the form of a rescaled or reparameterized output.

[0082] parameter This represents the noise term during the diffusion process. Velocity. It is a measure of how fast the diffusion process moves in a certain direction. In the context of diffusion, This could be random fluctuations or noise driving the diffusion process. Diffusion process speed. Based on noise and positioning The function determines it. For example, ,in Indicates the current angle. Angle Noise is defined Direction, amplitude and positioning In other words, with the angle Changes, noise and positioning speed The contribution also changes, allowing the diffusion process to move in different directions and at different speeds.

[0083] At box 504, the UNet architecture is used to process the output from e to v at box 502. The UNet architecture is named as such because the network is U-shaped, comprising both downsampling and upsampling paths. The UNet architecture in conventional state-of-the-art (SOTA) models employs a combination of attention layers and convolutional layers at each level of the UNet. The conventional UNet architecture uses global self-attention and cross-attention operations across all spatial resolutions. Attention operations consume significant computational and memory resources, making inference on edge devices challenging, and on-device generation of artificial intelligence (AI) inference challenging as well.

[0084] Various aspects of this disclosure introduce a hardware-efficient architecture for generative models based on machine learning diffusion. In some aspects, the hardware-efficient architecture is an efficient UNet architecture. According to various aspects of this disclosure, only convolutional blocks are employed for the first stage of UNet. Therefore, in the first stage, which has the highest network resolution, cross-attention is not used to process low-level features of text or image information. Because high-resolution blocks primarily recover low-level details, global self-attention is not employed in these blocks with the highest resolution (e.g., the first stage of the architecture).

[0085] Two variants of the efficient architecture with minimal attention layers are introduced. In both architectures, textual semantics are injected only for spatial resolutions of 32×32 and lower. That is, only convolutional blocks are provided for the first level of the UNet architecture. In the first variant, two convolutional blocks are provided in the downsampling part of the network and three convolutional blocks are provided in the upsampling part. The second variant builds on the first variant architecture and further adopts one less convolutional block per level.

[0086] At box 506, guided conditionalization is specified. In the context of stable diffusion, guided conditionalization involves a process where, for each denoising step or iteration, two UNets are executed: one that merges the text embeddings (with conditionalization), and another that excludes the text embeddings (without conditionalization). Guided conditionalization simplifies the training process by reducing training to only one UNet forward pass, aiming to preserve the same output quality. Thus, with a single forward pass, the diffusion model maintains the quality that would be achieved in a regular system with two forward passes.

[0087] At box 508, step distillation reduces the number of denoising iterations or steps throughout the inference process. Some conventional systems use step distillation as a preprocessing step to refine the loss. In contrast, various aspects of this disclosure combine architectural compression with step distillation and guided conditionalization.

[0088] like Figure 5 As shown in the example, the architecture of the diffusion model can be compressed by using the UNet architecture (box 504). Figure 6A This is a block diagram illustrating a first variant of an efficient architecture for a diffusion-based generative model according to various aspects of this disclosure. Figure 6A The architecture shown is an example of UNet. Figure 6A In the architecture shown, the first level 602 receives textual semantic input. Figure 6AIn the example, the first stage 602 is a 64×64 input stage, but this disclosure is not limited to any such resolution. The first stage 602 includes two convolutional blocks 604 and 606. The first stage 602 does not include any self-attention or cross-attention layers. The second stage 608 has a lower resolution than the first stage 602 and is used to downsample the feature maps generated by the first stage 602. For example, the second stage 608 may have a 32×32 resolution. The second stage 608 includes two convolutional blocks 610 and 612, and an attention layer 614. For example, the attention layer 614 may be a 1024×1024 self-attention and / or cross-attention layer. The attention layer 614 injects textual semantic input into the second stage 608.

[0089] The third level 616 may also include two convolutional blocks and an attention layer (not labeled separately). The third level 616 and its attention layer have a lower resolution than the second level 608. For example, the third level 616 may have a 16×16 spatial resolution, and the attention layer in the third level 616 may be 256×256. The fourth level 618 (also called an intermediate block) includes two convolutional blocks (not labeled separately) and no attention layer. The fourth level 618 may have an 8×8 spatial resolution. The second level 608, the third level 616, and the fourth level 618 may be referred to as downsampling levels or downsampling blocks in the network architecture.

[0090] exist Figure 6A In the example, the upsampling stages (or upsampling blocks) of the architecture include stage 5 620, stage 6 628, and stage 7 630. Stage 5 620 can also be referred to as the intermediate block and includes three convolutional blocks 622, 624, and 626. The spatial resolution of stage 5 620 can be 8×8. Stage 6 628 includes three convolutional blocks and one attention layer (not labeled separately). Similarly, stage 7 630 includes three convolutional blocks and one attention layer (not labeled separately). Stage 6 628 and stage 7 630 can have corresponding resolutions of 16×16 and 32×32, respectively. Stage 8 632 is the output layer and includes three convolutional blocks (not labeled separately) and no attention layer. Therefore, it can be seen that attention is only provided in stages with a resolution of 32×32 or lower. Each stage has two convolutional blocks, i.e., each stage has self-attention and cross-attention, providing good information preservation and simplified optimization capabilities.

[0091] Not only does each stage receive input from the immediately preceding stage, but each upsampling stage also receives input from the corresponding downsampling stage via a skip connection. For example, the output stage (e.g., stage 8 632) receives input from stage 1 602 (via skip connection 640) and also from stage 7 630. Note that the second intermediate block (e.g., stage 5 620) receives input from stage 4 618 without any additional input. Compared to the baseline model, Figure 6AThe efficient architecture shown reduces device latency with almost no impact on output quality.

[0092] Figure 6B This is a block diagram illustrating a second variant of an efficient architecture for a diffusion-based generative model according to various aspects of this disclosure. Figure 6B The second variant shown is in Figure 5 This is built upon the architecture shown and can be considered a pruned version of variant one. The second variant uses one less block of convolution, self-attention, and cross-attention per downsampling level (excluding the input level). Figure 6B In the architecture shown, the first level 652 receives textual semantic input. Figure 6B In the example, the first stage 652 is a 64×64 input stage, but this disclosure is not limited to any such resolution. The first stage 652 includes a convolutional block 654. The first stage 652 does not include any self-attention or cross-attention layers. The second stage 658 has a lower resolution than the first stage 652 and is used to downsample the feature map of the first stage 652. For example, the second stage 658 may have a 32×32 resolution. The second stage 658 includes a convolutional block 660 and an attention layer 664. For example, the attention layer 664 may be a 1024×1024 self-attention and / or cross-attention layer. The attention layer 664 injects textual semantic input into the second stage 658.

[0093] Level 3 690 may also include a convolutional block and an attention layer (not labeled separately). Level 3 690 and its attention layer have a lower resolution than Level 2 658. For example, Level 3 690 may have a 16×16 spatial resolution, and the attention layer in Level 3 690 may be 256×256. Level 4 668 (also called an intermediate block) includes a convolutional block (not labeled separately) and no attention layer. Level 4 668 may have an 8×8 spatial resolution. Level 2 658, Level 3 690, and Level 4 668 may be referred to as downsampling levels or downsampling blocks in the network architecture.

[0094] exist Figure 6B In the example, the architecture's upsampling stages (or upsampling blocks) include stage 5 670, stage 678, and stage 7 680. Stage 5 670 can also be referred to as the intermediate block and includes two convolutional blocks 672 and 674. The spatial resolution of stage 5 670 can be 8×8. Stage 678 includes two convolutional blocks and an attention layer (not labeled separately). Similarly, stage 7 680 includes two convolutional blocks and an attention layer (not labeled separately). Stages 678 and 680 can have corresponding resolutions of 16×16 and 32×32, respectively. Stage 8 682 is the output layer and includes two convolutional blocks (not labeled separately) and no attention layer. Therefore, it can be seen that attention is only provided in stages with a resolution of 32×32 or lower.

[0095] Not only does each stage receive input from the immediately preceding stage, but each upsampling stage also receives input from the corresponding downsampling stage via a skip connection. For example, the output stage (e.g., stage 8 682) receives input from stage 1 652 (via skip connection 640) and also from stage 7 680. Note that the second intermediate block (e.g., stage 5 670) receives input from stage 4 668 without any additional input. Compared to the baseline model and also with... Figure 6A Compared to the first variant shown, Figure 6B The efficient architecture shown reduces device-side latency with minimal impact on generation quality. For example, although variant two may discard some features, the results are similar and it is computed faster than variant one.

[0096] like Figure 5 As shown in the example, architectural compression (box 504) is followed by guided conditionation (box 506). Guided conditionation can be used in teacher-student frameworks within the field of machine learning. In such frameworks, a more complex or knowledgeable model, referred to as the teacher, facilitates the training of a less complex model, referred to as the student. The goal is to make the student mimic the teacher's behavior as accurately as possible. In this context, the term "guided" refers to additional cues or information provided by the teacher to the student during the training phase. This guidance helps the student make more accurate and faster predictions. On the other hand, "conditionation" refers to modulating the student model's learning trajectory based on the guidance received at the student model, thereby fine-tuning the student model's behavior to align with the teacher's input.

[0097] Figure 7 This is a block diagram illustrating examples of guided conditionalization 700 according to various aspects of this disclosure. For example... Figure 7 As shown in the example, guided conditionalization 700 uses a teacher model 702 and a student model 704. The teacher model 702 can receive two prompts: an empty string (e.g., an unconditional prompt) and a conditional prompt (in...). Figure 7 (The example shown is "dogs in the park"). Output based on unconditional cues can be called unconditional output, and output based on conditional cues can be called conditional output. Additionally, for student model 704, in addition to the guiding embedding 706, it receives a cue that combines the conditional and unconditional cues with a guiding value associated with the output of teacher model 702. Guiding embedding 706 may be based on features extracted from the output of teacher model 702. In some examples, student model 704 uses the guiding value to modulate its output, thereby attempting to match teacher model 702.

[0098] The loss function measures the difference between the output of student model 704 and the sum of the unconditional and conditional outputs of teacher model 702. The loss is backpropagated through student model 704 to update its weights and / or parameters, thereby moving the output of student model 704 closer to the output of teacher model 702. In some examples, the output of student model 704 may be aligned with the unconditional and conditional outputs of teacher model 702 during a single forward pass. This alignment becomes very significant downstream, especially when formulating the loss function. Guided conditionalization can be performed at each training time step.

[0099] The validity of guided conditionalization 700 can be based on the inherent consistency of unconditional prompts, which are independent of conditional prompts. That is, conditional prompts change, while unconditional prompts remain constant, exhibiting global, consistent behavior. Regarding the output space, a scaling type is performed, which can be based on a scaling factor conditional on the guided value. In some examples, the guided value is processed through certain layers of the student model 704, such as sine, cosine, and / or projection layers. The output of the student model 704 can be based on the processing of the guided value through various layers. During intermediate levels, the guided value is used as a modulator aligned with the unconditional prompt. That is, the guided value modulates or scales the output of the student model 704, where the degree of modulation is determined by the guided embedding 706. Modulation can occur in the intermediate activation space of the student model 704, such that a second teacher model is not queried.

[0100] In some examples, guided conditionation can be used for high-level or low-level tuning. In some such examples, a guided embedding is added to an intermediate activation within a ResNet block (in the current architecture), or any conditionalized or modulated layer that processes time-step embeddings can also process guided value embeddings, and downstream layers of that particular block then process the guided value associated with the guided embedding. This process is not limited to a single block; instead, it extends across all blocks within the student model 704. By applying this conditionation across all blocks, the student model 704 achieves more uniform and comprehensive modulation, allowing the tuning facilitated by the embeddings to be processed throughout the student model 704.

[0101] refer to Figure 7 The described guided conditionation 700 can be distinguished from regular guided conditionation and is more like time-step conditionation. In some examples, guided conditionation 700 uses SNR loss only in later steps of the training process, compared to regular guided conditionation which uses signal-to-noise ratio (SNR) across all training steps, making the training scheme more adaptive. That is, the use of SNR loss is schedulable.

[0102] Furthermore, conventional guided conditionation tends to focus on specific scenarios, such as "dogs in the park." In contrast, guided conditionation 700 exhibits a wider range, even during training. Specifically, guided conditionation 700 also considers the unconditional case, where an empty string is taken as input and the guidance is set to zero. By doing so, the training process prompts the student model 704 to behave in accordance with the unconditional output of the teacher model 702, thus training a more general student model 704 capable of handling both conditional and unconditional scenarios. This type of training helps to preserve the unconditional scenario, a feature not prioritized in conventional guided conditionation, which focuses solely on the conditional aspect. The focus of conventional guided conditionation on the conditional scenario can lead to inconsistencies in the vector field, a key component in the diffusion process. Unconditional training is used to train the baseline diffusion model, which improves upon obtaining satisfactory gradient field or score function estimates across the entire space. Unconditionally training the student model 704 mitigates the inconsistencies in the vector field.

[0103] As shown in training pipeline 500, step distillation (box 508) is specified after guided conditionalization (box 506). Step distillation is the process of moving from the student model to the teacher model. Specifically, step distillation uses model compression to encapsulate the performance of the teacher model within a simpler model structure. This process can be performed in a sequence of training steps or levels, where the order of these steps is specified for efficient knowledge transfer from the teacher model to the student model. In some examples, at each step in this sequence, the student model learns from the output of the teacher model, which serves as a form of “soft label” or guidance to help the student model discern underlying patterns within the data. A significant feature of step distillation is the reduction in the number of iterative or denoising steps required to achieve the desired performance.

[0104] In most cases, step distillation uses a specialized loss function to measure the difference between the outputs of the teacher and student models at each step. This loss metric guides the training process, helping the student model gradually improve its performance. For example, in scenarios involving image recognition, a complex teacher model initially trained on a large dataset can guide the training of a simpler student model. Through step distillation, the student model is trained on subsets of data over a series of steps, each designed to refine the student model's performance based on the teacher's knowledge. The loss function measures the performance gap at each step, guiding the student model to adjust its parameters to enhance performance. This process ultimately yields a student model that incrementally learns from the teacher model with fewer iterations, thus becoming proficient in tasks such as image recognition with less computational resources.

[0105] Figure 8 This is a block diagram illustrating example 800 of step distillation according to various aspects of this disclosure. Figure 8 In the example, two teacher models 802 and one student model 806 are specified. The step distillation shown in Example 800 can be performed progressively in a multi-stage manner. For example, in a six-step process, the process can start with a 24-step teacher-initiated process, transition to 12 steps, and finally to six steps. In some examples, student model 806 can simultaneously simulate two steps of teacher model 802. For example, given the current estimate of the latent variables in the diffusion process, denoted as the current latent variable... This latent variable can be updated to the first latent variable. Second hidden variable In teacher model 802, from the current latent variables... Transition to the first hidden variable Second hidden variable At that time, student model 806 directly predicted the second latent variable. Instead of predicting (Because the weights are shared and / or the student model 806 is initialized with the weights of the teacher model 802, it will typically be predicted.) Step distillation in Example 800 allows the student model 806 to scale its process such that two teacher steps correspond to one student step. The teacher model 802 can be initiated with any number of steps, such as, but not limited to, 24, 48, 50, 100, or 200. According to various aspects of this disclosure, two steps of the teacher model 802 are distilled into one step of the student model 806. Step distillation reduces computational effort, making the student model 806 more efficient, while simulating the performance of the teacher model 802 with the reduced number of steps.

[0106] Conventional classifier-free guided (CFG) perceptual step distillation can be performed based on the following function:

[0107] (1)

[0108] In equation 1, This represents a score function estimation network in the v (velocity) space. The score function estimation network estimates the gradient of the log-likelihood in the diffusion model. Additionally, Indicates the time step within the diffusion process. Represents the time step within the diffusion process The intermediate hidden variables. This represents the text conditionalization used for sampling from conditional diffusion models or score estimates, and This represents the null conditionalization used for sampling from the unconditional diffusion model or score estimate. In the context of step distillation, CFG guides the training of both the teacher and student models. This is achieved by... (The sentence is incomplete and requires further context to translate accurately.) The results of the teacher model and the student model are compared for guidance. In Equation 1, the score function estimation network can be conditionally defined using text. and / or null conditionalization To handle intermediate implicit variables Weighting factor The contributions of conditional and unconditional components in the estimate can be adjusted. For step distillation, the teacher can use an ordinary differential equation (ODE) or stochastic differential equation (SDE) solver to expand to two sampling steps in the reverse process, and the guided conditional student model is forced to match the teacher's more steps within the student's fewer steps.

[0109] Conventional CFG-aware distillation may perform two forward passes in each iteration, which can increase latency. Latency can increase further when the guided conditional model is used as the teacher model during step-by-step distillation, as the guided conditional model leads to error accumulation across successive levels, impacting the accuracy of the distillation process. Furthermore, the specific implementation of guided conditionalization is associated with performance degradation. Conventional systems can use conditional sampling for both guided and step-by-step distillation. However, conventional systems ignore the benefits of unconditional sampling. Given that unconditional sampling finds its application within CFG and has an impact on the network's overall coverage and performance (especially in later stages of training), it may be desirable to maintain unconditional sampling across all levels. In addition to these challenges, a common practice in conventional diffusion models is to clip the input space to the range (-1, 1). However, when this clipping occurs within the two denoising diffusion implicit model (DDIM) steps of the teacher, inconsistencies arise in the default equations. This inconsistency poses a challenge to achieving a reliable and efficient distillation process. These outlined issues highlight the complexities faced in conventional CFG-aware distillation processes.

[0110] Various aspects of this disclosure relate to improving guided student distillation by reducing the time associated with the learning process while maintaining the accuracy level of the student model. Figure 9 This is a block diagram illustrating example 900 of a step-by-step distillation process according to various aspects of this disclosure. Figure 9 In Example 900, during the training process, the step distillation process alternates between a guided conditional teacher model 902 (using one UNet model per step) and a CFG-aware teacher model 904 (using two UNet models per step). By learning from the CFG-aware teacher model 904, the step-distilled student model 906 avoids error accumulation across multiple levels, which could occur during a regular step distillation process.

[0111] In some examples, the number of steps of the pruning can be specified to prune the input space represented by x within the teacher's objective, to maintain consistency between the teacher's two steps, as described by the equation transformation. In Equation 2, and The time-dependent noise level coefficients represent the forward (and backward) processes in the control diffusion model, which are captured by the latent variable z as the input space x and noise. The function is defined as ,and

[0112] (2)

[0113] Additionally, unconditional training used for guided conditionalization can be maintained, and step distillation with dropout can be introduced. This strategy aligns the overall behavior of the step-distilled student model 906 with the overall behavior of both the CFG teacher and the GC teacher, thereby improving performance and preventing mode collapse.

[0114] In some examples, signal-to-noise ratio (SNR) loss can be used to guide conditionalization. In such examples, instead of applying SNR loss across all steps, a training time schedule can be specified to incorporate SNR weighting, such that the student model 906 of the step distillation is trained in line with the default model during the early stages of distillation, resulting in a gradual and structured learning curve. Specifically, the step distillation process described in Example 900 consistently maintains unconditional training, which improves regularization and overall model behavior. This process differs from conventional step distillation processes that typically apply SNR weighting. SNR weighting can shift focus towards the data space, especially early in the diffusion process. However, if not carefully managed, SNR weighting can lead to undesirable model behavior. As discussed, various aspects of this disclosure implement learning scheduling instead of applying conventional SNR loss. In some examples, the step distillation process initially avoids applying SNR loss, allowing the model to focus uniformly on each stage of the diffusion process. As training progresses, SNR loss is gradually incorporated, shifting the model's focus more towards the data space. This phased approach, similar to course learning, aims to balance model generalization with quality preservation, a balance that has not been efficiently achieved in existing methods.

[0115] Finally, end-to-end fine-tuning of the efficient architecture is specified to obtain a consistent score function estimate and refine the SNR loss towards the tail of the training process. After distillation, this fine-tuning process regularizes the step-distilled student model 906, making it a reliable and consistent score function estimator, thereby improving performance and accuracy. That is, while architecture compression and bootstrapping conditionalization are instrumental, they can sometimes propagate unwanted artifacts from the teacher model to the student model. To mitigate the propagation of unwanted artifacts, end-to-end fine-tuning can be performed at each stage after architecture distillation and bootstrapping conditionalization, where unconditional sampling is slightly emphasized. This fine-tuning process manages the vector field, avoids the propagation of any unwanted features, and facilitates a more reliable and efficient learning process.

[0116] The step distillation process described in Example 900 reduces hyperparameter sensitivity and overall training necessity. As mentioned above, this modification results in better-performing step distillation, making the training scheme more manageable and efficient. Various aspects of this disclosure utilize a unique combination of a CFG-aware teacher model 904 and a guided step distillation student model 906, compared to a conventional step distillation process. This pairing resolves and corrects inconsistencies observed during conventional step distillation at training time. To address these inconsistencies, a transformation to the X space is implemented, with necessary corrections performed before switching back to the Z space. This step in the distillation process corrects misalignments, thus contributing to a more stable training process.

[0117] As discussed, the specific implementation of the training process described in Training Pipeline 500 differs from a conventional training process. For example, layer-by-layer distillation (box 502) can be specified for architecture compression. Layer-based distillation can be used for guided optimization paths. In contrast, conventional training systems attempt to prune the architecture in an end-to-end manner. Furthermore, in conventional training systems, a performance degradation is observed when guided conditionation is implemented, which can propagate to the step-by-step distillation stage. To mitigate this performance degradation, although the student model is a guided conditional student—meaning it runs only one UNet per iteration—the initial step-by-step distillation objective is adjusted to account for the teacher model without guided conditionation. This adjustment helps avoid error accumulation across sequential training levels, thereby minimizing the reduction in accuracy. Additionally, in some examples, the teacher model without guided conditionation is associated with two UNet forward passes per iteration, whereas the student model is associated with only one UNet forward pass per iteration.

[0118] Various aspects of this disclosure can also improve the training and inference of control neural networks. ControlNet can be an example of a control neural network. A control neural network is a neural network that controls a diffusion model by integrating additional semantic guidance into a text-to-image diffusion model. For ease of explanation, the various aspects discussed below will use the example of ControlNet. The aspects of this disclosure are not limited to ControlNet, as other types of control neural networks can be used. Unlike conventional diffusion models, ControlNet facilitates the generation of images with higher levels of detail and accuracy by utilizing supplementary guiding cues such as edge information, depth perception, segmentation maps, and / or pose data. ControlNet can be used for a variety of tasks or applications, such as, but not limited to, moving cameras, data generation, image generation, video generation, and / or image editing.

[0119] Figure 10 This is a block diagram of the ControlNet architecture 1000 based on various aspects of this disclosure. Figure 10In the example, the ControlNet architecture 1000 includes a diffusion model 1002. The diffusion model 1002 may be pre-trained. The diffusion model 1002 may be a baseline diffusion model, where each level of the diffusion model 1002 is represented by a different neural network block. In some examples, the ControlNet model 1004 receives the input x from the diffusion model 1002 and combines the input x with an additional input c used as a conditional factor for the creation of the final image (e.g., the final output y). This additional input c can range from simple doodles to full-fledged depth or edge maps. The combined input can be received at a trainable copy 1006 of the ControlNet model 1004. The output of the trainable copy 1006 can modulate or update the output y from the diffusion model 1002 before the output y is passed down to subsequent layers. The additional input c and the output of the trainable copy 1006 can be processed by a zero-convolutional layer 1008, where instead of initializing the convolutional layer with random weights, all weights or parameters are initialized with zeros. By conditioned on these inputs (x and c), ControlNet model 1004 guides diffusion model 1002 to produce an image that is closely aligned with the additional input c.

[0120] Training the ControlNet model 1004 is non-intrusive relative to the diffusion model 1002. During the training phase, the ControlNet model 1004 does not change the weights or parameters of the diffusion model 1002. This strategy maintains the basic characteristics of the diffusion model 1002, while the ControlNet model 1004 contributes its modulation to enhance the output, thereby guiding the generation process in a controlled manner.

[0121] ControlNet model 1004 is constructed to be built on top of baseline diffusion model 1002, thereby ensuring that text-to-image generation capabilities remain intact without altering baseline diffusion model 1002. Instead, ControlNet model 1004 enhances the generation process by processing additional information. Because the original weights of baseline diffusion model 1002 are frozen, ControlNet model 1004 operates essentially in parallel, leveraging the existing architecture without directly modifying baseline diffusion model 1002.

[0122] Figure 11 This is a block diagram illustrating examples of a ControlNet training pipeline 1100 according to various aspects of this disclosure. Figure 11 In the example, the ControlNet training pipeline 1100 is mounted on the reference... Figure 5 The described stable diffusion training pipeline is 500. (Example) Figure 11As shown, each stage of the ControlNet training pipeline 1100 (also known as ControlNet distillation) corresponds to a UNet derived from the stable diffusion training pipeline 500. The process begins at box 1102 with a compressed UNet from which layers can be removed or replaced to initiate a new, efficient architectural version of ControlNet. This efficient architecture performs the standard functions of ControlNet. In some respects, the efficient ControlNet architecture at box 1102 can be based on a model trained from scratch or a model derived from an existing UNet architecture. In some examples, the efficient UNet at box 504 is trained followed by the efficient ControlNet at box 1102. The transition to an efficient or compressed ControlNet is achieved by using a compressed UNet architecture. This simplified version aims to maintain the functionality of standard ControlNet while operating on a more efficient framework.

[0123] like Figure 11 As shown, for guided conditionation, a guided conditional student version of either UNet or the baseline diffusion model is used (box 506). At box 1104, the ControlNet from the previous stage (box 1102) is taken and initialized using guided conditionation specifically designed for ControlNet models. Because UNet is already guided conditional, minimal training can be specified to obtain a guided conditional ControlNet that mimics the behavior of a regular ControlNet.

[0124] In some cases, four forward passes can be specified for a regular ControlNet, two for ControlNet and two for UNet, each with and without text embeddings. However, according to various aspects of this disclosure, based on the training pipeline 1100, two forward passes of the teacher model can be simulated through a single forward pass of ControlNet. This efficiency is achieved by reducing the computational load while maintaining the desired output quality.

[0125] At each stage of the ControlNet training pipeline 1100, the output of the ControlNet from the previous stage and the corresponding UNet from the stable diffusion training pipeline 500 are obtained. If bootstrap conditionation is being applied, the bootstrap conditional unit is obtained, and the ControlNet from the previous stage is fine-tuned. Similarly, for step distillation at box 1106, the step distillation unit (box 508) and the bootstrap conditional ControlNet are obtained and fine-tuned to simulate the desired behavior.

[0126] Regardless of guided conditionation and step distillation, the goal of the ControlNet training pipeline 1100 is to enable ControlNet to mimic the behavior of the baseline diffusion model. This process is described as obtaining the final checkpoint of a specific stage in the stable diffusion training pipeline 500, and ControlNet acquiring the output from the previous stage in the ControlNet training pipeline 1100.

[0127] It should be noted that the steady-state diffusion model is frequently mentioned throughout this description. However, this disclosure contemplates any diffusion model and is not limited to the steady-state diffusion model or any other particular diffusion model.

[0128] Figure 12 This is a flowchart illustrating an example of a process 1200 for training a diffusion model according to various aspects of this disclosure. Process 1200 may be described by reference to... Figure 1 The described SOC 100 is executed. (As follows) Figure 12 As shown, process 1200 begins at box 1202 by compressing the diffusion model by removing one or more model parameters and / or one or more gigabit multiply-accumulate (GMAC) operations. At box 1204, process 1200 performs guided conditionalization to train the compressed diffusion model, which combines the conditional and unconditional outputs from the corresponding teacher model. At box 1206, process 1200 performs step distillation on the compressed diffusion model after guided conditionalization.

[0129] Figure 13 This is a flowchart illustrating an example of process 1300 for a CFG-sensing step-counting distillation process according to various aspects of this disclosure. Process 1300 may be derived from, as referenced... Figure 1 The described SOC 100 is executed. (As follows) Figure 13 As shown, process 1300 begins at box 1302 by randomly selecting a teacher model from a set of teacher models for each iteration of the step distillation training process. At box 1304, process 1300 applies a clipped input space within the step distillation of the randomly selected teacher model at each iteration. At box 1306, process 1300 updates the parameters of the diffusion model at each iteration based on guidance from the randomly selected teacher model.

[0130] Figure 14 This is a flowchart illustrating an example of a process 1400 for training a ControlNet model according to various aspects of this disclosure. Process 1400 may be described by reference to... Figure 1 The described SOC 100 is executed. (As follows) Figure 14As shown, process 1400 begins at block 1402 by initializing a baseline diffusion model for training the control neural network, with each stage of the control neural network training pipeline corresponding to an element of the baseline diffusion model. At block 1404, process 1400 trains the control neural network in a step-by-step manner, with each stage of the control neural network training pipeline receiving input from the previous stage of the control neural network training pipeline and the corresponding element of the diffusion model.

[0131] Specific implementation examples are described in the following numbered clauses:

[0132] Clause 1. A method for training a diffusion model, the method comprising: compressing the diffusion model by removing at least one of: one or more model parameters or one or more gigabyte multiply-accumulate (GMAC) operations; performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining conditional and unconditional outputs from a corresponding teacher model; and performing step distillation on the compressed diffusion model after the guided conditionalization.

[0133] Clause 2. The method according to Clause 1, wherein: the conditional output is based on a text string received at the first teacher model; and the unconditional output is based on an empty string received at the second teacher model.

[0134] Clause 3. The method according to any one of Clauses 1 to 2, wherein: the compressed diffusion model receives a guiding value from the guiding embedding during the guiding conditionation; and the guiding value modulates the output of the compressed diffusion model.

[0135] Clause 4. The method according to any one of Clauses 1 to 3, wherein: the step distillation comprises two sequential teacher models and a compressed diffusion model; and the step distillation comprises a step of distilling two steps associated with the two sequential teacher models into a compressed diffusion model.

[0136] Clause 5. The method according to any one of Clauses 1 to 4, the method further comprising an ε-to-velocity conversion prior to compressing the diffusion model.

[0137] Clause 6. The method according to any one of Clauses 1 to 5, wherein the diffusion model includes the UNet architecture.

[0138] Clause 7. An apparatus comprising: one or more processors; and one or more memories coupled to the one or more processors and storing instructions which, when executed by the one or more processors, are operable to cause the apparatus to perform any one of Clauses 1 to 6.

[0139] Clause 8. An apparatus comprising at least one component for performing any one of Clauses 1 to 6.

[0140] Clause 9. A computer program comprising code for causing a device to perform any one of Clauses 1 to 6.

[0141] The various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components with similar numbering plus functional components.

[0142] As used, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, choosing, building, etc.

[0143] As used, the phrase "at least one of the items in the list" refers to any combination of these items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0144] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic component, discrete hardware component, or any combination thereof designed to perform the described functions. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0145] The steps or algorithms of the methods described in this disclosure may be directly embodied in hardware, software modules executed by a processor, or a combination of both. The software modules may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. Software modules may include a single instruction or multiple instructions and may be distributed across several different code segments, across different programs, and across multiple storage media. The storage medium may be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor.

[0146] The disclosed method includes one or more steps or actions for implementing the described method. The steps and / or actions of the method may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0147] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. This processing system may utilize a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. In some respects, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0148] A processor may be responsible for managing the bus and general-purpose processing, including executing software stored on a machine-readable medium. A processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. By way of example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. A machine-readable medium may be embodied as a computer program product. A computer program product may include packaging material.

[0149] In a hardware implementation, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media, or any portion thereof, can be external to the processing system. By way of example, machine-readable media may include transmit lines, carrier waves modulated by data, and / or computer components separate from the device, all accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or a general-purpose register file. Although the various components discussed may be described as having a specific location, such as local components, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0150] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of machine-readable medium, all of which are linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron and nervous system models. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, bus interface, user interface, supporting circuitry, and at least a portion of machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality of the processing system depends on the specific application and the overall design constraints imposed on the system as a whole.

[0151] Machine-readable media may include multiple software modules. These software modules include instructions that, when executed by a processor, enable the processing system to perform various functions. Software modules may include send and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard disk drive into RAM. During the execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a general-purpose register file for processor execution. When the functionality of a software module is referred to below, it will be understood that such functionality is implemented by the processor when executing the instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0152] If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. A storage medium can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and is accessible to a computer. Additionally, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then such coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. The disks and optical discs used include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs. ® Optical discs, where magnetic disks typically reproduce data magnetically, and optical discs reproduce data optically using lasers. Therefore, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0153] Therefore, certain aspects may include a computer program product for performing the presented operations. For example, such a computer program product may include a computer-readable medium on which instructions are stored (and / or encoded) that can be executed by one or more processors to perform the described operations. In some aspects, the computer program product may include packaging material.

[0154] Furthermore, it should be understood that modules and / or other suitable components for performing the described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such devices can be coupled to a server to facilitate the delivery of components for performing the described methods. Alternatively, the various methods described can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that the user terminal and / or base station can obtain the various methods once the storage component is coupled to or provided to the device. In addition, any other suitable techniques suitable for providing the described methods and techniques to the device may be utilized.

[0155] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, variations, and alterations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. An apparatus for training a diffusion model, the apparatus comprising: One or more processors; and One or more memories, coupled to the one or more processors and storing instructions that, when executed by the one or more processors, can operate the device to: The diffusion model is compressed by removing at least one of the following: one or more model parameters or one or more giga multiply-accumulate operations (GMAC). Guided conditionalization is performed to train the compressed diffusion model, wherein the guided conditionalization combines the conditional and unconditional outputs from the corresponding teacher model; as well as Step distillation is performed on the compressed diffusion model after the induction conditionation.

2. The apparatus according to claim 1, wherein: The conditional output is based on the text string received at the first teacher model; and The unconditional output is based on the empty string received at the second teacher model.

3. The apparatus according to claim 1, wherein: The compressed diffusion model receives boot values ​​from the boot embedding during the boot conditionation; and The output of the diffusion model is modulated by the guided value.

4. The apparatus according to claim 1, wherein: The step distillation includes two sequential teacher models and a compressed diffusion model; and The step distillation includes a step that distills two steps associated with the two sequential teacher models into a compressed diffusion model.

5. The apparatus of claim 1, wherein execution of the instructions further causes the apparatus to perform an ε-to-velocity conversion before compressing the diffusion model.

6. The apparatus of claim 1, wherein the diffusion model comprises the UNet architecture.

7. A method for training a diffusion model, the method comprising: The diffusion model is compressed by removing at least one of the following: one or more model parameters or one or more giga multiply-accumulate operations (GMAC). Guided conditionalization is performed to train the compressed diffusion model, wherein the guided conditionalization combines the conditional and unconditional outputs from the corresponding teacher model; as well as Step distillation is performed on the compressed diffusion model after the induction conditionation.

8. The method according to claim 7, wherein: The conditional output is based on the text string received at the first teacher model; and The unconditional output is based on the empty string received at the second teacher model.

9. The method according to claim 7, wherein: The compressed diffusion model receives boot values ​​from the boot embedding during the boot conditionation; and The output of the diffusion model is modulated by the guided value.

10. The method according to claim 7, wherein: The step distillation includes two sequential teacher models and a compressed diffusion model; and The step distillation includes a step that distills two steps associated with the two sequential teacher models into a compressed diffusion model.

11. The method of claim 7, further comprising performing an ε-to-velocity conversion before compressing the diffusion model.

12. The method of claim 7, wherein the diffusion model comprises the UNet architecture.

13. An apparatus for training a diffusion model, the apparatus comprising: Components used to compress the diffusion model by removing at least one of the following: one or more model parameters or one or more gigabit multiply-accumulate operations (GMAC). A component for performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining the conditional and unconditional outputs from the corresponding teacher model; and A component for performing step distillation on the compressed diffusion model after the guided conditionation.

14. The apparatus according to claim 13, wherein: The conditional output is based on the text string received at the first teacher model; and The unconditional output is based on the empty string received at the second teacher model.

15. The apparatus according to claim 13, wherein: The compressed diffusion model receives boot values ​​from the boot embedding during the boot conditionation; and The output of the diffusion model is modulated by the guided value.

16. The apparatus according to claim 13, wherein: The step distillation includes two sequential teacher models and a compressed diffusion model; and The step distillation includes a step that distills two steps associated with the two sequential teacher models into a compressed diffusion model.

17. The apparatus of claim 13, further comprising a component for performing an ε-to-velocity conversion prior to compressing the diffusion model.

18. The apparatus of claim 13, wherein the diffusion model comprises the UNet architecture.

19. A non-transitory computer-readable medium having program code for training a diffusion model recorded thereon, the program code being executed by a processor and comprising: Program code used to compress the diffusion model by removing at least one of the following: one or more model parameters or one or more gigabit multiply-accumulate operations (GMAC); Program code for performing guided conditionalization to train the compressed diffusion model, the guided conditionalization combining conditional and unconditional outputs from the corresponding teacher model; and Program code for performing step distillation on the compressed diffusion model after the guided conditionation.

20. The non-transitory computer-readable medium according to claim 19, wherein: The conditional output is based on the text string received at the first teacher model; and The unconditional output is based on the empty string received at the second teacher model.

21. The non-transitory computer-readable medium according to claim 19, wherein: The compressed diffusion model receives boot values ​​from the boot embedding during the boot conditionation; and The output of the diffusion model is modulated by the guided value.

22. The non-transitory computer-readable medium according to claim 19, wherein: The step distillation includes two sequential teacher models and a compressed diffusion model; and The step distillation includes a step that distills two steps associated with the two sequential teacher models into a compressed diffusion model.

23. The non-transitory computer-readable medium of claim 19, wherein the program code further comprises program code for performing an ε-to-velocity conversion prior to compressing the diffusion model.

24. The non-transitory computer-readable medium of claim 19, wherein the diffusion model includes the UNet architecture.

Images