Training generative networks using auxiliary forward diffusion

Abstract

Techniques for training a machine-learned model are provided. In particular, the present disclosure is directed to a training scheme that enables fast sampling for a generative model during inference without compromising sample diversity and quality.

Description

TRAINING GENERATIVE NETWORKS USING AUXILIARY FORWARD DIFFUSION CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to United States Provisional Patent Application No.63 / 507,313, filed June 9, 2023, the contents of which are incorporated herein by reference in their entirety. FIELD

[0002] The present disclosure relates generally to generative models such as denoising diffusion models. More particularly, the present disclosure relates to utilizing both a discriminator network and a regressor network for training a generative network. BACKGROUND

[0003] Generative models are used in various domains, such as image generation, video synthesis, audio generation, and point cloud generation, among others. Generative adversarial networks (“GANs”) include a min-max game framework to implicitly model data distribution and enable one-step generation. Denoising diffusion probabilistic models (“DDPMs”), also known as score-based generative models, recover an original data distribution through iterative denoising from an initial random Gaussian noise vector.

[0004] However, these models face a challenge in ensuring high-quality sampling, mode coverage, and fast sampling speed simultaneously. For example, while DDPMs excel in generating high-quality samples compared to other types of generative models, they typically require thousands of iterative steps to obtain the highest-quality results. These long sampling steps are based on the assumption that the reversed diffusion distribution can be approximated by Gaussian distributions when the noise addition in the forward diffusion process is small. However, if the noise addition is significant, the reversed diffusion distribution can become a non-Gaussian multimodal distribution. Consequently, reducing the number of sampling steps for faster generation would violate the assumptions made by existing models and introduce bias in generated samples.SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure is directed to a computer- implemented method for training a machine-learned model. The method includes processing a noised training example using at least one denoising generative model to obtain a denoised output. The method includes providing the denoised output to a discriminator model and a regressor model. The method includes receiving a first output from the discriminator model based on the denoised output. The method includes receiving a second output from the regressor model based on the denoised output, wherein the regressor model is trained to minimize a negative conditional entropy term. The method includes training the denoising generative model based on a combination of the first output and the second output.

[0007] In some implementations, the method further comprises performing an auxiliary forward diffusion step on the denoised output to generate a forward-noised output. In some implementations, the negative conditional entropy term comprises a difference between the second output provided by the regressor model and the forward-noised output.

[0008] In some implementations, training the denoising generative model comprises training the denoising generative model to maximize the negative conditional entropy term.

[0009] In some implementations, the first output from the discriminator model is evaluated using a generative adversarial network loss.

[0010] In some implementations, the discriminator model and the regressor model share at least a plurality of layers.

[0011] In some implementations, the first output is a classification output that indicates that the discriminator network was fooled by the denoised output.

[0012] In some implementations, the second output comprises an attempt by the regressor model to predict a forward-noised output generated by application of an auxiliary forward diffusion step on the denoised output.

[0013] In some implementations, training the denoising generative model based on the combination of the first output and the second output comprises minimizing a marginal distribution and optimizing a conditional distribution.

[0014] In some implementations, training the denoising generative model based on the combination of the first output and the second output comprises training the denoising generative model to maximize the negative conditional entropy term.

[0015] Another example aspect is directed to a computing system configured to perform the method described above.

[0016] Another example aspect is directed to a non-transitory computer-readable medium storing computer-executable instructions to perform the method described above.

[0017] Another example aspect is directed to a non-transitory computer-readable medium storing a denoising generative model trained according to the method described above.

[0018] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0019] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0021] Figure 1A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[0022] Figure 1B depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0023] Figure 1C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0024] Figure 2 depicts a block diagram of an example training scheme for an example generative model according to example embodiments of the present disclosure.

[0025] Figure 3 depicts a flow chart diagram of an example method to train a generative model according to example embodiments of the present disclosure.

[0026] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations. DETAILED DESCRIPTION Overview

[0027] Generally, the present disclosure is directed to training a machine-learned model. In particular, the present disclosure is directed to a training scheme that enables fast sampling for a generative model during inference without compromising sample diversity and quality.

[0028] Diffusion models contain two processes: forward diffusion and reversion, or reverse diffusion. Forward diffusion gradually generates corrupted data given a starting point by interpolating between sampled data and added noise (e.g., Gaussian noise) over a number of steps. The reversion process, or reverse diffusion, attempts to “reverse” the forward diffusion process by predicting a less noisy version of an image at a diffusion step given the image at a later diffusion step.

[0029] Current denoising diffusion probabilistic models (“DDPMs”) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. Denoising diffusion generative adversarial networks (“DDGANs”) attempt to circumvent this limitation by integrating a generative adversarial network (“GAN”) for larger jumps in the diffusion process. However, DDGANs encounter scalability limitations when applied to large datasets.

[0030] Current DDGANs attempt to indirectly match a conditional distribution of reverse diffusion via matching joint distributions of noisy samples and the paired denoised samples under the constraint of forward diffusion sampling.

[0031] Aspects of the present disclosure provide advantages over current DDGANs by matching implicit and explicit factors. A training objective of a DDGAN can be similar to the training objective presented in Equation 1 provided below, where the DDGAN’s formulation indirectly matches the conditional distribution ^^^^( ^^^^^^^^−1| ^^^^^^^^) and ^^^^^^^^( ^^^^^^^^−1| ^^^^^^^^) by matching the joint distribution between ^^^^(^^^^ ^^^^−1|^^^^ ^^^^)and ^^^^ ^^^^( ^^^^ ^^^^−1| ^^^^ ^^^^) under a sampling strategy^^^^^^^^( ^^^^0) ^^^^( ^^^^ ^^^^−1| ^^^^0) ^^^^( ^^^^ ^^^^| ^^^^ ^^^^−1). ^^^^))direction to obtain ^^^^( ^^^^^^^^−1, ^^^^^^^^) = ^^^^( ^^^^^^^^| ^^^^^^^^−1) ^^^^( ^^^^^^^^−1) and ^^^^^^^^( ^^^^^^^^−1, ^^^^^^^^) = ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1) ^^^^^^^^( ^^^^^^^^−1).These denoising distributions are decomposed into two components: a pair of marginal distributions ^^^^( ^^^^^^^^−1), ^^^^^^^^( ^^^^^^^^−1) of denoised sampled data and a pair of conditional forward diffusion distributions ^^^^( ^^^^^^^^| ^^^^^^^^−1), ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1), also referred to as "auxiliary forward diffusion.” Together, these formulate the denoising distribution at each step of the denoising process.distributions do not have explicit forms, the marginal distributions can be matched implicitly by minimizing the Jensen-Shannon divergence (“JSD”) via adversarial learning. For the conditional distribution of forward diffusion, because the terms have explicit forms, matching can be performed using Kullback-Leibler (“KL”) divergence.

[0034] Two training objectives can be used: an implicit GAN objective and an explicit L2 reconstruction loss. The implicit GAN objective is applied to the marginal distribution, while the L2 loss is adopted for the conditional distribution. The process of matching conditional distributions can be considered auxiliary forward diffusion (“AFD”).

[0035] It is possible to approximately match a joint distribution from a traditional DDGAN by matching these pairs of distributions. For example, let ^^^^(xt−1, xt) and pθ(xt−1, xt) denote the data distribution from forward diffusion and the denoising distribution specified by a denoiser Gθ, respectively. This results in the inequality provided in Equation 2.

[0036] Equation 2: ^^^^ ^^^^ ^^^^ ( ^^^^( ^^^^ ^^^^−1, ^^^^ ^^^^), ^^^^ ^^^^( ^^^^ ^^^^−1, ^^^^ ^^^^)) ≤ 2 ^^^^1�2 ^^^^ ^^^^ ^^^^ ( ^^^^( ^^^^ ^^^^−1), ^^^^ ^^^^( ^^^^ ^^^^−1)) +2 ^^^^2�2 ^^^^ ^^^^ ( ^^^^ ^^^^( ^^^^ ^^^^| ^^^^ ^^^^−1be determined as shown in Equation 3.

[0039] Equation 3: m ^^i^^n max ^^^^ ^^^^^^^^( ^^^^^^^^ ^^^^ ^^^^ 0) ^^^^( ^^^^ ^^^^−1| ^^^^0) ^^^^( ^^^^ ^^^^| ^^^^ ^^^^−1)[ ^^^^^^^^ ^^^^ ^^^^( ^^^^( ^^^^^^^^−1)|| ^^^^^^^^( ^^^^^^^^−1)) + ^^^^^^^^ ^^^^ ^^^^^^^^ ^^^^� ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1) ^^^^( ^^^^^^^^| ^^^^^^^^−1)

[0040] In Equation 3, ^^^^^^^^ ^^^^ ^^^^is the weight for the matching of auxiliary forward diffusion and the adversarial part ^^^^^^^^ ^^^^ ^^^^is the standard GAN objective. To match the distributions of auxiliary forward diffusion via ^^^^ ^^^^ divergence, Equation 3 can be expanded as shown in Equation 4.

[0041] Equation 4: ^^^^ ^^^^ ( ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1)|| ^^^^( ^^^^^^^^| ^^^^^^^^−1)) =∫^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1)log ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1) − ^^^^^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ = − ^^^^ ^^^^^^^^^^^^ ^^^^ + ^^^^ ^^^^^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ and^^^^.

[0043] To match the distributions of AFD using the KL divergence, the KL divergence can be created using a combination of the negative entropy of forward diffusion with the cross entropy between the sampled forward diffusion from the data and the predictions of model. Using Equation 4, optimizing the cross-entropy between an empirical distribution and a Gaussian distribution can be performed using mean square error between empirical and Gaussian distributions, which is possible because forward diffusion follows a Gaussian distribution. On the other hand, the negative entropy term is intractable. However, the parameterized forward diffusion can be estimated on samples from a denoiser. This conditional entropy can be approximated using an adversarial training objective of less complexity than the overall adversarial training objective of the model. For example, Equation 5 illustrates an adversarial training objective that can be used to maximize the conditional entropy.

[0044] Equation 5: min max ^^^^^^^^ ( ^log ^^^^ ( ^^^^ | ^^^^ ). ^^^^ ^^^^^^^^ ^^^ ^^^^| ^^^^ ^^^^−1) ^^^^ ^^^^ ^^^^−1

[0045] Thisan iterative min-max game between a generator and a conditional estimator that minimizes negative conditional entropy as a reconstruction loss. This adversarial process can be performed as long as the system has access to the likelihood of ^^^^^^^^( ^^^^^^^^| ^^^^^^^^−1), which is forward diffusion and follows Gaussian distribution.

[0046] The GAN can also be utilized to minimize the JSD of marginal distributions, which gives us both a JSD term and the reconstruction loss as parameters for training the model. For example, similarly to DDGANS, the GAN can define ^^^^^^^^( ^^^^^^^^−1| ^^^^^^^^): = ^^^^( ^^^^^^^^−1| ^^^^^^^^, ^^^^0= ^^^^^^^^( ^^^^^^^^, ^^^^)) using the posterior distribution. In the distribution matching objective, the GAN can be applied to minimize the JSD of the marginal distributions and the L2 reconstruction to optimize cross entropy. In some embodiments, ^^^^′^^^^−1can be defined as the data sampled from the newly defined distribution, and ^^^^′^^^^are sampled from ^^^^′^^^^−1via forward diffusion. Thus, one example final training objective is provided in Equation 6.

[0047] Equation 6: min ^ m ^^ax ∑^^^^>0^^^^^^^^( ^^[[−log( ^^^^ ( ^^^^ , ^^^^))] + ^^^^^ ^^^^^^, ^^^^ ^^^^ 0) ^^^^( ^^^^ ^^^^−1| ^^^^0) ^^^^( ^^^^ ^^^^| ^^^^ ^^^^−1) ^^^^ ^^^^−1�−log�1 − ^^^^(1− ^^^^ )‖ ^^^ ‖2 � �2^^^^ ^′ ^^^^ − ^^^^ − ^^^^ ( ^^^^′ )− ^^^^′^^^^( ^^^^′^^^^−1, ^^^^) + ^^^^−1 ^^^^−1 ^^^^ ^^^^−1 ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^] to minimize thethe discriminator and the regression model can be

[0049] Thus, in some implementations, a computing system can include a Semi-Implicit Denoising Diffusion Model (SIDDM) that integrates both implicit and explicit training objectives to enhance the generative capabilities of the model while maintaining computational efficiency. Specifically, the SIDDM can employ an implicit GAN objective applied to the marginal distribution of the model outputs. This allows the model to effectively learn and replicate the complex, high-dimensional data distributions encountered in practical applications such as image and audio generation. Concurrently, an explicit L2 reconstruction loss can be adopted for the conditional distribution, which aids in refining the fidelity of the generated outputs by minimizing the error between the denoised outputs and their corresponding ground truth data.

[0050] As such, some example implementations abandon purely adversarial training objectives and, instead decompose the training objectives into marginal and conditional distributions, where the conditional distribution can be optimized with a less complex training objective, which leads to stable training for updating the denoising model. Additionally, in some implementations, the model structure for the proposed GAN can share a model structure with DDPMs and can be improved based on advanced DDPM network structures. This combination ensures superior training convergence without introducing additional computational overhead as compared to existing DDGANs.

[0051] Furthermore, in some implementations, the discriminator component can utilize a UNet-like structure, which is beneficial for enhancing the detail and quality of the generative outputs. This architectural choice leverages the ability of UNet structures to capture and utilize contextual information from both local and global perspectives within the data, thereby improving the model's ability to produce highly detailed and accurate generative outputs. The integration of a UNet-like structure in the discriminator not only supports more effective trainingby providing robust feature extraction capabilities but also contributes to the overall stability and performance of the generative model during inference tasks.

[0052] Additionally, a regularization technique involving an auxiliary denoising task can be implemented to stabilize the training of the discriminator without incurring any additional computational burden. A denoising output can be obtained from the discriminator and reconstructed with the ground truth of the training data.

[0053] In some implementations, the proposed techniques can utilize JSD and / or KL divergence to effectively match distributions during the model training process. JSD can be beneficial for minimizing the discrepancy between the marginal distributions of the model outputs and the target data in an adversarial learning context. Meanwhile, KL divergence can be employed for matching conditional distributions, where it helps in quantifying the difference between the predicted distribution and the actual distribution of the data, thus aiding in the refinement of the model's accuracy.

[0054] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the proposed techniques provide enhanced sampling efficiency. In particular, the proposed techniques can significantly accelerate the sampling process during inference. For example, this can be achieved by not enforcing a parametric distribution for the reverse step, allowing the model to take larger steps and thus reduce the total number of required iterations. For example, in image generation, this means that high-quality images can be generated in fewer steps compared to traditional methods such as DDPMs. Performing fewer steps corresponds to a reduced consumption of computational resources such as processor cycles, memory usage, network bandwidth, etc.

[0055] As another example technical effect, the systems and methods of the present disclosure can provide improved scalability. For example, the proposed technology can handle large-scale datasets more effectively than some existing models like DDGANs. This scalability can be facilitated by the semi-implicit approach which optimizes the matching of distributions through less computationally intensive objectives, thus accommodating larger and more complex datasets without a proportional increase in computational demand.

[0056] As yet another example technical effect, the proposed techniques enable stabilization of training. In particular, the proposed regularization technique involving an auxiliary denoising task helps stabilize the training of the discriminator. This technical effecthelps to maintain the model's performance over extensive training periods and across diverse datasets, thereby ensuring consistent output quality. Improving the stability of training can result in fewer training failures (e.g., failure to converge). Therefore, improving the stability of training can result in an overall reduction in consumption of computational resources such as processor cycles, memory usage, network bandwidth, etc.

[0057] The systems and methods of the present disclosure can be applied to a number of different use cases or applications. For example, the trained models can be utilized in image generation, where they can transform textual descriptions into high-fidelity visual content. As another example, they can be employed in video synthesis to generate or alter video content dynamically, which can be beneficial for the entertainment industry, virtual reality experiences, and training simulations.

[0058] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail. Example Devices and Systems

[0059] Figure 1A depicts a block diagram of an example computing system 100 that performs generative modeling according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0060] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0061] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0062] In some implementations, the user computing device 102 can store or include one or more generative models 120. For example, the generative models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi- headed self-attention models (e.g., transformer models). Example generative models 120 are discussed with reference to Figures 2 and 3.

[0063] In some implementations, the one or more generative models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single generative model 120.

[0064] More particularly, the generative models 120 can be used to generate new content, such as new images or text, from a random noise sample given a prompt.

[0065] Additionally or alternatively, one or more generative models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the generative models 140 can be implemented by the server computing system 140 as a portion of a web service (e.g., a generative service). Thus, one or more models 120 can be stored and implemented at the user computing device 102 and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0066] The user computing device 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0067] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0068] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0069] As described above, the server computing system 130 can store or otherwise include one or more generative models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi- headed self-attention models (e.g., transformer models). Example models 140 are discussed with reference to Figures 2 and 3.

[0070] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0071] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0072] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0073] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0074] In particular, the model trainer 160 can train the generative models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, input images or text.

[0075] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0076] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or moresets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0077] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0078] The machine-learned models described in this specification may be used in a variety of tasks, applications, and / or use cases.

[0079] In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

[0080] Figure 1A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computingdevice 102 can implement the model trainer 160 to personalize the models 120 based on user- specific data.

[0081] Figure 1B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0082] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0083] As illustrated in Figure 1B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0084] Figure 1C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0085] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0086] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0087] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API). Example Model Arrangements

[0088] Figure 2 depicts a block diagram of an example scheme for training a generative model 200 according to example embodiments of the present disclosure. Generative model 200 receives a training example 202, such as an image, and applies one or more forward diffusion steps 204 to the training example 202 to generate a noised training example 206. The noised training example 206 is then provided to a denoising generative model 208, which de-noises the noised training example 206 into a denoised output 210, which is the best approximation the denoising generative model 208 can approximate of the original training example 202.

[0089] This denoised output 210 is then provided to a discriminator model 212 and a regressor model 214. The discriminator model 212 attempts to determine if the denoised output 210 is a true image or a fake image. In other words, the denoising generative model 208 attempts to “fool” the discriminator model 212 into thinking the denoised output 210 is the training example 202 and not an output generated from noised training example 206. The discriminator model 212 can provide a first output. For example, the first output can be a classification of the denoised output as real or fake. The first output can be input into or evaluated with a GAN loss term 216. The GAN loss term 216 can indicate whether or not the discriminator model 212 was fooled by the denoising output 210 or not. This GAN loss term 216 can then be used (e.g., via backpropagation) to further train and refine the discriminator model 212, the regressor model 214 and / or the denoising generative model 208.

[0090] The regressor model 214 receives the denoised output 210 and performs processing on the denoised output 210 to minimize a negative conditional entropy term 218 for the denoised output. This negative conditional entropy term 218 is a term that quantifies the amount of information needed to describe the outcome of a random variable given the value ofanother random variable in the denoised. The regressor model 214 works to minimize this quantity. In turn, the negative conditional entropy term 218 is used (e.g., via backpropagation) to further train and refine the discriminator model 212, the regressor model 214 and / or the denoising generative model 208. By reducing this value, training can be stabilized for the denoising generative model 208 despite the use of larger steps in diffusion that come from utilizing the discriminator model 212.

[0091] In some embodiments, the discriminator model 212 and the regressor model 214 can share one or more layers that perform similar functionality for both models.

[0092] The denoising output 210 can also undergo one or more auxiliary forward diffusion steps 220 in an attempt to reconstruct the original noised training example 206. This forward-noised output 222 can then be provided as a second factor into minimization of the negative conditional entropy term 218 by the regressor model 214.

[0093] The denoising generative model 208 can be trained on both the GAN loss term 216 and the negative conditional entropy term 218, which provides both larger steps in the training process while being more stable in training than traditional models.

[0094] The denoising generative model 208 can also be trained based on a reconstruction loss term 224 that involves a comparison between the original training example 202 and the denoised output 210 (e.g., a comparison to see how close the denoised output 210 is to the original training example 202). Example Methods

[0095] Figure 3 depicts a flow chart diagram of an example method 300 to training a denoising generative model according to example embodiments of the present disclosure. Although Figure 300 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method 300 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0096] At step 302, a computing system processes a noised training example through at least one denoising generative model to produce a denoised output. In some implementations,this step can include performing an auxiliary forward diffusion step on the denoised output, generating a forward-noised output.

[0097] At step 304, the denoised output is provided to both a discriminator model and a regressor model. The discriminator model, in some implementations, evaluates the denoised output to determine its authenticity, for example providing an indication of whether the output mimics a real training example or not. In particular, the output from this evaluation, received at step 306, can be a classification indicating whether the discriminator model was deceived by the denoised output.

[0098] Concurrently, at step 308, the regressor model processes the denoised output to minimize a negative conditional entropy term. The difference between a second output provided by the regressor model and the forward-noised output can constitute the negative conditional entropy term. The regressor model can operate to refine the model's predictive accuracy regarding the forward-noised output. The second output from the regressor model can be used to further train and stabilize the denoising generative model.

[0099] Finally, at step 310, the denoising generative model is trained based on a combination of the first and second outputs from the discriminator and the regressor models. This training can include optimizing both a marginal distribution and a conditional distribution, enhancing the model's overall performance. In some implementations, step 310 can include maximizing the negative conditional entropy term to achieve a more robust generative model. [000100] In some implementations, the discriminator and regressor models share several layers, promoting efficiency and reducing computational overhead. This shared architecture can facilitate a more integrated approach to evaluating and adjusting the denoised outputs. Moreover, in some implementations, the method 300 includes evaluating the first output from the discriminator model using a generative adversarial network loss, which helps in fine-tuning the discriminator’s ability to evaluate outputs effectively. [000101] The method 300, as illustrated, offers a flexible and effective approach to training denoising generative models, accommodating various implementations that cater to specific needs and objectives within the field of generative modeling.Additional Disclosure [000102] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel. [000103] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for training a machine-learned model, the method comprising: processing a noised training example using at least one denoising generative model to obtain a denoised output; providing the denoised output to a discriminator model and a regressor model; receiving a first output from the discriminator model based on the denoised output; receiving a second output from the regressor model based on the denoised output, wherein the regressor model is trained to minimize a negative conditional entropy term; and training the denoising generative model based on a combination of the first output and the second output.

2. The method of claim 1, wherein the method further comprises: performing an auxiliary forward diffusion step on the denoised output to generate a forward-noised output; and wherein the negative conditional entropy term comprises a difference between the second output provided by the regressor model and the forward-noised output.

3. The method of claim 1, wherein training the denoising generative model comprises training the denoising generative model to maximize the negative conditional entropy term.

4. The method of claim 1, wherein the first output from the discriminator model is evaluated using a generative adversarial network loss.

5. The method of claim 1, wherein the discriminator model and the regressor model share at least a plurality of layers.

6. The method of claim 1, wherein the first output is a classification output that indicates that the discriminator network was fooled by the denoised output.

7. The method of claim 1, wherein the second output comprises an attempt by the regressor model to predict a forward-noised output generated by application of an auxiliary forward diffusion step on the denoised output.

8. The method of claim 1, wherein training the denoising generative model based on the combination of the first output and the second output comprises minimizing a marginal distribution and optimizing a conditional distribution.

9. The method of claim 1, wherein training the denoising generative model based on the combination of the first output and the second output comprises training the denoising generative model to maximize the negative conditional entropy term.

10. A computing system configured to perform the method of any of claims 1-9.

11. A non-transitory computer-readable medium storing computer-executable instructions to perform the method of any of claims 1-9.

12. A non-transitory computer-readable medium storing a denoising generative model trained according to the method of any of claims 1-9.

Images