Diffusion model having improved accuracy and reduced consumption of computational resource

Abstract

To provide a computer-implemented method for use of a diffusion model having improved accuracy.SOLUTION: A method comprises: obtaining input data, the input data comprising one or more channels; and providing the input data to a machine-learned diffusion model. The machine-learned diffusion model comprises: a noising model comprising a plurality of noising stages, the noising model configured to introduce noise to receive the input data and produce intermediate data in response to receipt of the input data; and a denoising model configured to reconstruct output data from the intermediate data. The method also comprises receiving the output data from the machine-learned diffusion model. The diffusion model can include a learned noise schedule. Additionally and / or alternatively, input to the denoising model can include a set of Fourier features. Additionally and / or alternatively, the diffusion model can be trained based at least in part on a continuous-time loss for an evidence lower bound.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 210,314, filed June 14, 2021. U.S. Provisional Patent Application No. 63 / 210,314 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to machine learning and, more particularly, to diffusion models with improved accuracy. [Background technology]

[0003] Likelihood-based generative modeling is a central task in machine learning that is fundamental for a wide range of applications. Autoregressive models have long been the dominant model class for this task due to their tractability in likelihood and representability. Diffusion models are a class of machine-learned models that includes noisy and denoising models. Diffusion models have not yet rivaled autoregressive models in density estimation benchmarks. Summary of the Invention [Means for solving the problem]

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computing system that utilizes Fourier features for improved fine-scale prediction. The computing system includes one or more processors and one or more non-transitory computer-readable media, the non-transitory computer-readable media collectively storing at least a denoising model of a machine-learned diffusion model, the diffusion model including a plurality of noiseification stages, the denoising model configured to receive input data and generate latent data in response to receiving the input data, and a denoising model configured to reconstruct output data from the latent data, wherein an input to the denoising model includes a set of Fourier features including a linear projection of a channel of at least one of the plurality of noiseification stages, and instructions that, when executed by the one or more processors, cause the computing system to execute the denoising model to process the latent data to generate output data.

[0006] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using an improved diffusion model, the method including: acquiring, by a computing system comprising one or more computing devices, input data, the input data including one or more channels; providing, by the computing system, the input data to a machine-learned diffusion model, the machine-learned diffusion model including a plurality of noiseification stages, the noiseification model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data; and a denoising model configured to reconstruct output data from the intermediate data, the input to the denoising model including a set of Fourier features comprising a linear projection of channels of at least one of the plurality of noiseification stages; and receiving, by the computing system, output data from the machine-learned diffusion model.

[0007] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store at least a noisization model of a diffusion model, the diffusion model including a noisization model including a plurality of noisization stages, the noisization model configured to introduce noise to input data according to a noise schedule to produce intermediate data, and a denoising model configured to reconstruct output data from the intermediate data, the noise schedule being a learned noise schedule including one or more learned parameter values.

[0008] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using an improved diffusion model, the method including: acquiring, by a computing system comprising one or more computing devices, input data, the input data including one or more channels; providing, by the computing system, the input data to a machine-learned diffusion model, the machine-learned diffusion model including a plurality of noisization stages, the noisization model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data, and a denoising model configured to reconstruct output data from the intermediate data, the diffusion model including a learned noise schedule; and receiving, by the computing system, output data from the machine-learned diffusion model.

[0009] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for training a diffusion model while consuming fewer computational resources, the method including: acquiring, by a computing system comprising one or more computing devices, training data, the training data including one or more channels; providing, by the computing system, the training data to a machine-learned diffusion model, the machine-learned diffusion model including a plurality of noisization stages, the noisization model configured to receive the training data and introduce noise to produce intermediate data in response to receiving the training data, and a denoising model configured to reconstruct output data from the intermediate data; and determining, by the computing system, a training loss based at least in part on use of the machine-learned diffusion model with the training data, the diffusion model being trained by optimizing parameters of the machine-learned diffusion model toward an evidence lower bound, the evidence lower bound comprising a continuous-time loss.

[0010] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using an improved diffusion model, the method including receiving compressed data and decompressing the compressed data with a denoising model trained as part of the diffusion model, wherein at least one of: 1) an input to the denoising model includes one or more Fourier features; 2) the denoising model includes a learned noise schedule; or 3) the denoising model is trained using a continuous-time loss function.

[0011] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0013] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein, which refer to the accompanying figures. [Brief explanation of the drawings]

[0014] [Figure 1A] FIG. 1 is a block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary diffusion model, according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 1 is a flowchart diagram of an exemplary method according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Reference numbers that are repeated among the figures identify the same features in various implementations.

[0016] In general, the present disclosure is directed to a flexible family of diffusion-based models (e.g., generative models) that achieve state-of-the-art performance, such as state-of-the-art likelihood in image density estimation benchmarks. Models according to exemplary aspects of the present disclosure provide efficient optimization of the noise schedule in conjunction with the rest of the diffusion model. Furthermore, the evidence lower bound (ELBO) for the model can be simplified to a relatively short expression in terms of the signal-to-noise ratio (or noise schedule). This improved understanding enables the development of continuous-time evidence lower bounds that can be optimized to improve model accuracy. Furthermore, the continuous-time ELBO may be invariant to the noise schedule except at the signal-to-noise ratio endpoints. This allows for learning a noise schedule that minimizes the variance of the resulting ELBO estimator, providing faster optimization. Exemplary models according to exemplary aspects of the present disclosure may further include a set of Fourier features at the input of the denoising model, which may improve accuracy results.

[0017] Likelihood-based generative modeling is a central task in machine learning, particularly for applications that can be applied to a wide range of applications, including speech synthesis, translation, and compression. For these tasks, autoregressive models have proven themselves due to their likelihood and representation tractability. Diffusion models have shown impressive results in image and audio generation, especially for perceptual quality, but traditional methods have not been able to match them in density estimation benchmarks. Exemplary aspects of the present disclosure provide technical contributions to improve the competitiveness of diffusion models in this area.

[0018] One exemplary aspect of the present disclosure is directed to a diffusion-based generative model that incorporates Fourier features into the diffusion model. Another exemplary aspect of the present disclosure is directed to a diffusion-based generative model that utilizes a learnable specification of the diffusion process, referred to herein as a "noise schedule." Another exemplary aspect of the present disclosure is directed to a diffusion-based generative model that utilizes a novel continuous-time evidence lower bound (ELBO). This aspect enables these models to achieve performance improvements, including new state-of-the-art log-likelihood results, in image density estimation benchmarks without data augmentation.

[0019] Exemplary aspects of the present disclosure are directed to diffusion probabilistic models (DPMs), also known as diffusion models. A diffusion model may include a noisy model that introduces noise into input data to obscure the input data after several stages, or "time steps." The model may be or include a finite number of steps T or an infinite number of steps as T→∞. Exemplary aspects of the present disclosure recognize that a larger number of steps provides improved performance relative to likelihood. For example, a noisy model may include multiple noisy stages (e.g., layers), with each successive layer being noisier than the previous layer. For example, a noisy model may be configured to introduce noise into input data to produce intermediate data.

[0020] The diffusion model may additionally and / or alternatively include a denoising model that can produce samples that match the input data after several stages. For example, the diffusion model may include a Markov chain in the noised model and / or the denoising model. The diffusion model may be implemented in discrete time, e.g., each layer corresponds to a time step. The diffusion model may also be implemented in arbitrarily deep (e.g., continuous) time. For example, the model may be fully Gaussian so that an unbiased estimate of the objective function can be obtained from a single layer. Thus, it is possible to avoid computing intermediate layers. The diffusion model may be conceptually similar to a variational autoencoder (VAE), whose structure and loss function enable efficient training of arbitrarily deep (e.g., infinitely deep) models. The diffusion model can be trained using variational inference. Specifically, exemplary aspects of the present disclosure recognize an evidence lower bound for the diffusion model and utilize the continuous-time evidence lower bound to train a diffusion model with improved performance.

[0021] A diffusion model can be or include one or more latent variables forming a latent variable model, which can be extended to multiple observed variables, estimating conditional densities (e.g., p(x|y)), etc. A diffusion model can include a diffusion process (e.g., a noisy model) that is inverted to obtain a generative model (e.g., a denoised model).

[0022] According to an exemplary aspect of the present disclosure, a computer-implemented method for using a diffusion model may include acquiring input data by a computing system including one or more computing devices. In some implementations, the input data may include one or more channels. For example, a case of generative modeling involves a dataset of observations of x, where the task is to estimate the marginal distribution p(x). For example, a diffusion model may receive data x and estimate a latent variable z given x. tThe time step t can be advanced from time t=0 to t=1. The latent variable z at time step t t The distribution of

[0023]

number

[0024] where α is the mean of the marginal distribution and

[0025]

number

[0026] is the variance of the marginal distribution. The mean and / or variance may be smooth so that their derivatives with respect to t are finite. Furthermore, the signal-to-noise ratio (SNR)

[0027]

number

[0028] , also referred to as the noise schedule, may be monotonically decreasing over t. More generally, the noise schedule may additionally and / or alternatively refer separately to the mean and / or variance, and / or any other suitable ratio of those quantities. The joint distribution of the latent variables at subsequent time steps is distributed as a first-order Markov chain.

[0029] The diffusion model may further include a denoising model configured to reconstruct the output data from the intermediate data. For example, the diffusion process may be inverted to produce a generative model (e.g., the denoising model). For example, the generative model may be a hierarchical model that samples a sequence of latent variables as time progresses backward from t=1 to t=0.

[0030] Model parameters can be optimized by maximizing a lower bound on the variation of the marginal log-likelihood, also known as the evidence lower bound (ELBO). According to exemplary aspects of the present disclosure, the inference model parameters that define the forward diffusion process can be jointly optimized with the rest of the model. An exemplary negative marginal log-likelihood is bounded by the sum of a prior loss, a reconstruction loss, and a diffusion loss. The prior loss is a KL diffusion between two Gaussians and can be calculated in closed form. The denoising model loss can be evaluated and optimized using a reparameterized gradient. The diffusion loss can depend on the number of time steps.

[0031] According to an exemplary aspect of the present disclosure, the input to the denoising model includes a set of Fourier features including a linear projection of a channel of at least one of the multiple noiseing stages. The set of Fourier features may include a linear projection of each channel of the multiple noiseing stages. For example, the set of Fourier features may include a linear projection of at least one of the multiple noiseing stages onto a set of periodic basis functions with high frequencies. Intuitively, the set of Fourier features may improve understanding of fine details of the input data. In some implementations, the set of Fourier features may include four channels. For example, the set of Fourier features may include:

[0032]

number

[0033] where q is the frequency index of the Fourier feature, i and j are position indices, k is the channel index, and z i,j,k are the network inputs at the position index and channel index. Additionally and / or alternatively, the set of Fourier features can be

[0034]

number

[0035] where q is the frequency index of the Fourier feature, i and j are position indices, k is the channel index, and z i,j,k are the network inputs at the position index and channel index. These Fourier features may have one or more frequencies based on the selection of q. In some implementations, for example, the input data may have a bit length, and the set of Fourier features may include Fourier features with each frequency index from 1 to the bit length. Additionally and / or alternatively, in some implementations, the input data may have a bit length of 8 or more, and the set of Fourier features may include Fourier features with each frequency index from 7 to the bit length. As an example, if the input data is represented in an 8-bit byte, the bit length may be 8, and the four channels of Fourier features may be 2 7 and 2 8 It may include f and g features with frequencies higher (e.g., 2 7 Using only frequencies greater than can provide faster training of the model.

[0036] According to exemplary aspects of the present disclosure, the diffusion model may include a learned noise schedule. The noise schedule may include at least one or both of the mean and / or variance of the marginal distribution of the diffusion model. For example, in some implementations, the learned noise schedule includes the ratio of the mean squared marginal distribution of the diffusion model to the squared variance of the marginal distribution. For example, the learned noise schedule may be a signal-to-noise ratio function. The learned noise schedule may be jointly trained with the diffusion model, such as jointly trained with the noise generation model and / or the denoising model.

[0037] In some implementations, the learned noise schedule can be parameterized by a monotonically increasing function. For example, in some implementations, the signal-to-noise ratio is SNR(t)=exp(-γ η(t)), where γ η (t) is a monotonically increasing function. The parameters η of the monotonically increasing function may be learned jointly with the diffusion model. For example, in some implementations (e.g., in the discrete-time case), the parameters may be learned by maximizing the ELBO together with other model parameters. In the continuous-time case, where the diffusion loss relative to the signal-to-noise ratio is invariant except at its endpoints, the parameters are learned by optimizing with respect to the endpoints (e.g., rather than the parameters of a schedule that interpolates between them). For example, in some implementations, the parameters may be learned by minimizing the variance, such as by performing stochastic gradient descent on the squared diffusion loss. This gradient may be calculated with relatively low computational overhead as a by-product of calculating the gradient of the ELBO.

[0038] The monotonically increasing function may be any suitable function according to example aspects of the present disclosure. In some implementations, the monotonically increasing function may be a monotonically increasing neural network. In some implementations, the monotonically increasing neural network includes one or more linear layers that are constrained to be positive. For example, in some implementations, the monotonically increasing neural network is

[0039]

number

[0040] In the above equation, l i where (t) is the ith layer of the monotonically increasing neural network at time step t, and φ is a sigmoid function. In some implementations, the l2 layer can have 1024 outputs, and other layers can have a single output. In some implementations, the monotonically increasing neural network is

[0041]

number

[0042] In the above formula, the range of the monotonically increasing neural network is [SNR min ,SNR max ], so that γ0 = -log(SNR max ) and γ = -log(SNR min ), where γ0 and γ1 are jointly optimized with the parameters of the denoising model. For example, post-processing can bound the range of the neural network.

[0043] In some implementations, the derivative of the loss function with respect to the noise schedule is calculated together with the gradients of other parameters of the diffusion model without a second backpropagation pass through the denoising model. In some implementations, the parameters of the learned noise schedule are learned by maximizing an evidence lower bound together with other parameters of the diffusion model. In some implementations, the diffusion model is a continuous-time diffusion model, and the parameters of the learned noise schedule are learned by optimizing an evidence lower bound with respect to the endpoints of the learned noise schedule. In some implementations, the parameters of the learned noise schedule are learned by minimizing the variance by performing stochastic gradient descent on a squared diffusion loss.

[0044] According to example aspects of the present disclosure, the diffusion model can be trained by optimizing parameters of the machine-learned diffusion model toward an evidence lower bound, where the evidence lower bound includes a continuous-time loss. In some implementations, the continuous-time loss is approximated using an unbiased estimator of the continuous-time loss. In some implementations, the unbiased estimator includes a Monte Carlo estimator. In some implementations, the continuous-time loss includes infinite depth.

[0045] For example, exemplary aspects of the present disclosure allow the evidence lower bound to improve for a larger number of time steps. In the continuous-time case (e.g., therefore with infinite depth), the ELBO may thus achieve improved performance. When the time step is infinite, the diffusion loss may be simplified, thus providing a practical implementation, as described in U.S. Provisional Patent Application No. 63 / 210,314. In some implementations, an unbiased Monte Carlo estimator may be used instead of the diffusion loss, since evaluating the infinite integral may be computationally intractable.

[0046] In some implementations, the input data may include data to be compressed, and the output data may be reconstructed input data. For example, the input data may be provided to a diffusion model (e.g., a noisy model). The intermediate data may be stored as a compressed representation. A different device and / or the same device may then access the intermediate data and provide it as input to a denoising model. The output data from the denoising model may therefore be reconstructed input data. As an example, a diffusion model may be used for image compression. For example, the input data and / or output data may include image data. A diffusion model may also be used for compressing other types of data, such as audio data, text data, multimodal data, etc. A diffusion model may be used for signal (e.g., image) generation. For example, a denoising model may be utilized as a generative model to generate data (e.g., image data) from an input signal (e.g., an input distribution). A denoising model may be trained with a noisy model and then utilized as a standalone model. For example, the denoising model may be useful for noisy images, image filling or other image manipulation, audio / text filling or noisiness or other manipulation, translation, text to image, image to text, speech transcription, or other predictions of high-dimensional signals conditional on the input signal.

[0047] Systems and methods according to exemplary aspects of the present disclosure may provide several technical effects and benefits, including improvements to computational techniques. For example, diffusion models according to exemplary aspects of the present disclosure may achieve improved performance, such as improved likelihood. For example, in data compression, systems and methods according to exemplary aspects of the present disclosure may provide improved compression ratios, such as requiring fewer bits to store compressed data relative to existing systems.

[0048] As another exemplary technical effect and benefit, systems and methods according to exemplary aspects of the present disclosure may enable reduced consumption of computational resources when training a diffusion model. For example, the learned noise schedule described herein may minimize the variance of the resulting ELBO estimator, leading to faster optimization. Faster optimization results in lower consumption of computational resources, such as memory usage, processor usage, etc. Similarly, the inclusion of Fourier features may also enable faster training. As another example, the use of continuous time loss may enable training to occur in which fewer time steps need to be performed and / or evaluated. Performing and / or evaluating fewer time steps results in lower consumption of computational resources, such as memory usage, processor usage, etc.

[0049] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in more detail.

[0050] 1A illustrates a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0051] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0052] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0053] In some implementations, the user computing device 102 can store or include one or more diffusion models 120. For example, the diffusion models 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models can leverage attention mechanisms, such as self-attention. For example, some exemplary machine learning models can include multi-headed self-attention models (e.g., transformer models). Exemplary diffusion models 120 are discussed with reference to FIGS. 2-3.

[0054] In some implementations, one or more diffusion models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single diffusion model 120.

[0055] Additionally or alternatively, one or more diffusion models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the diffusion model 140 may be implemented by the server computing system 140 as part of a web service (e.g., an image generation service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0056] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0057] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0058] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0059] As described above, the server computing system 130 may store or otherwise include one or more diffusion models 140. For example, the models 140 may be or otherwise include various machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models may utilize attention mechanisms such as self-attention. For example, some exemplary machine learning models may include multi-headed self-attention models (e.g., transformer models). Exemplary models 140 are discussed with reference to FIGS. 2-3.

[0060] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0061] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0062] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters for several training iterations.

[0063] In some implementations, performing error backpropagation may include performing truncated backpropagation over time. The model trainer 160 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0064] In particular, model trainer 160 may train diffusion models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, image data, audio data, etc.

[0065] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some instances, this process may be referred to as model personalization.

[0066] Model trainer 160 includes computer logic used to provide the desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0067] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communications over network 180 may be carried over any type of wired and / or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0068] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0069] In some implementations, an input to a machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an upscaled image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0070] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language-encoded output. As another example, the machine learning model may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model may process the text or natural language data to generate a transformation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a predicted output.

[0071] In some implementations, the input to a machine learning model of the present disclosure may be audio data. The machine learning model may process the audio data to generate an output. As an example, the machine learning model may process the audio data to generate a speech recognition output. As another example, the machine learning model may process the audio data to generate a speech conversion output. As another example, the machine learning model may process the audio data to generate a latent embedding output. As another example, the machine learning model may process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning model may process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning model may process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning model may process the audio data to generate a predicted output.

[0072] In some implementations, the input to the machine learning based model of the present disclosure may be latent coding data (e.g., a latent space representation of the input, etc.). The machine learning based model may process the latent coding data to generate an output. As an example, the machine learning based model may process the latent coding data to generate a recognition output. As another example, the machine learning based model may process the latent coding data to generate a reconstruction output. As another example, the machine learning based model may process the latent coding data to generate a search output. As another example, the machine learning based model may process the latent coding data to generate a reclustered output. As another example, the machine learning based model may process the latent coding data to generate a predicted output.

[0073] In some implementations, input to a machine learning model of the present disclosure may be statistical data. The statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate a recognition output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0074] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0075] In some cases, the machine learning model can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), and the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0076] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images show an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, the likelihood that the region shows an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel in one of the input images, the scene motion indicated at that pixel between the images in the network input.

[0077] In some cases, the input includes audio data representing speech and the task is a speech recognition task. The output may include text output that is mapped to the speech. In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes a microprocessor-implemented task such as branch prediction or memory address translation.

[0078] 1A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems may be used. For example, in some implementations, a user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0079] 1B illustrates a block diagram of an exemplary computing device 10 for implementing an exemplary embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0080] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0081] 1B , each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0082] 1C illustrates a block diagram of an exemplary computing device 50 for implementing in accordance with an exemplary embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0083] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0084] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, a respective machine learning model may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0085] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 1C , the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0086] 2 shows a block diagram of an example diffusion model 200 according to an example embodiment of the present disclosure. In some implementations, the diffusion model 200 receives a set of input data 210 and is trained to provide output data 220 as a result of receiving the input data 210. The diffusion model may include a noisization model 202 and a denoising model 204.

[0087] 3 shows a flowchart diagram of an exemplary method for using an improved diffusion model, according to an exemplary embodiment of the present disclosure. While FIG. 3 shows steps performed in a particular order for purposes of explanation and discussion, the methods of the present disclosure are not limited to the order or sequence specifically shown. Various steps of method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0088] The method 300 may include, at 302, acquiring, by a computing system comprising one or more computing devices, input data, the input data including one or more channels. For example, the input data may be image data.

[0089] The method 300 may include providing, by a computing system, input data to a machine learning diffusion model at 304. The diffusion model may include a noisy model including multiple noisy stages, the noisy model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data, and a denoising model configured to reconstruct output data from the intermediate data.

[0090] The method 300 may include receiving, by the computing system, output data from the machine learning diffusion model, at 306. For example, the output data may be the reconstructed input data.

[0091] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0092] While the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation, not limitation, of the present disclosure. Those skilled in the art, upon understanding the above, will be able to readily create modifications, variations, and equivalents of such embodiments. Accordingly, the present disclosure does not exclude the inclusion of such changes, variations, and / or additions to the present subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet a further embodiment. Accordingly, it is intended that the present disclosure cover such modifications, variations, and equivalents. [Explanation of symbols]

[0093] 100 Computing Systems 102 User Computing Devices 112 processors 114 Memory, computing device memory 120 Diffusion model, machine learning model 122 User Input Components 130 Server Computing System 132 processors 134 memory 140 Model, Diffusion Model, Machine Learning Model 150 Training Computing System 152 processors 154 memory 160 Model Trainer 162 training data 180 Network 200 Diffusion Model 202 Noisy Model 204 Noise Reduction Model 210 Input Data 220 Output Data

Claims

1. 1. A computing system that exploits Fourier features for improved fine-scale prediction, comprising: one or more processors; one or more non-transitory computer-readable media; the non-transitory computer-readable medium comprising: At least a noise removal model of a machine learning diffusion model, wherein the machine learning diffusion model is a noisy model including a plurality of noisy stages, the noisy model configured to receive input data and to generate latent data in response to receiving the input data; the denoising model configured to reconstruct output data from the latent data; Including, a denoising model, the input to which comprises a set of Fourier features comprising linear projections of channels of at least one of the plurality of noise stages; instructions that, when executed by the one or more processors, cause the computing system to execute the denoising model to process the latent data to generate the output data; A computing system that stores all of the above information.

2. The computing system of claim 1 , wherein the set of Fourier features comprises a linear projection of the channel of each of the plurality of noise stages.

3. The computing system of claim 1 , wherein the set of Fourier features comprises a linear projection of at least one of the plurality of noise stages onto a set of periodic basis functions having high frequencies.

4. The computing system of claim 1 , wherein the set of Fourier features includes four channels.

5. The set of Fourier features may be [Equation 1] where q is the frequency of said Fourier feature, i and j are position indices, k is a channel index, and z i,j,k 10. The computing system of claim 1, wherein: is a network input at the location index and the channel index.

6. The set of Fourier features may be [Equation 2] where q is the frequency of said Fourier feature, i and j are position indices, k is a channel index, and z i,j,k 10. The computing system of claim 1, wherein: is a network input at the location index and the channel index.

7. the input data includes a bit length; The computing system of claim 1 , wherein the set of Fourier features includes a Fourier feature having each frequency from 1 to the bit length.

8. the input data has a bit length of 8 or more; The computing system of claim 1 , wherein the set of Fourier features includes Fourier features having frequencies from 7 to the bit length.

9. The computing system of claim 1 , wherein the input data comprises image data.

10. the latent data includes a condensed representation of the input data; The computing system of claim 1 , wherein the output data comprises a decompressed representation of the input data.

11. 1. A computer-implemented method for use of an improved diffusion model, comprising: acquiring, by a computing system comprising one or more computing devices, input data, the input data comprising one or more channels; providing, by the computing system, the input data to a machine learning diffusion model, a noiser model including a plurality of noiser stages, the noiser model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data; a denoising model configured to reconstruct output data from the intermediate data; and Including, an input to the denoising model comprising a set of Fourier features comprising linear projections of channels of at least one of the plurality of noise stages; receiving, by the computing system, the output data from the machine learning diffusion model; 11. A computer-implemented method comprising:

12. The computer-implemented method of claim 11 , wherein the set of Fourier features comprises a linear projection of the channel of each of the multiple noise stages.

13. The computer-implemented method of claim 11 , wherein the set of Fourier features comprises a linear projection of at least one of the plurality of noise stages onto a set of periodic basis functions with high frequencies.

14. The computer-implemented method of claim 11 , wherein the set of Fourier features includes four channels.

15. The set of Fourier features may be [Equation 3] where q is the frequency of said Fourier feature, i and j are position indices, k is a channel index, and z i,j,k 12. The computer-implemented method of claim 11, wherein: is a network input at the position index and the channel index.

16. The set of Fourier features may be [Equation 4] where q is the frequency of said Fourier feature, i and j are position indices, k is a channel index, and z i,j,k 12. The computer-implemented method of claim 11, wherein: is a network input at the position index and the channel index.

17. the input data includes a bit length; The computer-implemented method of claim 11 , wherein the set of Fourier features includes a Fourier feature having each frequency from 1 to the bit length.

18. the input data has a bit length of 8 or more; The computer-implemented method of claim 11 , wherein the set of Fourier features includes Fourier features having each frequency from 7 to the bit length.

19. The computer-implemented method of claim 11 , wherein the input data includes image data.

20. The computer-implemented method of claim 11 , wherein the output data comprises reconstructed input data.

21. One or more non-transitory computer-readable media collectively storing at least a noisy model of a diffusion model, said diffusion model comprising: the noise model comprising a plurality of noise stages, the noise model being configured to introduce noise to the input data according to a noise schedule to produce intermediate data; a denoising model configured to reconstruct output data from the intermediate data; and Including, One or more non-transitory computer-readable media, wherein the noise schedule is a learned noise schedule that includes one or more learned parameter values.

22. 22. The one or more non-transitory computer-readable media of claim 21 , wherein the learned noise schedule comprises a ratio of a mean squared marginal distribution of the diffusion model to a squared variance of the marginal distribution.

23. 22. The one or more non-transitory computer-readable media of claim 21, wherein the learned noise schedule is learned jointly with the diffusion model.

24. 22. The one or more non-transitory computer-readable media of claim 21, wherein the learned noise schedule comprises a signal-to-noise ratio function.

25. 22. The one or more non-transitory computer-readable media of claim 21, wherein the learned noise schedule is parameterized by a monotonically increasing function.

26. 26. The one or more non-transitory computer-readable media of claim 25, wherein the monotonically increasing function comprises a monotonically increasing neural network.

27. 27. The one or more non-transitory computer-readable media of claim 26, wherein the monotonically increasing neural network includes one or more linear layers that are constrained to be positive.

28. The monotonically increasing neural network is [Equation 5] In the above equation, l i 28. The one or more non-transitory computer-readable media of claim 27, wherein (t) is the i-th layer of the monotonically increasing neural network at time step t, and φ is a sigmoid function.

29. The monotonically increasing neural network is [Equation 6] where the monotonically increasing neural network range is [SNR min ,SNR max ], so that γ 0 =-log(SNR max ) and γ 1 =-log(SNR min ) and γ 0 and γ 1 30. The one or more non-transitory computer-readable media of claim 28, wherein: is jointly optimized with parameters of the denoising model.

30. 22. The one or more non-transitory computer-readable media of claim 21 , wherein the derivative of the loss function with respect to the noise schedule is calculated together with gradients of other parameters of the diffusion model without a second backpropagation pass through the denoising model.

31. 22. The one or more non-transitory computer-readable media of claim 21, wherein the parameters of the learned noise schedule are learned by maximizing an evidence lower bound together with other parameters of the diffusion model.

32. the diffusion model is a continuous-time diffusion model; 22. The one or more non-transitory computer-readable media of claim 21, wherein parameters of the learned noise schedule are learned by optimizing a lower evidence bound with respect to endpoints of the learned noise schedule.

33. 22. The one or more non-transitory computer-readable media of claim 21, wherein the parameters of the learned noise schedule are learned by minimizing variance by performing stochastic gradient descent on a squared diffusion loss.

34. 1. A computer-implemented method for use of an improved diffusion model, comprising: acquiring, by a computing system comprising one or more computing devices, input data, the input data comprising one or more channels; providing, by the computing system, the input data to a machine learning diffusion model, a noiser model including a plurality of noiser stages, the noiser model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data; a denoising model configured to reconstruct output data from the intermediate data; and Including, the machine learning diffusion model includes a learned noise schedule; receiving, by the computing system, the output data from the machine learning diffusion model; 10. A computer-implemented method comprising:

35. 35. The computer-implemented method of claim 34, wherein the learned noise schedule comprises a ratio of a mean squared marginal distribution of the machine learning diffusion model to a squared variance of the marginal distribution.

36. 35. The computer-implemented method of claim 34, wherein the learned noise schedule is jointly learned with the machine learning diffusion model.

37. 35. The computer-implemented method of claim 34, wherein the learned noise schedule comprises a signal-to-noise ratio function.

38. 35. The computer-implemented method of claim 34, wherein the learned noise schedule is parameterized by a monotonically increasing function.

39. 39. The computer-implemented method of claim 38, wherein the monotonically increasing function comprises a monotonically increasing neural network.

40. 40. The computer-implemented method of claim 39, wherein the monotonically increasing neural network includes one or more linear layers that are constrained to be positive.

41. The monotonically increasing neural network is [Equation 7] In the above equation, l i 41. The computer-implemented method of claim 40, wherein (t) is the i-th layer of the monotonically increasing neural network at time step t, and φ is a sigmoid function.

42. The monotonically increasing neural network is [Equation 8] where the monotonically increasing neural network range is [SNR min ,SNR max ], so that γ 0 =-log(SNR max ) and γ 1 =-log(SNR min ) and γ 0 and γ 1 42. The computer-implemented method of claim 41, wherein is jointly optimized with parameters of the denoising model.

43. 35. The computer-implemented method of claim 34, wherein the derivative of the loss function with respect to the noise schedule is calculated along with gradients of other parameters of the diffusion model without a second backpropagation pass through the denoising model.

44. 35. The computer-implemented method of claim 34, wherein the parameters of the learned noise schedule are learned by maximizing an evidence lower bound together with other parameters of the diffusion model.

45. the diffusion model is a continuous-time diffusion model; 35. The computer-implemented method of claim 34, wherein parameters of the learned noise schedule are learned by optimizing lower evidence bounds with respect to endpoints of the learned noise schedule.

46. 35. The computer-implemented method of claim 34, wherein the parameters of the learned noise schedule are learned by minimizing variance by performing stochastic gradient descent on a squared diffusion loss.

47. 1. A computer-implemented method for training a diffusion model while consuming fewer computational resources, comprising: acquiring, by a computing system comprising one or more computing devices, training data, the training data comprising one or more channels; providing, by the computing system, the training data to a machine learning diffusion model, a noisier model including a plurality of noisier stages, the noisier model configured to receive the training data and to introduce noise to produce intermediate data in response to receiving the training data; a denoising model configured to reconstruct output data from the intermediate data; and and determining, by the computing system, a training loss based at least in part on use of the machine learning diffusion model with the training data; the machine learning diffusion model is trained by optimizing parameters of the machine learning diffusion model toward an evidence lower bound; the evidence lower bound includes continuous time loss; 10. A computer-implemented method comprising:

48. 48. The computer-implemented method of claim 47, wherein the continuous time loss is approximated using an unbiased estimator of the continuous time loss.

49. 49. The computer-implemented method of claim 48, wherein the unbiased estimator comprises a Monte Carlo estimator.

50. 48. The computer-implemented method of claim 47, wherein the continuous-time loss comprises infinite depth.

51. 1. A computer-implemented method for use of an improved diffusion model, comprising: receiving compressed data; decompressing the compressed data with a denoising model trained as part of a diffusion model; Including, 1) the input to the denoising model includes one or more Fourier features; 2) the denoising model includes a learned noise schedule; or 3) Is the denoising model trained using a continuous-time loss function? The computer-implemented method is at least one of:

52. 52. The computer-implemented method of claim 51, wherein the compressed data includes at least one of image data, audio data, text data, or incomplete data.

Images