Diffusion model with improved accuracy and reduced consumption of computing resources

Abstract

A computer-implemented method for use of an improved diffusion model includes obtaining input data, the input data including one or more channels, providing the input data to a machine learning diffusion model, the machine learning diffusion model including a noiser model including a plurality of noiser stages, the noiser model configured to receive the input data and introduce noise to produce intermediate data in response to receiving the input data, and a denoising model configured to reconstruct output data from the intermediate data, and receiving output data from the machine learning diffusion model by a computing system. The diffusion model may include a learned noise schedule. Additionally and / or alternatively, the input to the denoising model may include a set of Fourier features. Additionally and / or alternatively, the diffusion model may be trained based at least in part on a continuous time loss for evidence lower bound.

Description

Technical Field

[0001] Related Applications This application claims priority and the benefit thereof to U.S. Provisional Patent Application No. 63 / 210,314, filed on Jun. 14, 2021. U.S. Provisional Patent Application No. 63 / 210,314 is hereby incorporated by reference in its entirety.

[0002] This disclosure generally relates to machine learning. More particularly, this disclosure relates to diffusion models with improved accuracy.

Background Art

[0003] Likelihood-based generative modeling is a central task in machine learning that is fundamental to a wide range of applications. Autoregressive models have long been the dominant model class in this task due to their ease of handling likelihood and expressiveness. Diffusion models are a class of machine-learned models that include noising models and denoising models. Diffusion models have not yet matched autoregressive models in density estimation benchmarks.

Summary of the Invention

Means for Solving the Problems

[0004] Aspects and advantages of embodiments of this disclosure will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computing system that utilizes Fourier features for improved fine-scale prediction. The computing system includes one or more processors and one or more non-transitory computer-readable media. The non-transitory computer-readable media includes at least a noise removal model of a machine learning-based diffusion model. The diffusion model is a noise addition model that includes a plurality of noise addition stages and is configured to receive input data and generate latent data in response to receiving the input data. The noise removal model is configured to reconstruct output data from the latent data. The input to the noise removal model includes a set of Fourier features that includes a linear projection of a channel of at least one of the plurality of noise addition stages. The non-transitory computer-readable media stores instructions that, when executed by one or more processors, cause the computing system to execute the noise removal model to process the latent data to generate output data.

[0006] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using a diffusion model with improved accuracy. The method includes obtaining, by a computing system including one or more computing devices, input data that includes one or more channels; providing, by the computing system, the input data to a machine learning-based diffusion model. The machine learning-based diffusion model includes a noise addition model that includes a plurality of noise addition stages and is configured to receive input data and introduce noise to generate intermediate data in response to receiving the input data, and a noise removal model that is configured to reconstruct output data from the intermediate data. The input to the noise removal model includes a set of Fourier features that includes a linear projection of a channel of at least one of the plurality of noise addition stages; and receiving, by the computing system, the output data from the machine learning-based diffusion model.

[0007] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store at least a denoising model of a diffusion model, the diffusion model being a denoising model that includes a plurality of denoising steps and is configured to introduce noise into input data according to a noise schedule so as to generate intermediate data, and a denoising model configured to reconstruct output data from the intermediate data, the noise schedule being a learned noise schedule that includes one or more learned parameter values.

[0008] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using an improved diffusion model, the method comprising: obtaining, by a computing system comprising one or more computing devices, input data, the input data comprising one or more channels; providing, by the computing system, the input data to a machine learning-based diffusion model, the machine learning-based diffusion model being a denoising model that includes a plurality of denoising steps and is configured to receive the input data and introduce noise so as to generate intermediate data in response to receiving the input data, and a denoising model configured to reconstruct output data from the intermediate data, the diffusion model including a learned noise schedule; and receiving, by the computing system, output data from the machine learning-based diffusion model.

[0009] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for training a diffusion model while consuming fewer computing resources. The method includes: obtaining training data by a computing system including one or more computing devices, where the training data includes one or more channels; providing the training data to a machine learning-based diffusion model by the computing system, where the machine learning-based diffusion model includes a noise model including a plurality of noise levels, the noise model being configured to receive the training data and introduce noise to generate intermediate data in response to receiving the training data, and a denoising model configured to reconstruct output data from the intermediate data; and determining a training loss by the computing system based at least in part on the use of the machine learning-based diffusion model with the training data, where the diffusion model is trained by optimizing the parameters of the machine learning-based diffusion model towards the evidence lower bound, and the evidence lower bound includes a continuous time loss.

[0010] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for using a diffusion model with improved accuracy. The method includes receiving compressed data and decompressing the compressed data with a denoising model trained as part of the diffusion model, where at least one of: 1) the input to the denoising model includes one or more Fourier features; 2) the denoising model includes a learned noise schedule; or 3) the denoising model is trained using a continuous time loss function.

[0011] Other aspects of the present disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of the various embodiments of the present disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the relevant principles.

[0013] A detailed description of embodiments directed to those of ordinary skill in the art is set forth herein, with reference to the accompanying figures.

Brief Description of the Drawings

[0014]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Best Mode for Carrying Out the Invention

[0015] Reference numerals repeated throughout the several figures identify like features in various implementations.

[0016] Generally, the present disclosure is directed to flexible families of diffusion-based models (e.g., generative models) that achieve state-of-the-art performance, such as state-of-the-art likelihood in image density estimation benchmarks. Models according to example aspects of the present disclosure provide efficient optimization of the noise schedule, jointly with the rest of the diffusion model. Further, the evidence lower bound (ELBO) for the model can be simplified to a relatively short expression with respect to the signal-to-noise ratio (or noise schedule). This improved understanding enables the development of a continuous-time evidence lower bound that can be optimized to improve model accuracy. Further, the continuous-time ELBO can be invariant to the noise schedule, except at the endpoints of the signal-to-noise ratio. By being so, it is possible to learn a noise schedule that minimizes the variance of the resulting ELBO estimator, and can provide faster optimization. Exemplary models according to example aspects of the present disclosure can further include a set of Fourier features in the input of the denoising model, whereby accuracy results can be improved.

[0017] Likelihood-based generative modeling is a central task in machine learning that can be applied, in particular, to a wide range of applications including, for example, speech synthesis, translation, and compression. For these tasks, autoregressive models have been proven by the ease of handling likelihood and representation. Diffusion models have shown excellent results, particularly for perceptual quality, in image generation and audio generation, but in conventional approaches, they do not compete in density estimation benchmarks. Exemplary aspects of the present disclosure provide a technical contribution for improving the competitiveness of diffusion models in this area.

[0018] One exemplary aspect of the present disclosure is directed to a diffusion-based generative model that incorporates Fourier features into a diffusion model. Another exemplary aspect of the present disclosure is directed to a diffusion-based generative model that utilizes a learnable specification of a diffusion process, herein referred to as a "noise schedule". Another exemplary aspect of the present disclosure is directed to a diffusion-based generative model that utilizes a novel continuous-time evidence lower bound (ELBO). This aspect enables these models to achieve performance improvements, including a new state-of-the-art log-likelihood in an image density estimation benchmark, without data augmentation.

[0019] Exemplary aspects of the present disclosure are directed to a diffusion probability model (DPM), also referred to as a diffusion model. The diffusion model can include a noising model that introduces noise into input data to obscure the input data after a number of stages, or "time steps". This model can have or include a finite number of steps T or an infinite number of steps as T→∞. Exemplary aspects of the present disclosure recognize that a larger number of steps results in performance improvements with respect to likelihood. For example, the noising model can include a plurality of noising stages (e.g., layers), where each successive layer is more noisy than the previous layer. For example, the noising model can be configured to introduce noise into the input data to produce intermediate data.

[0020] The diffusion model may include, in addition and / or as an alternative, a denoising model that can produce samples that match the input data after several steps. For example, the diffusion model may include a Markov chain in the noise addition model and / or the denoising model. The diffusion model may be implemented at discrete times, for example, each layer corresponding to a time step. The diffusion model may optionally be implemented at arbitrarily deep (e.g., continuous) times. For example, the model may be fully Gaussian so that an unbiased estimate of the objective function can be obtained from a single layer. Thus, it is possible to avoid computing intermediate layers. The diffusion model may conceptually be similar to a variational autoencoder (VAE) whose structure and loss function enable efficient training of arbitrarily deep (e.g., infinitely deep) models. The diffusion model can be trained using variational inference. Specifically, the exemplary embodiments of the present disclosure recognize the evidence lower bound of the diffusion model and utilize the continuous-time evidence lower bound to train a diffusion model with improved performance.

[0021] The diffusion model may be or include one or more latent variables that form a latent variable model. This can be extended to a plurality of observed variables, such as estimating a conditional density (e.g., p(x|y)). The diffusion model can include a diffusion process (e.g., a noise addition model), which is inverted to obtain a generative model (e.g., a denoising model).

[0022] According to an exemplary embodiment of the present disclosure, a computer-implemented method for using a diffusion model may include obtaining input data by a computing system including one or more computing devices. In some implementations, the input data may include one or more channels. For example, in some cases of generative modeling, there is a dataset of observations of x, where the task is to estimate the marginal distribution p(x). For example, the diffusion model receives data x and given the latent variable z for x tThe sequence can be sampled. The time step t can advance from time t = 0 to t = 1. The distribution of the latent variable z at time step t t is given by

[0023]

Number

[0024] which can be given by, where in the above formula, α is the mean of the marginal distribution,

[0025]

Number

[0026] is the variance of the marginal distribution. The mean and / or variance may be smooth such that their derivatives with respect to t are finite. Further, the signal-to-noise ratio (SNR)

[0027]

Number

[0028] which is also called the noise schedule, may be monotonically decreasing over t. More generally, the noise schedule may separately refer to the mean and / or variance, and / or any other appropriate ratio of those quantities, additionally and / or alternatively. The joint distribution of the latent variables at subsequent time steps is distributed as a first-order Markov chain.

[0029] The diffusion model may further include a noise removal model configured to reconstruct output data from intermediate data. For example, the diffusion process may be inverted to yield a generative model (e.g., a noise removal model). For example, the generative model may be a hierarchical model that samples a sequence of latent variables as time advances from t = 1 to t = 0.

[0030] The model parameters can be optimized by maximizing the lower bound of the variational lower bound of the log-likelihood. This is also called the evidence lower bound (ELBO). According to an exemplary aspect of the present disclosure, the inference model parameters defining the forward diffusion process can be optimized jointly with the rest of the model. An exemplary negative log-likelihood is bounded by the sum of the prior loss, the reconstruction loss, and the diffusion loss. The prior loss is the KL divergence between two Gaussians and can be calculated in closed form. The noise removal model loss can be evaluated and optimized using the reparameterization gradient. The diffusion loss may depend on the number of time steps.

[0031] According to an exemplary aspect of the present disclosure, the input to the noise removal model includes a set of Fourier features including a linear projection of the channels of at least one of the plurality of noise addition stages. The set of Fourier features may include a linear projection of the channels of each of the plurality of noise addition stages. For example, the set of Fourier features may include a linear projection of at least one of the plurality of noise addition stages onto a set of periodic basis functions with high frequencies. Intuitively, the set of Fourier features may improve the understanding of the fine details of the input data. In some implementations, the set of Fourier features may include four channels. For example, the set of Fourier features may include

[0032]

Number

[0033] at least one Fourier feature of the form, where q is the frequency index of the Fourier feature, i and j are position indices, k is the channel index, and z i,j,k is the network input at the position index and channel index. Additionally and / or alternatively, the set of Fourier features may include

[0034]

Number

[0035] may include at least one Fourier feature of the form, where q is the frequency index of the Fourier feature, i and j are position indices, k is the channel index, and z i,j,k is the network input at the position index and the channel index. These Fourier features may have one or more frequencies based on the selection of q. In some implementations, for example, the input data can have a bit length, and the set of Fourier features can include Fourier features having each frequency index from 1 to that bit length. Additionally and / or alternatively, in some implementations, the input data can include a bit length of 8 or more, and the set of Fourier features can include Fourier features having each frequency index from 7 to that bit length. As an example, if the input data is represented by 8-bit bytes, the bit length can be 8, and the four channels of Fourier features can include f and g features having frequencies of 2 7 and 2 8 . By using only higher (e.g., greater than 2 7 ) frequencies, faster training of the model can be provided.

[0036] According to an exemplary aspect of the present disclosure, the diffusion model may include a learned noise schedule. The noise schedule may include at least one or both of the mean of the marginal distribution of the diffusion model and / or the variance of the marginal distribution. For example, in some implementations, the learned noise schedule includes the ratio of the root mean square of the marginal distribution of the diffusion model to the root mean square of the variance of the marginal distribution. For example, the learned noise schedule may be a signal-to-noise ratio function. The learned noise schedule may be co-learned with the diffusion model, such as in conjunction with the noise addition model and / or the noise removal model.

[0037] In some implementations, the learned noise schedule can be parameterized by a monotonically increasing function. For example, in some implementations, the signal-to-noise ratio is SNR(t)=exp(-γ ηcan be parameterized by γ(t), where γ η (t) is a monotonically increasing function. The parameter η of the monotonically increasing function can be learned jointly with the diffusion model. For example, in some implementations (e.g., in the discrete-time case), the parameter can be learned by maximizing the ELBO together with other model parameters. In the continuous-time case where the diffusion loss with respect to the signal-to-noise ratio is invariant except at its endpoints, the parameter is learned by optimizing with respect to the endpoints (e.g., not the parameters of the schedule that interpolates between them). For example, in some implementations, the parameter can be learned by minimizing the variance, such as by performing stochastic gradient descent on the squared diffusion loss. This gradient can be calculated with relatively low computational overhead as a byproduct of calculating the gradient of the ELBO.

[0038] The monotonically increasing function can be any suitable function according to the exemplary embodiments of the present disclosure. In some implementations, the monotonically increasing function can be a monotonically increasing neural network. In some implementations, the monotonically increasing neural network includes one or more linear layers that are restricted to be positive. For example, in some implementations, the monotonically increasing neural network is

[0039]

Number

[0040] represented by, where in the above formula, l i (t) is the i-th layer of the monotonically increasing neural network at time step t, and φ is the sigmoid function. In some implementations, the l 2 layer can have 1024 outputs, and other layers can have a single output. In some implementations, the monotonically increasing neural network is

[0041]

Number

[0042] is post - processed such that, in the above equation, the range of the monotonically increasing neural network is limited to [SNR min , SNR max , where γ 0 =-log(SNR max ) and γ 1 =-log(SNR min ), and γ 0 and γ 1 are optimized jointly with the parameters of the noise removal model. For example, the post - processing can bound the range of the neural network.

[0043] In some implementations, the derivative of the loss function with respect to the noise schedule is computed together with the gradients of the other parameters of the diffusion model without a second backpropagation path through the noise removal model. In some implementations, the parameters of the learned noise schedule are learned by maximizing the evidence lower bound together with the other parameters of the diffusion model. In some implementations, the diffusion model is a continuous - time diffusion model, and the parameters of the learned noise schedule are learned by optimizing the evidence lower bound with respect to the endpoints of the learned noise schedule. In some implementations, the parameters of the learned noise schedule are learned by minimizing the variance by performing stochastic gradient descent on the squared diffusion loss.

[0044] According to an exemplary aspect of the present disclosure, a diffusion model can be trained by optimizing the parameters of a machine - learning - type diffusion model towards the evidence lower bound, where the evidence lower bound includes a continuous - time loss. In some implementations, the continuous - time loss is approximated using an unbiased estimator of the continuous - time loss. In some implementations, the unbiased estimator includes a Monte Carlo estimator. In some implementations, the continuous - time loss includes an infinite depth.

[0045] For example, the exemplary embodiments of the present disclosure enable the evidence lower bound to improve for a larger number of time steps. In the continuous time case (e.g., thus having infinite depth), the ELBO can thus achieve performance improvement. When the number of time steps is infinite, the diffusion loss may be simplified and thus a practical implementation can be provided as described in U.S. Provisional Patent Application No. 63 / 210,314. In some implementations, since it may be computationally difficult to evaluate the infinite integral, an unbiased Monte Carlo estimator can be used instead of the diffusion loss.

[0046] In some implementations, the input data may include data to be compressed, and the output data may be the reconstructed input data. For example, the input data can be provided to a diffusion model (e.g., a noise model). The intermediate data can be stored as a compressed representation. Different devices and / or the same device can then access the intermediate data and provide the intermediate data as input to a denoising model. The output data from the denoising model can thus be the reconstructed input data. As an example, the diffusion model can be used for image compression. For example, the input data and / or the output data can include image data. The diffusion model can be used for other types of data compression, such as audio data, text data, multimodal data, etc. The diffusion model can also be used for generating a signal (e.g., an image). For example, the denoising model can be utilized as a generative model for generating data (e.g., image data) from an input signal (e.g., an input distribution). The denoising model can be trained with a noise model and then utilized as a stand-alone model. For example, the denoising model can be useful for denoising images, image inpainting, or other image operations, audio / text inpainting or denoising or other operations, translation, text-to-image, image-to-text, speech transcription, or other predictions of high-dimensional signals conditioned on the input signal.

[0047] The systems and methods according to the exemplary embodiments of the present disclosure can provide several technical effects and benefits, including improvements to computing technology. For example, the diffusion model according to the exemplary embodiments of the present disclosure can achieve performance improvements such as an increase in likelihood. As an example, in data compression, the systems and methods according to the exemplary embodiments of the present disclosure can provide an improved compression ratio, such as requiring fewer bits to store compressed data relative to existing systems.

[0048] As another exemplary technical effect and benefit, the systems and methods according to the exemplary embodiments of the present disclosure can enable a reduction in the consumption of computing resources when training a diffusion model. For example, the learned noise schedule described herein can minimize the variance of the resulting ELBO estimator, leading to faster optimization. As a result of faster optimization, there is less consumption of computing resources, such as memory usage, processor usage, and the like. Similarly, by including Fourier features, faster training can also be enabled. As another example, the use of continuous-time loss can enable training to occur where fewer time steps need to be implemented and / or evaluated. As a result of implementing and / or evaluating fewer time steps, there is less consumption of computing resources, such as memory usage, processor usage, and the like.

[0049] Reference is now made to the drawings, and the exemplary embodiments of the present disclosure will be discussed in further detail.

[0050] FIG. 1A shows a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0051] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0052] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operably connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0053] In some implementations, the user computing device 102 can store or include one or more diffusion models 120. For example, the diffusion model 120 can be various machine learning models such as a neural network (e.g., a deep neural network) or other types of machine learning models including non-linear models and / or linear models, or it can include those machine learning models. The neural network can include a feed-forward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some exemplary machine learning models can utilize attention mechanisms such as self-attention. For example, some exemplary machine learning models can include a multi-head self-attention model (e.g., a transformer model). The exemplary diffusion model 120 will be discussed with reference to FIGS. 2-3.

[0054] In some implementations, one or more diffusion models 120 are received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then can be used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single diffusion model 120.

[0055] Additionally or alternatively, one or more diffusion models 140 may be included in a server computing system 130 that communicates with the user computing device 102 according to a client - server relationship, or otherwise may be stored and implemented by the server computing system 130. For example, the diffusion model 140 may be implemented by the server computing system 140 as part of a web service (e.g., an image generation service). Thus, one or more models 120 may be stored and implemented on the user computing device 102, and / or one or more models 140 may be stored and implemented on the server computing system 130.

[0056] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch - sensitive component (e.g., a touch - sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch - sensitive component can serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which a user can provide user input.

[0057] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operably connected. The memory 134 can include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0058] In some implementations, the server computing system 130 includes one or more server computing devices or, alternatively, is implemented by a server computing device. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0059] As described above, the server computing system 130 can store one or more diffusion models 140, or alternatively, can include the model 140. For example, the model 140 can be, or alternatively can include, various machine learning models. Exemplary machine learning models include neural networks or other multi-layer non-linear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, regression neural networks, and convolutional neural networks. Some exemplary machine learning models can utilize attention mechanisms such as self-attention. For example, some exemplary machine learning models can include multi-head self-attention models (e.g., transformer models). The exemplary model 140 will be discussed with reference to FIGS. 2-3.

[0060] The user computing device 102 and / or the server computing system 130 can train the model 120 and / or 140 through interaction with a training computing system 150 communicatively coupled via the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a part of the server computing system 130.

[0061] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operably connected. The memory 154 can include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or, alternatively, is implemented by one or more server computing devices.

[0062] The training computing system 150 can include a model trainer 160 that trains a machine learning model 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, error backpropagation. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters for a number of training iterations.

[0063] In some implementations, performing backpropagation of errors may include performing reduced backpropagation over time. The model trainer 160 can implement some generalization techniques (such as weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0064] In particular, the model trainer 160 can train the diffusion model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, image data, audio data, etc.

[0065] In some implementations, when the user gives consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 with respect to user-specific data received from the user computing device 102. In some cases, this process may be referred to as model personalization.

[0066] The model trainer 160 includes computer logic used to provide the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0067] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication via Network 180 can be carried over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0068] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0069] In some implementations, the input to the machine learning models of the present disclosure may be image data. The machine learning models may process the image data to generate an output. By way of example, the machine learning models may process the image data to generate an image recognition output (e.g., recognition of the image data, embedding of the potential of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning models may process the image data to generate an image segmentation output. As another example, the machine learning models may process the image data to generate an image classification output. As another example, the machine learning models may process the image data to generate an image data modification output (e.g., modification of the image data, etc.). As another example, the machine learning models may process the image data to generate an encoded image data output (e.g., encoded and / or compressed representation of the image data, etc.). As another example, the machine learning models may process the image data to generate an upscaled image data output. As another example, the machine learning models may process the image data to generate a prediction output.

[0070] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. By way of example, the machine learning model may process natural language data to generate a language encoding output. As another example, the machine learning model may process text or natural language data to generate a latent text embedding output. As another example, the machine learning model may process text or natural language data to generate a conversion output. As another example, the machine learning model may process text or natural language data to generate a classification output. As another example, the machine learning model may process text or natural language data to generate a text segmentation output. As another example, the machine learning model may process text or natural language data to generate a semantic intent output. As another example, the machine learning model may process text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process text or natural language data to generate a prediction output.

[0071] In some implementations, the input to the machine learning model of the present disclosure may be audio data. The machine learning model may process the audio data to generate an output. As an example, the machine learning model may process the audio data to generate an audio recognition output. As another example, the machine learning model may process the audio data to generate an audio conversion output. As another example, the machine learning model may process the audio data to generate a latent embedding output. As another example, the machine learning model may process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning model may process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning model may process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning model may process the audio data to generate a prediction output.

[0072] In some implementations, the input to the machine learning model of the present disclosure may be latent encoded data (e.g., a latent space representation of the input, etc.). The machine learning model may process the latent encoded data to generate an output. As an example, the machine learning model may process the latent encoded data to generate a recognition output. As another example, the machine learning model may process the latent encoded data to generate a reconstruction output. As another example, the machine learning model may process the latent encoded data to generate a search output. As another example, the machine learning model may process the latent encoded data to generate a reclustering output. As another example, the machine learning model may process the latent encoded data to generate a prediction output.

[0073] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. Statistical data can be data that is calculated and / or derived from some other data source, represents this, or otherwise includes this. The machine learning model can process the statistical data to generate an output. By way of example, the machine learning model can process the statistical data to generate a recognition output. As another example, the machine learning model can process the statistical data to generate a prediction output. As another example, the machine learning model can process the statistical data to generate a classification output. As another example, the machine learned model can process the statistical data to generate a segmentation output. As another example, the machine learning model can process the statistical data to generate a visualization output. As another example, the machine learning model can process the statistical data to generate a diagnostic output.

[0074] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model can process the sensor data to generate an output. By way of example, the machine learning model can process the sensor data to generate a recognition output. As another example, the machine learning model can process the sensor data to generate a prediction output. As another example, the machine learning model can process the sensor data to generate a classification output. As another example, the machine learning model can process the sensor data to generate a segmentation output. As another example, the machine learning model can process the sensor data to generate a visualization output. As another example, the machine learning model can process the sensor data to generate a diagnostic output. As another example, the machine learning model can process the sensor data to generate a detection output.

[0075] In some cases, the machine learning model can be configured to perform tasks that include encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0076] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images show an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, the likelihood that the region shows the object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, the respective likelihoods for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel in one of the input images, the motion of the scene shown at the pixel between the images in the network input.

[0077] In some cases, the input includes audio data representing speech, and the task is a speech recognition task. The output may include a text output mapped to the speech. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes microprocessor-implemented tasks such as branch prediction or memory address translation.

[0078] FIG. 1A shows one exemplary computing system that can be used to implement the present disclosure. Other computing systems may be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained locally and used on the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to customize the model 120 based on user-specific data.

[0079] FIG. 1B shows a block diagram of an exemplary computing device 10 implemented in accordance with an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0080] The computing device 10 includes several applications (e.g., applications 1 to N). Each application includes its own machine learning library and machine learning-based model. For example, each application may include a machine learning-based model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like.

[0081] As shown in FIG. 1B, each application can communicate with some other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0082] FIG. 1C shows a block diagram of an exemplary computing device 50 implemented in accordance with an exemplary embodiment of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0083] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0084] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, each machine learning model can be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model to all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0085] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As shown in FIG. 1C, the central device data layer can communicate with some other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0086] FIG. 2 shows a block diagram of an exemplary diffusion model 200 according to an exemplary embodiment of the present disclosure. In some implementations, the diffusion model 200 is trained to receive a set of input data 210 and provide output data 220 as a result of receiving the input data 210. The diffusion model can include a noise model 202 and a noise removal model 204.

[0087] FIG. 3 shows a flowchart diagram of an exemplary method for using an improved diffusion model according to an exemplary embodiment of the present disclosure. FIG. 3 shows steps implemented in a specific order for purposes of explanation and discussion, but the methods of the present disclosure are not limited to the specific order or arrangement shown. The various steps of method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0088] Method 300 may include, at 302, obtaining input data by a computing system comprising one or more computing devices, the input data including one or more channels. For example, the input data may be image data.

[0089] Method 300 may include, at 304, providing the input data to a machine learning-based diffusion model by the computing system. The diffusion model is a noise model including a plurality of noise steps, and includes a noise model configured to receive the input data and introduce noise to generate intermediate data in response to receiving the input data, and a noise removal model configured to reconstruct output data from the intermediate data.

[0090] Method 300 may include, at 306, receiving the output data from the machine learning-based diffusion model by the computing system. For example, the output data may be the reconstructed input data.

[0091] The technology described in this specification refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent between such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described in this specification can be implemented using a single device or component, or multiple devices or components acting in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0092] The subject matter has been described in detail with respect to various specific exemplary embodiments, but each example is provided by way of illustration and not limitation of the disclosure. Those skilled in the art, upon understanding the foregoing, can readily make modifications, variations, and equivalents of such embodiments. Accordingly, the disclosure is not intended to exclude such changes, variations, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to further result in additional embodiments. Accordingly, the disclosure is intended to cover such modifications, variations, and equivalents.

Explanation of Reference Numerals

[0093] 100 Computing System 102 User Computing Device 112 Processor 114 Memory, Computing Device Memory 120 Diffusion Model, Machine Learning Model 122 User Input Component 130 Server Computing System 132 Processor 134 Memory 140 model, diffusion model, machine learning model 150 computing system for training 152 processor 154 memory 160 model trainer 162 training data 180 network 200 diffusion model 202 noise addition model 204 noise removal model 210 input data 220 output data

Claims

1. A computing system that utilizes Fourier features for improving fine-scale prediction, comprising: one or more processors; one or more non-transitory computer-readable media wherein the non-transitory computer-readable media stores, at least, a noise removal model of a machine learning-based diffusion model, and the machine learning-based diffusion model includes: a noise addition model including a plurality of noise addition stages, configured to receive input data and generate latent data in response to the reception of the input data; the noise removal model configured to reconstruct output data from the latent data; wherein the input to the noise removal model includes a set of Fourier features including a linear projection of channels of at least one of the plurality of noise addition stages; instructions that, when executed by the one or more processors, cause the computing system to execute the noise removal model to process the latent data to generate the output data; wherein the latent data includes a compressed representation of the input data and the output data includes a decompressed representation of the input data.

2. A computing system that utilizes Fourier features for improving fine-scale prediction, comprising: one or more processors; one or more non-transitory computer-readable media wherein the non-transitory computer-readable media stores, at least, a noise removal model of a machine learning-based diffusion model, and the machine learning-based diffusion model includes: a noise addition model including a plurality of noise addition stages, configured to receive input data and generate latent data in response to the reception of the input data; the noise removal model configured to reconstruct output data from the latent data; wherein the input to the noise removal model includes a set of Fourier features including a linear projection of channels of at least one of the plurality of noise addition stages; instructions that, when executed by the one or more processors, cause the computing system to execute the noise removal model to process the latent data to generate the output data; wherein the latent data includes a compressed representation of the input data and the output data includes a decompressed representation of the input data. ​ ​ ​ ​ ​ The noise removal model is a generative model configured to generate image data from an input signal, the latent data includes an input distribution, and the output data includes the generated image data, a computing system.

3. The computing system according to claim 1 or 2, wherein the set of Fourier features includes a linear projection of each channel of each of the plurality of noise addition stages.

4. The computing system according to claim 1 or 2, wherein the set of Fourier features includes a linear projection of at least one of the plurality of noise addition stages onto a set of periodic basis functions having high frequencies.

5. The computing system according to claim 1 or 2, wherein the set of Fourier features includes four channels.

6. The set of Fourier features is 【Number 1】 includes at least one Fourier feature of the form, where in the above equation, q is the frequency of the Fourier feature, i and j are position indices, k is a channel index, and z i,j,k is the network input at the position index and the channel index, the computing system according to claim 1 or 2.

7. The set of Fourier features is 【Number 2】 including at least one Fourier feature of the form, where in the above formula, q is the frequency of the Fourier feature, i and j are position indices, k is a channel index, and z i,j,k The computing system according to claim 1 or 2, wherein z is a network input at the position index and the channel index.

8. The input data includes a bit length, The computing system according to claim 1 or 2, wherein the set of Fourier features includes Fourier features having each frequency from 1 to the bit length.

9. The input data includes a bit length of 8 or more, The computing system according to claim 1 or 2, wherein the set of Fourier features includes Fourier features having each frequency from 7 to the bit length.

10. The computing system according to claim 1 or 2, wherein the input data includes image data.

11. A computer-implemented method for using a diffusion model with improved accuracy, comprising: obtaining, by a computing system comprising one or more computing devices, input data, the input data including one or more channels; providing, by the computing system, the input data to a machine learning-based diffusion model, the machine learning-based diffusion model comprising: a noise addition model including a plurality of noise addition stages, the noise addition model configured to receive the input data and introduce noise to generate intermediate data in response to the reception of the input data; and a noise removal model configured to reconstruct output data from the intermediate data and The input to the noise removal model includes a set of Fourier features including a linear projection of channels at at least one of the plurality of noise addition stages, a step, receiving, by the computing system, the output data from the machine learning-based diffusion model, a step including, The intermediate data includes a compressed representation of the input data, and the output data includes a decompressed representation of the input data, a computer-implemented method. **Claim 12** A computer-implemented method for using a diffusion model with improved accuracy, obtaining, by a computing system comprising one or more computing devices, input data, the input data including one or more channels, a step, providing, by the computing system, the input data to a machine learning-based diffusion model, the machine learning-based diffusion model a noise addition model including a plurality of noise addition stages, configured to receive the input data and introduce noise to generate intermediate data in response to receiving the input data, and a noise removal model configured to reconstruct output data from the intermediate data including, The input to the noise removal model includes a set of Fourier features including a linear projection of channels at at least one of the plurality of noise addition stages, a step, receiving, by the computing system, the output data from the machine learning-based diffusion model, a step including, The noise removal model is a generation model configured to generate image data from an input signal, the intermediate data includes an input distribution, and the output data includes the generated image data, a computer-implemented method. **Claim 13** The set of Fourier features includes a linear projection of channels of each of the plurality of noise addition stages, the computer-implemented method according to claim 11 or 12. **Claim 14** The set of Fourier features includes a linear projection of at least one of the plurality of noise addition stages onto a set of periodic basis functions with high frequencies, the computer-implemented method according to claim 11 or 12. **Claim 15** The set of Fourier features includes four channels, the computer-implemented method according to claim 11 or 12. **Claim 16** The set of Fourier features is 【Mathematics 3】 including at least one Fourier feature of the form, where in the above formula, q is the frequency of the Fourier feature, i and j are position indices, k is a channel index, and z i,j,k The computer-implemented method according to claim 11 or 12, wherein i,j,k is the network input at the position index and the channel index.

17. The set of Fourier features, 【Number 4】 including at least one Fourier feature of the form, where in the above formula, q is the frequency of the Fourier feature, i and j are position indices, k is a channel index, and z i,j,k The computer-implemented method according to claim 11 or 12, wherein i,j,k is the network input at the position index and the channel index.

18. The input data includes a bit length, The set of Fourier features includes Fourier features having each frequency from 1 to the bit length, the computer-implemented method according to claim 11 or 12.

19. The input data includes a bit length of 8 or more, The set of Fourier features includes Fourier features having each frequency from 7 to the bit length, the computer-implemented method according to claim 11 or 12.

20. The input data includes image data, the computer-implemented method according to claim 11 or 12.

21. The output data includes the reconstructed input data, the computer-implemented method according to claim 11 or 12.

22. A computer-implemented method for use of a diffusion model with improved accuracy, Receiving compressed data, Decompressing the compressed data with a noise removal model trained as part of a diffusion model, wherein the output data of the noise removal model includes a decompressed representation of the compressed data, a step Including, The input to the noise removal model includes a set of Fourier features including a linear projection of channels of at least one stage among a plurality of noise addition stages, a computer-implemented method.

23. The compressed data includes at least one of image data, audio data, text data, or incomplete data, the computer-implemented method according to claim 22.

Images