Multistep consistency models

Abstract

Systems and methods, implemented as computer programs on one or more computers for training a consistency model for use in generating a frame of data, such as a frame of image data, and methods of using a trained consistency model to generate a frame of data. A consistency model is used to generate a frame of data by predicting a succession of de-noised frames starting with an initial, noisy frame at an initial time and ending with a final frame, without noise, at a final time. The consistency model is trained to generate self-consistent predictions over a trajectory of predicted frames corresponding to these times. Implementations of the described techniques divide the trajectory into segments and only require the model to generate self-consistent predictions over each segment. This can facilitate the rapid generation of high quality frames of data.

Description

DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT ApplicationMULTISTEP CONSISTENCY MODELS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This Application claims priority to U.S. Provisional Application No. 63 / 661,510, filedon June 18, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Neural networks are machine learning models that employ one or more layers ofnonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0004] Background material relating to diffusion model distillation can be found in Berthelotet al. “TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation”, arXiv: 2303.04248v1 March 2023. Background material relating to consistency models can be found in Luo et al., “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference”, arXiv:2310.04378v1, October 2023. SUMMARY

[0005] This specification describes systems and methods, implemented as computerprograms on one or more computers in one or more locations, for training a consistency model for use in generating a frame of data, such as a frame of image data, or of audio data. Methods of using a trained consistency model to generate a frame of data are also described.

[0006] In general a consistency model is used to generate a frame of data by predicting asuccession of de-noised frames starting with an initial, noisy frame at an initial time and ending with a final frame, without noise, at a final time. The consistency model can be trained to generate self-consistent predictions over a trajectory of predicted frames corresponding to these times.

[0007] Implementations of the described techniques divide the trajectory into segments andonly require the model to generate self-consistent predictions over each segment. This can facilitate the rapid generation of high quality frames of data.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0008] The specification describes computer-implemented methods of training a consistency model that comprises one or more neural networks. The methods are performed for each of a plurality of de-noising (time) steps and involve training the consistency model to generate the same predicted frame for different times within a de-noising step. The training uses a loss function that depends on a transformed teaching frame (^^^^^^,௧ೞ^^^) corresponding to an initial step time of a de-noising step, and on a training predicted frame (^^^) also obtained from the consistency model.

[0009] The specification also describes methods for generating a frame of data using a trained consistency model. The trained consistency model need not have been trained using the described training methods.

[0010] There is further described a system comprising one or more computers, and one or more storage devices communicatively coupled to the one or more computers. The storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the described methods.

[0011] There is still further described one or more non-transitory computer storage media storing instructions that when executed by one or more computers perform the operations of the described methods.

[0012] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0013] Some conventional diffusion models can require hundreds or thousands of de-noising steps to generate a high resolution data sample, which is computationally expensive. By contrast implementations of the described training techniques can provide consistency models that produce good quality samples with fewer than 100, 20 or 10 de-noising steps. For example, some consistency models trained as described herein can match a baseline diffusion model performance with only 8 or 16 inference steps.

[0014] The consistency model training techniques described herein also have a well-defined trade-off for the model when used in inference, between the quality of a generated data frame and the number of steps used to generate it, i.e. between computing power / speed and sample quality.

[0015] Improved inference techniques are also described; these can be used independently of the described training techniques. In particular the described inference techniques can correct a tendency to generate blurred samples, counter-intuitively by adding noise during the sample generation process.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0016] In general the described techniques can be much more computationally efficient than some conventional approaches when used to generate frames of data representing, e.g., still or moving images or audio.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIGS.1A to 1C show an example system for training a consistency model, and systems for using a consistency model to generate a frame of data.

[0019] FIG.2 is a flow diagram of an example process for training a consistency model.

[0020] FIG.3 is a flow diagram of an example process for using a consistency model to generate a frame of data.

[0021] FIG.4 is a flow diagram of another example process for using a consistency model to generate a frame of data.

[0022] FIG.5 is a flow diagram of a further example process for using a consistency model to generate a frame of data.

[0023] FIGS.6A and 6B illustrate the performance of consistency models using the described techniques.

[0024] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0025] FIG.1A shows a computer-implemented training system 100 for training a consistency model as described herein.

[0026] The training system 100 includes a consistency model 110 comprising one or more neural networks. The consistency model 110, in particular the neural network(s), is configured to process a time, and a noisy frame corresponding to the time, to generate a predicted frame 120 for the time, i.e. for a de-noising step or “segment”. In implementations of the described techniques there are multiple de-noising steps between the initial time and the final time each starting at a respective initial step time (^^^௧^^).DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0027] In some implementations the consistency model can also process a step identifier to facilitate identifying the de-noising step. However this is not essential and, for example, the step may be inferred from the time processed by the consistency model 110.

[0028] The consistency model 110 is so named because it is trained to generate the same predicted frame for different times within each de-noising step, i.e. it is trained to generate consistent predictions along a de-noising trajectory. Implementations of the described techniques use a particular approach to split the overall de-noising trajectory into a set of discrete steps or segments, enforcing consistent predictions for each step or segment.

[0029] The trajectory represents a solution trajectory of a probability flow ordinary differential equation (ODE), and the trained consistency model in effect learns a mapping that solves for the probability ODE path along time. The mapping can be sharp and difficult to learn, especially for complex distributions, and splitting the trajectory into segments as described herein can make the learning easier.

[0030] When used in inference, e.g. as shown in FIG.1B, the trained consistency model 110 is used to iteratively de-noise an initial noisy frame over a plurality of de-noising steps, each de-noising step generating a respective predicted frame 120, to generate a final, de-noised frame of data 130. As described later, the frame of data 130 can represent, e.g., an image, an audio waveform, or other data.

[0031] In general, references to sampling or processing a frame, or values of a frame of data, are to sampling or processing values of data elements defined by the frame of data, e.g. pixel values of an image frame, audio signal values (e.g. instantaneous amplitude values) of an audio frame, and so forth.

[0032] In some implementations the consistency model 110 is trained using examples of the frames of data that the consistency model 110 will be used to generate, e.g. example images; this is referred to as “consistency training”. In some implementations the consistency model 110 is trained from a diffusion model; this is referred to as “consistency distillation”.

[0033] As shown in FIGS.1A and 1B, optionally the consistency model 110 may be conditioned on content conditioning data 140, an input to the consistency model 110 that characterizes a content of the generated frame of data 130. Optionally the content conditioning data 140 can be processed by a (pre-trained) conditioning data encoder 142 to provide the conditioning data input to the consistency model 110.

[0034] In some implementations the content conditioning data characterizes a content of the generated frame of data, e.g. defining one or more properties of the generated of frame of data or features that the generated of frame of data should represent. For example, for anDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationimage frame the content conditioning input may comprise text in a natural or computer language, or features of text, or audio, e.g. speech, or features of audio, that the image frame should represent.

[0035] As an example, where the content conditioning data 140 comprises text or audio the conditioning data encoder 142 may comprise a (pre-trained) text or audio encoder such as a Transformer-based language model, e.g. a pre-trained text encoder of a T5 language model (Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, arXiv:1910.10683.2023), or a (pre-trained) audio encoder of an audio language model or of a speech recognition system such as BEST-RQ (Chine et al. arXiv:2202.01855).

[0036] As shown in FIG.1C, in some implementations the consistency model 110 is configured to process a reference frame of data 150, such as a reference image, to generate a modified frame of data 160, e.g. a modified image 160. That is the consistency model 110 is configured to perform a data frame processing task such as an image processing task. In this case the content of the modified frame of data 160 that is generated is guided by the reference frame of data 150, as described further later.

[0037] Generally, the described techniques involve a “diffusion process”, from data into noise, and a corresponding de-noising process, and split the process into a multiple segments, or steps, each with a respective intermediate target. In effect a separate consistency model is trained for each segment of the process. In some implementations the consistency model 110 may comprise a separate neural network for each segment. In practice, however, a single consistency model with shared parameters can be used; i.e. the same neural network can be used to make predictions for all the segments.

[0038] Neural network(s)

[0039] The one or more neural networks of the consistency model 110 can have any suitable architecture consistent with processing values, e.g. pixel values, of a frame of data as an input to generate a set of output values for a frame of data, in particular a set of corresponding output values. For example the neural network(s) may have a U-Net architecture or a variant thereof, or a Transformer architecture (characterized by having a succession of attention layers) or a variant thereof, or a combination of these. Processing a time generally involves processing data specifying the time, e.g. an embedding of the time.

[0040] In general the neural network may comprise one or more feedforward, convolutional, attention, normalization, or other neural network layers. An attention layer is typicallyDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationconfigured to apply an attention mechanism, such as QKV attention, over an attention layer input to generate an attention layer output for each element of the input.

[0041] Merely as one example, the neural network(s) may comprise a U-Net (Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, arXiv:1505.04597) or other architecture that maps one input of a given dimensionality to an output of the same dimensionality. This may include one or more ResNet blocks (He et al. “Deep residual learning for image recognition”, Proc. IEEE conference on computer vision and pattern recognition. pp.770–778, 2016) and one or more self-attention layers. As another example the neural network(s) may comprise a diffusion transformer (DiT) (Peebles et al., “Scalable Diffusion Models with Transformers” arXiv:2212.09748, 2023) or a transformer backbone. A diffusion transformer can operate on latent patches. As a further example example the neural network(s) may comprise a U-ViT (Hoogeboom et al., arXiv:2301.11093, 2023), a U-Net with MLP (MultiLayer Perceptron) blocks instead of convolutional layers when a block has self-attention. An example U-ViT can have, e.g., a set of ResNet blocks and a set of Transformer blocks in the down path, an up path that matches, and a middle path that including another set of Transformer blocks.

[0042] Where a neural network is configured to process a conditioning data input, e.g. content conditioning data, this may be done, e.g., by incorporating one or more cross- attention layers to attend to the conditioning data, or in some other way. Merely as one example the content conditioning data may be provided as tokens or an embedding representing a content conditioning input. For example a content conditioning input may be encoded into a sequence of embeddings using a text, image, audio, or multimodal Transformer model such as a language model or vision language model, e.g. based on T5.

[0043] In some implementations the training system 100 includes target version 112 of the consistency model that has the same architecture as the consistency model. In some implementations the target version of the consistency model can just be the consistency model 110. In some implementations, for stability, the target version of the consistency model can have parameters that are periodically updated using corresponding learnable parameters, e.g. weights, of the consistency model 110.

[0044] For example the target version of the consistency model can be a periodically updated copy of the consistency model or have parameters ^^′ that are an exponential moving average of the learnable parameters ^^ consistency model. As an example the an exponential movingaverage can be determined as ^^ି ← ^^^^ି ^ ^1 െ ^^^^^ with 0 ^ ^^ ^ 1.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0045] FIG.2 is a flow diagram of an example process for training the consistency model 110 of FIG.1. The process of FIG.2 may be implemented by one or more computers in one or more locations.

[0046] The process is performed for each training frame of a plurality of training frames. These training frames can be sampled from a training dataset (^^ௗ^௧^). In implementations each training frame (^^) comprises an example of the frame of data, in particular an example from a distribution of frames of data that the consistency model is trained to generate.

[0047] In general the process is performed for each of a plurality of de-noising (time) steps. However a different step can be sampled for each training frame, i.e. it is not necessary to perform the process for each de-noising step for each training image. Broadly, each de- noising step corresponds to one of the above described segments of the trajectory of predicted frames.

[0048] The process involves determining a training time (t) between the initial time and the final time (step 202). The training time is associated with one of the de-noising steps. The method also determines a teaching time (s) that is before the training time (t) and during thesame de-noising step as the training time; i.e. ^^^௧^^ ^ ^^ ^ ^^ where ^^^௧^^ is the initial step timeof the de-noising step (step 204).

[0049] For example, the process can involve comprises determining a de-noising step (index), e.g. by sampling from a distribution. The training time can be determined as a time that is (at or) after the initial step time of the de-noising step (^^^௧^^) and either before the initial step time of next de-noising step or before the final time step (when the de-noising step is a final de-noising step).

[0050] Here the “next” de-noising step is the adjacent de-noising step that has a next largest initial step time, i.e. it is next in the direction of increasing time (in inference time runs in the reverse direction). Conveniently the initial time can be defined as 1; the final time can be 0 or substantially zero (in some cases for numerical stability the final time can have a small positive value).

[0051] The process uses the target version of the consistency model 112 to generate a teaching predicted framefor the de-noising step based on the teaching time (s) and on the training frame (^^) (step 206).

[0052] The consistency model 110 is used to generate a training predicted frame (^^^) for the de-noising step based on the training time (t) (step 208).DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0053] The process trains the consistency model 110 to generate the same predicted frame for different times within each de-noising step, in particular by updating the learnable parameters of the consistency model (and not the parameters of the target version of the consistency model) using a loss function (step 210).

[0054] In general the loss function depends on a transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time, ^^^௧^^(an initial time of the de-noising step), and on the training predicted frame (^^^). As described later, the loss can be determined in either ^^-space or ^^-space.

[0055] In implementations the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time is defined by a deterministic mapping between the teaching predicted frameand the transformed teaching frameinitial step time, i.e. between^ ^^^^^and ^^^^^^,௧ೞ^^^. particular this deterministic mapping uses a noise schedule that defines a variation of a noise level in a frame (of the succession of de-noised frames) with time. For example, the teaching predicted frame can be a de-noised frame and the transformed teaching frame can be a frame that includes noise according to the noise schedule and the initial step time. Any deterministic mapping can be used; a particular illustrative example, “DDIM”, is described later.

[0056] It is not necessary for the transformed teaching frameto be determined explicitly, although it can be. As described later, in some implementationsdetermined (explicitly or implicitly) from a teaching frame (^^^).

[0057] In general the noise schedule defines a variation in signal to noise ratio (SNR) for a de-noising process from the final time to the initial time that the consistency model is trained to follow. In general the noise schedule defines the SNR for the training time (t) and teaching time (s) as well as for the initial step times of the de-noising steps. In general the SNR decreases monotonically from the initial time to the final time. The SNR variation can be linear or non-linear; there is no need to use any particular type of SNR variation.

[0058] In some implementations the noise schedule defines one or more parameters in a noise addition “diffusion” process, optionally a variance-preserving diffusion process. The diffusion process can be described as a data corruption process that adds noise to the data at a level that depends on a time value, and the noise schedule defines the level of noise that isadded. For example, the data corruption process can be described as ^^௧ ൌ ^^௧^^ ^ ^^௧^^௧ where^^ is a frame of data, ^^௧is a corrupted frame of data, ^^௧is the added noise (e.g. ^^௧~^^^0,1^), and parameters ^^௧and ^^௧define the noise variation with time i.e. the noise schedule. As anDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationexample, for ^^ → 1, ^^௧ can be approximately distributed as a standard normal, and for ^^ → 0^^௧can be approximately ^^.

[0059] The noise schedule can, for example, define a value of one of parameters ^^௧and ^^௧relative to the other. That is, in implementations the noise schedule can define a time- dependent mean scaling factor for a frame (^^௧) and a time-dependent variance scaling factorfor a frame (^^௧), and the SNR may be defined as SNR ൌOptionally, but notnecessarily, determining one of the mean scaling factor and the variance scaling factor maydetermine the other, e.g. according to ^^ଶ ଶ௧ ൌ 1 െ ^^௧ , which defines a variance-preservingnoise (SNR) schedule. Diffusion models have experimented with various noise schedules and any of these can be used with the presently described techniques.

[0060] In general a consistency model ^^^^^௧ , ^^^ is constrained to predict ^^ ൌ ^^^^^^, 0^ and canbe trained to be consistent by minimizing the consistency loss ฮ^^^^^௧ , ^^^ െ nograd൫^^^^^^, ^^^൯ฮwhere ^^^ ൌ ^^^^^ ^ ^^^^^ and ^^௧ ൌ ^^௧^^ ^ ^^௧^^ (with the same ^^ for both), and where ^^ ^ ^^ (^^ iscloser to the uncorrupted data). Here nograd^∙^ indicates that gradients are not backpropagated into the term inside the brackets (which term may be obtained from the target version of the consistency model), and ‖∙‖ denotes the L2 norm. The consistency model can be trained by consistency distillation by using a deterministic model, such as “DDIM”, to predict ^^^from ^^௧and ^^ (or an estimate of ^^); again any deterministic model can be used.The step size, or gap between model evaluations at ^^ and ^^, can be determined as ^^ െ ^^ ൌ1 / ^^^^^^ where ^^^^^^ is the total number of time steps. The total number of time steps can vary with training iteration ^^ to anneal the step size (large steps propagate a solution faster through diffusion time but incur a larger bias).

[0061] As previously mentioned, one example of a deterministic mapping as described above is “DDIM” (Song et al., “Denoising Diffusion Implicit Models”, arXiv:2010.02502v4, October 2022). An example of this mapping, which predicts ^^^from ^^௧and ^^ (which may be an estimate), is given (in a variance-preserving setting) by:

[0062] An inverse of this function which predicts ^^ from ^^௧and ^^^, can also be defined, as:

[0063] In implementations of the described techniques, instead of predicting ^^ ൌ ^^^^^^, 0^ ^^^^the targets, and consistency loss, are generalized to targets ^^௧ೞ^^^. The deterministicDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationmapping is used to map noisy frames back to ^^௧ೞ^^^at ^^^௧^^. Thus one example of a multi- step consistency loss can be expressed as:

[0064] In this example the transformed teaching frame, ^^^^^^,௧ೞ^^^, may be determined explicitly from a teaching frame, ^^^, at the teaching time ^^, using the deterministic mapping. For example if the deterministic mapping is a DDIM mapping,may be determinedas ^^^^^^,௧ೞ^^^ ൌ DDIM^→௧ೞ^^^൫nograd൫^^^^^^, ^^^൯, ^^^൯, where ^^^ is a teaching frame that can bedetermined from ^^, again using the deterministic mapping.

[0065] As one example ^^^ ൌ DDIM^→^^^^, ^^௧^. As another example the deterministicmapping described later and referred to as aDDIM^∙^can be used. The aDDIM^∙^mappingachieves an effect of noise addition in a deterministic way, and can lead to improved results.

[0066] The above example of a multi-step consistency loss is determined in ^^-space, i.e. a space of the noisy frames. Alternatively the multi-step consistency loss can be determined in ^^-space, i.e. a space of de-noised frames. This can be done by re-parameterizing the loss to be dependent on a difference,^^^ௗ^^^, between the transformed teaching predicted frame (^^^^^^,௧ೞ^^^) and the training predicted frame (^^^). As an example the difference can be determined as:

[0067] The multi-step consistency loss can depend on^ ^^ௗ^^^is many ways. As an examplethe multi-step consistency loss, ^^௧, can be determined as ^^௧ ൌ ^^௧ ∙where ^^௧ is aweight that weightsฮ, a distance between the true data point and the model prediction.The weight can be chosen in various ways; as one example ^^௧ ൌ SNR ^ 1 (a “^^-loss”, asdescribed in Salimans et al., arXiv:2022.00512v2). The above example uses the Euclidean(L2) distancebut other metrics, e.g. a squared Euclidean distance, can alsobe employed.

[0068] In some implementations using the consistency model to generate the training predicted frame (^^^) for the training time (t) involves generating and using a reduced noise training frame (^^௧) to generate the training predicted frame.

[0069] This can involve generating a noisy frame (^^), e.g. by sampling values for the dataframe from a distribution such as a Gaussian distribution, e.g. as ^^~^^^0, ^^^. The reducednoise training frame (^^௧) for the training time (t) is then determined (deterministically) fromDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationthe training frame (^^), the noisy frame (^^), and the noise schedule (e.g. from ^^௧, ^^௧; SNR).For example the reduced noise training frame can be determined as ^^௧ ൌ ^^௧^^ ^ ^^௧^^.

[0070] The reduced noise training frame (^^௧) and the training time (t) can then be processed using the consistency model to generate the training predicted frame (^^^) (based on thetraining time, t), e.g. as ^ ^^ ൌ ^^^^^௧ , ^^^, where ^^^∙,∙^ denotes the consistency model.

[0071] Where a step identifier is used, the reduced noise training frame (^^௧), the training time (t), and the step identifier for the de-noising step can then be processed using the consistency model to generate the training predicted frame (^^^).

[0072] In implementations using the target version of the consistency model to generate the teaching predicted framefor the de-noising step based on the teaching time (s) involves generating a teaching frame (^^^) from the reduced noise training frame (^^௧) and a teacher frame (^^௧^^^^^^), and using ^^^to generate the teaching predicted frame

[0073] More particularly this can involve applying the deterministic mapping to both (to a combination of) the teacher frame (^^௧^^^^^^) and the reduced noise training frame (^^௧) (obtained from the training frame, ^^), using the noise schedule, to determine the teaching frame (^^^) for the teaching time (s). For example this can involve determining the teachingframe as ^^^ ൌ DDIM^→^^^^௧^^^^^^ , ^^௧^ or as ^^^ ൌ aDDIM^→^^^^௧^^^^^^ , ^^௧ , ^^௩^^^ where ^^௩^^represents an estimate of the variance of the training data (that may be computedbeforehand). As one example ^^ can be estimated as ^^ ൌ ‖^^ ‖ଶ௩^^ ௩^^ ௧^^^^^^ െ ^^ / ^^ where ^^is the dimensionality (number of elements in) of the frame of data.

[0074] The teaching frame (^^^) and the teaching time (s) can then be processed using the target version of the consistency model to generate the teaching predicted frame (^^^^^^) (based on teaching time, s). For example the teaching predicted frame can be determined as^ ^^^^^ ൌ nograd൫^^^^^^, ^^^൯, or as ^ ^^^^^ ൌ ^^′^^^^, ^^^ where ^^′^∙,∙^ denotes the target version ofthe consistency model (which may be the same as the consistency model, i.e. in someimplementations ^^^∙,∙^ ൌ ^^′^∙,∙^).

[0075] Where a step identifier is used, the teaching frame (^^^), the teaching time (s), and the step identifier for the de-noising step can then be processed using the target version of the consistency model to generate the teaching predicted frame (^^^^^^).

[0076] The transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time can be defined by the deterministic mapping, using the noise schedule, from both of the teaching predictedDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationframe for the de-noising step and the teaching frame (^^^) to the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time.

[0077] In some implementations the transformed teaching frame is determined explicitly, by applying the deterministic mapping, using the noise schedule, to both the teaching predicted frame and the teaching frame (^^^) to generate the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time. For example the transformed teaching frame can bedetermined

[0078] In some implementations, when the multi-step consistency loss is determined in ^^- space, the difference,^ ^^ௗ^^^, between the transformed teaching predicted frame (^^^^^^,௧ೞ^^^) and the training predicted frame (^^^) can be determined by evaluating^^^൯, ^^௧^ without an explicit determination of ^^^^^^,௧ೞ^^^explicitly.

[0079] In some implementations the teacher frame (^^௧^^^^^^) comprises the training frame, to implement a version of “consistency training”.

[0080] In some implementations the process implements a version of “consistency distillation”, in which a pre-trained diffusion model is distilled into the consistency model. These implementations can involve obtaining the teacher frame (^^௧^^^^^^) by processing the training time (t) and the reduced noise training frame (^^௧) using a trained diffusion model neural networkto generate the teacher frame (^^௧^^^^^^), i.e. the teacher frame (^^௧^^^^^^) comprises a sample from the trained diffusion model neural network.

[0081] In general such a diffusion model neural network can be a neural network that has been trained to process a diffusion input comprising a current noisy data item and data specifying a current time to generate a diffusion output that defines an estimate (given the current time) of either a noise component of the current noisy data item, i.e. an estimate of the noise that has been added to an original data item to generate the current noisy data item; or of a de-noised version of the current noisy data item.

[0082] As previously mentioned, and as described further later, some implementations of the described techniques involve determining a variance estimate (^^௩^^) for the teacher frame (^^௧^^^^^^). This can then be used to add noise to the teaching frame (^^^) dependent on the variance value, and that increases with increasing variance value. Counter-intuitively this can improve the quality of the generated frames of data by reducing blurring.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0083] For example this can involve determining the teaching frame (^^^) as a weighted sum of the teacher frame (^^௧^^^^^^) and of a deterministic estimate of the noisy frame (^^^). The teacher frame can be scaled by a first weight and the deterministic estimate of the noisy frame can be scaled by a second weight, where the first and second weights depend on the noise schedule. It is not necessary to explicitly determine ^^^ (e.g. the deterministic estimate of the noisy frame can be obtained by including ^^^and ^^௧^^^^^^in the calculation of ^^^), although it can be explicitly determined.

[0084] Determining the teaching frame (^^^) as a weighted sum can involve including a contribution to a weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum that depends on a variance estimate (^^௩^^) for the teacher frame (^^௧^^^^^^). In particular the contribution can increase a weight of the deterministic estimate of the noisy frame, i.e. so that the weight of the deterministic estimate of the noisy frame increases with increasing estimated variance.

[0085] In some implementations the variance estimate for the teacher frame (^^௧^^^^^^) can be determined from a difference between the teacher frame (^^௧^^^^^^) and the training frame (^^). This can be scaled by a dimension of the data (d) as the difference will depend on this.

[0086] As previously mentioned, in some implementations the loss is determined in ^^-space, i.e. a space of the noisy frames. This can involve applying the deterministic mapping, using the noise schedule, to both the training predicted frame (^^^) and the teaching frame (^^^) to generate a transformed training predicted frame (^^^௧ೞ^^^) for the initial time step. The consistency model can then be trained using a loss function for the multi-step consistency loss that depends on a difference between the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time and the transformed training predicted frame (^^^௧ೞ^^^)) for the initial time step.

[0087] In some implementations the loss is determined in ^^-space, i.e. a space of de-noised frames. This can involve applying an inverse of the deterministic mapping (e.g. invDDIM^∙,∙^), using the noise schedule, to the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time and to the reduced noise training frame (^^௧) for the training time (t) togenerate a transformed teaching predicted frame (^^^^^^,௧ೞ^^^ ൌ invDDIM^→^౩౪^౦ ^^^^^^^,௧ೞ^^^ , ^^௧^,based on the training time (t). The consistency model can then be trained using a loss function that depends on a difference between the transformed teaching predicted framepredicted frame (^^^).DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0088] The inverse of the deterministic mapping can be defined by the property that applying the deterministic mapping, using the noise schedule, to both i) the transformed teaching predicted framefrom the inverse deterministic mapping and ii) the reduced noise training frame (^^௧) generates the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time.

[0089] In some implementations a value of the loss function can be scaled by a time- dependent weight, e.g. to de-emphasize (or emphasize), losses where the training signal is high, i.e. at low times, close to the training data. In some implementations the weight can be flat, i.e. no time-dependent weight is used.

[0090] Some examples of losses that can be used for a loss function for the multi-step consistency loss described herein include an L1 loss, an L2 loss, a Pseudo-Huber loss, a LPIPS (Learned Perceptual Image Patch Similarity) loss, and a loss with an SNR-dependent weighting such as a “truncated SNR” weighting or an “SNR+1” weighting (^^-loss).

[0091] Merely as an example, one implementation of a training process as described above is given below (where ^^ is the total number of training steps, ^^^^^^ in an iteration):As an illustration, the multistep consistency model can have, e.g., 2, 4, 8 or 16 steps; and ^^^^^^ can define a log-liner interpolated schedule from 64 total teacher steps to 1280 total teacher steps.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0092] Where the consistency model 110 is conditioned on a content conditioning data 140 input the model may be trained as described above using, e.g. so-called classifier free guidance (Ho and Salimans, “Classifier-free diffusion guidance”, arXiv.2207.12598, 2022). This can involve randomly masking out or otherwise removing the conditioning data from the consistency model input so as to train the neural network to generate the respective predicted frame both with and without guidance from the conditioning data.

[0093] As described later, the trained consistency model 110 can also be used to perform various image and other processing tasks without further training, as the training inherently enables the model to de-noise a noisy data frame.

[0094] Inference

[0095] FIG.3 is a flow diagram of an example process for using a consistency model trained as described above, e.g. the consistency model 110 of FIG.1, to generate a frame of data. The process of FIG.3 may be implemented by one or more computers in one or more locations.

[0096] As previously described the consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time, i.e. for a de- noising step. In general there are multiple de-noising steps between the initial time and the final time each starting at a respective initial step time.

[0097] The example process involves generating a noisy frame of data (step 302), e.g. by sampling from a distribution such as a Gaussian distribution, and determining, from the noisy frame, a current reduced noise frame (^^௧) for an initial de-noising step (step 304). The current reduced noise frame (^^^) for the initial de-noising step can be the noisy frame of data. Alternatively this can involve processing the noisy frame and the initial time (a time for the initial de-noising step) using the consistency model to generate an initial reduced noise frame (^^^) for use as the current reduced noise frame (^^^) for the initial de-noising step.

[0098] The process is performed iteratively, for each of a plurality of (further) de-noising steps until a final de-noising step.

[0099] In implementations a time for the de-noising step, e.g. the initial step time, ^^^௧^^, for the de-noising step, and the current reduced noise frame (^^௧), are processed using theconsistency model, to generate a current predicted frame for the de-noising step, e.g. as ^ ^^ ൌ^^^^^௧ , ^^^ (step 306).

[0100] The process applies a deterministic mapping, such as DDIM or aDDIM, to both the current predicted frame (^^^) for the de-noising step and the current reduced noise frame (^^௧)DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationfor the de-noising step, using a noise schedule that defines a variation of a noise level in a frame with time (over the succession of de-noised frames), to determine the current reduced noise frame (^^^) for the next de-noising step (^^^becomes ^^௧for the next step) (step 308).

[0101] At the final de-noising step the current reduced noise frame for the next de-noising step comprises the generated frame of data (step 310).

[0102] Merely as an example, one implementation of the data frame generation process of FIG. 3 is given below:

[0103] FIG. 4 is a flow diagram of another example process for using a consistency model e.g. the consistency model 110 of FIG. 1, to generate a frame of data. The process of FIG. 4 can be used with any consistency model, whether or not trained as described above. The process of FIG. 4 may be implemented by one or more computers in one or more locations.

[0104] In general the consistency model used in the process of FIG. 4 comprises one or more neural networks for generating a frame of data by predicting a succession of de-noised frames starting with an initial noisy frame at an initial time and ending with a final frame without noise at a final time. The consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time.

[0105] The process involves generating a noisy frame (of data), e.g. by sampling from a distribution such as a (multivariate) Gaussian distribution (step 402), and determining, from the noisy frame, a current reduced noise frame (^^௧) for an initial de-noising time (step 404). The current reduced noise frame (^^^) for the initial de-noising time can be the noisy frame (of data). Alternatively this can involve processing the noisy frame and the initial time using the consistency model to generate an initial reduced noise frame (^^^) for use as the current reduced noise frame (^^^) for the initial de-noising time.

[0106] The process is performed iteratively, for each of a plurality of (further) de-noising times until a final de-noising time.

[0107] In implementations the de-noising time and the current reduced noise frame (^^௧), are processed using the consistency model, to generate a current predicted frame for the de-noising time, e.g. as ^ ^^ ൌ ^^^^^௧ , ^^^ (step 406).

[0108] The current reduced noise frame for the next de-noising time (^^^) is determined from a weighted sum of the current predicted frame (^^^) for the de-noising step and an estimatedDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationnoisy frame (^^^) (step 408). That is, the current reduced noise frame, ^^௧, is used to obtain the current reduced noise frame for the next de-noising time, ^^^.

[0109] In implementations the estimated noisy frame (^^^) comprises a deterministic estimate of the noisy frame obtained from the current predicted frame (^^^) for the de-noising step and the current reduced noise frame (^^௧) for the de-noising time, e.g. using a noise schedule that defines a variation of a noise level in a frame with time. For example the estimated noisyframe can be determined as ^^^ ൌ ^^^௧ െ ^^௧^^^^ / ^^௧.

[0110] The determination of the current reduced noise frame for the next de-noising time (^^^) can include determining a positive addition to a weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum, where the positive addition depends on a variance of frames predicted by the consistency model (^^௩^^,௧). In some implementations the variance is predetermined (pre-computed). In some implementations the variance of frames predicted by the consistency model is dependent on the de-noising time.

[0111] At the final de-noising time the current reduced noise frame for the next de-noising time comprises the generated frame of data (step 410).

[0112] In broad terms, a data frame generation process using a consistency model can suffer from over-smoothing. This can be addressed by adding noise during the data frame generation process. More particularly this is done by increasing a contribution of the deterministic estimate of the noisy frame, ^^^, in particular in a way that depends on the variance of frames predicted by the consistency model (^^௩^^,௧). There are many ways of doing this. Whilst this is a heuristic, empirically it has been found to improve the quality of the generated data frames.

[0113] In general the variance can be determined based on a variance of ^^^given ^^௧, Var^^^^|^^௧^. As a particular example the current reduced noise frame can be determined as^^∙ ^^^ where tr^∙^ denotes the trace of a matrix.tr^Var^^^^|^^௧^^ can be determined empirically from the data by precomputingtr^Var^^^^|^^௧^^ ൌ ^^‖^^^^^^௧^ െ ^^‖ଶ where ^^ is a hyperparameter, e.g. ^^ ൌ 0.75; ortr^Var^ |^^௧^^ can be analytically,the data dimension and ^^ ^ 1 is a constant, e.g. ^^ ൌ 0.1, which can be interpreted as 10% ofthe posterior variance of ^^.

[0114] More generally, in some implementations the variance of frames predicted by the consistency model is determined based on the noise schedule, e.g. based on ^^^^^^^^^^. In someDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationimplementations the variance of frames predicted by the consistency model is determined from one or more example frames of data (^^).

[0115] The process can involve determining the estimated noisy frame (^^^) from the current predicted frame (^^^) for the de-noising time and the current reduced noise frame (^^௧) for the de-noising time. The current reduced noise frame (^^^) for the next de-noising time can then be obtained by determining the weighted sum of the current predicted frame (^^^) for the de- noising time and the estimated noisy frame (^^^).

[0116] In some implementations the process of FIG. 4 involves obtaining a variance (^^^,௩^^) for the current reduced noise frame (^^^) by modifying the variance of frames predicted by the consistency model (^^௩^^,௧) using the noise schedule, and determining the positive addition to the weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum from the variance for the current reduced noise frame (^^^,௩^^).

[0117] Merely as an example, one implementation of the data frame generation process of FIG. 4 is given below:

[0118] Frames of data

[0119] In general a frame of data as described herein can represent any type of data that can be represented as a frame in one, two, or more dimensions. In this context a frame can be a data structure of a defined size containing data elements with a defined order. The data elements can be, e.g. pixels of an image (or points of a point cloud), instantaneous or other audio signal values, atomic or molecular coordinates, data elements representing sensor readings, or any other type of data element.

[0120] As used herein an image can be a still or moving image, in monochrome or color (including in non-visible wavelengths), in 2D or in 3D; and can includes a LIDAR point cloud (a “pixel” may then be a point of the point cloud).

[0121] An image, or training image, as described herein may be an image that represents a real world environment. For example a training image may be captured from the real worldDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationby a camera or other image sensor. In a training image, or an image generated by a diffusion model trained as described herein, objects represented in the image may comprise physical real-world objects.

[0122] In some implementations the frames of data need not necessarily represent images and may, as some other examples, represent audio data, weather or climate data, or fluid mechanics data. In some implementations an image frame may encode another type of data, e.g. as a spectrogram.

[0123] As one particular example, audio or other data may be represented as a spectrogram. The audio or other data may be processed to generate a spectrogram representing the audio or other data by performing a time-frequency domain transform on an audio or other signal to generate a frequency domain representation of the audio or other signal for a range of frequencies. There are many suitable time-frequency domain transforms; as one example a short-time Fourier transform, STFT. The audio or other data (signal) may be decoded from a generated spectrogram by applying the inverse of the time-frequency domain transform, i.e. a frequency-time domain transform, e.g. an inverse STFT.

[0124] In general a spectrogram can be an image that represents the time-frequency domain transform, e.g. a representation with time on one axis, e.g. a horizontal axis, and frequency on another axis, e.g. a vertical axis. The location of a pixel of the image along the time axis can represent a time position in the audio or other data; the location along the frequency axis can represent a frequency at that time; the value of the pixel e.g. a luminance or color value, can represent a component of the audio or other signal at that time and at that frequency, e.g. a magnitude and / or a phase of the signal. In some time-frequency domain transforms the component of the signal is represented by a complex number. Generally, the spectrogram can represent a changing spectrum of an audio or other signal over time.

[0125] Merely as one example, a consistency model as described herein can be used to generate a moving image i.e. video by treating the temporal axis dimension as an additional spatial dimension.

[0126] Where a moving image is generated this can comprise, as some examples, a continuation of a previous video sequence, an edited version of a video sequence, or a video sequence generated to represent a text or audio conditioning input.

[0127] As another example, implementations of the system can be used to generate an image representing a predicted state or trajectory of a real-world physical system, such as a robot or vehicle, for use by a control algorithm in controlling the physical system. For example a video sequence may be captured by a camera and then the system used to generate a still orDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationmoving image that predicts a future state or configuration of the physical system, optionally conditioned on one or more variables relating to the physical system. This can be used in a model predictive control system to control a mechanical agent such as a robot to perform a particular task, by processing the predicted image using the control system to generate control signals to control the mechanical agent, in accordance with the generated image to perform the task.

[0128] As some other examples, a frame of data may comprise, e.g., sensor data such as data for an EEG (electroencephalogram) or ECG (electrocardiogram) trace; or medical imaging data such as CT (computed tomography) or MRI (magnetic resonance imaging) data; or data representing a 3D configuration of a molecule, e.g. as coordinates of atoms of the molecule.

[0129] A frame of data as described above can be generated based on the content conditioning data 140. For example the content conditioning data 140 can comprise text, or audio, or image, or other data, e.g. sensor data representing the configuration of a mechanical system as described above. Then one or more frames of data can be generated representing a still or moving image or audio that characterize or are described by the content conditioning data 140, e.g. to generate an image representing text, or speech audio representing text in a text-to-speech task.

[0130] Example training datasets

[0131] A consistency model as described herein may be trained using either consistency training or consistency distillation. The consistency training can use any dataset appropriate to the frames of data to be generated, and a corpus of training data may be collected accordingly. When trained from a diffusion model using consistency distillation the diffusion model can be any available trained model, or a diffusion model can be trained (using any appropriate training data) and then distilled into the consistency model.

[0132] Merely as an example, when generating frames of data that represent images a public dataset such as ImageNet or the Open Images Dataset can be used to obtain training data, and also to obtain content conditioning data if used (speech synthesis can be used to generate audio conditioning data from text data). When generating frames of data that represent audio any public dataset can be used to obtain training data, e.g. LibriTTS-R, https: / / www.openslr.org / 141 / .

[0133] Image and other processing

[0134] A consistency model trained as described above, and process of generating a frame of data as described above, can be used to perform various data frame processing tasks such as data frame interpolation (from one or more frames of data), data frame interpolation (betweenDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationframes of data), data frame resolution or other enhancement, data frame editing, data frame de-noising, compressed sensing, and data frame modification in general. In implementations this can involve obtaining a reference data frame to guide the data frame modification task, i.e. to guide to generation of a frame of data.

[0135] As an example, when the frame of data comprises an image a process of generating a frame of data similar to that described above can be used to perform an image modification task such as colorization, super-resolution, in-painting, out-painting, interpolation, or stroke- guided image generation. As another example a process of generating a frame of data similar to that described above can be used to perform a compressed sensing task, by generating a frame of data representing an image whilst imposing data consistency with compressed sensing measurements. For example, measurements for MRI images can be obtained in k- space (spatial frequency space), e.g. using compressed sensing and undersampling the k- space, and an MRI image reconstructed by using the consistency model to generate a frame of data representing an image whilst imposing data consistency with the measurements, e.g. as described in Peng et al., arXiv:2203.04292.

[0136] In general performing an image modification task can involve obtaining a reference image to guide the image modification task, and using the reference image to modify the current predicted frame at each iteration to guide generation of the current reduced noise frame. The reference image can define the image to be modified (which may have a lower resolution or have a different size or boundaries to the image represented by the generated frame of data). In some implementations a mask frame can be used to mask values of pixels that are to be imputed by the process. A corresponding process can be used to perform other data frame modification tasks, where a frame of data represents something other than an image, e.g. audio.

[0137] Some examples of using a reference image to guide an image modification task are described in, e.g. Song et al., “Consistency models”, in International Conference on Machine Learning (ICML), 2023, and these may be adapted for use with the presently described techniques.

[0138] As previously described, implementations of the described techniques can produce good quality samples with only a few inference steps. Because of this using a reference data frame to guide a data frame modification task as described above can result in artefacts.

[0139] One way to address this is to fine-tune the consistency model based on the particular task or tasks to be performed. That is, the consistency model can be (further) trained whilst it receives example reference data frames guiding the data frame modification task, e.g. whilstDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationit receives reference images whilst performing an image processing task, so that the model learns which data elements or pixels are to be modified and which are given by the task (and are to remain unmodified).

[0140] Another way to address this is to adapt the data generation process, e.g. as described below.

[0141] FIG.5 is a flow diagram of a further example process for using a consistency model to generate a frame of data. The process of FIG.5 can be used with any consistency model, whether or not trained as described above. It is particularly useful when using a reference data frame, e.g. a reference image, to guide a data frame modification task, e.g. an image processing task, but the process can also be used in other contexts. The process of FIG.5 may be implemented by one or more computers in one or more locations.

[0142] In general the consistency model used in the process of FIG.5 comprises one or more neural networks for generating a frame of data by predicting a succession of de-noised frames starting with an initial noisy frame at an initial step and ending with a final frame without noise at a final step. The consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time.

[0143] The process involves initialising a current reduced noise frame (step 502). This can involve generating a noisy frame (of data), e.g. by sampling from a distribution such as a (multivariate) Gaussian distribution, and determining from this a current reduced noise frame for an initial de-noising step. An initial reduced noise frame for use as the current reduced noise frame for the initial de-noising step can be the generated a noisy frame (of data). Alternatively this can involve processing the noisy frame and the initial time (a time for the initial de-noising step) using the consistency model to generate an initial reduced noise frame for use as the current reduced noise frame for the initial de-noising step.

[0144] The process is performed iteratively, for each of a plurality of (further) de-noising steps until a final de-noising step.

[0145] For each step a time for the de-noising step and the current reduced noise frame are processed, using the consistency model, to generate a current predicted frame (^^^) for the de- noising step (step 504). An initial current reduced noise frame for the next de-noising step is determined from the current predicted frame (^^^) for the de-noising step and, in particular, a noise schedule that defines a variation of a noise level in a frame with time (step 506). As some particular examples, this can be done as described above; other techniques can also be used.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0146] The initial current reduced noise frame is refined by performing one or more refining steps to determine the current reduced noise frame for the next de-noising step (step 508).

[0147] Performing each refining step involves adding noise to a reduced noise frame for refinement to obtain a re-noised frame (step 508a), and de-noising the re-noised frame, in particular using the noise schedule (step 508b). Some particular example implementations of this in the context of the previously described techniques are given later.

[0148] In a first refining step the reduced noise frame for refinement is the initial current reduced noise frame. In a subsequent refining step, if present, the reduced noise frame for refinement is the refined reduced noise frame from a previous refining step. The refined reduced noise frame after the last refining step provides the current reduced noise frame for used in the next de-noising step.

[0149] In some implementations multiple reefing steps are performed for each of some or all of the de-noising steps. Then each successive refining step (at any particular de-noising step) can add less noise to the reduced noise frame for refinement than a preceding refining step.

[0150] In broad terms this adds extra “refining” steps to the sampling procedure, byperforming a de-noising step, e.g. as previously described (^^ →and then a re-noisingstep, adding forward diffusion noise (e.g. ^^^௧^^ → ^^).

[0151] In some implementations multiple de-noising and re-noising refining steps are added. Then the amount of noise (the value of ^^) can be different for each re-noising step, in particular to add less noise at each successive re-noising step. Thus where, say, M steps areadded for each of N de-noising steps, the consistency model can become an ^^ ൈ ^^ stepconsistency model. Whilst this takes more steps, and hence slows down the inference process, it makes the incorporation of the constraint smoother and works for arbitrary constraints including, for example, the previously mentioned compressed sensing.

[0152] There now follow some example implementations of this technique in the context of the processes of FIGS.3 and 4.

[0153] In the context of the process of FIG.3, applying the deterministic mapping to determine the current reduced noise frame for the next de-noising step can involve applying the deterministic mapping to both the current predicted frame (^^^) for the de-noising step and the current reduced noise frame (^^௧) for the de-noising step, using the noise schedule, to determine an initial current reduced noise frame for the next de-noising step, and refining the initial current reduced noise frame by performing one or more refining steps to determine the current reduced noise frame for the next de-noising step.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application

[0154] Each refining step can then involve adding noise to a reduced noise frame for refinement to obtain a re-noised frame, and de-noising the re-noised frame using the noise schedule. More particularly adding noise to the reduced noise frame for refinement to obtain the re-noised frame can involve adding noise according to the noise schedule and a re-noising time (^^).

[0155] In implementations the re-noising time can define a time that is between a time for the de-noising step and a time for the next de-noising step. That is, the time for the re-noising step can be further from the final generated frame of data than, in implementations greater than, the time for the next de-noising step, but less than or equal to the time processed to generate the current predicted frame.

[0156] In implementations de-noising the re-noised frame using the noise schedule can involve comprises applying the deterministic mapping to both the re-noised frame and a frame (^^^) predicted, using the consistency model, from the re-noised frame and the re-noising time. For example this can be done using a re-arrangement of the above DDIM mapping(^^^ ൌ ^^^^^ ^ ^ఙೞఙ^ ^ ^^^௧ െ ^^௧^^^) to go from ^^^ (closer to the final data) to ^^௧.

[0157] In the context of the process of FIG.4, for each of some or all of the plurality of de- noising times the current reduced noise frame for the next de-noising time is refined by performing one or more refining steps.

[0158] Performing each refining step involves adding noise to the current reduced noise frame for the next de-noising time to obtain a re-noised frame, and determining a refined version of the current reduced noise frame for the next de-noising time from the re-noised frame using the consistency model. In implementations adding noise to the current reduced noise frame for the next de-noising time to obtain a re-noised frame can involve adding noiseaccording to the noise schedule and a re-noising time, e.g. using ^^^ ൌ ^^^௧ െ ^^௧^^^^ / ^^௧. Herethe re-noising time can be further from the final data, e.g. larger than, the next de-noising time.

[0159] In implementations determining the refined version of the current reduced noise frame for the next de-noising time from the re-noised frame using the consistency model can involve processing the re-noising time and the re-noised frame, using the consistency model, to generate a predicted frame (^^^) for the re-noising time. The refined version of the current reduced noise frame can then be determined from the predicted frame for the re-noising time and an estimated noisy frame (^^^) for the re-noising time. By analogy with FIG.4, theDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationestimated noisy frame can be a deterministic estimate of the noise in the frame obtained from the predicted frame for the re-noising step and the re-noised frame for the re-noising time.

[0160] Again, multiple refining steps can be used, and each successive refining step (for a de- noising time) can add less noise to the refined version of the current reduced noise frame than a preceding refining step.

[0161] Example results

[0162] FIG.6A shows a comparison between an image generated using a consistency model as descried herein (upper image) and one generated using a diffusion model (lower image). A 20B parameter diffusion model was trained on text-to-image pairs and the lower image was generated using a 100-step DDIM sampler. A 16-step consistency model was trained using the diffusion model (“consistency distillation”), using an implementation of the process of FIG.2, and this was used to generate the upper image, using an implementation of the process of FIG.3. FIG 6B corresponds to FIG.6A but used an implementation of the process of FIG.4. The same prompt (content conditioning data) was used for both images; and both images were generated from the same initial random noise.

[0163] The examples illustrates that the differences in image quality are minor and there is essentially no performance loss. However the images generated using the techniques described herein were generated more than 10 times faster.

[0164] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0165] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationreadable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0166] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Thus a system, artificial neural network, or trained artificial neural network as described herein, can be implemented in hardware using electronic circuitry, e.g. in a physical box. Similarly computer code as described herein can be code to emulate such hardware or code for a hardware description language.

[0167] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0168] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or moreDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationcomputers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0169] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0170] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0171] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0172] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from aDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationdevice that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0173] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0174] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.

[0175] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0176] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0177] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT ApplicationConversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0178] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0179] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT ApplicationCLAIMS 1. A computer-implemented method of training a consistency model, the consistency model comprising one or more neural networks for use in a system to generate a frame of data by predicting a succession of de-noised frames starting with an initial noisy frame at an initial time and ending with a final frame without noise at a final time, wherein the consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for a de-noising step, and wherein there are multiple de-noising steps between the initial time and the final time each starting at a respective initial step timethe method comprising: maintaining a target version of the consistency model that has the same architecture as the consistency model and parameters that are periodically updated using corresponding learnable parameters of the consistency model; and, for each training frame (^^) of a plurality of training frames: determining a training time (t) between the initial time and the final time, wherein the training time is associated with one of the de-noising steps; determining a teaching time (s) that is before the training time (t) and during the same de-noising step; using the target version of the consistency model to generate a teaching predicted frame for the de-noising step based on the teaching time (s) and on the training frame (^^); using the consistency model to generate a training predicted frame (^^^) for the denoising step based on the training time (t); and training the consistency model to generate the same predicted frame for different times within the de-noising step by updating the learnable parameters of the consistency model using a loss function that depends on a transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time of the de-noising stepand on the training predicted frame (^^^), wherein the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time is defined by a deterministic mapping, using a noise schedule that defines a variation of a noise level in a frame with time, between the teaching predicted frameand the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application2. The method of claim 1, wherein using the consistency model to generate the training predicted frame (^^^) for the training time (t) comprises: generating a noisy frame (^^); determining a reduced noise training frame (^^௧) for the training time (t) from the training frame (^^), the noisy frame (^^), and the noise schedule; and processing the reduced noise training frame (^^௧) and the training time (t) using the consistency model to generate the training predicted frame (^^^).

3. The method of claim 2, wherein using the target version of the consistency model to generate the teaching predicted framefor the de-noising step based on the teaching time (s) and on the training frame (^^) comprises: applying the deterministic mapping to both a teacher frame (^^௧^^^^^^) and the reduced noise training frame (^^௧), using the noise schedule, to determine a teaching frame (^^^) for the teaching time (s); and processing the teaching frame (^^^) and the teaching time (s) using the target version of the consistency model to generate the teaching predicted frame (^^^^^^); wherein the teacher frame (^^௧^^^^^^) is the training frame (^^) or is obtained from the training frame (^^).

4. The method of claim 3, comprising obtaining the teacher frame (^^௧^^^^^^) from the training frame (^^) by: processing the training time (t) and the reduced noise training frame (^^௧) using a trained diffusion model neural network to generate the teacher frame (^^௧^^^^^^).

5. The method of claim 3, wherein the teacher frame (^^௧^^^^^^) is the training frame (^^).

6. The method of any of claims 3-5, wherein applying the deterministic mapping to both the teacher frame (^^௧^^^^^^) and the reduced noise training frame (^^௧), using the noise schedule, to determine the teaching frame (^^^) for the teaching time (s) comprises: determining the teaching frame (^^^) as a weighted sum of the teacher frame (^^௧^^^^^^) and of a deterministic estimate of the noisy frame (^^^).

7. The method of claim 6, further comprising:DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationincluding a contribution to a weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum that depends on a variance estimate (^^௩^^) for the teacher frame (^^௧^^^^^^) and that increases a weight of the deterministic estimate of the noisy frame.

8. The method of claim 7, comprising determining the variance estimate for the teacher frame (^^௧^^^^^^) from a difference between the teacher frame (^^௧^^^^^^) and the training frame (^^).

9. The method of any of claims 3-8, wherein the transformed teaching frame (^^^^^^,௧ೞ^^^) for the initial step time is defined by the deterministic mapping, using the noise schedule, from both of the teaching predicted frame (^^^^^^) for the de-noising step and the teaching frame (^^^) to the transformed teaching framefor the initial step time.

10. The method of claim 9, comprising: applying the deterministic mapping, using the noise schedule, to both the teaching predicted frameand the teaching frame (^^^) to generate the transformed teaching framefor the initial step time.

11. The method of any of claims 3-10, further comprising: applying the deterministic mapping, using the noise schedule, to both the training predicted frame (^^^) and the teaching frame (^^^) to generate a transformed training predicted frame for the initial time step of the de-noising step; and training the consistency model using a loss function that depends on a difference between the transformed teaching framefor the initial step time and the transformed training predicted framefor the initial time step.

12. The method of any of claims 3-10, further comprising: applying an inverse of the deterministic mapping, using the noise schedule, to the transformed teaching framefor the initial step time of the de-noising step and to the reduced noise training frame (^^௧) for the training time (t) to generate a transformed teaching predicted frame (^^^^^^,௧ೞ^^^); andDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationtraining the consistency model using a loss function that depends on a difference between the transformed teaching predicted frame (^^^^^^,௧ೞ^^^) and the training predicted frame (^^^).

13. The method of any of claims 1-12, comprising: determining the de-noising step; determining the training time (t) as after the initial step time of the de-noising step and either before the initial step time of next de-noising step or before the final time step; and determining the teaching time (s) as at or after the initial step time for the de-noising step 14. The method of any of claims 1-13, wherein the consistency model is configured to process the time, the noisy frame corresponding to the time, and a step identifier that identifies the de-noising step, to generate the predicted frame for the de-noising step; wherein using the consistency model to generate the training predicted frame (^^^) includes processing the step identifier using the consistency model; and wherein using the target version of the consistency model to generate the teaching predicted frame includes processing the step identifier using the target version of the consistency model.

15. The method of any of claims 1-14, wherein the noise schedule defines a signal-to- noise ratio for a frame that monotonically decreases from the final time to the initial time.

16. The method of any of claims 1-15, comprising: using the same neural network for each of the multiple de-noising steps, to process the time and the noisy frame corresponding to the time to generate the predicted frame for a de- noising step.

17. A computer-implemented method of using a consistency model to generate a frame of data, wherein the consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time; the method comprising:DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationgenerating a noisy frame; determining, from the noisy frame, a current reduced noise frame (^^௧) for an initial de-noising step; and iteratively, for each of a plurality of de-noising steps until a final de-noising step: processing a time for the de-noising step and the current reduced noise frame (^^௧), using the consistency model, to generate a current predicted frame (^^^) for the de-noising step, and applying a deterministic mapping to both the current predicted frame (^^^) for the de-noising step and the current reduced noise frame (^^௧) for the de-noising step, using a noise schedule that defines a variation of a noise level in a frame with time, to determine the current reduced noise frame (^^^, ^^௧) for the next de-noising step; wherein, at the final de-noising step, the current reduced noise frame for the next de- noising step comprises the generated frame of data.

18. The method of claim 17, wherein applying the deterministic mapping to determine the current reduced noise frame (^^^, ^^௧) for the next de-noising step further comprises, for each of some or all of the plurality of de-noising steps: applying the deterministic mapping to both the current predicted frame (^^^) for the de- noising step and the current reduced noise frame (^^௧) for the de-noising step, using the noise schedule that defines the variation of the noise level in a frame with time, to determine an initial current reduced noise frame for the next de-noising step; and refining the initial current reduced noise frame by performing one or more refining steps to determine the current reduced noise frame for the next de-noising step; wherein performing each refining step comprises: adding noise to a reduced noise frame for refinement to obtain a re-noised frame, and de-noising the re-noised frame using the noise schedule; wherein in a first refining step the reduced noise frame for refinement is the initial current reduced noise frame; in a subsequent refining step, if present, the reduced noise frame for refinement is the refined reduced noise frame from a previous refining step; and wherein the refined reduced noise frame after the last refining step provides the current reduced noise frame for the next de-noising step.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application19. The method of claim 18, wherein adding noise to the reduced noise frame for refinement to obtain the re-noised frame comprises adding noise according to the noise schedule and a re-noising time; and wherein de-noising the re-noised frame using the noise schedule comprises applying the deterministic mapping to both the re-noised frame and a frame predicted using the consistency model from the re-noised frame and the re-noising time.

20. The method of claim 18 or 19, comprising, for each of some or all of the plurality of de-noising steps, performing a plurality of the refining steps, wherein each successive refining step adds less noise to the reduced noise frame for refinement than a preceding refining step.

21. The method of any of claims 17 to 20 wherein the consistency model is trained using the method of any one of claims 1-16.

22. A computer-implemented method of using a consistency model to generate a frame of data, wherein the consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time; the method comprising: generating a noisy frame; determining, from the noisy frame, a current reduced noise frame (^^௧) for an initial de-noising time; and iteratively, for each of a plurality of de-noising times until a final de-noising time: processing the de-noising time and the current reduced noise frame (^^௧), using the consistency model, to generate a current predicted frame (^^^) for the de-noising time; and determining the current reduced noise frame (^^^) for the next de-noising time from a weighted sum of the current predicted frame (^^^) for the de-noising step and an estimated noisy frame (^^^), wherein the estimated noisy frame (^^^) comprises a deterministic estimate of the noisy frame obtained from the current predicted frame (^^^) for the de-noising step and the current reduced noise frame (^^௧) for the de-noising time, and including determining a positive addition to a weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum, wherein the positive addition depends on a variance of frames predicted by the consistency model (^^௩^^,௧); andDeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationwherein at the final de-noising time the current reduced noise frame for the next de- noising time comprises the generated frame of data.

23. The method of claim 22, comprising determining the variance of frames predicted by the consistency model dependent on the de-noising time.

24. The method of claim 22 or 23, wherein the deterministic estimate of the noisy frame (^^^) is obtained from the current predicted frame (^^^) for the de-noising time and the current reduced noise frame (^^௧) for the de-noising time using a noise schedule that defines a variation of a noise level in a frame with time; the method comprising: determining the variance of frames predicted by the consistency model based on the noise schedule.

25. The method of claim 22 or 23, comprising determining the variance of frames predicted by the consistency model from one or more example frames of data (^^).

26. The method of any of claims 22-25, comprising: determining the estimated noisy frame (^^^) from the current predicted frame (^^^) for the de-noising time and the current reduced noise frame (^^௧) for the de-noising time; and determining the current reduced noise frame (^^^) for the next de-noising time by determining the weighted sum of the current predicted frame (^^^) for the de-noising time and the estimated noisy frame (^^^).

27. The method of any of claims 22-26, wherein the deterministic estimate of the noisy frame (^^^) is obtained from the current predicted frame (^^^) for the de-noising time and the current reduced noise frame (^^௧) for the de-noising time using a noise schedule that defines a variation of a noise level in a frame with time; the method comprising: obtaining a variance (^^^,௩^^) for the current reduced noise frame (^^^) by modifying the variance of frames predicted by the consistency model (^^௩^^,௧) using the noise schedule; and determining the positive addition to the weight of the deterministic estimate of the noisy frame (^^^) in the weighted sum from the variance for the current reduced noise frame.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application28. The method of any of claims 22-27, further comprising, for each of some or all of the plurality of de-noising times: refining the current reduced noise frame for the next de-noising time by performing one or more refining steps, wherein performing each refining step comprises: adding noise to the current reduced noise frame for the next de-noising time to obtain a re-noised frame; and determining a refined version of the current reduced noise frame for the next de- noising time from the re-noised frame using the consistency model.

29. The method of claim 28, wherein adding noise to the current reduced noise frame for the next de-noising time to obtain the re-noised frame comprises adding noise according to the noise schedule and a re-noising time; and wherein determining the refined version of the current reduced noise frame for the next de- noising time from the re-noised frame using the consistency model comprises: processing the re-noising time and the re-noised frame, using the consistency model, to generate a predicted frame (^^^) for the re-noising time; and determining the refined version of the current reduced noise frame from the predicted frame for the re-noising time and an estimated noisy frame (^^^) for the re-noising time.

30. The method of claim 28 or 29, comprising, for each of some or all of the plurality of de-noising times, performing a plurality of the refining steps, wherein each successive refining step adds less noise to the refined version of the current reduced noise frame than a preceding refining step.

31. The method of any of claims 17-30, further comprising using the method to perform an image modification task by: obtaining a reference image to guide the image modification task; and using the reference image to modify the current predicted frame at each iteration to guide generation of the current reduced noise frame.

32. A computer-implemented method of using a consistency model to generate a frame of data, wherein the consistency model is configured to process a time and a noisy frame corresponding to the time to generate a predicted frame for the time; the method comprising:DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Applicationinitialising a current reduced noise frame; and iteratively, for each of a plurality of de-noising steps until a final de-noising step: processing a time for the de-noising step and the current reduced noise frame, using the consistency model, to generate a current predicted frame for the de-noising step, and determining an initial current reduced noise frame for the next de-noising step from the current predicted frame for the de-noising step and a noise schedule that defines a variation of a noise level in a frame with time; and refining the initial current reduced noise frame by performing one or more refining steps to determine the current reduced noise frame for the next de-noising step, wherein performing each refining step comprises: adding noise to a reduced noise frame for refinement to obtain a re-noised frame, and de-noising the re-noised frame using the noise schedule; wherein in a first refining step the reduced noise frame for refinement is the initial current reduced noise frame; in a subsequent refining step, if present, the reduced noise frame for refinement is the refined reduced noise frame from a previous refining step; and wherein the refined reduced noise frame after the last refining step provides the current reduced noise frame for the next de-noising step.

33. The method of claim 32, wherein performing one or more refining steps comprises performing a plurality of the refining steps, wherein each successive refining step adds less noise to the reduced noise frame for refinement than a preceding refining step.

34. The method of claim 32 or 33, further comprising using the method to perform an image modification task by: obtaining a reference image to guide the image modification task; and using the reference image to modify the current predicted frame at each iteration to guide generation of the current reduced noise frame.

35. The method of any of claims 1-34, wherein processing a time and a noisy frame corresponding to the time using the consistency model to generate a predicted frame includes processing content conditioning data to generate a predicted frame that characterizes a content of the predicted frame.DeepMind Technologies LimitedF&R Ref.: 45288-0479WO1 PCT Application36. The method of any of claims 1-35 wherein the frame of data comprises an image frame that defines values of pixels of a still or moving image.

37. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-35.

38. A system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-36.

Images