Generating data items using diffusion neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a data item using a diffusion neural network or other generative neural network.

Description

[0001] GENERATING DATA ITEMS USING DIFFUSION NEURAL NETWORKS

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims priority to U.S. Provisional Patent Application No. 63 / 533,906, filed on August 21, 2023, the disclosure of which is hereby incorporated by reference in its entirety.

[0004] BACKGROUND

[0005] This specification relates to generating outputs conditioned on conditioning inputs using neural networks.

[0006] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to or more other layers in the network, i.e., one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates an output data item conditioned on a conditioning input.

[0009] Generally, the conditioning input characterizes one or more desired properties for the data item, i.e., characterizes one or more properties that the final data item generated by the system should have.

[0010] More specifically, the system can generate the output data item using a diffusion neural network.

[0011] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0012] Relative to conventional techniques that use diffusion neural networks to generate data items, the described system can modify one or more of (i) the training of the diffusion neural network, (ii) the inputs to the diffusion neural network, or (iii) how the diffusion neural network is used to generate output data items after training, to increase the quality of the output data items generated by the diffusion neural network.

[0013] As one example, the system can configure the diffusion neural network to, at each updating iteration, process a diffusion input for the updating iteration that includes a noisy data item for the updating iteration to generate a denoising output that, unlike other techniques, defines an estimate of a residual error between an analytic estimate of a noise component of the noisy data item and a true noise component of the noisy data item. That is, other techniques generally use diffusion neural networks that generate other types of denoising outputs, e.g., denoising outputs that are estimates of the noise component or denoising outputs that are estimates of the ground truth data item.

[0014] Generating a denoising output that defines an estimate of a residual error between an analytic estimate of a noise component of the noisy data item and a true noise component of the noisy data item can increase the quality of output data items, particularly when a high guidance weight for classifier-free guidance is being used as part of the generation process. For example, while using a high guidance weight can result in generated data items that align more strongly to the provided conditioning input, which is advantageous, using a high guidance weight can also reduce the overall quality of the generated data items. For example, when the data items are images, using a high guidance weight can result in highly-saturated images. By making use of the above denoising output, the system can generate data items that align with the conditioning input while maintaining overall quality, e.g., can reduce the saturation of generated images to levels that appear realistic.

[0015] As another example, the diffusion neural network can be configured to receive a context embedding that can represent either multiple different context data items or a single context data item. The context data items are generally of the same type as the output data item and are used to guide the generation process. By allowing a user to condition the generation process on a variable number of context data items, the system can use the diffusion neural network to generate an output data item that accurately reflects the context provided by the user, e.g., to match certain properties of a single data item or to reflect aggregated properties aggregated across multiple data items.

[0016] As yet another example, the system can employ bound conditioning. When employing bound conditioning, the system conditions the diffusion neural network on a lower bound or an upper bound of an input scalar value that represents a value of the particular property of the generated data item. That is, rather than requiring the diffusion neural network to generate an output data item that has the exact value of the particular property represented by the input scalar value, the system can give the diffusion neural network the flexibility of generating any appropriate data item that has a value of the particular property that is appropriately bounded by the scalar value. As yet another example, the system can employ normalized guidance when generating the output data item using the diffusion neural network. In particular, when the system uses normalized guidance, at each updating iteration, the system can compute a difference between a conditional denoising output and an unconditional denoising output or a negative denoising output for the updating iteration. The system can then, for example, determine the overall denoising output based on the direction, and not the magnitude, of the difference. As described above, using high guidance weights in traditional classifier-free guidance allows for better alignment with the conditioning input and for a more coherent data item but may result in degradation in data item quality. For example, when using classifier- free guidance with high guidance scales to generate images, the generation process can result in images that have significant color saturation and very bright or very dark images.

[0017] Employing normalized guidance reduces this saturation and the resulting undesirable effects while preserving the benefits of high guidance scales, e.g., in terms of text-image alignment, and image quality and coherence.

[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. l is a diagram of an example data generation system.

[0021] FIG. 2 is a flow diagram of an example process for generating a final data item using analytic estimates.

[0022] FIG. 3 is a flow diagram of an example process for conditioning a diffusion neural network on a variable number of context data items.

[0023] FIG. 4 is a flow diagram of an example process for training a diffusion neural network to be effectively conditioned on multiple context data items.

[0024] FIG. 5 is a flow diagram of an example process for conditioning a diffusion neural network on a bound for a property value.

[0025] FIG. 6 is a flow diagram of an example process for training a generative neural network to be conditioned on bounds for property values.

[0026] FIG. 7 is a flow diagram of an example process for determining a final denoising output at a given updating iteration using normalized guidance. Like reference numbers and designations in the various drawings indicate like elements.

[0027] DETAILED DESCRIPTION

[0028] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates an output data item conditioned on a conditioning input.

[0029] Generally, the conditioning input characterizes one or more desired properties for the data item, i.e., characterizes one or more properties that the final data item generated by the system should have.

[0030] The system can be configured to generate any of a variety of output data items conditioned on any of a variety of conditioning inputs.

[0031] For example, the system can be configured to generate audio data, e.g., a waveform of audio or a spectrogram, e.g., a mel-spectrogram or a spectrogram where the frequencies are in a different scale, of the audio.

[0032] In this example, the conditioning input can be text or features of text that the audio should represent, i.e., so that the system serves as a text-to-speech machine learning model that converts text or features of the text to audio data for an utterance of the text being spoken.

[0033] As another example, the conditioning input can identify a desired speaker for the audio, i.e., so that the system generates audio data that represents speech by the desired speaker.

[0034] As another example, the conditioning input can characterize properties of a song or other piece of music, e.g., lyrics, genre, and so on, so that the system generates a piece of music that has the properties characterized by the conditioning input.

[0035] As another example, the conditioning input can specify a classification for the audio data into a class from a set of possible classes, so that the system generates audio data that belongs to the class. For example, the classes can represent types of musical instruments or other audio emitting devices, i.e., so that the system generates audio that is emitted by the corresponding class, or types of animals, i.e., so that the system generates audio that represents noises generated by the corresponding animal, and so on.

[0036] As another particular example, the data item can be an image, such that the system can perform conditional image generation by generating the intensity values of the pixels of the image. In this particular example, the conditioning input can be a sequence of text and the output data item can be an image that describes the text, i.e., the conditioning input can be a caption for the output image.

[0037] As yet another particular example, the conditioning input can be an object detection input that specifies one or more bounding boxes and, optionally, a respective type of object that should be depicted in each bounding box.

[0038] As yet another particular example, the conditioning input can specify an object class from a plurality of object classes to which an object depicted in the output image should belong.

[0039] As another example, the conditioning input can specify one or more images.

[0040] For example, the conditioning input can specify an image at a first resolution and the output data item can include the image at a second, higher resolution.

[0041] For example, the conditioning input can specify an image and the output data item can comprise a de-noised, enhanced, stylized, or otherwise edited version of the image.

[0042] As yet another particular example, the conditioning input can specify an image including a target entity for detection, e.g. a tumor, and the output data item can comprise the image without the target entity, e.g., to facilitate detection of the target entity by comparing the images.

[0043] As yet another particular example, the conditioning input can be a segmentation that assigns each of a plurality of pixels of the output image to a category from a set of categories, e.g., that assigns to each pixel a respective one of the category.

[0044] As yet another example, the conditioning input can be a different type of structured input, e.g., a mesh or a graph that specifies properties of the image to be generated.

[0045] More generally, the conditioning input can include one or more different types of inputs of one or more different modalities, e.g., only text, only one or more images, both text and one or more images, and so on.

[0046] As yet another example, the output data item can be a video.

[0047] As a particular example, the conditioning input can include text and the output data item can be a video described by the text.

[0048] As yet another particular example, the conditioning input can include one or more images and the output data item can be a video that completes the one or images, e.g., video starting from the one or more images.

[0049] More generally, the task of generating the output data item can be any task that outputs continuous data conditioned on a conditioning input. For example, the output can be an output of a different sensor, e.g., a lidar point cloud, a radar point cloud, an electrocardiogram reading, and so on, and the conditioning input can represent the type of data that should be measured by the sensor. Where a discrete output is desired this can be obtained, e.g., by thresholding the outputs generated by the diffusion neural network.

[0050] In some applications, the output data item can be used in a control task to control an action of a mechanical agent acting in a real-world environment to perform a mechanical task. For example, the output data item can be processed by a policy neural network of the agent to select one or more actions to be performed by the agent as part of the task. The agent may then perform the one or more actions. The output data item (e.g., image) can, for example, characterize a state of the real-world environment that is predicted to be obtained by the agent performing the one or more actions.

[0051] In any of the above examples, the output data item generated using the diffusion neural network can either be an output data item in the output space, i.e., so that the values in the output data item are the values of a data item of the appropriate type, e.g., values of image pixels, amplitude values of an audio signal, and so on, or an output data item in a latent space, i.e., so that the values in the output data item are values in a latent representation of an output data item in the output space.

[0052] When the output data item is generated in a latent space, the system can generate a final output data item in output space by processing the output data item in the latent space using a decoder neural network, e.g., one that has been pre-trained in an auto-encoder framework. During training, the system can use an encoder neural network, e.g., one that has been pre-trained jointly with the decoder in the auto-encoder framework, to encode target data items in the output space to generate target outputs for the diffusion neural network in the latent space.

[0053] FIG. 1 is a diagram of an example data generation system 100. The data generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0054] The system 100 obtains a conditioning input 102 and uses the conditioning input 102 to generate an output (final) data item 112 that has the one or more desired properties characterized by the conditioning input 102.

[0055] In particular, to generate the output data item 112, the system 100 uses a diffusion neural network 110. More specifically, the system 100 uses the diffusion neural network 110 to perform a reverse diffusion process across multiple updating iterations to generate the output data item 112.

[0056] The diffusion neural network 110 can be any appropriate diffusion neural network that has been trained, e.g., by the system 100 or another training system, to, at any given updating iteration, process a diffusion input for the updating iteration that includes the current data item (as of the updating iteration) to generate a denoising output for the updating iteration.

[0057] In some implementations, the denoising output is an estimate of the noise component of the current data item, i.e., the noise that needs to be combined with, e.g., added to or subtracted to, a final data item, i.e., to the output data item 112 being generated by the system 100, to generate the current data item.

[0058] In some other implementations, the denoising output is an estimate of the final data item given the current data item, i.e., an estimate of the data item that would result from removing the noise component of the current data item.

[0059] In some other implementations, the denoising output defines a predicted residual between the true noise component of the current data item and an analytic estimate of the noise component, i.e., an estimate that has been computed analytically from the current data item. For example, the estimate of the noise component can be obtained using a predetermined mathematical function with arguments that correspond to elements of the (current) data item.

[0060] This type of denoising output is described in more detail below.

[0061] For example, the system 100 or another training system can have trained the diffusion neural network 110 on a set of training data items using a denoising score-matching objective to generate the denoising output.

[0062] The denoising score-matching objective can measure an error, e.g., a mean-squared error, an LI error, an L2 error or a different type of error, between (i) a denoising output generated by processing an input that includes a noisy data item generated by adding sampled noise to a training data item and (ii) a target denoising output generated from the training data item, from the sampled noise, or both.

[0063] For example, when the denoising output defines a predicted residual between the true noise component of the current data item and an analytic estimate of the noise component, the target denoising output can define the actual residual between the sampled noise the analytic estimate of the component of the noise component. As another example, when the denoising output is an estimate of the true noise component of the current data item, the target denoising output can be the sampled noise.

[0064] As another example, when the denoising output is an estimate of the target data item, the target denoising output can be the target data item.

[0065] The diffusion neural network 110 can have any appropriate architecture that allows the neural network to map a diffusion input that includes an input data item that has the same dimensionality as the output data item 112 to a denoising output that also has the same dimensionality as the output data item 112.

[0066] For example, when the output data item is an audio signal or an image, the diffusion neural network 110 can be a convolutional neural network, e.g., a U-Net or other architecture that maps one input of a given dimensionality to an output of the same dimensionality.

[0067] As another example, the diffusion neural network 110 can be a Transformer neural network that processes the diffusion input through a set of self-attention layers to generate the denoising output.

[0068] The neural network 110 can be conditioned on the conditioning input 102 in any of a variety of ways.

[0069] As one example, the system 100 can use an encoder neural network to generate one or more embeddings that represent the conditioning input 102 and the diffusion neural network 110 can include one or more cross-attention layers that each cross-attend into the one or more embeddings.

[0070] An embedding, as used in this specification, is an ordered collection of numerical values, e.g., a vector of floating point values or other types of values.

[0071] For example, when the conditioning input is text, the system can use a text encoder neural network, e.g., a Transformer neural network, to generate a fixed or variable number of text embeddings that represent the conditioning input.

[0072] When the conditioning input is an image, the system can use an image encoder neural network, e.g., a convolutional neural network or a vision Transformer neural network, to generate a set of embeddings that represent the image.

[0073] When the conditioning input is audio, the system can use, e.g., an audio encoder neural network, e.g., an audio encoder neural network that has been trained jointly with a decoder neural network as part of a neural audio codec, to generate one or more embeddings that encode the audio. When the conditioning input is a scalar value, the system can use, e.g., an embedding matrix to map the scalar value or a one-hot representation of the scalar value to an embedding.

[0074] In some cases, the conditioning input 102 includes multiple different types of inputs, e.g., two or more of text, images, bound values, or context embeddings.

[0075] In some of these cases, the system 100 can generate one or more initial embeddings for each of the different types of inputs, i.e., using an appropriate encoder neural network as described above, and then process the initial embeddings for all of the different types of inputs using a Transformer encoder neural network to update each of the initial embeddings to generate a set of final embeddings. The one or more cross-attention layers within the diffusion neural network 110 can then cross-attend into the set of final embeddings.

[0076] In others of these cases, different cross-attention layers within the diffusion neural network 110 can cross-attend into embeddings of different types of conditioning inputs.

[0077] In yet others of these cases, the system 100 can concatenate the initial embeddings of the different types of inputs along the sequence dimension and then the one or more crossattention layers can cross-attend into the concatenated set of final embeddings.

[0078] As another example, the diffusion neural network 110 can include one or more other types of neural network layers that are conditioned on the one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gated activation functions, and so on.

[0079] The diffusion input at any given updating iteration can also include data defining a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linear function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the diffusion neural network 110 as described above for the conditioning input.

[0080] At each updating iteration, the system 100 uses the denoising output generated by the diffusion neural network 110 to update the current data item as of the updating iteration.

[0081] For example, the system can determine an initial estimate of the final data item using the denoising output and then apply an appropriate diffusion sampler to the initial estimate to update the current data item. As another example, the system can use classifier-free guidance or negative guidance to adjust the denoising output, determine an initial estimate of the final data item using the adjusted denoising output and then apply an appropriate diffusion sampler to the initial estimate to update the current data item. Classifier-free guidance is described in, for example, Ho and Salimans, arXiv:2207.12598.

[0082] The system can use any appropriate diffusion sampler to update data item, e.g., the DDPM (Denoising Diffusion Probabilistic Model) sampler, the DDIM (Denoising Diffusion Implicit Model) sampler or another appropriate sampler, to the estimate to generate the updated current data item. DDPMs are, for example, discussed in Ho et al. arXiv:2006: 11239.

[0083] When the denoising output is a prediction of the data item, the system can directly use the denoising output (or the adjusted denoising output) as the estimate.

[0084] When the denoising output is a prediction of the noise component, the system can determine the initial estimate from the current data item, the denoising output, and the noise level for the current updating iteration.

[0085] Optionally, after the last iteration, the system can refrain from using the diffusion sampler and can instead use the initial estimate as the updated current data item.

[0086] After the last updating iteration, the system 100 outputs the current data item as the final output data item 112.

[0087] For example, the system 100 can provide the data item 112 for presentation or play back to a user on a user computer or store the data item 112 for later use.

[0088] In some implementations, the diffusion neural network 110 is one of a sequence of diffusion neural networks, e.g., a hierarchy or a cascade of diffusion neural networks, that the system 100 uses to generate the final data item. For example, each diffusion neural network in the sequence can receive as input the output data item generated by the preceding diffusion neural network in the sequence and generate an output data item that has an increased resolution, e.g., an increased spatial resolution, an increased temporal resolution, or both, relative to the preceding diffusional neural network in the sequence. In these implementations, all of the neural networks in the sequence can receive the conditioning input 102 or only a proper subset of the diffusion neural networks in the sequence can receive the conditioning input 102, e.g., only the diffusion neural networks at one or more earliest positions in the sequence.

[0089] Generally, the system 100 can modify one or more of the training of the diffusion neural network 110, the inputs to the diffusion neural network 110, or how the diffusion neural network 110 is used to generate output data items after training to increase the quality of the output data items generated by the diffusion neural network 110.

[0090] As one example, and as indicated above, the system 100 can configure the diffusion neural network 110 to, at each updating iteration, process a diffusion input for the updating iteration that includes a current, noisy data item for the updating iteration to generate a denoising output that defines an estimate of a residual error between an analytic estimate of a noise component of the noisy data item and a true noise component of the noisy data item.

[0091] This is described in more detail below with reference to FIG. 2.

[0092] As another example, the diffusion neural network 110 can be configured to receive, as the conditioning input 102 or as part of the conditioning input 102, a context embedding that can represent either multiple different context data items or a single context data item. The context data items are generally of the same type as the output data item and are used to guide the generation process.

[0093] This is described in more detail below with reference to FIGS. 3 and 4.

[0094] As yet another example, the system 100 can employ bound conditioning, in which, in addition to the conditioning input 102, the system 100 conditions the diffusion neural network 110 on a lower bound or an upper bound of an input scalar value that represents a value of the particular property of the generated data item. That is, rather than requiring the diffusion neural network 110 to generate an output data item that has the exact value of the particular property represented by the input scalar value, the system 100 can give the diffusion neural network 110 the flexibility of generating any appropriate data item that has a value of the particular property that is bounded by the scalar value.

[0095] Bound conditioning is described below with reference to FIGS. 5 and 6.

[0096] As yet another example, the system 100 can employ normalized guidance when generating the output data item using the diffusion neural network 100. In particular, when the system uses normalized guidance, at each updating iteration, the system can compute a difference between a conditional denoising output and an unconditional denoising output or a negative denoising output for the updating iteration and then e.g., determine the overall denoising output based on the direction, and not the magnitude, of the difference. An unconditional denoising output can be a denoising output that is not conditioned on any conditioning input, whilst a negative denoising output can be a denoising output that is conditioned on a negative conditioning input that indicates one or more properties that the denoised data item should not have.

[0097] Normalized guidance is described below with reference to FIG. 7. FIG. 2 is a flow diagram of an example process 200 for generating a final data item using a diffusion neural network. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0098] The system obtains a conditioning input (step 202).

[0099] The system initializes a data item (step 204).

[0100] Generally, the initialized data item is the same dimensionality as the final data item but has noisy values. That is, the initialized data item has the same number of elements as the final data item.

[0101] For example, the system can initialize the data item, i.e., can generate the first instance of the data item, by sampling a value for each element in the data item from a corresponding noise distribution, e.g., a Gaussian distribution or a different noise distribution. That is, the output data item includes multiple elements and the initial data item includes the same number of elements, with the value for each element being sampled from a corresponding noise distribution.

[0102] The system then generates the final output data item by updating the data item at each of a plurality of updating iterations. In other words, the final output data item is the data item after the last iteration of the plurality of updating iterations.

[0103] In some cases, the number of iterations is fixed. In other cases, the system or another system can adjust the number of iterations based on a latency requirement for the generation of the final output data item, i.e., can select the number of iterations so that the final output data item will be generated to satisfy the latency requirement. In yet other cases, the system or another system can adjust the number of iterations based on a computational resource consumption requirement for the generation of the final output data item, i.e., can select the number of iterations so that the final output data item will be generated to satisfy the requirement. For example, the requirement can be a maximum number of floating operations (FLOPS) to be performed as part of generating the final output data item.

[0104] As described above, the system performs a reverse diffusion process across the updating iterations by updating the current data item at each iteration. Each updating iteration corresponds to a different time point in a time interval, e.g., the interval between zero and one or another appropriate time interval. The time point is also referred to a time step t or a time index t. For example, the updating iterations can be evenly spaced across the time interval, i.e., at regular intervals with in the interval, or can be arranged within the time interval according to a different scheme.

[0105] In particular, when analytic estimates are used, at each updating iteration, the system performs steps 206-212 to update the data item.

[0106] The system determines an analytic estimate of the noise component of the current data item (step 206).

[0107] For the first updating iteration, the current data item is the noisy initial data item. For each subsequent updating iteration, the current data item is the data item after being updated at the preceding updating iteration.

[0108] As described above, the noise component of the current data item is the noise that has been added to a final data item in order to generate the current data item.

[0109] For example, at an iteration with time index / , i.e., the time point (“time step”) corresponding to the updating iteration is / , the current data item xtcan be expressed as xt= atxQ+ <jt£, where £ is the noise component, xQis the final data item, and atand <Jtare weights for the iteration with time index t. For example, atcan be equal to and atcan be a value between zero and one, inclusive. As another example, atcan be equal to 1 and <Jtcan be a value greater than zero. Generally, one or both of at, <jtcan be determined according to a fixed schedule across time indices t, e.g., a linear schedule, a quadratic schedule, a cosine schedule, and so on.

[0110] To generate the analytic estimate, the system can apply a scaling factor to the current data item.

[0111] Generally, the scaling factor is a time-dependent scaling factor that depends on the updating iteration, i.e., is different for different updating iterations.

[0112] For example, the analytic estimate can be equal to ktxt, where ktis the scaling factor.

[0113] Generally, the scaling factor depends on an estimated standard deviation stof the current data item at updating iteration t and is therefore different for different updating iterations.

[0114] For example, ktcan be equal an estimate of the standard deviation of the data distribution, i.e., of the distribution of final data items that can be generated by the system.

[0115] For example, the system can approximate odataasthe standard deviation of the training data items that were used in training the diffusion neural network. For example, <jdatacan be equal to the standard deviation across all elements of a set of training data items that were used in training the diffusion neural network, e.g., all of the training data items that were used in training, or a subset of the training data items, e.g., a randomly selected subset or another proper subset of the training data items.

[0116] As another example, the system can receive < dataasinput from the user. As a particular example, the system can allow the user to submit values for <jdatato allow the user to modify certain properties of the generated data items. For example, when the data items are images, by modifying the value for <jdata, the user can modify the saturation of the generated images. As a particular example, by setting the value for <jdatato a lower value, e.g., a value lower than the standard deviation of the training data items that were used in training the diffusion neural network, the user can cause the saturation of the images to be decreased.

[0117] The system processes a first diffusion input for the updating iteration that includes the current data item and a representation of the conditioning input using the diffusion neural network to generate a first denoising output for the updating iteration (step 208).

[0118] For example, prior to the first updating iteration, and as described above, the system can process the conditioning input using one or more embedding neural networks to generate one or more embeddings of the conditioning input.

[0119] The first diffusion input for any given updating iteration can then include the one or more embeddings of the conditioning input.

[0120] The first diffusion input can also include one or more of: data identifying the updating iteration (e.g., data identifying the corresponding time point for the updating iteration), data characterizing one or more context data items for use as context during the data item generation, scalar values for one or more properties of the generated data item, and so on. These types of inputs will be described in more detail below.

[0121] As described above, the first denoising output defines a prediction, given the first denoising input, of the residual error, i.e., the difference, between the noise component of the current data item and the analytic estimate of the noise component. That is, the first denoising output defines a prediction of E — ktxt.

[0122] For example, the first denoising output can be a prediction of a standardized residual rt, where rtis equal t

[0123] Optionally, i.e., when using classifier-free guidance, the system can also process one or more additional diffusion inputs for the updating iteration to generate a respective additional denoising output for the updating iteration for each additional diffusion input (step 210).

[0124] Each additional diffusion input also includes the current data item as of the updating iteration but includes a different conditioning input.

[0125] For example, one of the additional diffusion inputs can be an unconditional diffusion input that include a representation of a conditioning input that has been designated to indicate that the data item should be generated unconditionally (i.e., without conditioning on another conditioning input). For example, the representation of a conditioning input that has been designated to indicate that the data item should be generated unconditionally can be a predetermined, fixed embedding, e.g., an embedding that includes all zeros.

[0126] As another example, one of the additional diffusion inputs can be a negative diffusion input that includes a representation of a negative conditioning input that indicates properties that the generated data item should not have.

[0127] That is, the system can also receive a negative conditioning input that indicates properties that the generated data item should not have and can include a representation of the negative conditioning input, e.g., one or more embeddings generated from the negative conditioning input, in the negative diffusion input.

[0128] Each additional denoising output defines a prediction, given the corresponding additional denoising input, of the residual error, i.e., the difference, between the noise component of the current data item and the analytic estimate of the noise component.

[0129] The system determines a final denoising output for the updating iteration from the first denoising output and, when generated, the additional denoising output(s) (step 212).

[0130] When no additional denoising outputs are generated, the system can set the final denoising output equal to the first denoising output.

[0131] When one or more additional denoising outputs are generated, the system can combine the first denoising output and the final denoising outputs in accordance with a guidance weight w for the updating iteration. The guidance weight can be used to adjust the relative contributions of the first denoising output and the additional denoising output(s) to the final denoising output,

[0132] For example, the system can set the final denoising output equal to (1+w) * the first denoising output - w*the additional denoising output or, when there are multiple additional denoising outputs, the sum of the additional denoising outputs (where * denotes the multiplication operator). That is, the final denoising output can be determined from a difference between the first denoising output scaled by (1+w) and the sum of the one or more additional denoising outputs scaled by w.

[0133] As another example, the system can combine the denoising outputs using normalized guidance. Normalized guidance will be described in more detail below with reference to FIG. 7.

[0134] The system then updates the current data item using the final denoising output (step 214).

[0135] For example, the system can compute a final estimate of the noise component from the final denoising output and the analytic estimate and then use the final estimate of the noise component to update the current data item.

[0136] For example, the system can compute the final estimate E of the noise component as: E = ytft— ktxt, where rtis the final denoising output.

[0137] The system can then compute an initial estimate of the final data item using the final estimate of the noise component as follows: xt-o-te at

[0138] For the last updating iteration, the system can use the initial estimate as the updated data item.

[0139] For each updating iteration other than the last updating iteration, the system can apply an appropriate diffusion sampler to the initial estimate to generate the updated data item. Examples of diffusion samplers are given above with reference to FIG. 1.

[0140] Generating a denoising output that defines an estimate of a residual error between an analytic estimate of a noise component of the noisy data item and a true noise component of the noisy data item can increase the quality of output data items, particularly when a high guidance weight for classifier-free guidance is being used as part of the generation process. For example, while using a high guidance weight can result in generated data items that align more strongly to the provided conditioning input, which is advantageous, using a high guidance weight can also reduce the overall quality of the generated images. For example, when the data items are images, using a high guidance weight can result in highly-saturated images. By making use of the above denoising output, the system can generate data items that align with the conditioning input while maintaining overall quality, e.g., can reduce the saturation of generated images to levels that appear realistic.

[0141] When analytic estimates are not used, i.e., when the denoising output is an estimate of the noise component of the current data item or an initial estimate of the final data item, the system can compute the initial estimate of the final data item without incorporating the analytic estimates.

[0142] That is, when the denoising output is an estimate of the noise component, i.e., the system can compute the initial estimate where E is the final denoising output.

[0143] When the denoising output is an estimate of the final data item, the system can directly use the final denoising output as the initial estimate.

[0144] As described above, prior to using the diffusion neural network to generate data items, the system trains the diffusion neural network, e.g., on the denoising-score matching objective.

[0145] In particular, to train the diffusion neural network using the score matching objective, the system can sample (i) a data item from a set of training data items, (ii) one or more corresponding conditioning input(s) for the data item, (iii) a time step t for the training, e.g., uniformly at random from the time interval or according to a different distribution over the time interval, and (iv), and noise £ from the noise distribution.

[0146] The system can then generate a noisy data item xtby combining the target data item xQwith the sampled noise in accordance with the sampled time step, e.g., by setting the noisy data item xtto be xt= atxQ+ ats.

[0147] The system can then process an input that includes the noisy data item, data specifying the time step, and the conditioning input(s) using the diffusion neural network to generate a denoising output.

[0148] The system can then compute an error between the denoising output and a target denoising output and use the error to train the diffusion neural network, e.g., by determining gradients of the error and then using the gradients to update the parameters of the diffusion neural network by applying an optimizer to (at least) the gradients.

[0149] As a particular example, the denoising score-matching objective can measure an error, e.g., a mean-squared error, an LI error, an L2 error or a different type of error, between (i) the denoising output and (ii) the target denoising output.

[0150] For example, when the denoising output defines a predicted residual between the true noise component of the current data item and an analytic estimate of the noise component, the target denoising output can define the actual residual between the sampled noise the analytic estimate of the component of the noise component.

[0151] As another example, when the denoising output is an estimate of the true noise component of the current data item, the target denoising output can be the sampled noise. As another example, when the denoising output is an estimate of the target data item, the target denoising output can be the target data item.

[0152] FIG. 3 is a flow diagram of an example process 300 for conditioning a diffusion neural network on a variable number of context data items. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0153] The system obtains one or more context data items (step 302).

[0154] For example, a user can provide the context data item(s) to the system or can identify one or more data items that are maintained by the system for use as the context data item(s).

[0155] The context data items are generally the same type of data item as the final data item to be generated by the system and provide context for the data item to be generated by the system. For example, when the generated data item is an image, the context data items can each be images. When the generated data item is audio, the context data items can each be audio samples.

[0156] The system processes each context data item using an embedding neural network to generate a respective embedding of each of the context data items (step 304).

[0157] As described above, an embedding is an ordered collection of numerical values. For example, an embedding can be a vector of numerical values, e.g., a vector of floating point values or other numerical values. As another example, the embedding can be a collection, e.g., a sequence or a two-dimensional grid, of vectors of numerical values.

[0158] The embedding neural network can be any appropriate type of neural network that can map an input of the corresponding type to an embedding. Examples of architectures of embedding neural networks are described above with reference to FIG. 1.

[0159] Generally, the embedding neural network can have been trained jointly with the diffusion neural network or can have been pre-trained prior to the training of the diffusion neural network, e.g., on a self-supervised representation learning objective. Examples of representation learning objectives include contrastive learning and masked generative modeling objectives.

[0160] When there are multiple context data items, the system combines, e.g., averages, the embeddings of the context data items to generate a combined embedding. In some implementations, the system can receive a user input that assigns a respective weight to each of the multiple context data items. For example, the respective weights can represent the degree to which each context data item should influence the content of the generated data item. In these implementations, the system can compute a weighted sum of the embeddings of the context data items in accordance with the respective weights for the corresponding context data items.

[0161] Optionally, the system can then multiply the generated embedding by a non-negative scalar value to control the strength of the effect of the context embedding on the generation of the output data item. The scalar value can be, e.g., predetermined or received as input from the user.

[0162] The system processes the generated embedding and, optionally, an additional conditioning input using a diffusion neural network to generate an output data item (step 306), e.g., by performing a reverse diffusion process across multiple updating iterations as described above with reference to FIGS. 1 and 2.

[0163] When there are multiple context data items, the system processes the combined embedding. When there is a single context data item, the system processes the embedding of the single context data item.

[0164] That is, the system uses the embedding to provide additional context to the diffusion neural network to guide the diffusion neural network during generation of the output data item.

[0165] Thus, the system allows the diffusion neural network to be conditioned on a variable number of user-specified context data items, i.e., because the embedding that is processed as input by the diffusion neural network can either represent a single data item or can be an aggregated embedding that represents multiple different data items.

[0166] In particular, the system can use the diffusion neural network to generate the output data item across multiple updating iterations, as described above with reference to FIGS. 1 and 2. As described above, at any given updating iteration, the diffusion neural network can be conditioned on the generated embedding in any of a variety of ways, e.g., by virtue of including one or more cross-attention layers that cross-attend into the generated embedding.

[0167] In some implementations, the system modifies the training of the diffusion neural network to improve the performance of the diffusion neural network in conditioning on variable numbers of context data items. One example of such a training process is described in more detail below with reference to FIG. 4. FIG. 4 is a flow diagram of an example process 400 for training a diffusion neural network so that the diffusion neural network can be conditioned on variable numbers of context data items. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0168] The system maintains a respective embedding for each of a plurality of context data items and data clustering the embeddings into a plurality of clusters (step 402). These context data items will also be referred to as “training” context data items and can be the same context data items referred to above with reference to FIG. 3 or a different set of context data items that are used only for training the neural network.

[0169] As one example, the embeddings can be clustered using a k-means clustering method.

[0170] As another example, the embeddings can be clustered based on similarities between the corresponding data items. As a particular example, in the case of images, different clusters can represent images of different objects. That is, the images within each cluster can include images of a corresponding object for the cluster.

[0171] In the case of audio, different clusters can represent different genres of music or music by different artists.

[0172] More generally, each cluster can correspond to a different value of a property and can include embeddings of data items having the corresponding property value.

[0173] The embeddings have been generated by processing the context data items using the embedding neural network described above, e.g., after the embedding neural network has been pre-trained.

[0174] The system obtains an input specifying a target data item from the plurality of context data items and, optionally, a conditioning input characterizing the target data item (step 404). For example, the system can randomly sample the target data item from the plurality of context data items or can receive an input specifying which target data item to use to train the neural network.

[0175] The system selects a context embedding for training the diffusion neural network (step 406).

[0176] In particular, the system randomly selects either (i) the embedding of the target context data item or (ii) the centroid of the cluster to which the embedding of the target context data item belongs. That is, with probability p, the system selects the embedding of the target context data item and, with probability 1- / ?, the system selects the centroid of the cluster. The value of the probability p can be pre-configured or received as input by the system from a user.

[0177] The system trains the diffusion neural network using the selected context embedding and the target data item (step 408).

[0178] That is, the system uses the denoising score-matching objective to train the diffusion neural network on the target data item using the selected context embedding as context.

[0179] In particular, to train the diffusion neural network using the score matching objective, the system can sample a time step t for the training, e.g., uniformly at random from the time interval or according to a different distribution over the time interval, and sample noise £ from the noise distribution.

[0180] The system can then generate a noisy data item xtby combining the target data item xQwith the sampled noise in accordance with the sampled time step, e.g., by setting the noisy data item xtto be xt= atxQ+ cts.

[0181] The system can then process an input that includes the noisy data item, the selected context embedding (and, optionally, an additional conditioning input) to generate a denoising output.

[0182] The system can then compute an error between the denoising output and a target denoising output and use the error to train the diffusion neural network, e.g., by determining gradients of the error and then using the gradients to update the parameters of the diffusion neural network by applying an optimizer to (at least) the gradients.

[0183] As a particular example, the denoising score-matching objective can measure an error, e.g., a mean-squared error, an LI error, an L2 error or a different type of error, between (i) the denoising output and (ii) the target denoising output.

[0184] For example, when the denoising output defines a predicted residual between the true noise component of the current data item and an analytic estimate of the noise component, the target denoising output can define the actual residual between the sampled noise the analytic estimate of the component of the noise component.

[0185] As another example, when the denoising output is an estimate of the true noise component of the current data item, the target denoising output can be the sampled noise.

[0186] As another example, when the denoising output is an estimate of the target data item, the target denoising output can be the target data item.

[0187] By repeatedly performing the process 400 on different context data items, the system can train the diffusion neural network to effectively generate data items conditioned on variable numbers of context data items. That is, because whether the selected context embedding represents (i) the embedding of the target context data item or (ii) the centroid of the cluster to which the target context data item belongs is selected randomly, the diffusion neural network is provided both types of inputs during training and learns to effectively incorporate context from both single context items and collections of multiple context items.

[0188] In other words, as a result of this training and taking images as an example, the diffusion neural network learns to accurately denoise an image conditioned on either (i) an embedding of the image or (ii) an embedding representative of the cluster to which the image belongs. Thus, after training, the diffusion neural network can be used to generate a high- quality new image conditioned on an embedding of a single context image or on an embedding aggregated from embeddings of multiple different context images.

[0189] In some cases, the system can perform the process 400 after pre-training the diffusion neural network on an objective that does not incorporate context data items or only incorporates embeddings that represent a single context data item.

[0190] FIG. 5 is a flow diagram of an example process 500 for conditioning a generative neural network on a bound for a property value. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0191] For example, the generative neural network can be the diffusion neural network described above or can be a different type of generative neural network, e.g., an autoregressive generative neural network, e.g., a Transformer-based generative neural network, or a neural network that first generates a latent representation of a data item and then decodes the latent representation to generate the data item. For example, the latent representation can represent the data item as a set of discrete latent vectors or as a set of continuous latent vectors.

[0192] For each of one or more properties of the output data item, the system receives an input scalar value from a range of scalar values that each represent different values of the property (step 502).

[0193] The one or more properties of the output data item can be any appropriate properties of data items of the type that the diffusion neural network is configured to generate, e.g., properties that users may desire to modify. Generally, the properties in the set of one or more properties are predetermined while the scalar values can be different for different output data items. When a property does not have a corresponding scalar value for a given output data item, i.e., when the system does not receive an input scalar value for a given output data item, the system can set the scalar value to a predetermined value.

[0194] Examples of such properties can include realism, brightness, quality (as determined by an appropriate quality metric for the type of data item), coherence, and so on of the generated data item.

[0195] The input scalar values can be received, e.g., as inputs from a user or from another system. In some cases, a user can configure desired values of the properties for each generated data item while, in other cases, the system can receive a high-level context input from the user, e.g., a natural language input or other input, and map the context input to respective values for the data item, e.g., through applying a set of heuristics or using a learned model.

[0196] For each of the one or more properties of the output data item, the system determines whether the input scalar value is an upper bound or a lower bound on the desired value of the property for the output data item (step 504).

[0197] For example, when the received input values are part of a “positive” prompt, i.e., part of a positive conditioning input, the system can determine that the input scalar values are each a lower bound on the desired value of the corresponding property. That is, because a given value is part of a “positive” prompt that represents desired properties of the output data item, the system can determine that the prompt will be satisfied if the output data item has a value for the corresponding property that is at least as high as the received input value for the property.

[0198] When the received input values are part of a “negative” prompt, i.e., part of a negative conditioning input being used for classifier-free guidance, the system can determine that the input scalar values are each an upper bound on the desired value of the corresponding property. That is, because a given value is part of a “negative” prompt that represents undesirable properties of the output data item, the system can determine that the prompt will be satisfied if the output data item has a value for the corresponding property that is at least as low as the received input value for the property.

[0199] As another example, the system can receive, with the input scalar value, an indication of whether the input scalar value is an upper bound or a lower bound for the corresponding property value. For each of the one or more properties of the output data item, the system determines an indicator value that indicates whether the input scalar value is an upper bound or a lower bound on the desired value of the property for the output data item (step 506). That is, for each property, the indicator value can be equal to a first value, e.g., one, when the input scalar value is a lower bound and to a second value, e.g., zero or negative one, when the input scalar value is an upper bound.

[0200] The system processes an input that includes, for each of the one or more properties, the input scalar value for the property and the respective indicator value for the property, i.e., that indicates whether input scalar value is an upper or lower bound, using the generative neural network to generate the output data item (step 508).

[0201] For example, when the neural network is a diffusion neural network, the system can perform the process 200 with the values for the properties being some of or all of the conditioning input to generate the output data item.

[0202] By expressing the input values as bounds for property values rather than as exact targets, the diffusion neural network is afforded more flexibility to generate high quality data items that will nonetheless satisfy the request. That is, generating a data item that has a property with a value that exactly matches a target value is a difficult task, while generating a data item that has a value for the property that is appropriately bounded by the target value is more readily achievable while respecting the constraints imposed by the remainder of the conditioning input.

[0203] In some implementations, the system modifies the training of the generative neural network to improve the performance of the generative neural network in conditioning on bounds. One example of such a training process is described in more detail below with reference to FIG. 6.

[0204] FIG. 6 is a flow diagram of an example process 600 for training a generative neural network to be conditioned on bounds for property values. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600.

[0205] The system obtains a target data item and, for each of one or more properties of the target data item, a ground truth value for the property (step 602). The ground truth value for a given property is a scalar value that represents the actual value of the given property for the target data item. For each property, the system samples a scalar value for the property, e.g., uniformly from the range of possible values for the property from another appropriate distribution over the range of possible values for the property (step 604).

[0206] The system processes a model input that includes, for each property, (i) the sampled value for the property and (ii) an indicator value that indicates whether the sampled value for the property is greater than or less than the ground truth value for the property using the generative neural network to generate a model output for the model input (step 606).

[0207] For example, the indicator value can be equal to one when the sampled value is greater than the target value and equal to zero or negative one when the sampled value is less than the target value.

[0208] As another example, the indicator value can be equal to zero or negative one when the sampled value is greater than the target value and equal to one when the sampled value is less than the target value.

[0209] When the sampled value is equal to the target value, the system can set the indicator value equal to one, zero, or negative one.

[0210] When the generative neural network is a diffusion neural network, e.g., as described above, the model input can also include a noisy version of the target data item and the model output can be any of the denoising outputs described above.

[0211] The system trains the generative neural network on a loss function that measures the quality of the model output relative to a target model output for the target data item (step 608).

[0212] For example, when the generative neural network directly generates data items, e.g., auto-regressively or in one forward pass, the target model output can be the target data item.

[0213] As another example, when the generative neural network generates a latent representation of an output data item, e.g., generates tokens representing the output data item, the target model output can be a latent representation of the target data item.

[0214] For example, when the generative neural network is a diffusion neural network, e.g., as described above, the loss function can be the denoising-score matching objective between the denoising output and the target denoising output as described above.

[0215] The system can repeatedly perform the process 600 on different target data items in order to train the generative neural network.

[0216] Thus, by repeatedly performing the process 600, although the system has available ground truth values for the training data items, the system nonetheless trains the generative neural network to generate data items through bound conditioning rather than training the generative neural network to attempt to exactly replicate the ground truth values.

[0217] FIG. 7 is a flow diagram of an example process 700 for determining a final denoising output at a given updating iteration using normalized guidance. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data generation system, e.g., the data generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 700.

[0218] The system generates a first denoising output for the updating iteration and an additional denoising output for the updating iteration as described above (step 702).

[0219] For example, the first denoising output can be a conditional denoising output and the additional denoising output can be an unconditional denoising output or a negative denoising output.

[0220] As described above, each denoising output can, e.g., be a respective prediction of a noise component of a noisy (“current”) data item for the given updating iteration, a respective prediction of the difference between the noise component and an analytic estimate of the noise component, or a respective prediction of the final data item.

[0221] The system computes, for each element of the final denoising output, a difference between the value of the corresponding element in the first denoising output and the value of the corresponding element in the additional denoising output (step 704).

[0222] The system determines, for each element of the final denoising output, the sign, i.e., positive, negative, or none (when the difference is zero), of the difference computed for the element (step 706).

[0223] The system determines, for each element of the final denoising output, the value of the element in the final denoising output based on the sign of the difference computed for the element and the guidance weight for the given updating iteration (step 708).

[0224] For example, the system can set the value equal to zero when the difference is zero and has no sign, equal to the guidance weight when the sign is positive, and equal to the negative of the guidance weight when the sign is negative.

[0225] Thus, the system determines the values of the elements in the final denoising output using only the direction, and not the magnitude, of the difference between the first denoising output and the additional denoising output.

[0226] The above description describes making use of a guidance weight w when generating a data item across multiple update iterations. Generally, the guidance weight w for each updating iteration is a scalar value. As a particular example, the guidance weight can be a positive scalar value.

[0227] In some implementations, the system uses the same guidance weight (also referred to as a guidance scale) for each updating iteration.

[0228] In some other implementations, the system sets the guidance weight for each updating iteration as a function of the time point t corresponding to the updating iteration.

[0229] The function can be any appropriate function that maps from t to a guidance weight.

[0230] As a particular example, the function can set different guidance weights for different time bins. That is, the function maps each of multiple intervals of t to a corresponding guidance weight and, to determine the guidance weight for any given t, the system identifies the interval to which t belongs and sets the guidance weight for t to be equal to the guidance weight to which the function maps the interval to which t belongs. For example, for t in [0, 0.25] the function can set a weight of 10, t in [0.25, 0.5] a weight of 25, and so on.

[0231] Generally, classifier-free guidance may have different effects at different diffusion time points / , e.g., when generating images, at high t it affects large coarse structures in the image, and at low t it can accentuate finer details and edges. Thus, the optimal guidance weights (in terms of output data item quality) are not uniform across time steps. Setting the guidance weight as a function of t allows the system to control for this and increase the quality of output data items.

[0232] When the system uses classifier-free guidance, e.g., either traditional classifier-free guidance or normalized classifier-free guidance, during training, the system can periodically perform unconditional

[0233] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0234] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0235] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0236] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0237] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0238] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0239] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0240] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0241] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0242] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0243] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.

[0244] Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow framework or a Jax framework.

[0245] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0246] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0247] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0248] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0249] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0250] Aspects of the present disclosure may be as set out in the following clauses: Clause 1. A method performed by one or more computers, the method comprising: initializing a data item; obtaining a conditioning input characterizing one or more desired properties for the data item; and generating a final data item having the one or more desired properties, the generating comprising, at each of a plurality of updating iterations: identifying a current data item as of the updating iteration; determining, from the current data item, an analytic estimate of a noise component of the current data item; processing a first diffusion input for the updating iteration that comprises the current data item and a representation of the conditioning input using a diffusion neural network to generate a first denoising output for the updating iteration, wherein the first denoising output defines a prediction of a residual error between the noise component of the current data item and the analytic estimate of the noise component; generating, from at least the first denoising output, a final denoising output for the updating iteration; and updating the current data item using the final denoising output and the analytic estimate of the noise component.

[0251] Clause 2. The method of clause 1, wherein determining, from the current data item, an analytic estimate of a noise component of the current data item comprises: determining a scaling factor for the updating iteration, wherein the scaling factor is a time-dependent scaling factor that depends on the updating iteration; and applying the scaling factor to the current data item to generate the analytic estimate.

[0252] Clause 3. The method of clause 2, wherein determining the scaling factor comprises: determining the scaling factor based on an estimate of a standard deviation of a distribution of final data items that can be generated by the diffusion neural network.

[0253] Clause 4. The method of clause 3, further comprising: receiving the estimate of the standard deviation as input from a user.

[0254] Clause 5. The method of clause 3, further comprising: setting the estimate of the standard deviation to be a standard deviation of training items that were used in training the diffusion neural network.

[0255] Clause 6. The method of any preceding clause, wherein the first denoising output is a prediction of a standardized residual error between the noise component of the current data item and the analytic estimate of the noise component. Clause 7. The method of any preceding clause, wherein updating the current data item using the final denoising output comprises: computing a final estimate of the noise component from the final denoising output and the analytic estimate; and using the final estimate of the noise component to update the current data item.

[0256] Clause 8. The method of any preceding clause, wherein generating, from at least the first denoising output, a final denoising output for the updating iteration comprises: processing one or more additional diffusion inputs for the updating iteration that each comprise the current data item and a representation of a different conditioning input using the diffusion neural network to generate a respective additional denoising output for the updating iteration for each additional diffusion input; and combining the first denoising output and the one or more respective additional denoising outputs for the updating iteration to generate the final denoising output for the updating iteration.

[0257] Clause 9. The method of clause 8, wherein combining the first denoising output and the one or more respective additional denoising outputs for the updating iteration to generate a final denoising output for the updating iteration comprises: combining the first denoising output and the one or more respective additional denoising outputs for the updating iteration in accordance with a guidance weight for the updating iteration to generate the final denoising output for the updating iteration.

[0258] Clause 10. The method of clause 9, wherein combining the first denoising output and the one or more respective additional denoising outputs for the updating iteration in accordance with a guidance weight for the updating iteration to generate the final denoising output for the updating iteration comprises: combining the first denoising output and the one or more respective additional denoising outputs for the updating iteration in accordance with a guidance weight for the updating iteration using normalized guidance to generate the final denoising output for the updating iteration.

[0259] Clause 11. The method of any preceding clause, further comprising: setting the final data item to be the current data item after being updated at a last updating iteration of the plurality of updating iterations. Clause 12. The method of any preceding clause, wherein the data item or target data item comprises image, video, or audio data.

[0260] Clause 13. A method performed by one or more computers, the method comprising: initializing a data item; obtaining a conditioning input characterizing one or more desired properties for the data item; and generating a final data item having the one or more desired properties, the generating comprising, at each of a plurality of updating iterations: obtaining a guidance weight for the updating iteration; identifying a current data item as of the updating iteration; processing a first diffusion input for the updating iteration that comprises the current data item and a representation of the conditioning input using a diffusion neural network to generate a first denoising output for the updating iteration; processing an additional diffusion input for the updating iteration that comprises the current data item using the diffusion neural network to generate an additional denoising output for the updating iteration; generating, from the first denoising output and the additional denoising output, a final denoising output for the updating iteration, comprising: for each element of the final denoising output, determining a difference between a value of a corresponding element in the first denoising output and a value of a corresponding element in the additional denoising output; for each element of the final denoising output, determining a sign of the difference between the value of the corresponding element in the first denoising output and the value of the corresponding element in the additional denoising output; and for each element of the final denoising output, determining a value of the element in the final denoising output based on the sign of the difference for the element and the guidance weight for the updating iteration; and updating the current data item using the final denoising output. Clause 14. The method of clause 13, wherein: the first denoising output defines a prediction of a residual error between a noise component of the current data item and an analytic estimate of the noise component given the first diffusion input; and the additional denoising output defines a prediction of the residual error between the noise component of the current data item and the analytic estimate of the noise component given the additional diffusion input.

[0261] Clause 15. The method of clause 13, wherein: the first denoising output defines a prediction of a noise component of the current data item given the first diffusion input; and the additional denoising output defines a prediction of a noise component of the current data item given the additional diffusion input.

[0262] Clause 16. The method of any one of clauses 13-15, wherein determining a value of the element in the final denoising output based on the sign of the difference for the element and the guidance weight for the updating iteration comprises: setting the value of the element equal to the guidance weight when the sign is positive; and setting the value of the element equal to a negative of the guidance weight when the sign is negative.

[0263] Clause 17. The method of any one of clauses 13-16, wherein the additional diffusion input for the updating iteration comprises the current data item and a representation of a negative conditioning input.

[0264] Clause 18. The method of any one of clauses 13-16, wherein the additional diffusion input for the updating iteration comprises the current data item and a representation of a conditioning input that has been designated to indicate unconditional generation.

[0265] Clause 19. The method of any one of clauses 13-18, wherein the guidance weight is the same for each updating iteration. Clause 20. The method of any one of clauses 13-18, wherein obtaining the guidance weight comprises: applying a function to a time step corresponding to the updating iteration to determine the guidance weight for the updating iteration.

[0266] Clause 21. The method of clause 20, wherein the time step corresponding to the updating iteration is selected from a time window, and wherein the function maps each of multiple intervals of the time window to a corresponding guidance weight.

[0267] Clause 22. The method of any preceding clause, wherein the data item or target data item comprises image, video, or audio data.

[0268] Clause 23. A method performed by one or more computers, the method comprising: receiving a respective input scalar value for each of one or more properties of an output data item; for each of the one or more properties, determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item; for each of the one or more properties, generating a respective indicator value that indicates whether the respective input scalar value for the property is an upper bound or a lower bound on a desired value of the property for the output data item; and processing an input that comprises, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using a generative neural network to generate the output data item.

[0269] Clause 24. The method of clause 23, wherein: the generative neural network is a diffusion neural network, and generating the output data item comprises updating a current data item at each of a plurality of updating iterations by, at each updating iteration, processing a diffusion input for the updating iteration that comprises the current data item and a representation of, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the diffusion neural network to generate a denoising output for the updating iteration. Clause 25. The method of clause 24, wherein the denoising output defines a prediction of a residual error between a noise component of the current data item and an analytic estimate of the noise component given the diffusion input.

[0270] Clause 26. The method of clause 24 or clause 25, wherein generating the output data item comprises: for each of the one or more properties, determining one or more embeddings representing the respective input scalar value for the property and the respective indicator value for the property; and wherein the diffusion input for the updating iteration comprises the current data item and, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the diffusion neural network to generate a denoising output for the updating iteration.

[0271] Clause 27. The method of any one of clauses 23-26, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: receiving an input indicating whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item.

[0272] Clause 28. The method of any one of clauses 23-26, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: determining that the respective input scalar for the property is a lower bound on a desired value of the property for the output data item when the respective input scalar is specified as part of a positive prompt.

[0273] Clause 29. The method of any one of clauses 23-26, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: determining that the respective input scalar for the property is an upper bound on a desired value of the property for the output data item when the respective input scalar is specified as part of a negative prompt. Clause 30. A method performed by one or more computers and for training a generative neural network, the method comprising: obtaining a target data item; obtaining, for each of one or more properties of the target data item, a ground truth value for the property; for each the one more properties, sampling a value for the property; processing a model input that includes, for each property, (i) the sampled value for the property and (ii) an indicator value that indicates whether the sampled value for the property is greater than or less than the ground truth value for the property using the generative neural network to generate a model output for the model input; and training the generative neural network on a loss function that measures the quality of the model output relative to a target model output for the target data item.

[0274] Clause 31. The method of clause 30, wherein: the generative neural network is a diffusion neural network, the model input further comprises a noisy data item generated from the target data item; the model output is a denoising output; and the target model output for the target data item is a target denoising output for the target data item.

[0275] Clause 32. The method of clause 31, wherein the denoising output defines a prediction of a residual error between a noise component of the noisy data item and an analytic estimate of the noise component given the model input.

[0276] Clause 33. The method of any one of clauses 30-32, further comprising: after the training: receiving a respective input scalar value for each of one or more properties of an output data item; for each of the one or more properties, determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item; for each of the one or more properties, generating a respective indicator value that indicates whether the respective input scalar value for the property is an upper bound or a lower bound on a desired value of the property for the output data item; and processing an input that comprises, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the generative neural network to generate the output data item.

[0277] Clause 34. The method of any preceding clause, wherein the data item or target data item comprises image, video, or audio data.

[0278] Clause 35. A method performed by one or more computers, the method comprising: receiving a first input identifying a plurality of context data items for generating a first output data item; obtaining a respective embedding of each of the plurality of context data items; combining the respective embeddings of each of the plurality of context data items to generate a combined embedding; and processing an input comprising the combined embedding using a diffusion neural network to generate the first output data item.

[0279] Clause 36. The method of clause 35, further comprising: receiving a second input identifying a particular data item for generating a second output data item; obtaining an embedding of the particular context data item; and processing an input comprising the embedding of the particular context data item using the diffusion neural network to generate the second output data item. Clause 37. The method of clause 36, wherein obtaining an embedding of the particular context data item comprises: processing the particular context data item using an embedding neural network to generate the embedding of the particular context data item.

[0280] Clause 38. The method of any one of clauses 35-37, wherein combining the respective embeddings of each of the plurality of context data items to generate a combined embedding comprises: averaging the respective embeddings.

[0281] Clause 39. The method of any one of clauses 35-37, wherein combining the respective embeddings of each of the plurality of context data items to generate a combined embedding comprises: receiving a user input specifying a respective weight for each of the context data items; and computing a weighted sum of the respective embeddings of the context data items in accordance with the respective weights for the corresponding context data items.

[0282] Clause 40. The method of any one of clauses 35-39, wherein obtaining a respective embedding of each of the plurality of context data items comprises: processing each of the plurality of particular context data items using an embedding neural network to generate the respective embeddings.

[0283] Clause 41. The method of any one of clauses 35-40, wherein the diffusion neural network has been trained by performing operations comprising: maintaining a respective embedding for each of a plurality of training context data items; maintaining data clustering the respective embeddings for the plurality of training context data items into a plurality of clusters; obtaining an input specifying a target data item from the plurality of context data items; selecting a context embedding for training the diffusion neural network, comprising selecting either (i) the embedding of the target context data item or (ii) a centroid of the cluster to which the embedding of the target context data item belongs; and training the diffusion neural network using the selected context embedding and the target data item.

[0284] Clause 42. The method of clause 41, wherein selecting either (i) the embedding of the target context data item or (ii) a centroid of the cluster to which the embedding of the target context data item belongs comprises: with probability p selecting (i) the embedding of the target context data item; and with probability -p selecting (ii) the centroid of the cluster to which the embedding of the target context data item belongs.

[0285] Clause 43. The method of any one of clauses 35-42, wherein processing an input comprising the combined embedding using a diffusion neural network to generate the first output data item comprises: updating a current data item at each of a plurality of updating iterations by, at each updating iteration, processing a first diffusion input for the updating iteration that comprises the current data item and the combined embedding using the diffusion neural network to generate a first denoising output for the updating iteration.

[0286] Clause 44. The method of clause 43, wherein the first denoising output defines a prediction of a residual error between a noise component of the current data item and an analytic estimate of the noise component given the first diffusion input. Clause 45. The method of clause 43 or clause 44, wherein updating the current data item at each of the plurality of updating iterations further comprises, at each updating iteration, processing an additional diffusion input for the updating iteration that comprises the current data item and does not include the combined embedding using the diffusion neural network to generate an additional denoising output for the updating iteration, combining the first diffusion output and the additional denoising output for the updating iteration in accordance with a guidance weight for the updating iteration to generate a denoising output, and updating the current data item using the denoising output.

[0287] Clause 46. The method of any preceding clause, wherein the data item or target data item comprises image, video, or audio data.

[0288] Clause 47. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of clauses 1-46.

[0289] Clause 48. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of clauses 1-46.

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving a respective input scalar value for each of one or more properties of an output data item; for each of the one or more properties, determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item; for each of the one or more properties, generating a respective indicator value that indicates whether the respective input scalar value for the property is an upper bound or a lower bound on a desired value of the property for the output data item; and processing an input that comprises, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using a generative neural network to generate the output data item.

2. The method of claim 1, wherein: the generative neural network is a diffusion neural network, and generating the output data item comprises updating a current data item at each of a plurality of updating iterations by, at each updating iteration, processing a diffusion input for the updating iteration that comprises the current data item and a representation of, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the diffusion neural network to generate a denoising output for the updating iteration.

3. The method of claim 2, wherein the denoising output defines a prediction of a residual error between a noise component of the current data item and an analytic estimate of the noise component given the diffusion input.

4. The method of claim 2 or claim 3, wherein generating the output data item comprises: for each of the one or more properties, determining one or more embeddings representing the respective input scalar value for the property and the respective indicator value for the property; and wherein the diffusion input for the updating iteration comprises the current data item and, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the diffusion neural network to generate a denoising output for the updating iteration.

5. The method of any one of claims 1-4, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: receiving an input indicating whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item.

6. The method of any one of claims 1-4, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: determining that the respective input scalar for the property is a lower bound on a desired value of the property for the output data item when the respective input scalar is specified as part of a positive prompt.

7. The method of any one of claims 1-6, wherein determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item comprises: determining that the respective input scalar for the property is an upper bound on a desired value of the property for the output data item when the respective input scalar is specified as part of a negative prompt.

8. A method performed by one or more computers and for training a generative neural network, the method comprising: obtaining a target data item; obtaining, for each of one or more properties of the target data item, a ground truth value for the property; for each the one more properties, sampling a value for the property; processing a model input that includes, for each property, (i) the sampled value for the property and (ii) an indicator value that indicates whether the sampled value for the property is greater than or less than the ground truth value for the property using the generative neural network to generate a model output for the model input; and training the generative neural network on a loss function that measures the quality of the model output relative to a target model output for the target data item.

9. The method of claim 8, wherein: the generative neural network is a diffusion neural network, the model input further comprises a noisy data item generated from the target data item; the model output is a denoising output; and the target model output for the target data item is a target denoising output for the target data item.

10. The method of claim 9, wherein the denoising output defines a prediction of a residual error between a noise component of the noisy data item and an analytic estimate of the noise component given the model input.

11. The method of any one of claims 8-10, further comprising: after the training: receiving a respective input scalar value for each of one or more properties of an output data item; for each of the one or more properties, determining whether the respective input scalar for the property is an upper bound or a lower bound on a desired value of the property for the output data item; for each of the one or more properties, generating a respective indicator value that indicates whether the respective input scalar value for the property is an upper bound or a lower bound on a desired value of the property for the output data item; and processing an input that comprises, for each of the one or more properties, the respective input scalar value for the property and the respective indicator value for the property using the generative neural network to generate the output data item.

12. The method of any preceding claim, wherein the data item or target data item comprises image, video, or audio data.

13. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-12.

14. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-12.

Images