Denoising a neural network using a shared core subnetwork

Abstract

Methods, systems, and devices for training a denoising neural network, including computer programs encoded on computer storage media, and generating new images using the denoising neural network once it has been trained. In particular, the described technology includes obtaining data specifying a trained initial denoising neural network, obtaining training data, and training a target denoising neural network for a denoising target using the training data. The target denoising neural network includes a core subnetwork of the trained initial denoising neural network, but also includes a target encoder neural network and a target decoder neural network that are not included in the trained initial denoising neural network. By training a target neural network that includes the core subnetwork of the trained initial denoising neural network, the system enables stable training of a large-scale target denoising neural network that can be used to generate high-resolution images.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 638,870, filed April 25, 2024. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This manual relates to the use of machine learning models to generate images.

[0004] As an example, a neural network is a machine learning model that uses one or more layers of non-linear units to predict an output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to another layer in the network (e.g., the next hidden layer or the output layer). Each layer of the network generates an output from the received input based on the current values ​​of a corresponding set of weights. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which trains a denoising neural network for use in generating images. Once trained, the system can use the denoising neural network to generate new images, for example, conditionally using a conditional input—such as one or more codes of a text sequence, another image, class labels, etc.

[0006] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0007] Training a denoising neural network to generate new, high-resolution images that are accurately aligned with a given conditional input is a challenging task. This is particularly difficult when training the denoising neural network requires optimizing both the aesthetic quality of the new images and the computational time required to generate them.

[0008] Some challenges include training instability of denoising neural networks as the size of the denoising neural network and / or the resolution of the target image increases, high training computational resource requirements (both in terms of memory and computation), and the need for a large number of high-resolution training images.

[0009] To mitigate these problems, techniques such as those requiring cascaded models (i.e., a series of diffusion models, each aiming for progressively higher resolution and trained independently) and techniques relying on latent diffusion models have been developed, where image generation is initially performed in a low-dimensional latent representation, and a high-resolution image is subsequently generated from that latent representation via a pre-trained latent decoder.

[0010] However, these other techniques are not without their drawbacks. Specifically, both of these techniques (the use of cascaded models and latent diffusion models) require disjoint and independent training of neural networks that do not share components, which is computationally expensive and complex, and potentially introduces additional approximation errors compared to training a single neural network.

[0011] For example, when designing cascaded models, it is challenging to identify potential sources of quality degradation and distortion within the generated new images from the design decisions made by the many neural networks included in the cascaded model. As a specific example, a well-known problem with cascaded models is the distribution shift between training and inference, where the inputs to the super-resolution or decoder neural network during training are obtained by downsampling or encoding the training images, but during inference, they are generated from other neural networks and may therefore deviate from the training distribution. This deviation can lead to an amplification of unnatural distortions generated by the neural networks early in the cascade.

[0012] Similar to previous examples of cascaded systems, techniques using latent diffusion models require training multiple neural networks independently without sharing components. For example, the latent diffusion model is trained independently of the super-resolution / latent decoder neural network and does not share any components with it, even though both are necessary for generating high-resolution images. This introduces additional costs, complexity, and error potential compared to training a single neural network.

[0013] This specification describes a system for addressing the aforementioned challenges. Specifically, this specification describes a technique comprising: obtaining data for a designated trained initial denoising neural network, the initial denoising neural network being configured to receive an initial denoising input comprising a noisy initial image (e.g., pixel data) having an initial resolution; and the initial denoising neural network being configured to process the initial denoising input to generate an initial denoising output defining an estimate of the noise components of the noisy initial image. The initial denoising neural network includes: (i) an initial encoder subnetwork configured to map the noisy initial image to an initial representation of the noisy initial image having a core resolution; (ii) a core subnetwork configured to process the initial representation having the core resolution to generate an updated representation having the core resolution; and (iii) an initial decoder subnetwork configured to map the updated representation having the core resolution to the initial denoising output. After obtaining data for a specified trained initial denoising neural network, the technique further includes obtaining training data comprising multiple training examples, each including a corresponding target image having a target resolution higher than the initial resolution. The technique then further includes training a target denoising neural network using the training data for the denoising target. The target denoising neural network is configured to receive a target denoising input comprising a noisy target image having the target resolution, and to process the target denoising input to generate a target denoising output defining an estimate of the noise components of the noisy target image. Furthermore, the target denoising neural network includes: (i) a target encoder subnetwork configured to map the noisy target image to an initial representation of the noisy target image having the core resolution; (ii) a core subnetwork; and (iii) a target decoder subnetwork configured to map an updated representation of the noisy target image having the core resolution, generated by the core subnetwork, to the target denoising output. Therefore, the updated representation can be considered as providing an initial estimate of the target denoising output.

[0014] The described technique has the advantage of being able to train a single, large-scale, non-cascaded pixel-based model (i.e., the target denoising neural network) with the goal of generating high-resolution images, since the training of the target denoising neural network includes and depends on the core subnetworks (i.e., shared components) of the already trained initial denoising neural network. In other words, training the components of the target denoising neural network individually but not independently allows training a denoising neural network that can generate high-resolution images and also exhibits high conditional input-to-image alignment. This is because the core subnetwork of the initial denoising neural network has already learned conditional input-to-image alignment, i.e., using conditional inputs (e.g., embeddings of text sequences) to generate denoised outputs, while the encoder-decoder subnetworks of the target denoising neural network learn high-resolution generation.

[0015] In addition, the described technique has the advantage of being able to stably train denoising neural networks using data including multiple resolutions (i.e., the target encoder-decoder subnetwork is trained using the target resolution, while the core subnetwork is trained using images with both the initial resolution (obtained previously by an initial denoising neural network that has been trained) and the target resolution (obtained using a target denoising neural network), thereby mitigating the challenges that may arise from the scarcity of high-resolution image availability.

[0016] Furthermore, the described technique has the advantage of being able to improve the aesthetic value of the generated images by further training the denoising neural network on a selected subset of training images.

[0017] Furthermore, the described techniques can be used to distill denoising neural networks to accelerate image generation. For example, a trained target denoising neural network can be used to train a consistency neural network that mimics the denoising output of the target denoising neural network.

[0018] Examples of the performance of the described technology include human preference studies that show that images generated using the described technology are preferred by users in a two-to-one ratio compared to images generated by state-of-the-art technologies.

[0019] As another example of the performance of the described technique, it can faithfully train a single denoising neural network for use in generating high-resolution images that are 2 to 8 times higher than those produced by state-of-the-art techniques, and is perfectly aligned with text prompts (i.e., conditional input), and the training batches are also much smaller.

[0020] The accompanying drawings and the following description illustrate details of one or more embodiments of the subject matter of this specification.

[0021] According to a first aspect, a method performed by one or more computers is provided. The method includes: obtaining data for a designated trained initial denoising neural network, the trained initial denoising neural network being configured to: receive an initial denoising input, the initial denoising input comprising a noisy initial image having an initial resolution; and processing the initial denoising input to generate an initial denoising output, the initial denoising output defining estimates of noise components of the noisy initial image. The initial denoising neural network includes: (i) an initial encoder subnetwork configured to map the noisy initial image to an initial representation of the noisy initial image having a core resolution; (ii) a core subnetwork configured to process the initial representation having the core resolution to generate an updated representation having the core resolution; and (iii) an initial decoder subnetwork configured to map the updated representation having the core resolution to the initial denoising output. After obtaining data for the designated trained initial denoising neural network, the method includes obtaining training data comprising a plurality of training examples, each training example comprising a corresponding target image having a target resolution higher than the initial resolution. The method then concludes by training a target denoising neural network using training data for the denoising target. This target denoising neural network is configured to receive a target denoising input comprising a noisy target image with a target resolution, and to process the target denoising input to generate a target denoising output that defines an estimate of the noise components of the noisy target image. Furthermore, the target denoising neural network includes: (i) a target encoder subnetwork configured to map the noisy target image to an initial representation of the noisy target image with the core resolution; (ii) a core subnetwork; and (iii) a target decoder subnetwork configured to map the updated representation of the noisy target image with the core resolution, generated by the core subnetwork, to the target denoising output.

[0022] In some cases, the core resolution is smaller than both the initial resolution and the target resolution.

[0023] In some implementations, training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while keeping the parameters of the core subnetwork fixed at the corresponding values ​​in the initial denoising neural network.

[0024] In some cases, this core subnetwork includes multiple self-attention layers.

[0025] In some cases, the initial denoising input and the target denoising input each include a corresponding conditional input, and each training example further includes a corresponding training conditional input that characterizes one or more attributes of the corresponding target image in the training example.

[0026] Furthermore, in some cases, the core subnetwork includes one or more conditional layers, each of which updates its input representation based at least on the representation of the corresponding conditional input.

[0027] In some cases, the one or more conditionalization layers are cross-attention layers.

[0028] In some cases, the corresponding conditional input includes a corresponding encoded representation of each of one or more context inputs, which include one or more embeddings.

[0029] In some cases, the initial denoising neural network and the target denoising neural network each include a context encoder neural network configured to process a corresponding encoded representation of each of the one or more context inputs to generate a representation of the conditional input.

[0030] In some cases, training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while keeping the parameters of the core subnetwork and the field encoder neural network fixed at the corresponding values ​​in the initial denoising neural network.

[0031] Furthermore, in some cases, the scene encoder neural network includes an attention pooling subnetwork followed by a multilayer perceptron (MLP).

[0032] In some cases, the initial denoising input and the target denoising input each include a corresponding temporal input that specifies a time step and defines the noise level of the corresponding noisy image. Additionally, one or more of the conditionalization layers are conditional on the representation of the corresponding temporal input.

[0033] In some cases, the initial denoising neural network and the target denoising neural network each include a time encoder neural network configured to process the corresponding time input to generate a representation of the corresponding time input.

[0034] In some cases, training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while updating the parameters of the temporal encoder neural network starting from the corresponding values ​​in the initial denoising neural network.

[0035] In some cases, the initial encoder subnetwork includes a downsampled convolutional subnetwork configured to map the noisy initial image to a downsampled representation with the core resolution. The initial encoder subnetwork also includes a convolutional encoder subnetwork configured to map the downsampled representation to an initial representation of the noisy initial image with the core resolution.

[0036] In some cases, the initial decoder subnetwork includes: a convolutional decoder neural network configured to map the updated representation to the decoded representation having the core resolution; and an upsampling convolutional subnetwork configured to map the decoded representation having the core resolution to the initial denoised output.

[0037] In some cases, obtaining data for the specified initial denoising neural network includes training the initial denoising neural network against the denoising target using initial training data, and the initial training data includes multiple initial images with the initial resolution.

[0038] In some implementations, the method further includes, after training the target denoising neural network, using the target denoising neural network to perform a backdiffusion process to generate a new image with the target resolution.

[0039] In some cases, using the target denoising neural network to perform a backdiffusion process to generate a new image with that target resolution involves performing the backdiffusion process conditioned on a new conditional input.

[0040] According to a second aspect, a method performed by one or more computers is provided. The method includes receiving a conditional input. A representation of a new image is then initialized by sampling noise values ​​from a noise distribution. The representation of the new image is then updated at each of a plurality of back-diffusion steps. This update process includes processing the denoising input of the back-diffusion step at each back-diffusion step using a denoising neural network conditioned on the conditional input to generate a denoised output, the denoising input including the representation of the new image, the denoising output defining estimates of the noise components of the representation of the new image. The denoising neural network is a target denoising neural network as described in any method of the first aspect. After updating the representation of the new image at each of the plurality of back-diffusion steps, the method includes using the denoised output to update the representation of the new image. Finally, after updating the representation of the new image at each of the plurality of back-diffusion steps, a new image is generated from the representation of the new image.

[0041] Furthermore, in some cases of the second aspect, the conditional input comprises a text sequence, and the method further comprises processing the text sequence using each of a plurality of text encoder neural networks to generate a corresponding encoded representation of the text sequence for each of the text encoder neural networks. The corresponding encoded representations of the text sequence for each of the text encoder neural networks are then combined to generate a combined encoded representation, wherein the denoising input for each backdiffusion step includes the combined encoded representation.

[0042] Furthermore, in some cases of the second aspect, combining the corresponding encoded representations of the text sequences for each of the text encoder neural network to generate a combined encoded representation includes concatenating the corresponding encoded representations of the text sequences for each of the text encoder neural network to generate a concatenated encoded representation. The concatenated encoded representation is then processed using a context encoder neural network to generate the combined encoded representation.

[0043] In some cases of the second aspect, the multiple text encoder neural networks have been trained independently of each other.

[0044] In some cases of the second aspect, the denoising neural network has been trained on a first training data comprising multiple first training images for the denoising target.

[0045] Furthermore, in some cases of the second aspect, after training on the first training data, the denoising neural network has been further trained on the second training data, which is generated from a mixed image set including the first training image and multiple second training images that have been classified as aesthetic images.

[0046] Furthermore, in some cases of the second aspect, the second training image is a proper subset of the training images that have been classified by the image classification neural network as conforming to aesthetic and compositional properties.

[0047] Furthermore, in some cases of the second aspect, after further training on the second training data, the denoising neural network has been further trained on the third training data for the uniform distillation target.

[0048] In some implementations of the second aspect, generating the new image includes updating the representation of the new image at each of the plurality of backdiffusion steps and then outputting it as the new image.

[0049] According to a third aspect, a method of the first or second aspect is provided, which is executed by a system comprising: one or more computers; and one or more storage devices storing operable instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the corresponding method.

[0050] According to the fourth aspect, a method of the first or second aspect is provided, which is executed by one or more computer storage media, the one or more computer storage media being encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the corresponding method.

[0051] Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. Attached Figure Description

[0052] Figure 1 An image generation system is shown.

[0053] Figure 2 This is a flowchart of an example process for training a target denoising neural network.

[0054] Figure 3 This is a flowchart of an example process for updating the trainable parameters of a target denoising neural network.

[0055] Figure 4 This is a flowchart of an example process for generating a new image using a backdiffusion process.

[0056] Figure 5 These are examples of the performance of the described technology.

[0057] Figure 6 These are examples of the performance of the described technology.

[0058] Figure 7 These are examples of the performance of the described technology. Detailed Implementation

[0059] Figure 1 An example image generation system 100 is shown. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0060] System 100 trains target denoising neural network 102 using an already trained initial denoising neural network (i.e., data 104 specifying the trained initial denoising neural network). Besides generating new images, the trained target denoising neural network 102 can be used in many other ways, such as removing noise from an input image (provided as a conditional input to the trained target denoising neural network), or achieving other improvements to the quality of other images. As another example, another image can be processed to provide inpainting, super-resolution, video generation (e.g., generating a series of video frames based on the input), or image-to-image conversion. The system can also be used for temporal data modeling, for example, predicting and modeling temporal data based on the input.

[0061] The trained initial denoising neural network is configured to: receive an initial denoising input, which includes a noisy initial image with an initial resolution; and process the initial denoising input to generate an initial denoising output, which defines an estimate of the noise components of the noisy initial image.

[0062] The initial denoising neural network includes: (i) an initial encoder subnetwork (not shown) configured to map the noisy initial image to an initial representation of the noisy initial image having a core resolution (i.e., the resolution to which the core subnetwork is configured to process); (ii) a core subnetwork 128 configured to process the initial representation having the core resolution to generate an updated representation having the core resolution; and (iii) an initial decoder subnetwork (not shown) configured to map the updated representation having the core resolution to the initial denoising output.

[0063] Therefore, an initial denoising neural network can be used to generate images with an initial resolution. That is, using an appropriate algorithm (e.g., any suitable diffusion sampler for a backdiffusion process), an initial denoising neural network can be used to generate images with an initial resolution.

[0064] In some cases, the initial denoising neural network is a network that System 100 (or another system) has trained for the denoising target using initial training data including multiple initial images with initial resolutions.

[0065] For example, the initial image can be obtained using one or more cameras and / or other passive or active image sensors. As an example, the image could be a medical image obtained using a medical sensor. If the image is a medical image, a trained target denoising neural network can be used in medical procedures, such as processing MRI data, to generate images that clearly show medically relevant information such as the location of a tumor.

[0066] The denoising objective targeted by the initial denoising neural network trained by System 100 can be any of a variety of suitable objectives, such as the mean squared error objective (to minimize the difference between the predicted estimate of the noise component of the noisy initial image and the true noise component) or the fractional matching objective (to estimate the fractional function (defined as the gradient of the log density) of the distribution of perturbation data at different noise levels).

[0067] However, the target denoising neural network 102 may need to generate an image with a target resolution higher than the initial resolution. To this end, the system 100 utilizes the core subnetwork 128 of the initial neural network to initialize the target neural network 102, while still allowing the target denoising neural network 102 to generate an image with the target resolution.

[0068] Specifically, the target denoising neural network 102 is configured to receive a target denoising input 108 including a noisy target image 110 with a target resolution, and process the target denoising input 108 to generate a target denoising output 134, which defines an estimate of the noise components of the noisy target image 110.

[0069] In order to share the core subnetwork 128 between the initial denoising neural network and the target denoising neural network 102, the target denoising neural network 102 includes: (i) a target encoder subnetwork 116 configured to map the noisy target image 110 to an initial representation of the noisy target image 122 with core resolution; (ii) a core subnetwork 128; and (iii) a target decoder subnetwork 132 configured to map the updated representation of the noisy target image 130 with core resolution, generated by the core subnetwork 128, to the target denoising output 134.

[0070] Then, system 100 obtains training data 106, which includes multiple training examples, each of which includes a corresponding target image with a target resolution higher than the initial resolution. The system then uses the training data 106 to train a target denoising neural network 102 for the denoising target.

[0071] In some implementations, during this training period, system 100 can train the target encoder subnetwork 116 and the target decoder subnetwork 132 while keeping the parameters of the core subnetwork 128 fixed at the corresponding values ​​in the initial denoising neural network.

[0072] In some other implementations, system 100 can continue to update the parameters of the core subnetwork 128 during training.

[0073] In some cases, the target denoising neural network 102 is used to generate new images conditioned on a conditional input 112, which represents one or more attributes of the new image. In these cases, the initial denoising input and the target denoising input 108 each include a corresponding conditional input 112, and each training example includes a corresponding training conditional input, which represents one or more attributes of the corresponding target image in the training example.

[0074] Furthermore, in some cases, the core subnetwork 128 includes one or more conditional layers, each of which updates its representation input with at least the representation 124 of the corresponding conditional input as a condition.

[0075] In some cases, the corresponding conditional input 112 includes a corresponding encoded representation of each of one or more context inputs, which include one or more embeddings; the context inputs may be images, text, class labels, audio samples, video, sensor data, etc. Additionally, in some cases, the initial denoising neural network and the target denoising neural network 102 each include a context encoder neural network 118, which is configured to process the corresponding encoded representation of each of the one or more context inputs to generate a representation 124 of the conditional input, and this context encoder neural network is shared between the initial denoising neural network and the target denoising neural network 102.

[0076] In some cases, the initial denoising input and the target denoising input 108 each include a corresponding temporal input 114, which specifies a time step that defines the noise level of the corresponding noisy image (e.g., noisy target image 110). Additionally, one or more of the conditionalization layers can be conditional on a representation 126 of the corresponding temporal input.

[0077] In some cases, the initial denoising neural network and the target denoising neural network 102 each include a time encoder neural network 120, which is configured to process the corresponding time input 114 to generate a representation 126 of the corresponding time input, and the time encoder neural network is shared between the initial denoising neural network and the target denoising neural network 102.

[0078] In some cases, the training target denoising neural network 102 includes training the target encoder subnetwork 116 and the target decoder subnetwork 132, while updating the parameters of the time encoder neural network 120 starting from the corresponding values ​​in the initial denoising neural network.

[0079] After training the target denoising neural network 102, the system 100 can use the target denoising neural network 102 to perform a backdiffusion process to generate a new image with the target resolution.

[0080] In order to perform the back diffusion process, system 100 can receive conditional input (i.e., new conditional input).

[0081] Then, system 100 can initialize a representation of the new image by sampling noise values ​​from the noise distribution. This representation can have a target resolution.

[0082] Then, system 100 can update the representation of the new image at each of the multiple back-diffusion steps.

[0083] As part of the update, at each backdiffusion step, system 100 uses target denoising neural network 102 to process the denoising input of the backdiffusion step conditioned on the conditional input to generate a denoising output, the denoising input including a representation of the new image, the denoising output defining an estimate of the noise component of the representation of the new image.

[0084] In some cases, the denoising input also includes a temporal input, and the system 100 further imposes conditional constraints on the target denoising neural network 102 based on the temporal input to generate a denoising output that defines an estimate of the noise components in the representation of the new image.

[0085] Optionally, system 100 may use classifier-less guidance at each back-diffusion step. When using classifier-less guidance, system 100 uses a target denoising neural network but without conditionalizing the input to process another denoising input for the back-diffusion step to generate another denoising output, which includes a representation of the new image. The system then combines the conditional and unconditional denoising outputs according to the guidance weights of the back-diffusion step to generate the final denoising output.

[0086] At each backdiffusion step, system 100 then uses the denoised output to update the representation of the new image.

[0087] For example, system 100 can use the denoised output to determine an initial estimate of the final image, and then apply an appropriate diffusion sampler—such as a DDPM (Denoising Diffusion Probability Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another appropriate sampler—to the initial estimate of the final image to update the current representation of the new image. At the final backdiffusion step, the system can use the initial estimate of the final image as the updated representation of the new image.

[0088] After updating the representation of the new image at each of the multiple back-diffusion steps, system 100 generates a new image from the representation of the new image. For example, because the representation has a target resolution, system 100 can directly output the updated final representation as the new image.

[0089] As described above, each denoising output defines an estimate of the noise component of the corresponding representation of the image. This estimate can be defined in any of a variety of ways.

[0090] In some implementations, the denoising output is an estimate of the noise component of the current representation (i.e., the noise that needs to be combined with the final representation (e.g., added to or subtracted from the final representation) to generate the noise in the current representation).

[0091] In some other implementations, the denoised output is an estimate of the final representation given the current representation, i.e., an estimate of the representation that will be produced by removing the noise components from the current representation.

[0092] In some other implementations, the denoising output defines the predicted residual between the true noise component of the current representation and the analytical estimate of the noise component—that is, the estimate calculated analytically from the current representation.

[0093] In some other implementations, the denoised output is a v-parameterized estimate of the noise components.

[0094] Figure 2 This is a flowchart of an example process 200 for training a target denoising neural network. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification... Figure 1 Image generation system 100, image generation system executable process 200.

[0095] The system obtains data for a specified, trained initial denoising neural network (step 202). For example, the system can obtain data from system-maintained data. As another example, the system can obtain data from a user or another system through any of a variety of methods—such as using a network connection, for example, a cloud-based network, the Internet, or a local network. For example, the system can use a user device, such as a laptop computer, via a network—for example, by establishing a network connection with the user device—to receive data from the user.

[0096] The data specifying the initial trained denoising neural network includes any data required to implement or use the initial denoising neural network. For example, this data may include details of the initial denoising neural network architecture, such as the number of layers, the type of each layer (e.g., fully connected layers, convolutional layers, attention-based layers, etc.), layer activation functions, and connectivity between layers. Additionally, this data may include trained values ​​for the trainable parameters included in the layers.

[0097] The initial denoising neural network specified by the data is configured to: receive an initial denoising input, which includes a noisy initial image with an initial resolution; and be configured to process the initial denoising input to generate an initial denoising output, which defines an estimate of the noise components of the noisy initial image.

[0098] Generally, a noisy initial image is an image that includes pixel values ​​that have random deviations from the original pixel values, and these random deviations distort the image.

[0099] The term "resolution" refers to the number of pixels that make up an image. An initial resolution can be any number of pixels. For example, resolution can be expressed as the number of pixels that defines the "height" and "width" of the image. Therefore, an initial image could be, for example, 16×16 (i.e., 256 pixels), 32×32 (i.e., 1,024 pixels), 64×64 (i.e., 4,096 pixels), and so on. While the resolution examples here and below have two dimensions of equal size, more generally, resolution can be defined using any number of dimensions, each with its own size.

[0100] Pixel values ​​can include one or more numerical values, which represent the intensity depending on the image type.

[0101] For example, if the image is a grayscale image, the pixel values ​​can include numerical values ​​between 0 and 255 representing the grayscale intensity.

[0102] As another example, if the image is a red-green-blue (RGB) image, the pixel values ​​can include three numerical values ​​representing the intensity of the respective color channel (one value for each red, green, or blue channel).

[0103] An estimate of the noise component of a noisy initial image can be an estimate of the random deviation of the pixel values ​​that distort the image.

[0104] For example, for an undistorted image, the pixel intensity value is 123, while a distorted image can have an intensity value of 255. Therefore, in this case, the random deviation of the pixel value is 123 - 255 = -132.

[0105] The initial denoising neural network included in the data comprises: (i) an initial encoder subnetwork configured to map the noisy initial image to an initial representation of the noisy initial image having a core resolution; (ii) a core subnetwork configured to process the initial representation having the core resolution to generate an updated representation having the core resolution; and (iii) an initial decoder subnetwork configured to map the updated representation having the core resolution to the initial denoising output.

[0106] The initial encoder subnetwork can have any of a variety of neural network architectures. That is, the initial encoder subnetwork can have any suitable architecture in any appropriate configuration that can map the noisy initial image to an initial representation of that noisy initial image with core resolution, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, etc., as the case may be.

[0107] For example, the initial encoder subnetwork may include one or more convolutional blocks alternating with pooling layers or downsampling layers, which map the noisy initial image to an initial representation of the noisy initial image with core resolution.

[0108] In some cases, the initial encoder subnetwork includes: a downsampled convolutional subnetwork configured to map the noisy initial image to a downsampled representation with core resolution; and a convolutional encoder subnetwork configured to map the downsampled representation to the initial representation of the noisy initial image with the core resolution.

[0109] The initial encoder subnetwork may include, for example, a downsampled convolutional subnetwork configured with a convolutional kernel and stride that maps a noisy initial image to a downsampled representation with kernel resolution. Alternatively, the example encoder subnetwork may include a convolutional encoder subnetwork comprising a residual block containing two convolutional layers with a swish activation function.

[0110] The core subnetwork can be any of a variety of neural network architectures. That is, the core subnetwork can be any suitable architecture in any appropriate configuration that can process an initial representation with core resolution to generate an updated representation with that core resolution, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, depending on the situation.

[0111] For example, the core subnetwork may include multiple self-attention layers. For example, the core subnetwork may include one or more Transformer block layers (each Transformer block layer includes a self-attention layer) that process an initial representation with core resolution to generate an updated representation with core resolution.

[0112] The initial decoder subnetwork can be any of a variety of neural network architectures. That is, the initial decoder subnetwork can have any suitable architecture in any appropriate configuration that can map the updated representation with core resolution to the initial denoised output, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, etc., as the case may be.

[0113] For example, the initial decoder subnetwork may include one or more upsampling layers.

[0114] In some cases, the initial decoder subnetwork includes: a convolutional decoder neural network configured to map the updated representation to the decoded representation having the core resolution; and an upsampling convolutional subnetwork configured to map the decoded representation having the core resolution to the initial denoised output.

[0115] For example, the initial decoder subnetwork may include a convolutional decoder neural network comprising residual blocks containing two convolutional layers with swish activation functions, which maps the updated representation to the decoded representation with core resolution. Alternatively, this example initial decoder subnetwork may include an upsampled convolutional subnetwork, which is a subpixel convolutional neural network layer that maps the decoded representation with core resolution to the initial denoised output.

[0116] Specific layers within such initial denoising neural networks can perform disparate functions. For example, a text encoding layer can be configured to process one or more text, character, or multimodal representations (e.g., derived from models such as T5-XXL, CLIP (ViT-H14), or ByT5) and project these representations into an embedding space suitable for processing by the initial core subnetwork. Such layers can utilize operations such as pooling, followed by a multilayer perceptron (MLP). Furthermore, layers belonging to the initial core subnetwork may include hidden layers that interface with embedded representations via a cross-attention mechanism. These may include layers operating at a specific resolution (e.g., lower resolution in a UNet architecture), where features are combined with embedded text representations via cross-attention. Additionally, a temporal encoding layer can be configured, for example, to map diffused time-step inputs to an embedding space using a sinusoidal position encoder followed by a shallow MLP.

[0117] As an example of how components of an initial denoising neural network can be constructed, certain implementations can utilize specific architectures during the initial training phase. One such architecture comprises an initial encoder subnetwork and an initial decoder subnetwork operating on a fixed spatial grid (e.g., 16x16 pixels). The encoder and decoder can each utilize a single residual block and can operate without upsampling or downsampling layers, thereby sharing the spatial grid resolution with some initial core subnetwork layers. A first convolutional layer can project the received input image onto this predetermined spatial grid, potentially expanding the input channel dimension (e.g., to 256). A corresponding upsampling head (e.g., involving subpixel convolutions) can map the representation from the predetermined grid to the target output resolution. Variations in the number of Transformer blocks (e.g., 6, 8, 12, 16), the size of the hidden layers (e.g., 1536 to 4096), and the MLP channel dimension within the representation layers can produce models with different parameter counts (e.g., approximately 672M to 7.7B parameters).

[0118] In some cases, when the system obtains data for a specified trained initial denoising neural network, the system uses the initial training data to train the initial denoising neural network against the denoising target. This initial training data includes multiple initial images with an initial resolution.

[0119] The system obtains training data, which includes multiple training examples, each of which includes a corresponding target image with a target resolution higher than the initial resolution (step 204). The system can obtain the training data from system-maintained data, users, or another system in any of a variety of ways; for example, the system can obtain training data from another system via a network connection.

[0120] A target resolution higher than the initial resolution typically means that the target resolution has more pixels than the initial resolution. For example, if the initial resolution is 16×16 (i.e., 256 pixels), the target resolution could be 32×32 (i.e., 1024 pixels), 64×64, 128×128, 256×256, 512×512, 1024×1024, and so on.

[0121] In some cases, the core resolution is smaller than both the initial resolution and the target resolution. For example, if the initial resolution is 32×32 and the target resolution is 1,024×1,024, the core resolution could be 16×16. In other cases, the core resolution is equal to the initial resolution. For example, if the initial resolution is 32×32, the core resolution is 32×32; or if the initial resolution is 16×16, the core resolution is 16×16, and so on.

[0122] The system uses training data to train a target denoising neural network for the denoising target (step 206).

[0123] The target denoising neural network is configured to receive a target denoising input including a noisy target image with a target resolution, and process the target denoising input to generate a target denoising output that defines an estimate of the noise components of the noisy target image.

[0124] The target denoising neural network includes: (i) a target encoder subnetwork configured to map the noisy target image to an initial representation of the noisy target image having the core resolution; (ii) a core subnetwork; and (iii) a target decoder subnetwork configured to map the updated representation of the noisy target image having the core resolution and generated by the core subnetwork to the target denoising output.

[0125] While the core subnetwork of the target denoising neural network is the same as the core subnetwork of the initial denoising neural network, the target encoder subnetwork and the target decoder subnetwork are not the same. In this way, the system uses the core subnetwork in the target denoising neural network to process the representation with core resolution to generate an updated representation with core resolution, just as it does with the initial denoising neural network. However, the system uses the target encoder / decoder subnetwork of the target denoising neural network for target resolution encoding / decoding. That is, the system uses the target encoder subnetwork to generate an initial representation with core resolution from the noisy target image with target resolution, and uses the target decoder subnetwork to map the updated representation of the noisy target image with core resolution to a target denoising output that defines an estimate of the noise components of the noisy target image with target resolution.

[0126] A denoising target can be any target that, when used to train a target denoising neural network, leads to an improvement (relative to before training) in the target denoising neural network's generated target denoising output, which defines an estimate of the noise components of the noisy target image. For example, the following equation... Define the objective function, where This represents a time step in the diffusion process, which includes... Each time step This represents the expectation of the time steps sampled from T diffusion time steps. Represents the target image. Represents the quality probability of the target image in the training data. Take the expectation of the target image from the training data. This represents the noise component added to the target image, which is responsible for... Mapped to (time step) place (noisy target image), This represents the expectation of noise components sampled from a Gaussian normal distribution with mean 0 and isotropic covariance of 1, and... Represents true noise Target denoising neural network estimate of the noise component of a noisy target image The square norm of the difference between (where) This indicates that the target denoising neural network includes trainable parameters. When the system minimizes this objective function by training the target denoising neural network, the result is that the denoised output more accurately defines the noise components of the corresponding noisy target image generated by the denoising neural network.

[0127] In some embodiments, the specific process for training the target denoising neural network may involve a multi-stage process. In a first stage, certain components intended for the target denoising neural network, such as layers of the core subnetwork and text / temporal encoding layers, may be trained. This training may potentially occur within the surrogate architecture using a set of text-image pairs (e.g., the CC12M dataset) at a first resolution (e.g., 64x64).

[0128] In the second stage, the target denoising neural network architecture can be constructed (e.g., an architecture including a UNet structure operating at multiple resolutions). Components trained in the first stage (e.g., layers of the core subnetwork, text encoding layers) can be initialized using the component parameters obtained from the first stage. Additional components, such as encoder and decoder subnetwork layers operating at different resolutions, can be added, and these additional components can be randomly initialized. During training in this second stage, the parameters of the previously trained layers of the core subnetwork and text encoding layers can remain frozen. The parameters of the temporal encoding layers and newly added encoder / decoder subnetwork layers can be updated (tuned). The second stage training typically uses a training dataset that includes images at the target resolution (e.g., 512x512 or 1024x1024). The spread logSNR (signal-to-noise ratio) offset can be optionally adjusted for the target resolution (e.g., an adjustment factor of 2 log(64 / d) according to arXiv:2301.11093). Training in this phase can be performed using a specific batch size (e.g., a batch size of 256) without utilizing certain forms of regularization such as dropout or multi-scale loss.

[0129] In the optional third stage, the parameters previously fixed in the second stage can be unfrozen, and all parameters of the target denoising neural network can be updated jointly.

[0130] In implementing the training process described above, specific configurations and data parameters can be used. For example, for text conditionalization, embeddings from multiple text encoders such as T5-XXL (e.g., sequence length 128) and CLIP (e.g., ViT-H14, sequence length 77) can be generated from the cue, concatenated (e.g., producing a sequence of length 205), and processed by the text encoding layer. Additional encoders such as ByT5 (e.g., sequence length 256) can also be incorporated, resulting in longer combined sequences (e.g., 461). The training data may differ between stages; for example, the first stage might use the CC12M dataset (approximately 12M images) at 64x64 resolution, while the second stage might use a subset (e.g., 8.7M images) to filter for a target resolution such as 512x512. Training can be performed for a predetermined number of iterations (e.g., 2M iterations for the first stage and 1M iterations for the second stage). Aspect ratio bucketing can be used to handle image dimensions, such as (1024,1024), (768,1376), and (1376,768).

[0131] In some cases, the system trains the target denoising neural network by training the target encoder subnetwork and the target decoder subnetwork while keeping the parameters of the core subnetwork fixed at the corresponding values ​​in the initial denoising neural network.

[0132] In some cases, the system uses an initial denoising neural network and a target denoising neural network (respectively) to generate the initial denoised output and the target denoised output, respectively, with each neural network conditioned on its corresponding additional available input.

[0133] In some cases, for example, the initial denoising input and the target denoising input each additionally include a corresponding conditional input, and each training example further includes a corresponding training conditional input that characterizes one or more attributes of the corresponding target image in the training example.

[0134] In some implementations, the corresponding conditional input includes a corresponding encoded representation of each of one or more context inputs, which include one or more embeddings. For example, the context input may include one or more embeddings (or one or more encoded representations) representing one or more of text, images, sensor data, bounding boxes, semantic graphs, etc., and the context input may be generated using a pre-trained neural network model (e.g., a pre-trained text or image embedding model).

[0135] In some implementations, the core subnetwork (belonging to both the initial denoising neural network and the target denoising neural network) may include one or more conditional layers, each conditional layer updating its input representation conditioned on at least the representation of the corresponding conditional input. For example, the one or more conditional layers may be cross-attention layers.

[0136] As a specific example, the core subnetwork may include one or more cross-attention layers, each of which cross-attentions its input representation with the representation of conditional input received by the system from the text sequence.

[0137] The text sequence can be any text sequence. For example, a text sequence can be a description that affects the resulting image generated by an image generation system. For example, a text sequence can be a label, a description of visual features to be included, a style to be adopted for the image, a requested effect to be presented in the image, etc. Some examples of text sequences include “A futuristic street train—a rainy street at night in an old European city. Painting by David Friedrich, Claude Monet, and John Tenniel.”, “Realistic photograph of a cute otter zebra mouse in a field at sunset, tall grass, macro 35mm film,” or “Silver 1963 Ferrari 250 GTO in profile racing along a beach front road. Bokeh, high-quality 4K photograph.”

[0138] In other implementations, the initial denoising neural network and the target denoising neural network each include a context encoder neural network configured to process a corresponding encoded representation of each of the one or more context inputs to generate a representation of the conditional input.

[0139] A field encoder neural network can be any of a variety of neural network architectures. That is, a field encoder neural network can have any suitable architecture in any appropriate configuration that can process the corresponding encoded representation of each of one or more field inputs to generate a representation of the conditional input, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, depending on the situation.

[0140] As a specific example, a context encoder neural network may include one or more pooling layers and multilayer perceptron layers, or an attention pooling subnetwork followed by a multilayer perceptron (MLP). The context encoder neural network can process conditional inputs comprising one or more encoded representations of text, characters, and / or multimodal expressions to generate a representation of those conditional inputs. For example, the system could process a text cue by first lexicalizing it, then using pre-trained models T5-XXL (sequence length 128) and CLIP (VIT-H14) (sequence length 77) to generate two embedding sequences of the text cue (i.e., two contextual inputs constituting the conditional input), and then using the context encoder neural network to process that conditional input to generate a representation of it.

[0141] In some cases, when the system trains the target denoising neural network, it trains the target encoder subnetwork and the target decoder subnetwork, while keeping the parameters of the core subnetwork and the field encoder neural network fixed at the corresponding values ​​in the initial denoising neural network.

[0142] In some implementations, the initial denoising input and the target denoising input each include a corresponding temporal input, which specifies a time step and defines the noise level of the corresponding noisy image. Furthermore, in these implementations, one or more of the conditionalization layers are conditional on the representation of the corresponding temporal input.

[0143] In some implementations, the initial denoising neural network and the target denoising neural network may each include a time encoder neural network configured to process the corresponding time input to generate a representation of the corresponding time input.

[0144] A temporal encoder neural network can have any of a variety of neural network architectures. That is, a temporal encoder neural network can have any suitable architecture in any appropriate configuration that can process the corresponding temporal input to generate a representation of that corresponding temporal input, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, etc., as the case may be.

[0145] For example, a time encoder neural network could be a sinusoidal position encoder followed by a shallow MLP.

[0146] In some cases, when the system trains the target denoising neural network, it trains the target encoder subnetwork and the target decoder subnetwork, while updating the parameters of the time encoder neural network starting from the corresponding values ​​in the initial denoising neural network.

[0147] The following is for reference. Figure 3 More details describe how to update the trainable parameters of the target denoising neural network during training.

[0148] In some cases, after training the target denoising neural network, the system uses this network to perform a backdiffusion process to generate a new image with the target resolution. For example, the backdiffusion process may include: for each of a plurality of backdiffusion steps, the system updates the representation of the new image using the denoised output of the target denoising neural network. Then, after the final update in the last backdiffusion step, the system generates the new image.

[0149] Furthermore, in some cases, the system performs the back-diffusion process conditioned on new conditional inputs (e.g., text embeddings). That is, when the initial denoised input, the target denoised input, and each training example include conditional inputs (or training conditional inputs) representing one or more attributes, the system can perform the back-diffusion process conditioned on the corresponding conditional inputs.

[0150] The following is for reference. Figure 4 More details are provided regarding an example process for generating new images using the backdiffusion process.

[0151] Figure 3 This is a flowchart of an example process 300 for updating trainable parameters of a target denoising neural network. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification... Figure 1 The image generation system 100 has an executable process 300.

[0152] This system or another training system uses training data to repeatedly update the trainable parameters of the target denoising neural network. That is, the system can repeatedly execute the example process described below using training examples to repeatedly update the trainable parameters of the target denoising neural network from scratch, i.e., to train from randomly initialized parameters, or to fine-tune, i.e., to further update previously determined parameters.

[0153] As mentioned above, the set of trainable parameters is not necessarily all the parameters included in the denoising neural network. For example, for the components of the target denoising neural network (i.e., the target decoder subnetwork, the core subnetwork, the target encoder subnetwork, the context encoder neural network, the time encoder neural network, etc.), the system can configure the update of the trainable parameters to include only the trainable parameters belonging to a subset of these components.

[0154] The system obtains training data including training examples (302). As described above, the system can obtain training data from system-maintained data, users, or another system through any of a variety of methods.

[0155] As described above, the training data includes multiple training examples, and each training example includes a corresponding target image with a target resolution higher than the initial resolution to which the initial denoising neural network is configured to receive.

[0156] For each training example, the system combines the noise with the target image (step 304). The result is a noisy target image. The system can identify the noise and combine it with the target image using any of a variety of methods to generate a noisy target image.

[0157] For example, the system can sample noise from a noise distribution (e.g., a probability distribution, such as a Gaussian probability distribution) with the same number of dimensions as the number of pixels included in the target image (i.e., the number of dimensions of the target image), and sum the sampled noise element-wise with the pixel values ​​of the target image to generate a noisy target image.

[0158] In some cases, particularly when the time step defines the noise level of the corresponding noisy image, the system can determine the noise based on one or more time steps.

[0159] For example, the system can generate noisy target images using Markov processes, such as the forward process of DDPM. That is, given a target image, the number of diffusion steps (i.e., time steps), and the variance scheduling across diffusion steps, the system can generate noisy target images by sampling from a parameterized normal distribution.

[0160] For example, recursive equations This indicates how the system can generate time steps by repeatedly sampling a parameterized Gaussian distribution. Associated noisy target images ,in It is the number of diffusion steps. It is the variance scheduling value across the diffusion step. Is with the input image Identity matrices with the same dimensions It is a scaled noisy target image from the previous diffusion step, and This indicates the generation of a value with mean µ and covariance σ. It follows a normal distribution.

[0161] As another example, equation It also indicates how the system can generate noisy target images. ,in It is noise sampled from a Gaussian distribution, i.e. ,and .

[0162] Furthermore, the system can combine one or more different noises with the same target image to generate one or more corresponding noisy target images. For example, consider combining... The previous example of generating noisy target images by sampling allows the system to process images at the same time step. One or more time steps or both Sample one or more values ​​to generate one or more noisy target images.

[0163] For each training example, the system generates a denoised output of the target (step 308).

[0164] In other words, for each noisy target image generated during step 306, the system uses a denoising neural network to process at least that noisy target image to generate a target denoising output that defines an estimate of the noise components of the noisy target image.

[0165] When the system generates a noisy target image from the target image of the training samples using time steps, the system can use a target denoising neural network to generate a denoised output that further processes at least the corresponding time steps of the noisy target image.

[0166] As described above, in some cases, each training example further includes a corresponding training conditional input characterizing one or more attributes of the corresponding target image in that training example. In this case, the system can use a target denoising neural network to generate a denoised output, which further processes at least the conditional input.

[0167] The system uses all training examples and the corresponding target denoising output to evaluate the target (step 310).

[0168] Generally, the objective evaluation system uses the performance of the target denoising output generated by the target denoising neural network during step 308. For example, the objective (or objective function) may include the loss for each training example. For instance, the objective may be represented as an equation... in This indicates the target image (i.e., in This indicates the target image based on the training data. The expected value of the probability of sampling is taken, which takes into account oversampled noise (i.e., The expected value of the time step (i.e., ,in Indicates from arrive The expected value is taken from the uniform distribution of time steps. Additionally, the term... It is a noisy target image (as in the example above), and the item It refers to the estimated noise component of a noisy target image generated by a target denoising neural network.

[0169] In some cases, the objective includes one or more regularization terms that penalize higher values ​​of trainable parameters to reduce the risk of the target denoising model's trainable parameters overfitting the training data. For example, the regularization term may include an LP regularization term. , where λ is the regularization parameter, w is a vector of trainable parameters, and p is the norm order (e.g., p=1 represents L1 regularization, p=2 represents L2 regularization).

[0170] The system updates the trainable parameters to optimize the objective (step 312).

[0171] The system can update the trainable parameters of the target denoising neural network to optimize the target through any of a variety of methods, such as gradient-based methods, evolutionary algorithm-based methods, Bayesian optimization, etc.

[0172] For example, the system can use any of a variety of gradient descent techniques (e.g., batch gradient descent, stochastic gradient descent, or mini-batch gradient descent) to optimize the objective. These gradient descent techniques include using backpropagation to estimate the gradient of the loss with respect to the trainable parameters of the neural network and updating the trainable parameters accordingly.

[0173] Generally, the system repeats the above steps until one or more criteria are met (e.g., the system performs a predetermined number of iterations, the updates to the trainable parameters no longer exceed a predetermined range of change, the metrics on the validation dataset exceed a predetermined value, etc.).

[0174] Figure 4This is a flowchart of an example process 400 for generating a new image. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification. Figure 1 The image generation system 100 has an executable process 400.

[0175] The system receives conditional input (step 402). As mentioned above, conditional input generally represents one or more attributes of the new image, and examples include one or more encodings of a text sequence, another image, class labels, etc.

[0176] For an implementation where the conditional input includes a text sequence, the system can process the text sequence using each of multiple text encoder neural networks to generate a corresponding encoded representation of the text sequence for each of the text encoder neural networks. The system can then combine the corresponding encoded representations of the text sequence for each of the text encoder neural networks to generate a combined encoded representation. In this case, the denoising input for each backdiffusion step (described below) includes the combined encoded representation.

[0177] As an example, multiple text encoder neural networks can be three neural networks, such as T5-XXL (as described in arXiv:1910.10683), Clip (as described in arXiv:2103.00020), and ByT5 (as described in arXiv:2105.13626).

[0178] The system can combine the corresponding encoded representations of the text sequences for each element in the text encoder neural network to generate a combined encoded representation through any of a variety of methods. For example, the system can generate a combined encoded representation by first concatenating the corresponding encoded representations of the text sequences for each element in the text encoder neural network to generate a concatenated encoded representation. Then, the system processes the concatenated encoded representation using the context encoder neural network to generate the combined encoded representation.

[0179] In some cases, these multiple text encoder neural networks have been trained independently of each other. For example, the text encoder neural networks may each have been trained using different training settings, different training data, different objectives, and so on.

[0180] The system initializes the representation of the new image by sampling noise values ​​from a noise distribution (step 404). In other words, the system uses the noise values ​​of the sampled noise to set the pixel values ​​of the representation of the new image, where each dimension of the sampled noise represents a pixel value. The system can sample noise values ​​from each of various probability distributions—for example, a multivariate Gaussian distribution with isotropic covariance.

[0181] The system updates the representation of the new image at each of the multiple back-diffusion steps (step 406). Specifically, the system uses a denoising neural network to process the denoising input of the back-diffusion step conditioned on the conditional input to generate a denoised output, the denoising input including the representation of the new image, and the denoising output defining an estimate of the noise component of the representation of the new image; the denoising neural network used by the system can be any of the example target denoising neural networks described above. The system then uses the denoising output to update the representation of the new image.

[0182] For example, for a given diffusion step, when the system applies the DDPM diffusion sampler, the system can subtract the estimated noise component of the new image representation from the new image representation (e.g., subtract the estimated noise value of the estimated noise component from the pixel values ​​of the new image representation element-wise), and optionally add a small amount of noise. The result is an updated representation of the new image.

[0183] As another example, for a given diffusion step, when the system applies the DDIM diffusion sampler, the system can use the representation of the new image at the current diffusion step to determine an initial estimate of the final representation of the new image. The system can then use the initial estimate of the final representation of the new image and the representation of the new image at the current diffusion step to generate an updated representation of the new image. The result is the updated representation of the new image.

[0184] After generation, the updated representation of the new image can be included as the denoising input in subsequent diffusion steps.

[0185] As described above, the system uses classifier-less guidance at each back-diffusion step in some cases. When classifier-less guidance is used, the system uses a target denoising neural network but does not conditionally process the denoising input for the back-diffusion step to generate another denoised output, which includes a representation of the new image. The system then combines the conditional and unconditional denoising outputs according to the guidance weights of the back-diffusion step to generate the final denoised output.

[0186] For example, the following equation This indicates how the system can be guided without a classifier at each backdiffusion step, where the term... This indicates the final denoised output. Indicates the guiding weight. The representation of the new image. This represents conditional input, and This indicates the conditional denoising output, and This indicates unconditional noise reduction output.

[0187] Methods for selecting guiding weight hyperparameters to use in classifier-free guidance can include considering a variety of metrics. For example, metrics evaluating the distribution of image features (such as the Fréchet distance (FD-Dino) calculated based on Dino-v2 features or the CLIP maximum mean difference (CMMD)) can be considered alongside other metrics such as FID and CLIP scores, or as alternatives to them. The response curves generated by the resulting images according to these different metrics can be analyzed across different guiding weights. For example, minimizing FD-Dino has been observed to be associated with certain properties such as natural color distribution and object shape, while minimizing CMMD can be associated with properties such as the initial indication of color saturation. Guiding weights can be selected from a range defined by the minimum values ​​of these different metrics (e.g., the range between the minimum FD-Dino and the minimum CMMD). Based on such analysis, example guiding values ​​used in specific implementations are: 1.75 for some SU-derived models and 4.0 for some fully UViT-derived models.

[0188] Generally, the system performs a predetermined number of backdiffusion steps, set by the user, the system, or another system. For example, the system may receive the number of backdiffusion steps from the user.

[0189] In some cases, the denoising input for the backdiffusion step includes a corresponding temporal input. For such cases, the system can determine the number of backdiffusion steps to align, for example, with the number of time steps used to train the denoising neural network, such as the number of time steps sampled to generate a noisy target image from the target image, as described above in Example 300 regarding training the denoising neural network.

[0190] After updating the representation of the new image at each of the plurality of back-diffusion steps, the system generates the new image from the representation of the new image (step 408).

[0191] In some implementations, the system updates the representation of the new image at each of the multiple back-diffusion steps and outputs the representation of the new image as the new image. In other words, the system outputs the latest updated representation of the new image as the new image.

[0192] Before using the denoising neural network to generate a new image using, for example, the backdiffusion process of process 400 above, the system or another system trains the denoising neural network; the term "training system" will refer to either case.

[0193] In some cases, the training system trains the denoising neural network on a first training data consisting of multiple first training images for the denoising target.

[0194] The first training data can include images with any one or more styles, types, or characteristics, such as everyday scenes (e.g., scenes of natural or urban environments), extraordinary scenes (e.g., scenes of fantasy worlds), works of art of any genre, objects with any attributes, animals of any type, and so on. For example, the training system can use the first training data, which includes images randomly crawled from the Internet, to train a denoising neural network.

[0195] The denoising target can be any of various objectives, such as those mentioned above. Figure 3 Any of the example targets described.

[0196] Furthermore, in some cases, the training system trains the denoising neural network on first training data and then trains it on second training data, wherein the second training data is training data generated from a mixed image set, which includes the first training images and multiple second training images classified as aesthetic images. In other words, the training system further trains the denoising neural network using the second training data, which includes aesthetic images.

[0197] Additionally, in some cases, the second training images are a proper subset of the first training images classified by the image classification neural network as conforming to aesthetic and compositional attributes (e.g., "cartoon," "retro," or "sci-fi"). In other words, the training system uses the second training data to fine-tune the denoising neural network, which includes aesthetic images (e.g., image styles) of a specific class determined by the image classification neural network.

[0198] As a specific example, the training system can use an image classification neural network that distinguishes specific aesthetic and compositional attributes (e.g., a classifier neural network that distinguishes between images in a "newspaper cartoon" style and those in a non-"newspaper cartoon" style) to select a proper subset of images from the first training images as the second training images. The training system can then use this second training data to train a denoising neural network. Additionally, the training system can add special lexical units to the text prompts, which are used as conditional inputs when generating denoised output for the second training images during training. By including special lexical units in the text prompts for the second training images, the denoising neural network training will produce a denoising neural network particularly well-suited for generating images with the specific aesthetic and compositional attributes of the second training images. Furthermore, by "fine-tuning" the mixture of the first and second training images, the denoising neural network training can avoid "catastrophic forgetting" (i.e., forgetting previously learned information when learning new data) and the introduction of "artifacts" (i.e., unexpected or undesirable elements appearing in the generated output) during the training process.

[0199] In some cases, after further training the denoising neural network on the second training data, the training system further trains the denoising neural network on the third training data for the consistency distillation objective. In other words, the system uses the denoising neural network to train the consistency model, and then replaces the denoising neural network with the consistency model. The following is an example of this training.

[0200] For example, in some cases, the system trains a consistency neural network that replaces the denoising neural network and generates a new image in fewer back-diffusion steps (i.e., the system uses the consistency neural network to distill the denoising neural network). That is, in some cases, the system uses a consistency distillation objective to train the consistency neural network, which includes a loss across multiple back-diffusion steps between the denoised output of the consistency neural network and the denoised output of the target denoising neural network (i.e., the training examples included in the third training data). The system then replaces the target denoising neural network with a consistency neural network such that the consistency neural network receives a target denoising input comprising a noisy target image with a target resolution and processes the target denoising input to generate a target denoising output that defines an estimate of the noise components of the noisy target image.

[0201] Consistent neural networks can be any of a variety of neural network architectures. That is, a consistent neural network can have any suitable architecture in any appropriate configuration that can process a target denoising input including a noisy target image with a target resolution to generate a target denoising output that defines the noise components of the noisy target image, including fully connected layers, convolutional layers, recurrent layers, attention-based layers, etc., as the case may be.

[0202] Generally speaking, if a denoising neural network is used to generate a new image, it requires... If there are 10 backward diffusion steps, then generating a new image using a consistency neural network will require 10 steps. One reverse diffusion step, making .

[0203] For example, the system can train a consistency neural network that processes the target denoising input to generate a target denoising output that defines an estimate of the noise components of the noisy target image, such that a new image is generated using only 1, 2, 4, or 8 backdiffusion steps, whereas the denoising neural network requires more than 1, 2, 4, or 8 backdiffusion steps to generate a new image.

[0204] As another example, the system can train a consensus neural network comprising multiple subnetworks, each responsible for a separate, consecutive back-diffusion step when generating a new image, with all subnetworks sharing the same parameters. For instance, the system can train a multi-step consensus model as described in arXiv:2403.06807.

[0205] Generally, the goal of consistency distillation is that, after training, the denoised output of the consistency neural network better matches the output of the denoising neural network. For example, the goal of consistency distillation could be... The expected value is taken from the index. Various reverse diffusion steps and various target images For example, the target image in the training data of a denoising neural network; index s also represents the index of the backdiffusion step, making And the index of the diffusion step starts from... to ,in It is the first step, and It's the final step; This is the denoised output of a consistency neural network, which is indexed based on the back-diffusion step. To process the denoising input of the backdiffusion step, which includes the new image The representation of; This is the denoised output of a consistency neural network, which is indexed based on the back-diffusion step. To process the denoising input of the backdiffusion step, which includes the new image The representation, in which the system uses a denoising neural network to generate For example, using a DDIM sampler and a denoising neural network as described above for example process 400 to extract from... generate ,and This indicates that the term is non-differentiable. In some cases, the backdiffusion step... The value is 0 (that is, Corresponding to the final back-diffusion step), in other cases, the back-diffusion step The value is (Right now, This can correspond to any back-diffusion step that is not the last back-diffusion step.

[0206] The system can iteratively update the trainable parameters of the consistency neural network to optimize the consistency distillation objective using any of the following methods: gradient-based methods, evolutionary algorithm-based methods, Bayesian optimization, etc. For example, the system can use any of the following gradient descent techniques to optimize the consistency distillation objective: batch gradient descent, stochastic gradient descent, or mini-batch gradient descent. These gradient descent techniques involve using backpropagation to estimate the gradient of the loss with respect to the trainable parameters of the consistency neural network and updating the trainable parameters accordingly. Generally, the system iteratively updates the trainable parameters until one or more criteria are met (e.g., the system performs a predetermined number of iterations, updates to the trainable parameters no longer exceed a predetermined range of change, metrics on the validation dataset exceed predetermined values, etc.).

[0207] Providing further details about these optional steps, style fine-tuning can involve selecting a subset of training images (e.g., 3k images) based on aesthetic or compositional attributes using an image classifier. The target denoising neural network can then be further trained using a mixture of the original training data and this selected aesthetic subset (e.g., for 8k steps). During this mixed training, specified lexical units can be added to textual cues from the images of the aesthetic subset to guide conditionalization. Regarding distillation, consistency model-based techniques such as the Multi-Step Consistency Model (MCM) described by Heek et al. (2024) can be applied to the trained target denoising neural network. This yields a distilled model that requires fewer sampling steps (e.g., reduced from 256 steps to 16 steps) for image generation.

[0208] Figure 5 Example 500 shows the performance of the described technology.

[0209] Specifically, Example 500 is a table showing how image distribution metrics (i.e., FID or Fréchet Inception Distance as described in arXiv:1706.08500, FD-Dino or Fréchet Distance on Dino-v2 feature space as described in arXiv:2304.07193 and arXiv:2306.04675, CMMD or Clip Maximum Mean Discrepancy distance as described in arXiv:2401.09603, i.e., metrics that quantify the similarity of the statistical distribution of attributes of a group of images) and text-image alignment (i.e., text-image pairings) are evaluated for the MS-COCO image captioning dataset (which covers a wide variety of object categories and scene sets). This performs various subsets of features on the described techniques (Vermeer: ​​original model, + cue engineering, + style tuning, + distillation) and another technique (i.e., SDXL (v1.0)). That is, each row corresponds to one technique, and each column corresponds to a metric evaluated against a set of image-caption pairs, where the captions in the pair are used as a text sequence of conditional input to generate a new image that should correspond to the image in the image-caption pair.

[0210] The described techniques comprise various subsets of elements used to generate images using a denoising neural network, and include: (ii) an original model, which refers to a denoising neural network trained using a first training data set; (ii) a cueing engineering process identical to the original model, but which enhances the conditionalization of the input text sequence to include words to improve the aesthetic image quality of the new image; (iii) style tuning, which fine-tunes the style of the denoising neural network of the original model on a second training dataset comprising a carefully selected subset of 3K aesthetically pleasing images; and (iv) distillation, which trains a consistency model using the denoising neural network of the original model for a consistency distillation objective. Another technique, SDXL, is based on a latent diffusion model (as described in arXiv:2307.01952).

[0211] Example 500 shows that the original model minimizes image distribution metrics in the feature space using state-of-the-art features—namely FD-Dino and CMMD. Unlike FID, which relies on features from earlier Inception neural networks, FD-Dino and CMMD capture more subtle and harder-to-recognize features about the structure of objects and scenes in an image. Therefore, the better performance of FD-Dino and CMMD compared to FID on the MS-COCO dataset is significant because the newer metrics evaluate image quality in a semantically more meaningful space. Furthermore, since MS-COCO consists of a wide variety of real-world images captured using various cameras and sensors in diverse real-world environments, these low scores indicate that the statistical distribution of features in the images generated by the original model is closely aligned with the distribution found in this large corpus of real-world images, suggesting strong fidelity to real-world visual properties. In other words, images generated using the described techniques are more faithful to the semantic properties of real-world objects and scenes, indicating that images generated using the described techniques are highly realistic.

[0212] These metrics also highlight a significant deviation from the MSCOCO-commentary distribution after enhanced cues (+cue engineering), a deviation further amplified when combined with fine-tuning of the denoising neural network for aesthetically pleasing images (+style fine-tuning). This deviation from the MSCOCO-commentary distribution of +cue engineering and +style fine-tuning is intentional, as images generated using these techniques improve aesthetic appeal rather than fidelity to the MSCOCO image. An assessment of the improvement in aesthetic appeal is provided below. Figure 7 .

[0213] Example 500 also shows a CLIP score, which indicates a slight decrease in text-to-image alignment compared to SDXL. However, the CLIP score remains relatively consistent with the described technique, suggesting that the core subnetwork maintains most of the text-to-image alignment capability of the image generation system (i.e., its conditional input-to-image alignment).

[0214] Figure 6 Example 600 shows the performance of the described technology.

[0215] Specifically, Example 600 is a table illustrating how various subsets of features apply to the text-image alignment metric Davidsonian Scene Graph (DSG)-1k (as described in arXiv:2310.18235), an open-source evaluation benchmark including 1,060 cues, covering a wide range of fine-grained semantic categories with balanced distribution, an empirical evaluation framework inspired by formal semantics to perform the described techniques (Vermeer: ​​original model, +cue engineering, +style tuning, +distillation), and another technique (i.e., SDXL (v1.0), SD2.1, Muse, Imagen Cascade). Higher DSG scores indicate better performance of the technique.

[0216] Example 600 shows that the described technique presents performance that competes with SDXL and outperforms other techniques, including autoregressive techniques (i.e., Muse) and cascaded model techniques (i.e., Imagen Cascade).

[0217] Figure 7 Example 700 shows the performance of the described technology.

[0218] Specifically, Example 700 is a manually evaluated Likert scale plot of an image generated using the described technique (i.e., Vermeer) or by utilizing another technique (i.e., SDXL).

[0219] Human evaluation involved requiring human annotators to evaluate images generated by Vermeer and SDXL based on the same prompts. For this purpose, 495 prompts covering a range of skills were collected², including 160 from TIFA v1.0 designed to measure the faithfulness of the generated image to its text input covering 12 categories (objects, attributes, counts, etc.); 200 prompts sampled from 1600 Parti prompts, selecting those based on both the complexity and diversity of the challenge; and an additional 150 prompts either newly created for challenging cases or derived from more recent prompt strategies for challenging cases. Annotators were asked to consider image quality (y-axis aesthetics), fit to the prompts (y-axis consistency), or both (y-axis synthesis), and to use a 3-point Likert scale to indicate their preference: Vermeer preferred, uncertain, and SDXL preferred (model names have been anonymized). Neutral responses included cases where both images were equally good or equally bad. In the annotation user interface, annotators were shown the prompts along with two randomly arranged images. Each prompt was evaluated based on two aspects: aesthetics and consistency. Thirteen human scores were collected (26 scores per image).

[0220] Example 700 shows that annotators preferred Vermeer 44% of the time, while they chose SDXL 21.4% of the time, and the proportion choosing the neutral response was relatively lower (34.7%). Vermeer was significantly more favored due to its aesthetics, with a win rate of 61.4%, while the consistency gap between the two models was small, with a win rate difference of only 1.7%. Krippendorff's α was 0.27 for aesthetics and 0.41 for consistency, indicating moderate consistency among annotators.

[0221] In this specification, the term "configured" is used in connection with computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or combinations thereof that enable it to perform specific operations or actions during operation. For example, configuring a system may involve installing a software library with specific algorithms, updating firmware with new instructions to process data, or adding hardware components to gain enhanced processing power. Similarly, one or more computer programs are "configured" to perform those intended operations or actions when they contain instructions that, when executed by a computing device or hardware, cause the device to perform specific operations or actions.

[0222] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuit systems, software, firmware, computer hardware (covering the disclosed structures and their equivalents), or any combination thereof. The subject matter can be implemented as one or more computer programs, which are essentially modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a computing device or hardware or for controlling the operation of a computing device or hardware. The storage medium can be a storage device such as a hard disk drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions can be encoded on transmitted signals such as machine-generated electrical, optical, or electromagnetic signals, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations can leverage emerging technologies such as quantum computing or neuromorphic computing for specific applications and can be deployed in distributed or cloud-based environments, where components reside on different machines or within a cloud architecture.

[0223] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, computing devices or hardware may also include code that creates the execution environment for computer programs. This code can take the form of processor firmware, protocol stacks, database management systems, operating systems, or combinations of these elements. In a general-purpose computing on a graphics processing unit (GPGPU) scenario, embodiments can particularly benefit from leveraging the parallel processing capabilities of the GPU, employing code specifically designed for GPU execution, often referred to as kernels or shaders. Similarly, TPUs excel at running optimized tensor operations crucial to many machine learning algorithms. By utilizing these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in fields such as computer vision, natural language processing, and robotics.

[0224] Computer programs (also known as software, applications, modules, scripts, code, or simply programs) can be written in any programming language, including compiled or interpreted languages, as well as declarative or procedural languages. They can be deployed in various forms, such as standalone programs, modules, components, subroutines, or any other unit suitable for use in a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., a script within a markup language document), residing in a dedicated file, or distributed across multiple coordination files (e.g., files storing modules, subroutines, or code segments). Computer programs can execute on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected via data communication networks. The specific implementation of a computer program can involve a combination of traditional programming languages ​​and specialized languages ​​or libraries designed for GPGPU programming or TPU utilization, depending on the hardware platform chosen and the desired performance characteristics.

[0225] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is generally implemented as one or more software modules or components installed on one or more computers, which may be located at a single site or distributed across multiple locations. In some cases, one or more dedicated computers may be used for a particular engine, while in others, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions in the context of AI and machine learning include data preprocessing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of an engine will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0226] The processes and logic flows described in this specification can be executed by one or more programmable computers, which run one or more computer programs to perform functions by manipulating input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be used to implement aspects of these processes and logic flows concurrently, significantly accelerating execution. This approach offers significant advantages for computationally intensive tasks common in AI and machine learning applications, such as matrix multiplication, convolution, and other operations exhibiting high parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedup and efficiency gains can be achieved compared to relying solely on CPUs. Alternatively, or in combination with programmable computers and dedicated processors, these processes and logic flows can also be implemented using dedicated processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), to achieve higher performance or energy efficiency in specific use cases.

[0227] Computers capable of executing computer programs can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators typically work in conjunction with the CPU to handle specialized computations, while the CPU manages overall system operation and other tasks. Generally, the CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The components of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of the processing unit and memory will depend on factors such as the complexity of the AI ​​model, the amount of data being processed, and the desired performance and latency requirements. Implementations can be carried out on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. Systems may include storage devices such as hard disks, SSDs, or flash memory for persistent data storage.

[0228] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and storage devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or durability.

[0229] To facilitate user interaction, embodiments of the subject matter described herein can be implemented on computing devices equipped with display devices (such as liquid crystal displays (LCDs) or organic light-emitting diode (OLED) displays) for presenting information to users. Input can be provided by the user through various means, including keyboards, touchscreens, voice commands, gesture recognition, or other input modalities, depending on the specific device and application. Additional input methods may include sound, voice, or tactile input, while feedback to the user may take the form of visual, auditory, or tactile feedback. Furthermore, the computer can interact with the user by exchanging documents with the user's device or application. This may involve sending web content or data in response to a request, or sending and receiving text messages or other forms of messages via mobile devices or messaging platforms. The choice of input and output modalities will depend on the specific application and the desired form of user interaction.

[0230] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX. These frameworks provide comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0231] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These components may include: backend components, such as backend servers or cloud-based infrastructure; optional middleware components, such as middleware servers or application programming interfaces (APIs), for facilitating communication and data exchange; and frontend components, such as client devices having a user interface through which users can interact with the implemented subject matter, a web browser, or an app. For example, the described functionality may be implemented only on the client device (e.g., for on-device machine learning) or deployed as a combination of frontend and backend components for more complex applications. Where present, these components may be interconnected using any form or medium of digital data communication, such as communication networks like local area networks (LANs) or wide area networks (WANs), including the Internet. The specific system architecture and component selection will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0232] Computing systems can include geographically separated clients and servers that interact via communication networks. The specific type of network (such as a local area network (LAN), wide area network (WAN), or the Internet) will depend on the accessibility and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. Depending on the nature of the exchanged data and the system's security requirements, these protocols may include HTTP, TCP / IP, or other specialized protocols. In some embodiments, the server transmits data or instructions to a user device acting as a client, such as a computer, smartphone, or tablet. The client device can then process the received information, display the results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interaction between the user and the system, enabling a wide range of applications and functionalities.

[0233] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0234] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to be performed to achieve the desired result. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0235] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Obtaining data, the data specifying a trained initial denoising neural network, the trained initial denoising neural network being configured to: receive an initial denoising input, the initial denoising input including a noisy initial image having an initial resolution; and process the initial denoising input to generate an initial denoising output, the initial denoising output defining an estimate of the noise components of the noisy initial image, wherein the initial denoising neural network includes: (i) An initial encoder subnetwork configured to map the noisy initial image to an initial representation of the noisy initial image with core resolution; (ii) A core subnetwork configured to process the initial representation having the core resolution to generate an updated representation having the core resolution; and (iii) An initial decoder subnetwork configured to map the updated representation having the core resolution to the initial denoised output; Obtain training data, the training data including multiple training examples, each training example including a corresponding target image, the target image having a target resolution higher than the initial resolution; and The training data is used to train a target denoising neural network for a denoising target, wherein the target denoising neural network is configured to: receive a target denoising input, the target denoising input including a noisy target image having the target resolution; and process the target denoising input to generate a target denoising output, the target denoising output defining an estimate of the noise components of the noisy target image, and wherein the target denoising neural network includes: (i) A target encoder subnetwork configured to map the noisy target image to an initial representation of the noisy target image having the core resolution; (ii) the core subnetwork; and (iii) A target decoder subnetwork configured to map an updated representation of the noisy target image, having the core resolution and generated by the core subnetwork, to the target denoising output. Training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while keeping the parameters of the core subnetwork fixed at the corresponding values ​​in the initial denoising neural network.

2. The method of claim 1, wherein the core resolution is smaller than the initial resolution and the target resolution.

3. The method of claim 1, wherein the core subnetwork comprises a plurality of self-attention layers.

4. The method of claim 1, wherein: The initial denoising input and the target denoising input each include corresponding conditional inputs, and Each training example further includes a corresponding training conditional input, which characterizes one or more attributes of the corresponding target image in the training example.

5. The method of claim 4, wherein the core subnetwork comprises one or more conditional layers, each conditional layer updating the input representation of the conditional layer conditionally based on at least the representation of the corresponding conditional input.

6. The method of claim 5, wherein the one or more conditional layers are cross-attention layers.

7. The method of claim 5, wherein the corresponding conditional input comprises a corresponding encoded representation of each of one or more context inputs, the one or more context inputs comprising one or more embeddings.

8. The method of claim 7, wherein the initial denoising neural network and the target denoising neural network each comprise a context encoder neural network configured to process the corresponding encoded representation of each of the one or more context inputs to generate a representation of the conditional input.

9. The method of claim 8, wherein training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while keeping the parameters of the core subnetwork and the context encoder neural network fixed at the corresponding values ​​in the initial denoising neural network.

10. The method of claim 8, wherein the context encoder neural network comprises an attention pooling subnetwork followed by a multilayer perceptron (MLP).

11. The method of claim 5, wherein the initial denoising input and the target denoising input each include a corresponding temporal input, the temporal input specifying a time step that defines the noise level of the corresponding noisy image, and One or more of the conditionalization layers are conditional on the representation of the corresponding time input.

12. The method of claim 11, wherein the initial denoising neural network and the target denoising neural network each comprise a time encoder neural network configured to process the respective time input to generate a representation of the respective time input.

13. The method of claim 12, wherein training the target denoising neural network includes training the target encoder subnetwork and the target decoder subnetwork, while updating the parameters of the time encoder neural network starting from the corresponding values ​​in the initial denoising neural network.

14. The method of claim 1, wherein: The initial encoder subnetwork includes: a downsampled convolutional subnetwork configured to map the noisy initial image to a downsampled representation having the core resolution; and a convolutional encoder subnetwork configured to map the downsampled representation to the initial representation of the noisy initial image having the core resolution.

15. The method of claim 1, wherein: The initial decoder subnetwork includes: a convolutional decoder neural network configured to map the updated representation to a decoded representation having the core resolution; and an upsampled convolutional subnetwork configured to map the decoded representation having the core resolution to the initial denoised output.

16. The method of claim 1, wherein obtaining data for the specified trained initial denoising neural network comprises: The initial denoising neural network is trained using initial training data for the denoising target, the initial training data including a plurality of initial images having the initial resolution.

17. The method of claim 5, further comprising: After training the target denoising neural network, a backdiffusion process is performed using the target denoising neural network to generate a new image with the target resolution.

18. The method of claim 17, wherein performing a backdiffusion process using the target denoising neural network to generate a new image with the target resolution includes performing the backdiffusion process conditionally with a new conditional input.

19. A method executed by one or more computers, the method comprising: Receive conditional input; The representation of the new image is initialized by sampling noise values ​​from the noise distribution; as well as The representation of the new image is updated at each of the multiple back-diffusion steps, the update including at each back-diffusion step: A denoising neural network is used to process the denoising input of the backdiffusion step conditioned on the conditional input to generate a denoising output, the denoising input including the representation of the new image, the denoising output defining an estimate of the noise component of the representation of the new image, wherein the denoising neural network is the target denoising neural network as claimed in claim 5. as well as The denoised output is used to update the representation of the new image; as well as After updating the representation of the new image at each of the plurality of back-diffusion steps, the new image is generated from the representation of the new image.

20. The method of claim 19, wherein the conditional input comprises a text sequence, and wherein the method further comprises: The text sequence is processed using each of a plurality of text encoder neural networks to generate a corresponding encoded representation of the text sequence for each of the text encoder neural networks; as well as The corresponding encoded representations of the text sequences for each of the text encoder neural networks are combined to generate a combined encoded representation, wherein the denoising input for each backdiffusion step includes the combined encoded representation.

21. The method of claim 20, wherein combining the corresponding encoded representations of the text sequences of each of the text encoder neural networks to generate a combined encoded representation comprises: The corresponding encoded representations of the text sequences for each of the text encoder neural networks are concatenated to generate a concatenated encoded representation; as well as A field encoder neural network is used to process the encoded representation of the connection to generate the encoded representation of the combination.

22. The method of claim 20, wherein the plurality of text encoder neural networks have been trained independently of each other.

23. The method of claim 20, wherein the denoising neural network has been trained on first training data comprising a plurality of first training images for a denoising target.

24. The method of claim 23, wherein after training on the first training data, the denoising neural network has been further trained on second training data, the second training data being generated from a mixed image set, the mixed image set including the first training images and a plurality of second training images that have been classified as aesthetic images.

25. The method of claim 24, wherein the second training image is a proper subset of the training images that have been classified by the image classification neural network as conforming to aesthetic and compositional attributes.

26. The method of claim 24, wherein after further training on the second training data, the denoising neural network has been further trained on the third training data for a uniform distillation target.

27. The method of claim 20, wherein generating the new image comprises outputting the representation of the new image as the new image after it has been updated at each of the plurality of back-diffusion steps.

28. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to perform operations as described in any of the preceding claims.

29. One or more computer storage media, said computer storage media being encoded with instructions that, when executed by said one or more computers, cause said one or more computers to perform the operation of the corresponding method as claimed in any one of claims 1 to 27.

Images