Generating Images Using a Sequence of Generative Neural Networks

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for generating an image. In one aspect, the method includes receiving an input text prompt including a sequence of text tokens in a natural language, processing the input text prompt using a text encoder neural network to generate a set of context embeddings for the input text prompt, and processing the context embeddings through a sequence of a generative neural network to generate a final output image that depicts a scene described by the input text prompt.

Description

[Technical Field]

[0001] This specification relates to processing images using neural networks. [Background technology]

[0002] A neural network is a machine learning model that utilizes one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Colin Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140), 2020 [Non-patent document 2] Alexander Quinn Nichol and Prafulla Dhariwal, "Improved denoising diffusion probabilistic models", International Conference on Machine Learning, PMLR, 2021 [Non-patent document 3] Jonathan Ho, Ajay Jain, and Pieter Abbeel, "Denoising Diffusion Probabilistic Models", NeurIPS, 2020 [Non-patent document 4] Jiaming Song, Chenlin Meng, and Stefano Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 (2020) [Non-patent document 5] Tim Salimans and Jonathan Ho, "Progressive Distillation for Fast Sampling of Diffusion Models," ICLR, 2022 [Non-patent document 6] Chenlin Meng et al., “On distillation of guided diffusion models,” arXiv preprint arXiv:2210.03142 (2022) [Non-Patent Document 7] Karras, Tero et al., “Elucidating the design space of diffusion-based generative models,” arXiv preprint arXiv:2206.00364 (2022) Summary of the Invention [Means for solving the problem]

[0004] This specification describes an image generation system implemented as a computer program on one or more computers at one or more locations that uses a text encoder neural network and a sequence of generative neural networks to generate images from conditioned inputs. While the following description describes conditioned inputs in the form of text prompts (or a set of contextual embeddings of text prompts), in other implementations, the conditioned inputs can be different types of data, such as noise inputs sampled from a noise distribution, existing images, embeddings of existing images, video, video embeddings, numerical representations of desired object categories for the image, audio signals characterizing a scene the image should depict, audio signals containing speech describing the image, embeddings of audio signals, combinations of these, etc. The methods and systems disclosed herein can be applied to any conditioned image generation problem to generate high-resolution images.

[0005] In one aspect, a method implemented by one or more computers is provided. The method includes receiving an input text prompt including a string of text tokens in a natural language, processing the input text prompt using a text encoder neural network to generate a set of context embeddings for the input text prompt, and processing the context embeddings through a string of generative neural networks to generate a final output image depicting a scene described by the input text prompt. The string of generative neural networks includes an initial generative neural network and one or more subsequent generative neural networks. The initial generative neural network is configured to receive the context embeddings and process them to generate as output an initial output image having an initial resolution. The one or more subsequent generative neural networks are each configured to receive a respective input including (i) the context embedding and (ii) a respective input image having a respective input resolution and generated as output by a preceding generative neural network in the string, and to process the respective input to generate as output a respective output image having a respective output resolution higher than the respective input resolution.

[0006] In some implementations of the method, the text encoder neural network is a self-attention encoder neural network.

[0007] In some implementations of the method, the generative neural networks in the sequence are jointly trained on a set of training examples, each including (i) a respective training text prompt and (ii) a respective ground truth image depicting a scene described by the respective training text prompt, and the text encoder neural network is pre-trained and kept frozen during the joint training of the generative neural networks in the sequence.

[0008] In some implementations of the method, each generative neural network in the sequence is a diffusion-based generative neural network.

[0009] In some implementations of the method, the diffusion-based generative neural network uses classifier-free induction.

[0010] In some implementations of the method, for each subsequent diffusion-based generative neural network, processing each input to generate a respective output image as an output includes sampling a latent image having a respective output resolution and denoising the latent image across the sequence of steps into a respective output image. Denoising the latent images across the sequence of steps includes, for each step that is not the last step in the sequence of steps, receiving a latent image for that step, processing each input and latent image for that step to generate an estimated image for that step, dynamically thresholding pixel values ​​of the estimated image for that step, and generating a latent image for the next step using the estimated image for that step and randomly sampled noise.

[0011] In some implementations of the method, denoising the latent image across the sequence of steps includes, for a last step in the sequence of steps, receiving a latent image for the last step, and processing each input and latent image for the last step to generate a respective output image.

[0012] In some implementations of the method, processing each input and latent image for the step to generate an estimated image for the step includes: resizing each input image to generate a respective resized input image having a respective output resolution; concatenating the latent image for the step with each resized input image to generate a concatenated image for the step; and processing the concatenated image for the step with mutual attention to context embedding to generate an estimated image for the step.

[0013] In some implementations of the method, dynamically thresholding pixel values ​​of the estimated image for the step includes determining a limiting threshold based on pixel values ​​of the estimated image for the step, and thresholding pixel values ​​of the estimated image for the step using the limiting threshold.

[0014] In some implementations of the method, determining the limiting threshold based on pixel values ​​of the estimated image for that step includes determining the limiting threshold based on absolute pixel values ​​of a particular percentile in the estimated image for that step.

[0015] In some implementations of the method, thresholding pixel values ​​of the estimated image for that step using a limiting threshold includes limiting pixel values ​​of the estimated image for that step to a range defined by [−κ,κ], where κ is the limiting threshold.

[0016] In some implementations of the method, thresholding the pixel values ​​of the estimated image for that step using the limiting threshold further includes, after limiting the pixel values ​​of the estimated image for that step, dividing the pixel values ​​of the estimated image for that step by the limiting threshold.

[0017] In some implementations of the method, each subsequent generative neural network applies noise conditioning augmentation to the respective input image.

[0018] In some implementations of the method, the final output image is the output image of each of the last generating neural networks in the sequence.

[0019] In some implementations of the method, each subsequent generative neural network receives a respective k×k input image and generates a respective 4k×4k output image.

[0020] In a second aspect, a method implemented by one or more computers is provided. The method includes sampling a noise input from a noise distribution and processing the noise input through a train of generative neural networks to generate a final output image. The train of generative neural networks includes an initial generative neural network and one or more subsequent generative neural networks. The initial generative neural network is configured to receive the noise input and process the noise input to generate as output an initial output image having an initial resolution. The one or more subsequent generative neural networks are each configured to receive a respective input including (i) the noise input and (ii) a respective input image having a respective input resolution and generated as output by a preceding generative neural network in the train, and process the respective input to generate as output a respective output image having a respective output resolution that is higher than the respective input resolution.

[0021] In some implementations of the method, each generative neural network in the sequence is a diffusion-based generative neural network.

[0022] In some implementations of the method, the diffusion-based generative neural network uses classifier-free induction.

[0023] In a third aspect, a system is provided, the system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform any of the methods described above.

[0024] In a fourth aspect, a system is provided that includes one or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform any of the methods described above.

[0025] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0026] The described image generation system can generate high-resolution images that depict scenes described by text prompts. That is, the image generation system can effectively generate high-resolution images accurately titled by the text prompts. By utilizing a series (or "cascade") of generative neural networks (GNNs), each conditioned on a text prompt, the system can iteratively increase the resolution of the image, ensuring that high-resolution images can be generated without a single neural network having to directly generate the image at the desired output resolution. Cascading GNNs in this manner can greatly improve their sample quality while compensating for any artifacts that occur at lower resolutions, such as distortion, checkerboard artifacts, etc.

[0027] Due to the modular nature of the system, this iterative improvement procedure can be used to generate high-fidelity images at any desired resolution. The system can utilize any suitable number of GNNs, each implementing any suitable type of generative model and each having any suitable number of neural network layers, network parameters, and / or hyperparameters, to generate images at the desired resolution. In addition to improved performance during inference, the modularity of the system also realizes significant benefits during training. For example, the training engine can jointly train a sequence of GNNs in parallel, which facilitates advanced optimization and reduced training time. That is, each GNN in the sequence can be independently optimized by the training engine to impart certain properties to the GNN, such as specific output resolution, fidelity, perceptual quality, efficient decoding (or denoising), fast sampling, reduced artifacts, etc.

[0028] To achieve high-fidelity text-to-image synthesis with high-quality text-to-image alignment, the system can use a pre-trained text encoder neural network to process text prompts and generate a set (or sequence) of contextual embeddings of the text prompts. The text prompts can describe scenes (e.g., as a sequence of text tokens in natural language), and the contextual embeddings can represent the scenes in a computationally modifiable form (e.g., as a set or vector of numbers, alphanumeric values, symbols, or other coded representations). The training engine can also keep the text encoder frozen as the GNN sequence is trained to improve the alignment between the text prompts and the images generated during inference. A frozen text encoder can be particularly effective because, for example, the text encoder's bias toward specific scenes described by text-image training pairs can enable the GNN sequence to perform deep learning of linguistic encodings of scenes, which may not be feasible if the text encoders are trained in parallel. Furthermore, text-based training sets are generally richer and more sophisticated than currently available text-image training sets, which allows text encoders to be pre-trained and then implemented in a highly optimized manner. See, for example, the T5 text encoder provided by Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." JMLR, 21(140), 2020. Freezing the text encoder has several other advantages, such as offline computation of context embeddings, resulting in a negligible computational or memory footprint during training. In some implementations, the training engine fine-tunes the pre-trained text encoder after the GNN columns are trained, which may potentially enable even better text-to-image alignment.

[0029] The system can process context embeddings using a sequence of GNNs to generate a final output image depicting a scene described by the text prompt. Specifically, a first GNN can receive the context embedding of the text prompt. The first GNN can process the context embedding to generate an initial output image having an initial resolution. For example, the initial output image can be generated by the first GNN at a relatively low resolution (e.g., 64×64 pixels). The initial output image can be iteratively processed by each subsequent GNN in the sequence to generate respective output images with increasing resolution until a final output image having a desired final resolution is obtained. For example, the final output image can be generated by the last GNN in the sequence at a relatively high resolution (1024×1024 pixels). More specifically, each subsequent GNN can receive a respective input including the context embedding and a respective input image generated as output by the preceding GNN in the sequence, and process the respective input to generate a respective output image having a higher resolution than the respective input image. For example, the system may use a base image generation model for a first GNN and a super-resolution model for a subsequent GNN to increase the resolution of the output image relative to the input image. In some cases, the subsequent GNN may apply noise-conditioned augmentations to its input image, which slightly corrupt the input image. This may allow the subsequent GNN to correct errors and / or artifacts that the preceding GNN may have produced. The system may also provide a signal to the subsequent GNN that specifies the magnitude of the conditioned augmentations to be applied to its input image.

[0030] Each GNN in the train of GNNs may have any suitable neural network architecture that enables it to perform its described function, i.e., process a set of text prompts and / or context embeddings of each input image to generate a respective output image. Specifically, a GNN may include any suitable type of neural network layers (e.g., fully connected layers, convolutional layers, self-attention layers, etc.), connected in any suitable number (e.g., 5, 25, or 100 layers) and in any suitable configuration (e.g., as a linear series of layers).

[0031] In some implementations, the image generation system uses a diffusion-based model for each of the GNNs, although any combination of generative models, such as variational autoencoders (VAEs), generative adversarial networks (GANs), etc., may be utilized by the system. Diffusion models may be particularly useful in the context of modular systems due to their controllability and scalability. For example, compared to some generative models, diffusion models can be efficiently trained by a training engine for computationally tractable objective functions on a given training dataset. These objective functions may be simply optimized by the training engine to increase the speed and performance of the diffusion-based GNNs (DBGNNs) and to enable techniques such as classifier-less guidance and progressive distillation to further improve performance.

[0032] Among other aspects, this specification describes a method for extending an image generation system as a high-resolution text-to-image model. In DBGNN, v-parameterization can be implemented by the system for stability and to facilitate gradual distillation combined with classifier-free guidance for fast, high-quality sampling. This image generation system is not only capable of generating high-fidelity images, but also has a high degree of controllability and world knowledge, including the ability to generate diverse images and text in various artistic styles.

[0033] The image generation system described herein may be implemented in any suitable location, such as on a user device (e.g., a mobile device) or on one or more computers in a data center. The modularity of the image generation system allows multiple devices to implement individual components of the system separately from each other. Specifically, different GNNs in a sequence can run on different devices and can send their output and / or input to each other (e.g., via remote communication). As an example, a text encoder and a subset of the GNN may be implemented on a client device (e.g., a mobile device), and the remainder of the GNN may be implemented on a remote device (e.g., in a data center). The client device can receive input (e.g., a text prompt) and process the text prompt using the text encoder and the subset of the GNN to generate an output image of a particular resolution. The client device can then send the output (e.g., an output image and a set of context embeddings of the text prompt) that is received at the remote device as input. The remote device can then process the input using the remainder of the GNN to generate a final output image having a higher resolution than the received image.

[0034] A user can interact with the image generation system by providing input to the image generation system, for example, through an interface, e.g., a graphical user interface or an application programming interface (API). Specifically, the user can provide input including (i) a request to generate an image and (ii) a prompt (e.g., a text prompt) that describes the content of the image to be generated. In response to receiving the input, the image generation system can generate an image responsive to the request and provide the image to the user, for example, for display on the user's user device or for storage in a data storage device. In some cases, the image generation system can transmit the generated image to the user's user device, for example, over a data communications network (e.g., the Internet).

[0035] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0036] [Figure 1A] FIG. 1 is a block diagram of an exemplary image generation system capable of generating images from text prompts. [Figure 1B] 1 is a flow diagram of an exemplary process for generating an image from a text prompt. [Figure 2A] FIG. 1 is a block diagram of an example generative neural network column. [Figure 2B] 1 is a flow diagram of an example process for processing a set of context embeddings using a column of a generative neural network. [Figure 3A] FIG. 1 is a block diagram of an example training engine capable of jointly training columns of a generative neural network. [Figure 3B] 1 is a flow diagram of an exemplary process for jointly training columns of a generative neural network. [Figure 4] FIG. 1 is a block diagram of an exemplary U-Net architecture. [Figure 5A] FIG. 10 is a block diagram of an example ResNetBlock for the Efficient U-Net architecture. [Figure 5B] FIG. 1 is a block diagram of an example DBlock for the Efficient U-Net architecture. [Figure 5C] FIG. 1 is a block diagram of an exemplary UBlock for the Efficient U-Net architecture. [Figure 5D] FIG. 1 is a block diagram of an exemplary Efficient U-Net architecture implemented as a super-resolution model. [Figure 6A] FIG. 1 is a block diagram of an exemplary image generation system capable of generating an image from noise. [Figure 6B] 1 is a flow diagram of an exemplary process for generating an image from noise. [Figure 6C] 1 is a flow diagram of an exemplary process for processing a noisy input using a column of generative neural networks. [Figure 7] 1A-1C illustrate various images generated from text prompts by an image generation system. DETAILED DESCRIPTION OF THE INVENTION

[0037] Like reference numbers and designations in the various drawings indicate like elements.

[0038] This paper presents an image generation system that combines the power of a text encoder neural network (e.g., a large-scale language model (LLM)) with a series of generative neural networks (e.g., diffusion-based models) to achieve text-to-image generation with a high degree of photorealism, fidelity, and deep language understanding. In contrast to previous work that primarily uses image-text data for model training, the proposal described here is that context embeddings from a text encoder pre-trained on a text-only corpus are effective for text-to-image generation.

[0039] The example image generation system also demonstrates numerous advantages and insights regarding generative image modeling and generative modeling as a whole, including but not limited to the following: 1. The effectiveness of trains or "cascades" of generative neural networks (e.g., diffusion-based models) for generating high-resolution images. 2. Effectiveness of conditioning and classifier-free induction of text encoder freezing in diffusion-based models. 3. The effectiveness of a new diffusion-based sampling technique, called dynamic thresholding, for generating photorealistic and detailed images. 4. Additional design choices such as v-prediction parameterization and gradual distillation for guided diffusion models. 5. Several neural network architecture choices, including a new architecture called Efficient U-Net, which is fast-converging and memory-efficient.

[0040] As described below, an image generation system uses a train of generative neural networks (GNNs) to progressively increase the resolution of images depicting scenes described by text prompts. In this way, the system can generate images that closely match the distribution of natural and / or other images. For example, the train of GNNs can model a joint distribution across images of multiple resolutions conditioned on text prompts (or other conditioning inputs), which is based on the distribution of images in a training set used to train the train of GNNs (described in more detail below).

[0041] As used herein, the term “scene” generally refers to any collection of one or more objects or general “things,” which may or may not be interacting in some way. For example, a scene may include multiple objects interacting with each other in an environment, such as “a strawberry diving into a mug of coffee under a starry sky,” or “a brain aboard a spaceship bound for the moon,” or “a mug of strawberries filled with white sesame seeds, floating in a sea of ​​dark chocolate.” A scene may also include a single object without a background or backdrop or with a single-color background or backdrop, such as “a studio-taken photograph of a tiny, kinetic sculpture made from thin wire shaped like a bird, with a white background.” A scene may also include text or abstract art, such as colors, shapes, and lines, such as “a blue frame forming the word ‘Imagen.’” As shown in FIG. 7, the types of scenes that can be depicted in an image and described by a text prompt are diverse and can range from real-world environments to abstractions. Note that the text prompt does not have to explicitly depict all of the objects in the scene. For example, the text prompt may describe the mood the scene should evoke, such as "Happiness is a sunny day," or "Fear of the unknown." In general, the text prompt may include any text, regardless of whether it describes a visual characteristic.

[0042] When referring to an image, the term “resolution” generally refers to the spatial resolution of the image, i.e., how close two lines can be to each other while remaining visually distinguishable. That is, how close two lines can be to each other without appearing as a single line in the image. In some implementations, resolution may be specified in terms of pixel resolution, which in this case corresponds to the number of distinct pixels per unit length (or per unit area) for the image, not necessarily the total number of pixels per unit length (or per unit area) for the image. Specifically, a first image may have a higher number of pixels than a second image but still have a lower resolution than the second image. For example, simply upsampling the pixels of an image increases the number of pixels but does not increase the resolution. In general, relative length scales are also considered to have a clear comparison of resolution between images. For example, a digital image with 2048 x 1536 distinct pixels may appear to have a low resolution (approximately 72 pixels per inch (ppi)) when viewed on a 28.5-inch wide screen, but may appear to have a high resolution (approximately 300 ppi) when viewed on a 7-inch wide screen. Relative length scale generally refers to the length scale at which an image is viewed (e.g., on a display), and not necessarily the length scale of the scene depicted in the image. For example, an image depicting the motion of planets and an image depicting the motion of atoms may have different length scales in their respective scenes, but may have the same relative length scale when viewed.

[0043] 1A shows a block diagram of an exemplary image generation system 100. Image generation system 100 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0044] At the top level, the image generation system 100 includes a text encoder neural network 110, a sequence of generative neural networks (GNNs) 121, and, in some implementations, a post-processor 130. The system 100 is configured to receive a text prompt 102 as input and to generate a final image 108 as output.

[0045] More specifically, the system 100 can receive a text prompt (T) 102 that describes a scene. The text prompt 102 is generated by a set of multiple text tokens T in a natural language T=(T1, T2, ...). 1,2,… For example, as shown in FIG. 1A , the text prompt 102 may include a text string containing a "dragon fruit wearing a karate belt in the snow." In general, the text prompt 102 can describe any particular scene, and the system 100, when properly trained (e.g., by a training engine), is capable of generating high-resolution images that faithfully depict the scene. The text prompt 102 may also include text modifiers, such as "smooth," "studio lit," "pixel art," "in the style of Van Gogh," etc., that impart various styles, modifications, and / or characteristics to the final image 108 generated by the system 100. Moreover, the system 100 can generate a variety of different types of images, such as three-dimensional (3D) images, photorealistic images, cartoon images, abstract visualizations, point cloud images, medical images of various modalities, etc. For example, the system 100 can generate medical images, including, but not limited to, magnetic resonance imaging (MRI) images, computed tomography (CT) images, ultrasound images, x-ray images, etc.

[0046] The text encoder 110 is configured to process the text prompt 102 to generate a set of context embeddings (u) of the text prompt 102. In some implementations, the text encoder 110 is a pre-trained natural language text encoder, e.g., a T5 text encoder such as T5-XXL, CLIP text encoder, or large-scale language model (LLM), among others. For example, the text encoder 110 may be a self-attention encoder, such as a Transformer model, including a self-attention layer followed by a perceptron layer. The context embeddings 104 may also be referred to as encoded representations of the text prompt 102, providing a computationally modifiable representation for processing by the system 100. For example, the context embeddings 104 may be a set, vector, or array of values ​​(e.g., in UNICODE or Base64 encoding), alphanumeric values, symbols, or any convenient encoding.

[0047] The array of GNNs 121 includes a plurality of GNNs 120, each configured to receive a respective input (c). Each GNN 120 generates a respective output image

number

[0048] The system 100 processes the context embeddings 104 through a sequence 121 to generate a high-resolution output image 106 with few (if any) artifacts. The output image 106 is typically the final output image, i.e., the output image of each of the last GNNs in the sequence 121, but more generally, it can be provided by any GNN 120 in the sequence 121.

[0049] In some implementations, the output image 106 is further processed by a post-processor 130 to generate a final image (x) 108. For example, the post-processor 130 can perform transformations on the output image 106, such as image enhancement, motion blur, filtering, luminance, lens flare, brightening, sharpening, contrast, among other image effects. Some or all of the transformations performed by the post-processor 130 can also be performed by the column 121 when the GNN 120 is properly trained (e.g., by a training engine). For example, the GNN 120 can learn these transformations and associate them with each text modifier included in the text prompt 102. In some implementations, the system 100 does not include the post-processor 130, and the output image 106 generated by the column 121 is the final image 108. Alternatively, the system 100 can disable the post-processor 130, such that the transformations performed on the output image 106 by the post-processor 130 are equivalent to a match operation.

[0050] In some implementations, the post-processor 130 may perform analysis, such as image classification and / or image quality analysis, on the output image 106. The post-processor 130 may include one or more neural networks, such as a convolutional neural network (CNN), a recurrent neural network (RNN), and / or an image encoder for performing such classification and / or analysis. For example, the post-processor 130 may determine whether the output image 106 accurately depicts the scene depicted by the text prompt 102 by encoding the output image 106 into a set of visual embeddings and comparing them to the context embeddings 104. In these cases, the post-processor 130 may include an image encoder paired with the text encoder 110, such as a pre-trained text-image encoder pair, e.g., a CLIP text-image encoder pair. This also provides a means for zero-shot (or semi-supervised) training of the columns 121 by comparing the visual embeddings to the context embeddings 104. In other words, the columns 121 may be trained (e.g., by a training engine) to generate output images 106 from a text-based training set (rather than just a labeled text-image training set) by generating output images 106 that, when encoded into visual embeddings, faithfully reconstruct the contextual embeddings 104. As another example, the post-processor 130 may use CNNs and / or RNNs, as well as an objective image quality analysis (IQA), to determine whether the output images 106 have high resolution, high spatial coherence, few artifacts, etc.

[0051] The final image 108 depicts the scene described by the text prompt 102 and is output by the system 100 at the final resolution R. For example, as shown in FIG. 1A, the final image 108 is of a dragon fruit wearing a karate belt in the snow. Thus, the final image 108 is accurately titled by the corresponding text prompt 102 in FIG. 1A. FIG. 7 shows another example of an image that can be generated from a text prompt by the image generation system 100.

[0052] The final resolution R is a measure of the information content of the image 108, i.e., the dimensionality of the image 108. As mentioned above, the resolution may correspond to the pixel resolution, i.e., the number of distinct pixels over a predetermined length (or a predetermined area), R=N x ×N y Thus, an image is a collection of N pixel values ​​in a particular range (e.g., corresponding to RGB or CMYK color channels). x ×N y It may also include an array of sizes, e.g., pixel values ​​between [-1, 1], with more (distinct) pixels resulting in higher resolution. In many cases, the final resolution R of the final image 108 is equal to the resolution R of the output image 106, although these may differ in some implementations, for example if the post-processor 130 resizes the output image 106.

[0053] For reference, the exemplary images depicted in FIGS. 1A and 7 were generated at a resolution of 1024×1024 pixels. The exemplary images were generated by an image generation system implementing a sequence of three diffusion-based GNNs (DBGNNs), including an initial DBGNN utilizing a basic image generation model and two subsequent DBGNNs utilizing a super-resolution model. The initial DBGNN generates an initial output image at an initial resolution of 64×64, and the two subsequent DBGNNs successively increase the resolution by a factor of 4×4, such that the first subsequent DBGNN implements 64×64→256×256, and the second subsequent DBGNN implements 256×256→1024×1024. For a total of 3 billion neural network parameters, the initial DBGNN has 2 billion parameters, the first subsequent DBGNN has 600 million parameters, and the second subsequent DBGNN has 400 million parameters.

[0054] 1B is a flow diagram of an exemplary process 200 for generating a final image depicting a scene described by a text prompt. For convenience, process 200 is described as being performed by one or more computer systems at one or more locations. For example, an image generation system suitably programmed in accordance with this specification, such as image generation system 100 of FIG. 1A, can perform process 200.

[0055] The system receives an input text prompt (210) that includes a sequence of text tokens in a natural language.

[0056] The system processes the input text prompt using a text encoder neural network to generate a set of context embeddings for the input text prompt (220).

[0057] The system processes the context embeddings through a series of generative neural networks (230) to generate a final output image that depicts the scene described by the input text prompt.

[0058] In general, column 121 can utilize any of several types of generative models for GNN 120. Such generative models include, but are not limited to, diffusion-based models, generative adversarial networks (GANs), variational autoencoders (VAEs), autoregressive models, energy-based models, Bayesian networks, flow-based models, and hierarchical versions of any of these models (e.g., continuous-time or discrete-time), among others.

[0059] Broadly speaking, the goal of the sequence 121 is to generate new instances of high-resolution images that are highly controllable, i.e., strongly conditioned on conditioning inputs (e.g., text prompts). As explained above, each GNN 120 in the sequence 121 generates its own output image

number

[0060] In the context of sequence 121, the ability to generate conditioned images at multiple resolutions can be advantageous because it allows sequence 121 to train at multiple different spatial scales while keeping each individual GNN relatively simple. This can be important with respect to maintaining spatial coherence in the output image because features at different length scales can be captured at different stages in sequence 121. For example, the joint distribution of sequence 121, including the (i=0) initial GNN and (i=1, 2, ..., n) subsequent GNNs, can be represented as a Markov chain.

number

[0061] where x (i) is a specific resolution R (i) Corresponding to the image of R (i) >R (i-1) and p θ (x (i) |c (i) ) is c (i) =(x (i-1) , u) for each of the specific GNNs 120. We can then use a single GNN-generated image to compute the likelihood distribution for the highest resolution p θ (x (n) |u). The amount of data a single GNN uses in training can be orders of magnitude smaller than the columns of GNNs 121. Moreover, columns 121 allow data associated with each resolution to be trained in parallel. For brevity, superscript (i) identifying a particular GNN 120 is omitted unless otherwise significant.

[0062] To generate a strongly conditioned output image, the GNN 120 computes its likelihood distribution p as maximizing the conditional probability of corresponding pairs (x, c) of data, e.g., data derived from one or more text-image training sets. θ(x|c) can be parameterized. In other words, the GNN 120 maximizes the probability of the ground truth output image x given the corresponding training input c, or some objective function L that depends on the training data (x, c). θ A parameterization can be performed that at least optimizes (x, c), where θ is the respective set of network parameters of the GNN 120 that describe the functional form of the likelihood distribution. For clarity, the output image actually generated by the GNN 120 is the image x when

number

number

number

[0063] GNN 120 facilitates likelihood parameterization by modeling intermediate distributions over latent representations z of images x, also known as image embeddings, encodings, or "labels." For example, latent variables z can be used by GNN 120 to generate specific types of images as specified by particular conditioning inputs c. The latent space can also provide GNN 120 with a means to synthesize, blend, and compress information from different images, allowing column 121 to generate new instances of images that are superficially dissimilar to any present in the training set.

[0064] First, consider moving to the latent space: the likelihood p over the latent representation z. θ If we ignore (x|c), the integral relationship p θ (x|c)=∫p θ (x,z|c)dz pθ (x,z|c) is the joint distribution of x and z conditioned on c. Often, the dimension of the latent representation z is less than or equal to the dimension of the corresponding image x, i.e., the image resolution R, which allows for a compressed representation of the image. Using the chain rule, we can define the joint distribution as p θ (x,z|c)= p θ (x|z,c)p θ (z|c) and p θ (z|c) is the prior distribution z under c, while p θ (x|z,c) is the conditional distribution of x under z and c. The conditional distribution allows the GNN 120 to invert the image x under the latent representation z, while the prior allows the GNN 120 to realize a generative model of the latent representation itself. Modeling the prior can be advantageous, for example, when the GNN 120 tries to strongly correlate the conditioning input c with the latent representation z, thereby θ (z|c) is highly localized around c. The GNN 120 can model a variety of different priors, such as autoregressive priors, diffusion priors, and normal priors, among others.

[0065] Therefore, the output image

number

number

number

number

number

number

number

[0066] In some implementations, the GNN 120 is based on a standard normal distribution p θ We model the prior distribution as (z|c) = p(z) = N(z;0,I), and the normal distribution

number

number

number

number

number

number

number

[0067] The specific forms of the conditional and prior distributions generally depend on the generative model implemented by a particular GNN 120, as well as its assumptions, architecture, parameterization, and training scheme. For example, given the objective function L, θThe type of (x,c), the type and amount of training set, and the statistics of the training set can affect the convergence of a particular model. In either case, the training engine uses an expectation-maximization (EM) algorithm to determine the conditional and / or prior distributions of the likelihood p of the GNN 120 with respect to its network parameters θ. θ (x|c) can be maximized.

[0068] However, the EM algorithm and some objective functions L θ (x,c) can be computationally intractable in some cases, for example, when the training engine uses a fairly large training set, or when the prior and / or conditional distributions are particularly complex. In these cases, the training engine may use a posterior distribution q over the latent representation during training, which can speed up the computation, for example, when the training engine maximizes the evidence lower bound (ELBO). φ (z|x,c) can be simultaneously modeled. The posterior distribution describes how the data (x,c) is encoded into the latent representation z. Here, φ is another set of network parameters that can be included in the respective GNN 120 or another neural network, such as a discriminative neural network (DNN). The GNN 120 can sample from the posterior distribution instead of the prior distribution during training, which, for example, allows the training engine to calculate the objective function in terms of θ and φ.

number

number

number

[0069] In some implementations, the GNN 120 uses noise-conditioned extensions during image generation and / or training. Specifically, each subsequent GNN in the sequence 121 can apply a noise-conditioned extension to its respective input image, which corrupts the image to some extent. This can help facilitate parallel training of different GNNs 120 in the sequence 121 because it reduces sensitivity to domain gaps (e.g., due to artifacts) between the output image of one stage of the sequence 121 and the input used to train the subsequent stage. For example, the GNN 120 can apply Gaussian noise extensions (e.g., Gaussian noise and / or blur) to input images with a random signal-to-noise ratio during training. During inference, the GNN 120 can use a fixed signal-to-noise ratio (e.g., about 3 to 5) that represents a small amount of extension, which helps remove artifacts in the output image from the previous stage while preserving most of the structure. Alternatively, the GNN 120 can sweep across various values ​​of the signal-to-noise ratio during inference to determine the highest-quality estimate.

[0070] An example of a diffusion-based GNN (DBGNN) 120 that can generate strongly conditioned output images from latent representations is described below. Diffusion models generally have two variants: (i) a discrete time hierarchy and (ii) a continuous time hierarchy. Either approach can be implemented by the GNN 120. However, in a continuous time diffusion model, there can be fewer errors than in the discrete time version. For example, a continuous time diffusion model may have a better evidence lower bound (ELBO) than the discrete time version in some cases.

[0071] In continuous time, the latent representation is parameterized by a continuous time index z = {z t | t ∈ [0, 1]}. The forward (encoding) process is described by the posterior distribution q φ (z|x, c), which starts with the data (x, c) at t = 0 and ends with standard Gaussian noise at t = 1. The posterior distribution can be expressed as q φ (z|x, c) = q(z|x) = q(z s , z t |x) = q(z t |z s )q(z s |x) , where 0 ≤ s < t ≤ 1 is a truncated continuous time interval. q(z t |x) is the (forward) prior distribution of z t under x, which describes how the DBGNN 120 encodes an image into a latent representation. q(z t |z s ) is the forward transition distribution from z t to z s , which describes how the DBGNN 120 determines a new latent variable z s from z t for time t > s. In the DBGNN 120, the forward distribution is usually considered independent of φ and c. In other words, the forward (encoding) process is not usually learned by the DBGNN 120 and is a linear Gaussian

Number

number

number

[0072] In some implementations, the DBGNN 120 performs cosine noise scheduling (e.g., α t = cos(0.5πt)), which can be particularly effective in producing high-quality samples. A discussion of various noise schedules is given by Alexander Quinn Nichol and Prafulla Dhariwal, "Improved denoising diffusion probabilistic models," International Conference on Machine Learning, PMLR, 2021. In other implementations, DBGNN120 may use, for example, a variance

number

number

[0073] DBGNN 120 matches the forward process in the reverse time direction, starting at t=1 and ending at t=0. t Learning a generative model involves generating z for all t. t ~q(z t |x) and estimate it by removing noise.

number

number

[0074] where (x,c) is the image-input data pair, ε~N(0,I) is sampled from a standard normal distribution, and t~U(0,1) is sampled from a uniform distribution ranging from 0 to 1. W t is a weighting factor that can be used by DBGNN 120 to influence the quality of the estimate for a particular value of t. DBGNN 120 calculates the objective function

number

number

[0075] After learning the appropriate parameterization θ, DBGNN120 then outputs an image from the latent representation based on the conditional input c [Number] that can be generated. The backward (decoding) process is described by the joint distribution p θ (x,z|c), which starts with standard Gaussian noise at t = 1 and ends with an output image [Number] conditioned on c at t = 0. Note that s < t, and the joint distribution can be expressed as follows. p θ (x,z|c)=p θ (x,z s ,z t |c)=p θ (x|z s ,c)p θ (z s |z t ,c)pθ (z t |c)

[0076] p θ (z t (z|c) is the reverse prior distribution of z under c, which determines how the DBGNN 120 encodes c into the latent variable z. Due to the noise schedule, the reverse prior distribution converges to the standard normal distribution p t (z1|c) = p(z1) = N(z1; 0, I) at time t = 1, so it is not conditioned on c at the start of the reverse process. Similar to the forward process, p t (z θ |z θ (z s |z t ,c) is the reverse transition distribution from z t to z s under c, while p θ (x|z t ,c) is the conditional distribution of x under z t and c.

[0077] The reverse transition distribution can be determined from

Number

[0078] The reverse transition distribution describes, for time s < t and conditioned on c, how the DBGNN 120 determines a new latent variable z t from a given latent variable z s . In this case, q(z s |z t ,x) = q(z t |z s )q(z s |x) / q(z t |x) is the reversed description of the forward process and can be expressed in terms of a normal distribution of the following form.

Number

Number

number

number

[0079] Conditional distribution p θ (x|z t ,c) is the latent variable z based on the conditioning input c. t into image x. After completing the reversal process, DBGNN 120 obtains the conditional distribution p θ (x|z0,c) to output image

number

number

number

[0080] To sample latent variables during the regressive process, DBGNN120 can use a discrete-time ancestral sampler with sampling variances derived from lower and upper bounds on the entropy of the regressive process. Further details of the ancestral sampler are provided by Jonathan Ho, Ajay Jain, and Pieter Abbeel, "Denoising Diffusion Probabilistic Models," NeurIPS, 2020. The ancestral sampler starts with a regressive prior distribution z1~N(z1;0,I) at t=1 and samples the latent variables at time s <tについてp θ (z s |z t When calculating the transition using (c), the following update rules are followed:

number

[0081] ε is the standard Gaussian noise, γ is a hyperparameter that controls the randomness of the sampler, and s, t follow a uniformly spaced sequence from 1 to 0. The update rule is that DBGNN120 updates the previous latent variable z until it reaches z0. t and previous estimates

number

number

number

number

number

number

[0082] Instead of an ancestral sampler, DBGNN120 can use a deterministic denoising diffusion implicit model (DDIM) sampler as described by Jiaming Song, Chenlin Meng, and Stefano Ermon, "Denoising diffusion implicit models," arXiv preprint arXiv:2010.02502 (2020). The DDIM sampler is a numerical integration rule for stochastic flow ordinary differential equations (ODEs) that describes how samples from a standard normal distribution can be deterministically transformed into samples from an image data distribution using a denoising model.

[0083] In some implementations, the DBGNN 120 uses a v-prediction parameterization during image generation and / or training, in which case the DBGNN 120 estimates the image x.

number

number

number

[0084] In some implementations, the DBGNN 120 uses classifier-less guidance during image generation and / or training. Classifier-less guidance is the process of generating an output image for a given conditional input c.

number

number

number

[0085] ω is the induction weight,

number

number

number

[0086] Furthermore, large induction weights (e.g., greater than about 5, greater than about 10, greater than about 15) can improve text-image alignment but can also reduce fidelity, producing images that appear saturated, blank, or unnatural. For example, the estimation at a particular sampling step t

number

[0087] Static thresholding means that DBGNN120 estimates its

number

[0088] Dynamic thresholding is achieved by estimating the limiting threshold κ at each sampling step t.

number

number

number

[0089] Alternatively or additionally, one or more of the DBGNNs 120 may allow ω to oscillate between a high guiding weight (e.g., about 15) and a low guiding weight (e.g., about 1) at each sampling step t. Specifically, one or more of the DBGNNs 120 may use a constant high guiding weight for a certain number of initial sampling steps, and then oscillate between a high guiding weight and a low guiding weight. This oscillating method may be useful for reducing the size of the output image, particularly at the low-resolution stage in the sequence 121.

number

[0090] In implementations involving a DDIM sampler, incremental distillation, and classifier-free guidance, the DBGNN 120 may also incorporate a stochastic sampler to achieve a two-stage distillation approach. For reference, a one-stage incremental distillation approach distills a trained DDIM sampler into a diffusion model requiring fewer sampling steps without significantly compromising perceptual quality. In each iteration of the distillation process, the DBGNN 120 distills an N-step DDIM sampler into a new model with N / 2 steps. The DBGNN 120 repeats this procedure by halving the sampling step t in each iteration. Chenlin Meng et al., "On distillation of guided diffusion models," arXiv preprint arXiv:2210.03142 (2022), extended this one-stage approach to a sampler using classifier-free guidance as well as a new stochastic sampler. The DBGNN 120 can use a modified two-stage approach for improved image generation. Specifically, in the first stage, DBGNN 120 learns a single diffusion model that matches the combined output from the jointly trained conditional and unconditional diffusion models, with the combined coefficients determined by the induction weights. DBGNN 120 then applies incremental distillation to the single model in the second stage to produce a model with fewer sampling steps. After distillation, DBGNN 120 uses a stochastic N-step sampler. At each step, DBGNN 120 first applies one deterministic DDIM update with twice the original step size (i.e., the same step size as the N / 2-step sampler), and then performs one stochastic step backward with the original step size (i.e., perturbing the forward diffusion process with noise). This stochastic backward stepping is described in more detail by Karras, Tero, et al., "Elucidating the design space of diffusion-based generative models," arXiv preprint arXiv:2206.00364 (2022).Using this approach, DBGNN 120 can distill down to many fewer sampling steps (eg, about 8 steps) without any noticeable degradation in the perceptual quality of the output image.

[0091] 2A shows a block diagram of an exemplary GNN column 121. Column 121 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0092] Column 121, or "cascade," includes multiple GNNs 120.0-n, each configured to perform a specific step in the processing pipeline. Specifically, column 121 includes an initial GNN 120.0 that generates an initial output image 106.0, followed by one or more subsequent GNNs 120.1-n. The subsequent GNNs 120.1-n progressively increase the resolution of the initial image 106.0 by generating respective output images 106.i based on the received input images, until the final output image 106.n is reached.

[0093] The first GNN120.0 is the input c (0) =(u) and receives a set of context embeddings 104 of the text prompt 102. The first GNN 120.0 receives the first output image

number

[0094] As explained above, to generate the first output image 106.0, the first GNN 120.0 uses its prior distribution p θ (z (0) |c (0)) to the latent representation z (0) The initial GNN120.0 can then sample its conditional distribution p θ (x (0) |z (0) ,c (0) ) to the first output image

number

[0095] In some implementations, the initial GNN 120.0 is a DBGNN. As described above, the initial DBGNN 120.0 can perform a reversed process starting at t=1 and ending at t=0 to generate the initial output image 106.0. For example, the initial DBGNN 120.0 may use its (reverse) prior distribution t=1 to generate the initial output image 106.0.

number

number

number

number

number

Number

Number

Number

Number

[0096]

Number

Number

Number

Number

Number

number

number

number

[0097] As explained above, to generate an output image 106.i, a subsequent GNN 120.i uses its prior distribution p θ (z (i) |c (i) ) to the latent representation z (i) The subsequent GNN120.i can then sample its conditional distribution pθ (x (i) |z (i) ,c (i) ) to output image

Number

[0098] In some implementations, each subsequent GNN120.i is a DBGNN. As described above, the subsequent DBGNN120.i can execute a reversed process starting from t = 1 and ending at t = 0 to generate the output image 106.i. For example, the subsequent DBGNN120.i, at t = 1, its (reverse progression) prior distribution

Number

Number

Number

Number

Number

number

number

number

number

number

number

number

[0099] 2B is a flow diagram of an exemplary process 230 for processing a set of context embeddings of a text prompt using a column of generative neural networks. For convenience, process 230 is described as being performed by one or more computer systems located at one or more locations. For example, a column of generative neural networks, such as the column of generative neural network 121 of FIG. 2A, suitably programmed in accordance with this specification, can perform process 230.

[0100] The train of generative neural networks includes an initial generative neural network and one or more subsequent generative neural networks.

[0101] The first generative neural network receives the context embeddings (232).

[0102] A first generation neural network processes the context embedding to generate as output a first output image having a first resolution (234).

[0103] For each subsequent generative neural network:

[0104] The subsequent generative neural networks receive respective inputs including respective input images having respective input resolutions and produced as output by the preceding generative neural networks in the sequence (236). In some implementations, the respective inputs to one or more of the subsequent generative neural networks further include context embeddings. In some implementations, the respective inputs to each subsequent generative neural network include context embeddings.

[0105] A subsequent generative neural network processes each input to generate as output a respective output image having a respective output resolution that is higher than the respective input resolution (238).

[0106] In some implementations, each generative neural network in the sequence is a diffusion-based generative neural network.

[0107] 3 shows a block diagram of an example training engine 300 capable of jointly training columns 121 of a GNN. Training engine 300 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0108] The training engine 300 obtains a set of training examples 310, for example, from a publicly available training set or any appropriately labeled text-image training set. Each training example 310 includes (i) a respective training text prompt (T) 302 depicting a particular scene and (ii) a corresponding ground truth image (x) 306 depicting the particular scene. The text encoder neural network 110 processes each training text prompt 302 of each training example 310 to generate a corresponding set of context embeddings (u) 304 for the training text prompt 302. In some implementations, the text encoder 110 is pre-trained and kept frozen (111) by the training engine 300 during joint training of the GNNs 120.0-n. In other implementations, the text encoder 110 is pre-trained and fine-tuned on one or more of the training examples 310. For example, the training engine 300 can first train each GNN 120 in the sequence 121 using the text encoder 110 that is kept frozen (111), and then fine-tune the text encoder 110 on one or more of the training examples 30, which can potentially produce even better text-image matches. Specifically, the context embedding u = u ψ Because ψ depends on the network parameters ψ of the text encoder 110, the training engine 300 can further optimize any of the objective functions described herein with respect to ψ in order to fine-tune the text encoder 110.

[0109] The training engine 300 resizes the ground truth images 306 of each training example 310 to the appropriate input and output resolutions of the GNNs 120.0-n. This results in the correct resolution R for each GNN 120.i. (i) The ground truth output image x, scaled to (i) 306.i. For example, the training engine 300 can resize the ground truth image x to the appropriate resolution of the GNNs 120.0-n by spatial resizing and / or cropping. After resizing, the training engine 300 can train each GNN 120.i in the sequence 121 in parallel and / or individually. With this in mind, the training engine 300 can also use different optimization methods (e.g., stochastic gradient descent (SGD) methods) for the different GNNs 120 in the sequence 121 to update their respective network parameters θ. For example, the training engine 300 can use Adafactor for the first GNN 120.0 and Adam for subsequent GNNs 120.1 through 120.n, among other combinations. Other examples of SGD methods include, but are not limited to, momentum, RMSProp, a second-order Newton-Raphson algorithm, etc. The modularity of columns 121 allows training engine 300 to optimize the training scheme for each GNN 120.i to yield the best performance across the processing pipeline.

[0110] The training engine 300 is of the form (x (0) ,c (0) ) image-input data pairs, where x (0) is the first resolution R of the first GNN120.0 (0) The first output image of the ground truth is sized to 306.0, and c (0) =(u) is each training input including the context embedding 304 of the corresponding training text prompt 302. The initial GNN120.0 is then fitted with the appropriate prior distribution p θ (z (0) |c (0)) and / or the conditional distribution p θ (x (0) |z (0) ,c (0) ), the training engine 300 uses the EM algorithm to learn the network parameters θ of the initial GNN120.0. (0) The likelihood of the data with respect to p θ (x (0) |c (0) ) Alternatively or additionally, the training engine 300 may maximize x, e.g., using stochastic gradient descent (SGD). (0) and c (0) θ depends on (0) With respect to θ (x (0) ,c (0) In some implementations, the training engine 300 may optimize the posterior distribution q for the initial GNN120.0. φ (z (0) |x (0) ,c (0) ), for example, θ (0) and φ (0) When the initial GNN 120.0 is a DBGNN, the training engine 300 may minimize an objective function of the form:

number

[0111] The training engine 300 is of the form (x (i) ,c (i) ) image-input data pairs, where x (i) is the output resolution R of the subsequent GNN120.i(i) The ground truth output image 306.i is sized to c (i) =(x (i-1) ) is the output resolution R (i-1) The ground truth output image x of the preceding GNN in column 121, sized to (i-1) As explained above, the training engine 300 may apply noise conditioning extensions, e.g., Gaussian noise conditioning, to the training inputs x (i-1) In some implementations, the training input c (i) =(x (i-1) , u) also includes the context embedding 304 of the corresponding training text prompt 302. The subsequent GNN 120.i is then given an appropriate prior distribution p θ (z (i) |c (i) ) and / or the conditional distribution p θ (x (i) |z (i) ,c (i) ), the training engine 300 uses the EM algorithm to train the network parameters θ of the subsequent GNN120.i. (i) With respect to the likelihood of the data, p θ (x (i) |c (i) ) may be maximized. Alternatively or additionally, training engine 300 may maximize x, for example using SGD descent. (i) and c (i) θ depends on (i) With respect to θ (x (i) ,c (i) In some implementations, the training engine 300 may optimize each posterior distribution q for the subsequent GNN 120.i. φ (z (i) |x (i) ,c (i) ), for example, θ (i) and φ (i) When the subsequent GNN 120.1 is a DBGNN, the training engine 300 may minimize an objective function of the form:

number

[0112] ε~N(0,I) is sampled from a standard normal distribution, and t~U(0,1) is sampled from a uniform distribution ranging from 0 to 1. As described above, subsequent DBGNN120.i may use one or more of v-parametrization, gradual distillation, classifier-less induction, and / or static or dynamic thresholding during training. In an implementation with noise conditioning extension, subsequent DBGNN120.i may use x (i-1) training input that controls the strength of the conditioning extension applied to

number

number

[0113] 3B is a flow diagram of an exemplary process 400 for jointly training columns of a generative neural network. For convenience, process 400 is described as being performed by one or more computer systems at one or more locations. For example, a training engine suitably programmed in accordance with this specification, such as training engine 300 of FIG. 3A, can perform process 400.

[0114] The training engine obtains 410 a set of training examples, each of which includes (i) a respective training text prompt and (ii) a respective ground truth image depicting the scene described by the respective training text prompt.

[0115] The training engine processes each training text prompt for each training example using a text encoder neural network to generate a corresponding set of context embeddings for the training text prompt (420). In some implementations, the text encoder neural network is pre-trained and kept frozen by the training engine during the joint training of the generative neural network.

[0116] For each generative neural network in the sequence, the training engine resizes (430) the respective ground truth images of each training example to generate corresponding ground truth output images for the generative neural network.

[0117] The training engine jointly trains 440 a generative neural network on each set of context embeddings and the ground truth output images for each training example.

[0118] FIG. 4 illustrates a block diagram of an exemplary U-Net architecture that may be implemented by the GNN 120. For example, the initial GNN 120.0 may implement the U-Net architecture as a base image generative model, and one or more subsequent GNNs 120.1-n may implement the U-Net architecture as a super-resolution model. For simplicity, the U-Net architecture of FIG. 4 is described below with respect to a diffusion model and square-resolution images, but the architecture may be utilized for any type of generative model and rectangular-resolution images. For example, the architecture of FIG. 4 may be used to model the mean and variance of the conditional distribution of the GNN 120. As another example, the architecture of FIG. 4 may be adapted using different sizes of convolution kernels and strides to perform downsampling and upsampling for rectangular-resolution images. The dashed lines in FIG. 4 indicate optional layers.

[0119] As can be seen in Figure 4, DBGNN 120 calculates the image (zt )705.t 7. In this case, latent image 705.t has the same resolution (N×N) as the output image 106 for DBGNN 120. Subsequent DBGNNs 120.1-n may be conditioned on each input image 105 (of resolution M×M) by resizing the input image 105 and then concatenating it channel-by-channel into latent image 705.t to generate a concatenated image having the output resolution (N×N). For example, subsequent DBGNNs 120.1-n may upsample the input image 105 to its output resolution using bilinear or bicubic resizing before concatenating it channel-by-channel into latent image 705.t. Thus, subsequent DBGNNs 120.1-n process concatenated images that include each input image 105 that is “stacked” onto latent image 705.t. The initial DBNN 120.0 is unconditioned on the input image and therefore processes the latent image 705.t directly. Subsequent DBGNNs 120.1-n may be conditioned on their respective input images 105 in various ways, for example, by processing preprocessed versions of the input images 105 to sharpen or enhance certain features. Subsequent DBGNNs 120.1-n may also correct or alter the input images 105 in some specified way, for example, to change color.

[0120] Following the input block 512 is a series of K+1 downsampling blocks (DBlocks) 510-0 through 510-K. Each DBlock 510 is configured to receive a respective input image generated as output by a preceding block. Each DBlock 510 is configured to process the input image to generate a respective output image that is downsampled by a factor of 2×2 relative to the input image. Following the DBlock 510 is a series of K+1 upsampling blocks (UBlocks) 520-0 through 510-K. Each UBlock 520 is configured to receive a respective input image generated as output by a preceding block. Each UBlock 520 is configured to process the respective input image to generate a respective output image that is upsampled by a factor of 2×2 relative to the input image. The DBlocks 510 and UBlocks 520 can implement one or more convolutional layers with appropriate strides to downsample and upsample the respective input images. The UBlocks 520 can also receive, via skip connections, the output image of each DBlock 510 that corresponds to the input resolution of the UBlock 520. In some implementations, the skip connection

number

[0121] DBGNN 120 can also vary its network parameters from higher resolution blocks (e.g., blocks 510-0, 510-1, 520-0, and 520-1) to lower resolution blocks (e.g., blocks 510-(K-1), 510-K, 520-(K-1), and 520-K). Because lower resolution blocks typically have more channels, this allows DBGNN 120 to increase model capacity through more network parameters without significant memory and computational costs.

[0122] One or more of DBlock510 and UBlock520 may be conditioned on the context embedding (u) 104 via an attention mechanism (e.g., mutual attention) using one or more self-attention layers. Alternatively or additionally, one or more of DBlock510 and UBlock520 may be conditioned on the context embedding 104 via a pooled vector of the context embedding using one or more intermediate layers (e.g., one or more pooling layers). In some implementations, one or more of DBlock510 and UBlock520 may be conditioned on other visual features expected in the output image, such as, for example, the nature of a particular color or texture, or the position of an object, all of which may be obtained by the training engine 300 from the training images.

[0123] Finally, the output image of the last UBlock520-0 may be processed by an output block 522 (e.g., including one or more dense layers) to generate an estimated image 706.t for the sampling step t. The DBGNN120 can then use the current estimate 706.t to determine a latent image (z s ) 705.s for the next sampling step s < t using an update rule (e.g., an ancestral sampler update rule).

[0124] The DBGNN120 can repeat the above process for new latent variables 705.s, etc., before reaching the last sampling step t = 0, and then produce the current estimated image 706.t as the output image 106. Thus, the DBGNN120 can iteratively denoise a latent image z1 randomly sampled at the sampling step t = 1 to the output image 106 output at the last sampling step t = 0.

[0125] 5A-5C show block diagrams of exemplary neural network layer blocks for the Efficient U-Net architecture. The layer blocks of FIG. 5A-5C can be implemented in the U-Net architecture shown in FIG. 4 to improve memory efficiency, reduce inference time, and increase the convergence speed of the GNN 120 by utilizing such an architecture.

[0126] 5A is a block diagram of an example residual block (ResNetBlock) 500 for the Efficient U-Net architecture. ResNetBlock 500 processes an input image through a series of layers: a group normalization (GroupNorm) layer 502, a swish activation layer 504, a convolutional (Conv) layer 506-1, another GroupNorm layer 502, another swish layer 504, and another Conv layer 506-1. The Conv layer 506-2 is parallel to the series of layers that process the same input image. To generate the output image of ResNetBlock 500, the output images of each of the series of layers and the Conv layer 202 are summed. The hyperparameter of ResNetBlock 500 is the number of channels (channels:int).

[0127] FIG. 5B is a block diagram of an example downsampling block (DBlock) 210 for the Efficient U-Net architecture.

[0128] The DBlock 510 includes a sequence of layers: a Conv layer 506-3, a CombineEmbs layer 513, one or more ResNetBlocks 500 configured according to FIG. 5A, and a SelfAttention layer 514. The Conv layer 506-3 performs a downsampling operation for the DBlock 510. The CombineEmbs layer 513 (e.g., a pooling layer) can receive the conditional embeddings 103 (e.g., pooled vectors of context embeddings 104, a diffusion sampling step) to provide text prompt conditioning for the DBlock 510. One or more ResNetBlocks 500 perform convolutional operations for the DBlock 510. The SelfAttention layer 514 can perform attention mechanisms for the DBlock 510, such as mutual attention, on the context embeddings 104 to provide further text prompt conditioning. For example, the context embeddings 104 can be concatenated to key-value pairs in the SelfAttention layer 514.

[0129] The hyperparameters of DBlock 510 include the stride of DBlock 510 in case of downsampling (stride:Optional[Tuple[int,int]]), the number of ResNetBlocks 500 per DBlock 510 (numResNetBlocksPerBlock:int), and the number of channels (channels:int). The dashed blocks in Figure 5B are optional; e.g., not every DBlock 510 needs to be downsampled or self-attention.

[0130] Note that in a typical U-Net DBlock, the downsampling operation occurs after the convolution operation. In this case, the downsampling operation performed through Conv layer 506-3 occurs before the convolution operation is performed through one or more ResNetBlocks 500. This reversed order can significantly improve the speed of the forward pass of DBlock 210 with little or no performance degradation.

[0131] FIG. 5C is a block diagram of an example upsampling block (UBlock) 520 for the Efficient U-Net architecture.

[0132] UBlock 520 includes a sequence of layers: a CombineEmbs layer 513, one or more ResNetBlocks 500 configured according to FIG. 5, a SelfAttention layer 514, and a Conv layer 506-3. The CombineEmbs layer 513 (e.g., a pooling layer) can receive the conditional embeddings 103 (e.g., pooled vectors of context embeddings 104, diffusion time steps) to provide text prompt conditioning for UBlock 520. One or more ResNetBlocks 500 perform convolutional operations for UBlock 520. The SelfAttention layer 514 can perform attention mechanisms for UBlock 520, such as mutual attention on the context embeddings 104, to provide further text prompt conditioning. For example, the context embeddings 104 can be concatenated to the key-value pairs of the SelfAttention layer 514. The Conv layer 506-3 performs an upsampling operation for UBlock 520.

[0133] The hyperparameters of a UBlock 520 include the stride of the UBlock 520 in the presence of upsampling (stride:Optional[Tuple[int,int]]), the number of ResNetBlocks 500 per UBlock 520 (numResNetBlocksPerBlock:int), and the number of channels (channels:int). The dashed blocks in Figure 5C are optional; e.g., not every UBlock 520 needs to be downsampled or self-attention.

[0134] Note that in a typical U-Net UBlock, the upsampling operation occurs before the convolution operation. In this case, the upsampling operation performed through Conv layer 506-3 occurs after the convolution operation is performed through one or more ResNetBlocks 500. This reversed order can significantly improve the speed of the forward pass of the UBlock 510 with little or no performance degradation.

[0135] Figure 5D shows a block diagram of an exemplary Efficient U-Net architecture that may be implemented by a subsequent GNN 120.1-n as a super-resolution model for 64x64 to 256x256 input-to-output image upscaling. The architecture of Figure 5D is arranged similarly to Figure 4, with an input block 512 configured as a Conv layer 506-4, a series of five DB blocks 510-0 through 510-4 configured according to Figure 5B, a series of five UB blocks 520-0 through 520-4 configured according to Figure 5C, and an output block 522 configured as a dense layer 516.

[0136] 6A shows a block diagram of an exemplary image generation system 101 capable of generating an image from noise. Image generation system 101 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0137] While this specification is generally directed to text-to-image generation, the image generation systems disclosed herein are not so limited and may be applied to any conditioned image generation problem. For example, the image generation system 101 shown in Figure 6A can generate an image from noise, which is equivalent to modifying the conditioning input to the image generation system 100 of Figure 1A, e.g., replacing the text prompt 102 with noise input 114. In general, the conditioning input can be any desired input, such as an existing image, video, audio waveform, embedding any of these, a combination of these, etc.

[0138] The image generation system 101 can sample the noise input (v) 114 from a noise distribution p(v) 140. For example, the noise distribution 140 can be a binomial distribution, a normal distribution, a Poisson distribution, a beta distribution, a Coomaraswamy distribution, or any desired noise (or probability) distribution. The system 101 can sample the noise input 114 in response to a query, such as a user-generated query or an automatically generated query. For example, the system 101 can receive a query to generate a random image and then sample the noise input 114 from the noise distribution 140. The system 101 processes the noise input 114 through the GNN column 121 to generate a final output image 106.n, which, in some implementations, is further processed by a post-processor 130 to generate a final image 108. As described above, the post-processor 130 may apply transforms to the output image 106.n, perform image classification on the output image 106.n, and / or perform image quality analysis on the output image 106.n, etc. In this case, the final image 108 depicts a random scene because the column 121 is conditioned on the random noise input 114 rather than the text prompt 102 depicting a particular scene.

[0139] In summary, column 121 includes an initial GNN 120.0 and one or more subsequent GNNs 120.1-n. The initial GNN 120.0 is a conditioned input c(0) =(v) is the noise input. The first GNN120.0 is the first output image

number

number

number

number

number

number

number

number

[0140] A training engine (e.g., training engine 300 of FIG. 3A) extracts the output image from noise in a similar manner to text.

number

[0141] In this case, the training engine can jointly sample pairs of ground truth images and noise inputs from a joint distribution (x, v) ~ p(x, v). The joint distribution p(x, v) = p(x|v) p(v) describes the statistics of the data, where p(v) is the noise distribution 140 and p(x|v) is the likelihood of x under v. This likelihood can be modeled by the training engine in various ways to associate the randomly sampled noise input v with the ground truth image x. For example, the training engine may model the likelihood as a normal distribution p(x|v) = N(x; μ(v), ​​Σ(v)), such that x is localized around μ(v) and highly correlated with v. After sampling the data pair (x, v), the training engine can then resize the sampled ground truth image x to the appropriate input and output resolution of the GNN 120.0-n.

[0142] The training engine uses the form (x (0) ,c (0) ) sampled image-input pairs, where x (0) is the first resolution R of the first GNN120.0 (0) is the initial output image of the ground truth sized to c (0) =(v) is each training input including the corresponding noise input 114. The training engine generates training inputs of the form (x (i) ,c (i) ), where x (i) is the output resolution R of the subsequent GNN120.i (i) is the ground truth output image sized to c (i) =(x (i-1) ) is the output resolution R (i-1) The ground truth output image x of the preceding GNN in column 121, sized to (i-1) In some implementations, the training input c (i) =(x (i-1), v) also includes the corresponding noise input 114. The training engine can use any of the techniques described above for GNNs and DBGNNs to train the columns 121 on image-input data pairs.

[0143] 6B is a flow diagram of an exemplary process for generating an image from noise. For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, an image generation system 101 suitably programmed in accordance with this specification, such as the image generation system 101 of FIG. 6A, can perform process 600.

[0144] The system samples 610 a noise input from a noise distribution. The noise distribution may be, for example, a Gaussian noise distribution. The noise input may be a noise image, for example, where each pixel value in the noise image is sampled from the noise distribution.

[0145] The system processes the noise input through a bank of generative neural networks to generate a final output image (620). An exemplary process for generating an image from noise using a bank of generative neural networks is described in more detail below with reference to FIG. 6C.

[0146] 6C is a flow diagram of an exemplary process for generating an image from noise using a column of generative neural networks. For convenience, process 620 is described as being performed by one or more computer systems located at one or more locations. For example, a column of generative neural networks, such as the column of generative neural networks 121 of FIG. 6A, suitably programmed in accordance with this specification, can perform process 620.

[0147] The train of generative neural networks includes an initial generative neural network and one or more subsequent generative neural networks.

[0148] The initial generative neural network receives a noise input (622).

[0149] A first generation neural network processes the noise input to generate as output a first output image having a first resolution (624).

[0150] For each subsequent generative neural network:

[0151] The subsequent generative neural networks receive respective inputs having respective input resolutions and including respective input images produced as output by the preceding generative neural networks in the sequence (626). In some implementations, one or more respective inputs of the subsequent generative neural networks further include a noise input. In some implementations, the respective input of each subsequent generative neural network includes a noise input.

[0152] A subsequent generative neural network processes each input to generate as output a respective output image having a respective output resolution that is higher than the respective input resolution (628).

[0153] In some implementations, each generative neural network in the sequence is a diffusion-based generative neural network. Example implementations of diffusion-based generative neural networks are described throughout this specification, for example, with reference to FIG. 1B.

[0154] This specification uses the term "configured" in connection with systems and computer program components. Configuring one or more computer systems to perform a particular operation or activity means that software, firmware, hardware, or a combination thereof is installed on the system, which, during operation, causes the system to perform that operation or activity. Configuring one or more computer programs to perform a particular operation or activity means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform that operation or activity. Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by or to control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiver device for execution by a data processing device.

[0155] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Optionally, in addition to hardware, an apparatus may include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0156] A computer program, which may also be referred to or written as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer or on multiple computers, located at one site or distributed across multiple sites and interconnected by a data communications network.

[0157] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0158] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components that are installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine. In other cases, multiple engines may be installed on and run on the same one or more computers.

[0159] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0160] A computer suitable for running a computer program may be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from them, transfer data to them, or both. However, it is not necessary for a computer to have such devices. Moreover, a computer may be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or image player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name a few.

[0161] Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0162] To interact with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to achieve user interaction. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from that web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0163] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for processing common, computationally intensive portions of the machine learning training or deliverable, i.e., inference, workload.

[0164] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0165] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0166] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., acting as a client, for the purpose of displaying the data to and receiving user input from a user interacting with the user device. Data generated at the user device, e.g., a result of a user interaction, may be received from the device at the server.

[0167] While this specification contains many specific implementation details, these should not be considered as limitations on the scope of the invention or on the scope of what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, while features may be described above as operating in some combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be excluded from the combination, and a claimed combination may be directed to a subcombination or variations of the subcombination.

[0168] Similarly, although operations are described in the figures and claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the shown operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0169] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the corresponding processes in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0170] 100 Image Generation System 102 Text Prompt 103 Conditional Embedding 104 Context Embedding 106 output images 108 Final Image 110 Text Encoder Neural Network 120 Generative Neural Networks Column 121 130 Post Processor 140 Noise Distribution 144 Noise Input 302 Training Text Prompts 304 Context Embedding 306 ground truth images 310 Training Examples 500 ResNetBlock 502 GroupNorm layer 504 Swish Activation Layer 506 Conv layer 510 DBlock 512 input blocks 513 CombineEmbs layer 514 SelfAttention layer 516 Dense layer 520 UBlock 522 output block 705 Latent Images 706 Estimated Images

Claims

1. A method implemented by one or more computers, comprising: receiving an input text prompt comprising a sequence of text tokens in a natural language; processing the input text prompt using a text encoder neural network to generate a set of context embeddings for the input text prompt; processing the context embeddings through a sequence of generative neural networks to generate a final output image depicting the scene described by the input text prompt; Equipped with The sequence of the generative neural network is: receiving the context embedding; processing the context embedding to generate as output a first output image having a first resolution; an initial generative neural network, receiving a respective input comprising (i) the context embedding and (ii) a respective input image having a respective input resolution and produced as output by a preceding generative neural network in the sequence; processing said respective inputs to generate as output respective output images having a respective output resolution that is higher than said respective input resolution; one or more subsequent generative neural networks, each configured as follows: and each generative neural network in the sequence is a diffusion-based generative neural network; for each subsequent generation neural network, processing the respective input to generate as output the respective output image; sampling each latent image with the respective output resolution; denoising the respective latent images into the respective output images over a series of steps, wherein the denoising comprises, for each step in the series of steps that is not the last step: receiving a respective latent image for said step; processing the respective inputs and the respective latent images for the steps to generate respective estimated images for the steps; dynamically thresholding pixel values ​​of said respective estimated images for said steps; updating the respective latent images for the step using the estimated image for the step to generate respective latent images for a next step; A method comprising:

2. The method of claim 1 , wherein the text encoder neural network is a self-attention encoder neural network.

3. the generative neural networks in the sequence have been jointly trained on a set of training examples, each of which includes (i) a respective training text prompt and (ii) a respective ground truth image depicting a scene described by the respective training text prompt; 10. The method of claim 1, wherein the text encoder neural network is pre-trained and kept frozen during the joint training of the generative neural network in the sequence.

4. The method of claim 1 , wherein the generative neural network is trained using classifier-less induction.

5. For each subsequent generative neural network, denoising the respective latent image into the respective output image over the series of steps comprises, for the last step in the series of steps: receiving a respective latent image for said last step; processing said respective inputs and said respective latent images for said final step to generate said respective output images; The method of claim 1 further comprising:

6. For each subsequent generative neural network, processing the respective input and the respective latent image for each step to generate the respective estimated image for that step, resizing the respective input images to generate respective resized input images having the respective output resolutions; concatenating the respective latent image for the step with the respective resized input image to generate a respective concatenated image for the step; and processing the respective concatenated images for the steps with mutual attention to the context embedding to generate the respective estimated images for the steps.

7. For each subsequent generation neural network, dynamically thresholding the pixel values ​​of the respective estimated image for each step comprises: determining a limiting threshold based on the pixel values ​​of the respective estimated images for the step; and thresholding the pixel values ​​of the respective estimated images for the step using the limiting threshold.

8. determining the limiting threshold based on the pixel values ​​of the respective estimated images for the step; The method of claim 7 , comprising determining the limiting threshold based on absolute pixel values ​​of a particular percentile in the respective estimated image for the step.

9. thresholding the pixel values ​​of the respective estimated images for the steps using the limiting threshold; 9. The method of claim 8, comprising restricting the pixel values ​​of the respective estimated images for the step to a range defined by [-κ,κ], where κ is the restriction threshold.

10. thresholding the pixel values ​​of the respective estimated images for the steps using the limiting threshold; The method of claim 9 , further comprising, after limiting the pixel values ​​of the respective estimated images for the steps, dividing the pixel values ​​of the respective estimated images for the steps by the limiting threshold.

11. The method of claim 1 , wherein each subsequent generative neural network applies a noise-conditioned extension to the respective input image.

12. The method of claim 1 , wherein the final output image is the output image of each of the final generating neural networks in the sequence.

13. 10. The method of claim 1, wherein each subsequent generative neural network receives a respective kxk input image and produces a respective 4kx4k output image.

14. The method of claim 1, wherein each subsequent generative neural network denoises the respective latent image over the sequence of steps according to a stochastic sampler, a deterministic sampler, or a combination of a stochastic sampler and a deterministic sampler.

15. The method of claim 14, wherein the probabilistic sampler is an ancestor sampler and the deterministic sampler is a deterministic DDIM sampler.

16. The method of claim 1, wherein each generating neural network in the sequence is a convolutional neural network.

17. The method described in claim 16, wherein each generating neural network in the sequence has a U-Net architecture.

18. The method of claim 1, wherein the one or more subsequent generation neural networks are multiple subsequent generation neural networks.

19. one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 18.

20. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 18.

Images