Subject-driven Diffusion Neural Network

Abstract

A method, system, and apparatus for generating images of a new subject using a spreading neural network, comprising a computer program encoded on a computer storage medium.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 456,411, filed Mar. 31, 2023. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated into the disclosure of this application by reference.

Background Art

[0002] This specification relates to generating images using a neural network.

[0003] A neural network is a machine learning model that uses one or more layers of non - linear units to predict an output for an input received. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers within the network, i.e., one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters.

Summary of the Invention

Means for Solving the Problems

[0004] This specification describes a system implemented as a computer program on one or more computers that uses a diffusion neural network to generate an image conditioned on a text description.

[0005] Specifically, the system uses a diffusion neural network to perform subject - driven image generation, i.e., generate an image regarding a subject specified in an input to the system.

[0006] As used herein, a “subject” is a characteristic of a scene depicted in an image.

[0007] For example, the subject could be a specific object depicted in the scene, namely a specific animal, a specific person, or a specific inanimate object. Examples of specific objects might include a specific dog, shoes, or a backpack.

[0008] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages:

[0009] Recent text-to-image generative models have made significant progress in generating highly realistic, accurate, and diverse images from a given text prompt. While these models have achieved unprecedented success in generating highly accurate images, they cannot be customized for specific subjects such as dogs, shoes, or backpacks.

[0010] Therefore, some approaches attempt to perform subject-driven text-to-image generation, that is, the task of generating images that are highly customized with respect to a target subject.

[0011] Subject-driven image generation is related to text-driven image editing, but often requires more advanced transformations on the source image (e.g., view rotation, zoom in / out, subject posing changes, etc.) when generating a new image in order to align the new image with the desired text description of its content. As a result, existing image editing methods, i.e., methods not specifically tailored for subject-driven generation, are generally unsuitable for this new task.

[0012] Furthermore, existing subject-driven text-to-image generation approaches are time-consuming and expensive. For example, existing approaches require fine-tuning a specific model for a given subject for one or a few demonstrations. This further training typically requires significant additional time, such as 10-20 minutes, and computational resources, such as processor cycles and hardware accelerator usage, because it requires specializing text-to-image model checkpoints for a given subject. These approaches are time-consuming because they require backpropagation gradients across the entire model at hundreds or thousands of steps for each customization. Moreover, they consume space because they need to store subject-specific checkpoints for each subject. In other words, existing approaches require storing a separate fine-tuning model for each subject, which is highly memory-intensive.

[0013] To avoid excessive computational and memory costs while still performing high-precision generation, this specification describes a single subject-driven text-to-image generative model that allows for on-the-fly subject customization. That is, by taking demonstration pairs as input (in addition to the new text description) at inference time, a single diffuse neural network can be used to generate new images of many different specific subjects.

[0014] Specifically, this model can instantly adapt to invisible subjects, invisible descriptions, or even compositional descriptions with only 3 to 5 in-context demonstrations, minimizing latency—for example, under 30 seconds (on a cloud TPU v4)—without additional optimizations.

[0015] For example, this model can be trained with a novel apprentice learning algorithm described in this specification, which uses the model to mimic the behavior of a number of specialized expert models, allowing the model to perform well when generalizing to a new subject after training, providing only a few in-context demonstration pairs for the new subject.

[0016] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0017] [Figure 1] This is a diagram illustrating an exemplary neural network system. [Figure 2] This shows an example of how the system behaves after training. [Figure 3] This is a flowchart illustrating an exemplary process for generating a new image of a new subject. [Figure 4] This is a flowchart illustrating an exemplary process for generating new images of a new subject using a diffuse neural network. [Figure 5] This is an illustrative flowchart of the process for training a diffusion neural network. [Figure 6] This example demonstrates how to perform apprentice learning to generate training data for training a spreading neural network. [Modes for carrying out the invention]

[0018] Similar reference numbers and symbols in various drawings refer to the same elements.

[0019] FIG. 1 is a diagram of an exemplary neural network system 100. The neural network system 100 is an example of a system implemented as a computer program on one or more computers located in one or more locations, and the systems, components, and techniques described below can be implemented.

[0020] System 100 is a system that uses a diffusion neural network 110 to perform subject-driven image generation, that is, to generate an image 112 related to a subject specified in the input to system 100.

[0021] As used herein, a "subject" is a characteristic of a scene depicted in an image.

[0022] For example, a subject can be a specific object depicted in a scene, that is, a specific animal, a specific person, or a specific other inanimate object. Examples of specific objects can include a specific dog, shoes, a backpack, etc.

[0023] Therefore, subject-driven image generation requires generating an image of a given subject within the scene described by the text description when a text description corresponding to a given subject is provided.

[0024] For example, after training, system 100 can generate new images of new subjects that were not seen in the training data 120 of the diffusion neural network 110.

[0025] To perform subject-driven image generation for a new subject, system 100 receives a plurality of demonstration pairs 102 for the new subject.

[0026] Each demonstration pair 102 includes (i) an image 104 related to the new subject, for example, an image depicting a scene containing the new subject, and (ii) a text description 106 of the image 104 in the demonstration pair 102.

[0027] In some embodiments, system 100 receives both an image 104 and a corresponding text description 106 from the user of system 100.

[0028] In some other embodiments, system 100 receives images 104 from a user of system 100 and then processes the images 104 using a trained image caption neural network to generate a text description corresponding to each image 104.

[0029] System 100 also receives a new text description 108 related to the new subject. For example, System 100 may receive a new text description 108 from the same user as the demonstration pair 102, or from a different user. The new text description 108 generally describes the scene in which the new subject should be depicted in the new image.

[0030] Next, system 100 uses a spreading neural network 110 to process the demonstration pair 102 and the new text description 108 to generate a new image 112 described by the new text description 108.

[0031] Before generating a new image 112 using the diffuse neural network 110, system 100 or another training system trains the diffuse neural network 110 with training data 120.

[0032] For example, after training, the diffuse neural network 110 can effectively generate new images 112 of a new subject without being conditioned on only a small number of subject demonstration pairs 102, e.g., 2 to 10, or without being further trained on any of the new subject's demonstration images.

[0033] The spread neural network 110 can generally be any appropriate conditional spread neural network.

[0034] Specifically, the diffuse neural network 110 is configured to receive a diffuse input containing an input image and a conditioned input of a target image, and to process these inputs to generate a diffuse output for the initial image. Generally, the diffuse input also includes a time step that defines the noise level of the input image. The conditioned input generally represents a demonstration pair and a new text description.

[0035] In some embodiments, the diffuse neural network 110 performs a diffuse process in pixel space, thereby operating on the diffuse neural network, and the image generated by the diffuse neural network has values ​​for each pixel, for example, RGB values ​​or values ​​specifying another color coding scheme. In these examples, the diffuse output may be an estimate of the target image, or an estimate of the noise added to the target image to arrive at the input image.

[0036] In some other embodiments, the spread neural network 110 performs the spread process in a latent space, for example, a latent space that is lower dimensional than the pixel space. That is, the image manipulated by the spread neural network 110 is a latent image, and the pixel values ​​of the image are learned latent values ​​rather than color values. In these examples, the spread output may be an estimate of the latent representation of the target image, or an estimate of the noise added to the latent representation of the target image in the latent space to arrive at the input latent representation in the latent space.

[0037] In these embodiments, the trained diffuse neural network 110 can be associated with an image encoder that encodes an image into latent space, and a decoder neural network that receives an input including a latent representation of an image and decodes the latent representation to reconstruct the image. For example, the encoder and decoder may be co-trained for the purpose of image reconstruction, e.g., for VAE purposes, VQ-GAN purposes, or VQ-VAE purposes.

[0038] Therefore, in these examples, after training, system 100 can use the decoder neural network to generate output image 112 from images in latent space generated using the diffuse neural network 110.

[0039] The spreading neural network 110 can generally have any suitable neural network architecture.

[0040] For example, the spread neural network 110 can be a convolutional neural network such as a U-Net, having multiple convolutional layer blocks. In some of these cases, the spread neural network 110 may include one or more mutual attention layer blocks scattered among the convolutional layer blocks. As described below, some or all of the mutual attention blocks can be conditioned on a representation of the conditioned input.

[0041] As another example, the spreading neural network 110 can be a transformer neural network that processes a spreading input through a set of self-attention layers to produce a spreading output. In these examples, the neural network may also include one or more attention blocks that are conditioned on a representation of a conditioned input.

[0042] Specifically, the spreading neural network 110 may include a text encoder neural network that generates an encoded representation of a text sequence included in the conditioned input, and an image encoder neural network that generates an encoded representation of an image in the demonstration pair 102.

[0043] One or more layers of the spreading neural network 110, for example, one or more mutual attention layers or one or more self-attention layers, may then be conditioned on one or both of the encoded representations of the text description and the demonstration image 104.

[0044] An example of a spread neural network 110 architecture is described in more detail below.

[0045] Figure 2 shows an example of the operation of system 100 after training.

[0046] Specifically, Figure 2 shows three sets of demonstration pairs 102 for three different subjects ("Fancy Boots," "Canid Dogs," and "Duck Toys").

[0047] For each subject, the system 100 receives four new text descriptions 108, and for each new text description 108, and from the new text descriptions 108 and the subject demonstration pair 102, uses the spread neural network 110 to generate a new image 112 of the subject in the context described by the new text description, without performing any further training on the spread neural network 110 using either the image or text in the subject demonstration pair 102.

[0048] Therefore, as shown in Figure 2, system 100 can generate new images 112 of a particular subject without further training or optimizing the diffuse neural network 110, even when only a small number of demonstration pairs 102 are given.

[0049] Specifically, as shown in Figure 2, the new image 112 is a new rendition of a given subject, generated by a spread neural network 110, which is trained to be generalizable, i.e., to generalize to a new subject after training, and can learn in context, i.e., it can receive demonstration pairs 102 at inference time, extract context from those demonstration pairs 102 without further training, and generate a new image of the new subject without requiring additional training, thus providing instant customization.

[0050] Figure 3 is a flowchart of an exemplary process 300 for generating a new image of a new subject. For convenience, process 300 is described as being performed by one or more computer systems located in one or more locations. For example, a training system (e.g., the neural network system 100 shown in Figure 1) is appropriately programmed according to this specification and is capable of performing process 300.

[0051] The system receives multiple demonstration pairs (step 302).

[0052] Each demonstration pair includes (i) an image related to the input subject and (ii) a text description of the image within the demonstration pair.

[0053] The system receives a new text description related to the input subject (step 304). For example, the new text description may describe a new context in which the input subject is depicted.

[0054] The system uses a spreading neural network to process the demonstration pair and the new text description to generate a new image described by the new text description (step 306).

[0055] Specifically, the system generates new images across multiple dediffusion steps.

[0056] The generation of a new image using a diffuse neural network is explained with reference to Figure 4.

[0057] In some embodiments, the diffuse neural network generates an output image that is lower resolution than the required resolution for the new image.

[0058] In these embodiments, the system can generate a new image from a new image representation after the last despread step by processing the representation through one or more super-resolution neural networks. In a specific example, each super-resolution neural network may be a different spread neural network that is also conditioned on the new text description (but not on the demonstration pair).

[0059] Figure 4 is a flowchart of an exemplary process 400 for generating a new image of a new subject using a diffuse neural network. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a training system (e.g., the neural network system 100 shown in Figure 1) is appropriately programmed according to this specification and capable of performing process 400.

[0060] The system initializes a new image representation (step 402). For example, the system can sample each value in the representation from a specified noise distribution, such as a Gaussian distribution or another suitable distribution.

[0061] Next, the system updates the representation of the new image in each of several dediffusion steps. In each dediffusion step, the system performs steps 404 and 406 to update the representation.

[0062] Specifically, in each despreading step, the system processes the spread input for the despreading step, which includes the representation of the new image, using a spread neural network conditioned on (i) a text description of the new image and (ii) a demonstration pair that generates a spread output (step 404).

[0063] Specifically, a spreading neural network includes one or more layers that are conditioned on a representation of a text description, a representation of a demonstration pair, or both.

[0064] For example, a spreading neural network may include one or more attention layers, each paying attention to at least a portion of each representation of a text description, image, or both within a demonstration pair.

[0065] As a concrete example, a spreading neural network may include a first attention layer that pays mutual attention across each representation of each text description in a demonstration pair, and a second attention layer that pays mutual attention across each representation of each image in a demonstration pair.

[0066] As a specific example, a diffuse neural network may include a diffuse encoder that takes an image, text, and time step as input and generates a feature map as output. The feature map may have the same spatial resolution as the input image, or a different spatial resolution, for example, a downsampled spatial resolution relative to the input image. For example, a diffuse encoder may include a set of convolutional layers that process the image and one or more attention layers, such as a self-attention layer or a mutual attention layer, which update the image representation using a representation of the text, a representation of the time step, or both, respectively.

[0067] A diffuse neural network can also include a diffuse decoder that processes feature maps to generate a diffuse output. For example, a diffuse decoder can include a set of convolutional layers. Optionally, a diffuse decoder can also include one or more attention layers, such as self-attention layers or mutual attention layers, which update the image representation using a text representation, a time step representation, or both, respectively.

[0068] In this particular example, the system can process each demonstration pair using a spreading encoder (with the time step set to a default value, e.g., zero) to generate a feature map for each demonstration pair.

[0069] Next, in each despreading step, the spread neural network can use a spread encoder to process the new image representation, the new text description, and the time step corresponding to the despreading step to generate a feature map for the new image representation.

[0070] Next, the spread neural network can apply attention modules, such as a multi-head self-attention module or a mutual attention module, to the feature maps of the representation and the demonstration pair to generate updated feature maps. Then, the spread neural network can process the updated feature maps using a spread decoder to generate a spread output for the despread step.

[0071] Next, the system updates the representation using the diffusion output (step 406).

[0072] For example, the system can determine an estimate of the final representation, i.e., a noise-free representation, from the diffusion output, and then apply an appropriate diffusion sampler, such as a DDPM (Denoising Diffusion Probability Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another appropriate sampler, to the estimate to generate an updated representation.

[0073] Optionally, after the last backdiffusion iteration, the system may refrain from using the diffusion sampler and instead use the estimate as the updated representation.

[0074] To determine the estimated value, the system can use the diffusion output as the final diffusion output.

[0075] Alternatively, the system can utilize guidance that does not involve a classifier. In these cases, the system can use a diffuse neural network to process another diffuse input for a dediffusion step that includes a new image representation, thereby generating another diffuse output. When generating other diffuse outputs, the system can avoid conditioning the diffuse neural network on the text description and demonstration images.

[0076] The system can then combine the diffusion output and other diffusion outputs according to the guidance weights to generate the final diffusion output. That is, the system can determine the weighted sum (or weighted difference) between the diffusion output and other diffusion outputs, and these weights are determined by the guidance weights.

[0077] If the diffuse output is an estimate of the new image, the system can directly use the final diffuse output as the estimate. If the diffuse output is an estimate of the noise, the system can use the current representation x t , diffusion output ε θ , and the noise level corresponding to the despreading step (and defined by the time step t)

number

number

number

[0078] In some cases, the system uses the new image representation after the final dediffusion step as the new image.

[0079] In some other cases, the system performs further processing on the representation to generate a new image.

[0080] For example, the representation could be a low-resolution version of the new image. In this example, the system can generate a new image from the representation of the new image after the last despreading step by processing the representation through one or more super-resolution neural networks. In a specific example, each super-resolution neural network could be a different spread neural network that is also conditioned on the new text description.

[0081] As another example, when a representation is in latent space, the system can use the decoder described above to map the representation from latent space to pixel space.

[0082] Figure 5 is a flowchart of an exemplary process 500 for training a spreading neural network. For convenience, process 500 is described as being performed by one or more computer systems located in one or more locations. For example, a training system (e.g., the neural network system 100 shown in Figure 1) is appropriately programmed according to this specification and is capable of performing process 500.

[0083] The system obtains a training dataset to train a spreading neural network (step 502).

[0084] The training dataset includes one or more training examples for each of the multiple subjects.

[0085] Generally, each training example for a given subject includes (i) a set of training demonstration pairs, (ii) a training text description related to the subject, and (iii) a new image for training described by the training text description.

[0086] As described above, a demonstration pair for a given subject includes (i) an image related to the subject and (ii) a text description of the image in the training demonstration pair.

[0087] The system can generally acquire training datasets using one of several methods.

[0088] For example, the system can utilize apprenticeship to generate some or all of the training examples within the training dataset.

[0089] Specifically, when using apprentice learning, the system generates training data using one or more finely tuned diffuse neural networks, each starting with a pre-trained diffuse neural network.

[0090] A pre-trained diffusion neural network can be a suitable pre-trained text-conditional diffusion neural network. Examples of such diffusion neural networks include Imagen and Stable Diffusion.

[0091] As an example, to utilize apprenticeship learning as one of the given subjects, the system can execute steps 504-508.

[0092] The system fine-tunes a pre-trained diffuse neural network on training data containing demonstration pairs of a given subject to generate one or more fine-tuned diffuse neural networks for the subject (step 504).

[0093] To fine-tune a spread neural network pre-trained on training data containing demonstration pairs of a given subject, the system can train a spread neural network trained with denoising, for example, time-reweighted squared error loss as described below. Unlike spread neural network 110, the fine-tuned spread neural network does not receive demonstration pairs at inference and is conditioned only on text descriptions at inference.

[0094] For each finely tuned diffuse neural network for a given subject, and for each training text description for a subject, the system can use the finely tuned diffuse neural network to process the training text description for the subject to generate one or more new training images for candidates (step 506). That is, since the generation of new images begins with initializing a representation using sampled noise, by sampling different initial representations from the same distribution, the system can cause a given diffuse neural network to generate different output images.

[0095] The system can then select one or more candidate new training images to include in each of the subject's training examples (step 508).

[0096] For example, for each candidate new training image, the system can determine a quality score for the candidate new training image, which measures how well the new training image is described by the corresponding training text description.

[0097] To generate a quality score for a given new training image, the system can use an image encoder neural network to process the candidate new training images and generate image embeddings for the images.

[0098] Next, the system can use a text encoder neural network to process training text descriptions and generate text embeddings for images.

[0099] Next, the system can determine a quality score based on the similarity between the text embedding and the image embedding. The similarity can be, for example, the dot product or cosine similarity between the text embedding and the image embedding.

[0100] For example, image encoders and text encoders may be co-trained, for instance, by comparative learning or another unsupervised learning technique, such that the text embeddings and image embeddings are close to each other in the embedding space when the corresponding text describes the content of the corresponding image.

[0101] The system can then select one or more of the candidate new training images based on their quality scores.

[0102] For example, the system can select a new training image for each candidate that meets the threshold.

[0103] As another example, the system can calculate, for each new candidate training image, (i) a quality score for the new candidate training image and (ii) a difference quality score, which is the difference between the image embeddings of the demonstration images in the corresponding set of demonstration pairs and the highest quality score of the text embeddings of the new text embeddings. The system can then select any new candidate training image that satisfies a threshold, for example, one with a quality score greater than the threshold, to include in the training dataset. Thus, in these cases, the system leverages the demonstration images to verify the quality of any given new candidate image.

[0104] The system trains a diffusion neural network on the training dataset (step 510).

[0105] Specifically, the system trains a diffuse neural network for each training example to measure its performance in denoising noisy images generated from new training images in the training example. The diffuse neural network is conditioned on the training text description and the training demonstration pair in the training example.

[0106] For example, the objective could be time-reweighted squared error loss or another suitable diffusion model training loss. As an example, the loss can be expressed as follows:

number

number

number

number

[0107] As a result of being trained for a specific purpose, after training, the diffuse neural network can be effectively used to generate new images of a new subject when given a small set of demonstration images, without further training.

[0108] Specifically, Example 600 demonstrates how a pre-trained spreading neural network 610 can be fine-tuned to generate a different, finely tuned neural network ("expert") for each of three subjects: Subject A (kitchen mixer), Subject B (basket envelope handbag), and Subject C (London tourist landmarks).

[0109] After some fine-tuning, each expert can be used to sample new images of the corresponding subject when given a new text description (e.g., a kitchen mixer in red, a basket envelope handbag containing a bouquet of flowers, a London tourist landmark at sunset, etc.).

[0110] The sampled images (and their corresponding demonstrations) may then be used as supervisors during the training of the spread neural network 110. For example, as shown in Example 600, the spread neural network 110 is trained on a training example of Subject B which includes a demonstration pair of Subject B, the text description "a bouquet of flowers in..." for the sampled image of Subject B, and the sampled image of Subject B.

[0111] The system can generate a text description for use when sampling new images of a given subject, using one of several methods.

[0112] For example, for each subject, the system can generate a training text description for each training example of the subject using a text-generating neural network conditioned on one or more text descriptions in the training demonstration pairs within the subject's training examples. In specific examples, the text-generating neural network may be a large-scale language model, such as a PaLM model, a PaLM 2 model, or any other suitable transformer language model dedicated to a specific decoder.

[0113] The system can perform training of a finely tuned neural network and sampling of images from the finely tuned neural network using one of several methods.

[0114] For example, the system can perform training over multiple training iterations.

[0115] As an example of performing training over multiple training iterations, in each training iteration, the system can sample a batch containing multiple sets of training demonstration pairs, each set corresponding to a different subject. For each set of training demonstration pairs, the system can fine-tune a previously trained diffusion neural network for that set of training demonstration pairs to generate a finely tuned diffusion neural network for the corresponding subject. Thus, in this example, the system trains a separate, finely tuned diffusion neural network for each corresponding subject in each training iteration.

[0116] Once a finely tuned diffuse neural network is generated for a given subject, the system can use the finely tuned diffuse neural network for the corresponding subject to process a training text description for that subject and generate new training images for one or more candidates for that subject.

[0117] In some of these embodiments, the system can utilize parallelism to reduce the latency required to perform a given training iteration. For example, in each training iteration, each set of training demonstrations can be assigned to a different hardware accelerator, such as a different TPU, GPU, or other ASIC, and the fine-tuning and processing of the training text descriptions for the set of training demonstration pairs can be performed in parallel for each of the training demonstrations by the assigned hardware accelerator.

[0118] In some cases, the system can interleave the training of a diffusion neural network with the generation of a finely tuned neural network. That is, after a given number of training iterations have been completed to generate a given set of finely tuned neural networks, the system can perform a training step to train the diffusion neural network using the results of the training iterations obtained from the generation of the finely tuned neural networks. Optionally, the system can then reduce the memory requirements of the apprentice learning process by discarding the finely tuned neural networks and the images sampled using the finely tuned neural networks.

[0119] Another example of performing training over multiple training iterations is that in each training iteration, the system could sample a batch containing multiple sets of training demonstration pairs, each set corresponding to a different subject. The system could then fine-tune a pre-trained diffusion neural network on the batch of training demonstration pairs to generate a fine-tuned diffusion neural network that corresponds to all the subjects in each set of training demonstration pairs. Thus, in this example, the system trains a single fine-tuned diffusion neural network corresponding to each corresponding subject (in the batch) in each training iteration.

[0120] Once a finely tuned diffuse neural network is generated for the subjects in a batch, the system can use the finely tuned diffuse neural network to process a training text description for one of the given subjects and generate new training images for one or more candidates for the corresponding subject.

[0121] In some of these embodiments, the system can utilize parallelization to reduce the latency required to perform a given training iteration. For example, in each training iteration, the system can distribute training and sampling across multiple different hardware accelerators (e.g., TPUs, GPUs, or other ASICs).

[0122] This specification uses the term “configured” in relation to systems and computer program components. The configuration of one or more computer systems for a particular operation or action means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. The configuration of one or more computer programs for a particular operation or action means that, when executed by a data processing device, the program contains instructions that cause the device to perform that operation or action.

[0123] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, for example, one or more modules of computer program instructions, encoded in a tangible, non-temporary storage medium that is executed by or controls the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into artificially generated propagating signals, for example, mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0124] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0125] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required to, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or it can be located in one place or distributed across multiple locations and interconnected by a data communication network to run on multiple computers.

[0126] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0127] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0128] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0129] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0130] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0131] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device (for example, a smartphone running a messaging application) and then receiving a response message from the user.

[0132] Data processing devices for implementing machine learning models may also include dedicated hardware accelerator units for handling common computationally intensive parts of machine learning training or production, such as inference and workloads.

[0133] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0134] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application that enables a user to interact with embodiments of the subject matter described herein, or in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0135] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0136] While this specification includes details of many specific embodiments, these should not be construed as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, individually or in any preferred secondary combination. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover subcombinations or variations of subcombinations.

[0137] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0138] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require to be performed in the specific order or sequence shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, Receiving multiple demonstration pairs, each demonstration pair comprising (i) an image related to the input subject, and (ii) a text description of the image in the demonstration pair, Receiving a new text description related to the aforementioned input subject, Using a spreading neural network, the demonstration pair and the new text description are processed to generate a new image described by the new text description, Methods that include...

2. Using a spreading neural network to process the demonstration pair and the new text description to generate a new image described by the new text description, Initializing the representation of the new image, The process includes updating the representation of the new image in each of the multiple dediffusion steps, wherein the updating is performed in each dediffusion step. (i) processing the spread input of the iterations, including the new representation of the image, using the spread neural network conditioned on the text description of the image and (ii) the demonstration pair that generates the spread output, Updating the representation using the aforementioned diffusion output, including, The method according to claim 1.

3. The method according to claim 2, wherein the spreading neural network includes one or more attention layers, each paying attention to each other across at least a portion of each representation of the text description, image, or both within the demonstration pair.

4. The method according to claim 3, wherein the spreading neural network includes a first attention layer that pays mutual attention to each representation of each of the text descriptions in the demonstration pair.

5. The method according to claim 3 or 4, wherein the spreading neural network includes a second attention layer that mutually pays attention to each representation of each of the images in the demonstration pair.

6. Using a spreading neural network to process the demonstration pair and the new text description to generate a new image described by the new text description, To generate the new image from the representation of the new image after the last dediffusion step. The method according to any one of claims 2 to 5, further comprising:

7. The method according to claim 6, wherein the representation is a low-resolution version of the new image, and generating the new image from the representation of the new image after the last despreading step involves processing the representation via one or more super-resolution neural networks.

8. The method according to claim 7, wherein each ultra-high-resolution neural network is a different spreading neural network that is also conditioned on the new text description.

9. A method for training the spreading neural network according to any one of claims 1 to 8, Obtaining a training dataset containing one or more training examples for each of multiple subjects, Each subject's training example includes multiple training demonstration pairs, a training text description related to the subject, and a new image for training described by the training text description. Each of the aforementioned training demonstration pairs includes (i) an image related to the subject and (ii) a text description of the image in the training demonstration pair, For each training example, the purpose is to train the spreading neural network on the training dataset for the purpose of measuring the performance of the spreading neural network in denoising noisy images generated from the new images used for training in the training example, wherein the spreading neural network is trained conditioned on the training text description and the training demonstration pair in the training example. Methods that include...

10. The method according to claim 9, wherein the aforementioned objective is time-reweighted squared error loss.

11. Obtaining the aforementioned training dataset For each subject, To fine-tune a diffusion neural network pre-trained on training data including the aforementioned demonstration pair to generate one or more fine-tuned diffusion neural networks for the subject, For each finely tuned diffuse neural network for the subject, and for each training text description for the subject, the finely tuned diffuse neural network is used to process the training text description for the subject to generate one or more new candidate training images. Selecting one or more of the candidate new training images to include in each of the training examples of the subject, The method according to claim 9 or claim 10, including the method according to claim 9 or 10.

12. Selecting one or more of the candidate new training images to include in each of the training examples of the subject, For each of the candidate new training images, a quality score is determined to measure how well the new training image is described by the training text description. Based on the aforementioned quality score, one or more of the candidate new training images are selected. The method according to claim 11, including the method described in claim 11.

13. Determining the quality score of the candidate new training image to measure how well the new training image is described by the training text description, The process involves using an image encoder neural network to process the candidate new training images and generate image embeddings for those images. The process involves using a text encoder neural network to process the training text description and generate text embeddings for the image, The quality score is determined based on the similarity between the text embedding and the image embedding, The method according to claim 12, including the method described in claim 12.

14. The method according to claim 13, wherein the similarity is the dot product or cosine similarity.

15. For each subject, the training text description for each training example for the subject is generated using a text generation neural network conditioned on one or more of the text descriptions in the training demonstration pairs within the training example for the subject. The method according to any one of claims 9 to 14, further comprising:

16. The method according to claim 15, wherein the text generation neural network is a large-scale language model.

17. If dependent on claim 11, generating the one or more finely tuned neural networks for each of the subjects is: In each of the multiple training repetitions, Sampling a batch containing multiple sets of training demonstration pairs, each set corresponding to a different subject, For each set of training demonstration pairs, the pre-trained diffusion neural network for the set of training demonstration pairs is fine-tuned to generate a fine-tuned diffusion neural network for the corresponding subject. The method according to any one of claims 1 to 16, including the method described in any one of claims 1 to 16.

18. For each finely tuned diffuse neural network for the subject, and for each training text description for the subject, the finely tuned diffuse neural network is used to process the training text description for the subject to generate one or more new candidate training images. In each of the training iterations, for each set of training demonstration pairs in the batch for the training iteration, the fine-tuned spreading neural network for the corresponding subject is used to process a training text description for the corresponding subject and generate one or more candidate new training images for the corresponding subject. The method according to claim 17.

19. The method according to claim 18, wherein in each training iteration, each set of training demonstrations is assigned to a different hardware accelerator, and the fine-tuning and processing of the training text description for the set of training demonstration pairs are performed in parallel for each of the training demonstrations by the assigned hardware accelerator.

20. One or more computers, One or more storage devices that store instructions causing the one or more computers to perform each of the operations according to any one of claims 1 to 19 when executed by the one or more computers, A system that includes this.

21. One or more computer-readable storage media that, when executed by one or more computers, store instructions for causing one or more computers to perform each of the operations of the method according to any one of claims 1 to 19.

Images