Conditioning text-to-image generative neural networks on faces

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating images that depict faces using a diffusion neural network.

Description

[0001] CONDITIONING TEXT-TO-IMAGE GENERATIVE NEURAL NETWORKS ON FACES

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 504,698, filed on May 26, 2023, the contents of which are hereby incorporated by reference.

[0004] BACKGROUND

[0005] This specification relates to generating images using neural networks.

[0006] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to or more other layers in the network, i.e., one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers that generates images that depict faces, i.e., of faces of humans or other animals, using a diffusion neural network.

[0009] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0010] Recent text-to-image generation models have shown great progress in generating highly realistic, accurate, and diverse images from a given text prompt. Though achieving unprecedented success in generating highly accurate images, these models struggle with generating images that include faces, i.e., faces of humans or other animals.

[0011] This specification addresses these issues by introducing “face tokens” that are provided as input to the diffusion neural network when generating a given image.

[0012] Through the use of the face tokens, the described techniques instantaneously condition a text-to-image generation model on a face, in sample time, without any optimization procedures such as fine-tuning or inversions. In other w ords, by using face tokens, the techniques can. given a user-supplied face image and text description, generate an image of the user-supplied face modified as described by the text description with minimal latency. Moreover, the system can use the diffusion neural network to generate multiple different images of the same face in different contexts that are consistent with one another by, for each image, providing the same face tokens as input to the diffusion neural network along with different additional inputs, e.g., text prompts, that describe the respective context for each image.

[0013] Additionally, the system (or the user) can control the generated image through manipulating the text description and. optionally, by manipulating other factors, e.g.. manipulating the embedding that is used to generate the face tokens or manipulating various weights that are used as part of the generation process. Thus, the described techniques provide for high-quality, controllable generation of images that include face tokens with minimal computational overhead.

[0014] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1 is a diagram of an example neural network system.

[0017] FIG. 2 is a flow diagram of an example process for generating a new image.

[0018] FIG. 3 is a flow diagram of an example process for performing an update iteration.

[0019] FIG. 4 shows an example of training the diffusion neural network.

[0020] Like reference numbers and designations in the various drawings indicate like elements.

[0021] DETAILED DESCRIPTION

[0022] FIG. 1 is a diagram of an example neural network system 100. The neural netw ork system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0023] The system 100 is a system that uses a diffusion neural network 110 to generate face images 112. A face image is an image that includes a depiction of a face of an entity, e.g., a face of a human or an animal. Face images 112 are also referred to as "‘target images” in this specification. In particular, to generate an image 112 of a face using the diffusion neural network 110, the system 100 receives a target text description 102 that describes a target image to be generated and obtains face data 104 representing properties of a face to be depicted in the target image.

[0024] For example, the target text 102 and the face data 104 can be specified by an input received by the system 100 from a user of a user device.

[0025] For example, the input can specify the target text and (i) one or more images of faces or (ii) one or more larger images that the system 100 then crops to generate one or more images of faces.

[0026] The system 100 can then generate an embedding from the one or more images of faces that serves as the face data 104.

[0027] As another example, face data 104 can be randomly generated by the system, e.g., by sampling the embedding from a pre-determined distribution over the embedding space. By keeping the same randomly generated face data 104 fixed while generating multiple different images with different target text, the system 100 can ensure that certain properties of the target images are consistent with one another.

[0028] The system 100 processes the target text description 102 using a text encoder neural network 120 to generate one or more text token vectors 122 that represent the target text description in a token embedding space.

[0029] The system 100 also processes the face data 104 using a projection neural network 130 to generate one or more face token vectors 132 that represent the properties of the face to be depicted in the target image in the token embedding space.

[0030] The system 100 then generates a face image 112 from the face token vectors 132 and the text token vectors 122 using the diffusion neural network 110.

[0031] In particular, the system 100 uses the diffusion neural network 110 to perform a reverse diffusion process that updates an estimate of the face image 112 at each of multiple update iterations.

[0032] After the last update iteration, the system 100 can generate the face image 112 from the final estimate of the face image 112. i.e., the estimate after being updated at the last update iteration of the plurality of update iterations.

[0033] The diffusion neural network 110 can generally be any appropriate conditional diffusion neural network.

[0034] In particular, at any given update iteration, the diffusion neural network 110 is configured to receive a diffusion input that includes an estimate of the face image 112 and a conditioning input and to process the input to generate a denoising output for the update iteration. Generally, the diffusion input also includes a timestep that defines a noise level for the input image. The conditioning input can include the face token vectors 132, the text token vectors 122, or can be a combined conditioning input that includes both the face token vectors 132 and the text token vectors 122.

[0035] In some implementations, the diffusion neural network 110 performs the reverse diffusion process in pixel space, so that the estimates operated on and generated by the diffusion neural network are images that have values for each pixel that specify color values, e.g., RGB values or another color encoding scheme. In these implementations, the denoising output can generally be any appropriate output that defines a predicted noise component of the current estimate, i.e.. the noise that has been added to the target image to generate the current estimate. For example, the denoising output can be (i) an estimate of the target image (given the current estimate), (ii) an estimate of the noise that has been added to the target image to arrive at the current estimate, (iii) a v-parameterization of the target image and the noise, or (iv) another appropriate type of denoising output.

[0036] In some other implementations, the diffusion neural network 110 performs the reverse diffusion process in latent space, e.g., in a latent space that is lower-dimensional than the pixel space. In these implementations, the denoising output can generally be any appropriate output that defines a predicted noise component of the current estimate, i.e., the noise that has been added to a representation of the target image in the latent space to generate the current estimate. For example, the denoising output can be (i) an estimate of the latent representation of the target image (given the current estimate), (ii) an estimate of the noise that has been added to the latent representation of the target image to arrive at the current estimate, (iii) a v- parameterization of the latent representation of the target image and the noise, or (iv) another appropriate type of denoising output.

[0037] In these implementations, the trained diffusion neural network 110 can be associated with an image encoder to encode images into the latent space and a decoder neural network that receives an input that includes a latent representation of an image and decodes the latent representation to reconstruct the image. For example, the encoder and decoder can have been trained jointly on an image reconstruction objective, e.g., a VAE objective, a VQ-GAN objective, or a VQ-VAE objective.

[0038] Thus, in these examples, after training, the system 100 can use the decoder neural network to generate the face image 112 from the image in the latent space that has been generated using the diffusion neural network 110. The diffusion neural network 110 can generally have any appropriate neural network architecture.

[0039] For example, the diffusion neural network 110 can be a convolutional neural network, e.g., a U-Net, that has multiple convolutional layer blocks. In some of these cases, the diffusion neural network 110 can include one or more cross-attention layer blocks interspersed among the convolutional layer blocks. As will be described below, some or all of the cross-attention blocks can be conditioned on a representation of the conditioning input.

[0040] As another example, the diffusion neural network 110 can be a Transformer neural network that processes the diffusion input through a set of self-attention layers to generate the diffusion output. In these examples, the neural network can also include one or more attention blocks that are conditioned on a representation of the conditioning input.

[0041] Thus, one or more of the layers of the diffusion neural network 110, e.g., one or more cross-attention layers or one or more self-attention layers, can then be conditioned on one or both of the face token vectors 132 or the text token vectors 122. For example, when a layer is conditioned on both the face token vectors 132 or the text token vectors 122. one of the inputs to the layer can be a sequence that includes the face token vectors 132 and the text token vectors 122. When a layer is conditioned on the face token vectors 132 but not the text token vectors 122, one of the inputs to the layer can be a sequence that includes the face token vectors 132. When a layer is conditioned on the text token vectors 122, one of the inputs to the layer can be a sequence that includes the text token vectors 122.

[0042] After the system 100 has generated the face image 112 (also referred to as the target image), the system 100 can, e.g., provide the face image 112 for presentation on a user device, e.g., on the user device of the user that provided the text description 102. As another example, the system 100 can provide the face image 112 as input to another system or store the face image 112 for later use.

[0043] Additionally, as will be described below, the system 100 can use the face image 112 to refine subsequent face images that will be generated by the system 100, e.g., by using the face image 112 to update the face data before using the updated face data to generate another image.

[0044] FIG. 2 is a flow diagram of an example process 200 for generating a new image of a new face. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200. The system receives a target text description that describes a target image to be generated (step 202).

[0045] The system also obtains face data representing properties of a face to be depicted in the target image (step 204).

[0046] As described above, the face data is generally an embedding of the face to be depicted in the target image and represents the properties of the face.

[0047] The system can obtain this face data in any of a variety of ways.

[0048] As one example, the face data can be randomly generated by the system, e.g., by sampling the embedding from a pre-determined distribution over the embedding space.

[0049] As another example, the system can obtain a (single) image of an initial face. For example, a user can provide or identify the image. As yet another example, the user can provide a larger image and the system can crop the larger image to generate the image of the image of the face. For example, the system can process the larger image using a pre-trained face identification neural network that generates, as output, data specifying a region of the larger image that depicts a face, and the system can crop the specified region to generate the image of the initial face.

[0050] The system can then process the image using a face embedding neural network to generate an embedding of the face. The face embedding neural netw ork can generally be any appropriate neural network that can map from an image to an embedding. As one example, the face embedding neural network can include the initial layers of a larger convolutional neural network or vision Transformer neural network that has been trained on an image classification task or a contrastive learning task.

[0051] As yet another example, the system can obtain multiple images, with each image being an image of a respective face. As described, the system can either directly receive these images or can generate them by cropping a corresponding larger image.

[0052] The system can then process each of these images using the face embedding neural network to generate a respective embedding of the respective face in each image and combine the respective embeddings to generate a combined embedding to use as the face data.

[0053] For example, the system can combine the embeddings by computing an interpolation between the respective embeddings.

[0054] In some cases, the w eights for the embeddings in interpolation are fixed by the system, e.g., so that each embedding receives the same uniform weight.

[0055] In some other cases, however, prior to computing the interpolation, the system can receive a user input specifying the weights for each of the respective embeddings in the interpolation. That is, by allowing the user to specify the weights, the system can allow the user to control how much influence a given face depicted in any given one of the input images has on the face that will be depicted in the target image that will be generated by the system.

[0056] In some cases, the system can use the multiple images to refine the faces in images generated by the system. For example, of the images, one or more of the images can have been received as input from a user and another of the images can have been generated using the diffusion neural network conditioned on face tokens representing the respective faces in the one or more images that are received as input from the user. Thus, the system can add images generated by the diffusion neural network to the set of images that are used to generate face tokens in order to gradually adjust the appearance of the faces in the images generated by the neural network.

[0057] The system processes the target text description using a text encoder neural network to generate one or more text token vectors that represent the target text description in a token embedding space (step 206). The text encoder neural network can be any appropriate text processing neural network, e.g., a recurrent neural network (RNN) or a Transformer neural network, and can generally be pre-trained prior to the training of the diffusion neural network.

[0058] The system processes the face data using a projection neural network to generate one or more face token vectors that represent the properties of the face to be depicted in the target image in the token embedding space (step 208).

[0059] That is, the projection neural network processes the embedding in the face data to project the embedding into one or more token vectors that are in the same space as the text token vectors. The projection neural network can generally have any appropriate architecture, e.g., an MLP or other feed-forward neural network.

[0060] As will be described below, the system or another training system can train the projection neural network jointly with the diffusion neural network. Optionally, prior to this j oint training, the diffusion neural network can have been pre-trained on a general image generation objective, e.g., on a large data set of images, that is not specifically tailored to generating images of faces.

[0061] The system then initializes an estimate of the target image, e.g., using noise sampled from a noise distribution (step 210). That is, the system can sample each value in the estimate of the target image from a noise distribution, e.g., a Gaussian distribution. The system then updates the estimate of the target image at each of a plurality of update iterations using the diffusion neural network (step 212). That is, the system generates a final estimate from the initial estimate by, at each update iteration, updating the “current” estimate as of the update iteration.

[0062] As part of the updating, at each update iteration, the system generates a denoising output for the estimate of the image at least in part by processing a denoising input that includes the estimate of the target image, the text token vectors, and the face token vectors using a diffusion neural network to generate an initial denoising output for the target image.

[0063] The system then determines an update to the estimate of the target image based on the denoising output for the estimate of the target image.

[0064] Performing an update iteration is described in more detail below with reference to FIG. 3.

[0065] The system then generates the target image from the estimate of the target image after being updated at the last update iteration of the plurality of update iterations (step 214). As described above, when the estimate is in the pixel space, the system can use the estimate as the target image. When the estimate is in the latent space, the system can process the estimate using the decoder neural network to generate the target image.

[0066] As another example, in some implementations, the estimate is a lower-resolution version of the target image. In these implementations, the system can process the estimate using one or more upscaling neural networks to generate the final target image.

[0067] FIG. 3 is a flow diagram of an example process 300 for performing an update iteration. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0068] The system identifies the (“current”) estimate as of the update iteration (step 302). That is, for the first update iteration, the current estimate will be the initialized estimate. For each subsequent update iteration, the current estimate is the estimate after being updated at the preceding iteration.

[0069] The system then generates a final denoising output for the current estimate of the image (step 304).

[0070] Generally, the system can generate one or more denoising outputs and then combine the one or more denoising outputs to generate the final denoising output. Each denoising output is generated using a different form of conditioning. As one example, the system can process a denoising input that includes the current estimate of the target image, the text token vectors, and the face token vectors using the diffusion neural network to generate an initial denoising output for the target image (step 306). For example, as part of this, the system can generate a combined set of token vectors by combining the text token vectors and the face token vectors, e.g., by generating a sequence that includes both the text tokens and the face tokens, and process a diffusion input that includes the current estimate of the target image and the combined set of token vectors using the diffusion neural network to generate a jointly conditioned denoising output. Thus, this output is conditioned on both the face tokens and the text tokens.

[0071] As another example, the system can process an unconditional diffusion input that includes the current estimate of the target image and does not include the text token vectors or the face token vectors using the diffusion neural network to generate an initial unconditioned denoising output for the initial image (step 308). That is, this denoising output is an “unconditional'’ denoising output because the output is not dependent on either the text token vectors or the face token vectors.

[0072] As another example, the system can process a text-conditional diffusion input that includes the current estimate of the target image and the text token vectors but does not include the face token vectors using the diffusion neural network to generate a text conditioned denoising output (step 310).

[0073] As yet another example, the system can process an image-conditional diffusion input that includes the current estimate of the target image and the face token vectors but does not include the text token vectors using the diffusion neural network to generate a face conditioned denoising output (step 312).

[0074] Once the system has generated two or more of the above outputs, the system can combine them by assigning a respective weight to each of the outputs. That is, the system can compute a weighted sum of the denoising outputs, with each denoising output weighted by a corresponding weight.

[0075] In some implementations, the weights are fixed and determined by the system.

[0076] In some other implementations, however, prior to updating the estimate of the target image at any of the plurality of update iterations, the system receives a user input specifying the weight to assign to each of the generated denoising outputs. This allows the user to control how much influence the text tokens, the face tokens, or both have on the final target image that will be generated. For example, when only the initial denoising output and the initial unconditional denoising output are generated, the final denoising output etat update iteration t can satisfy:

[0077] 6t= W ■t(zt, d,f) + (1 - w) ■ 6t(zt), where w is a weight, fy(zt, d, f) is the initial unconditional denoising output generated by processing a current estimate zt, the text token vectors d, and the face token vectors , and et(zt) is the unconditional denoising output.

[0078] As another example, when the initial denoising output, the face conditioned denoising output and the text conditioned denoising output are all generated, the system can replace the initial unconditional denoising output fy(zt, d, f) in the above with the following combination: where a and c are weights. et(zt, d, f) is the jointly conditioned denoising output, et^ztf)' is the face conditioned denoising output, and is the text conditioned denoising output.

[0079] In some cases, one or two of the weights a, c, or w can be set to zero, reducing the number of denoising outputs that are generated.

[0080] Thus, as one example of the above scheme, the system can set the jointly conditioned denoising output equal to the initial denoising output, i.e., the jointly conditioned output can be the initial denoising output. As another example of this, the system can also generate the text conditioned denoising output, the face conditioned denoising output or both. The system can then combine the face conditioned denoising output and the text conditioned denoising output in accordance with a weight value to generate a separately conditioned denoising output and then combine the jointly conditioned denoising output and the separately conditioned denoising output in accordance with another weight value to generate the initial denoising output.

[0081] As yet another example of this scheme, rather than generate a jointly conditioned denoising output, the system can generate the text conditioned denoising output and the face conditioned denoising output. The system can then combine the face conditioned denoising output and the text conditioned denoising output in accordance with a weight value to generate the initial denoising output.

[0082] The system determines an update to the current estimate of the target image based on the final denoising output for the current estimate of the target image (step 314).

[0083] For example, the system can determine an initial estimate of the final estimate, i.e., the un-noised final estimate, from the diffusion output and then apply an appropriate diffusion sampler, e.g.. the DDPM (Denoising Diffusion Probabilistic Model) sampler, the DDIM (Denoising Diffusion Implicit Model) sampler or another appropriate sampler, to the initial estimate to generate an updated representation.

[0084] Optionally, after the last reverse diffusion iteration, the system can refrain from using the diffusion sampler and can instead use the initial estimate as the updated estimate.

[0085] When the denoising output is an estimate of the target image, the system can directly use the final denoising output as the initial estimate.

[0086] When the denoising output is an estimate of noise, the system can determine the initial estimate from the current estimate and the final denoising output. For example, when the denoising output is an estimate of noise, the system can determine the initial estimate x0from the current estimate xt. the denoising output eg. and a noise level atthat corresponds to the reverse diffusion step (and is defined by the time step t). For example, the system can make this determination as follows:

[0087] More generally, when the denoising output is not the estimate of the target image, e.g., is an estimate of noise or a v-parameterization output, the system can map the denoising output and the current estimate to the initial estimate using a mapping that corresponds to the type of denoising output that is generated by the system.

[0088] As described above, prior to using the diffusion neural network to generate new images, the system or another training system trains the diffusion neural network and the projection neural network on a set of training data.

[0089] As a particular example, the training system can train the diffusion neural network and the projection neural network jointly to optimize a diffusion objective on a training data set that includes a set of training images, a respective text description of each of the training images, and respective face data for each of the images that represents a face depicted in the training image. More specifically, during training, the system can generate the face data from the training image, i.e., the same image that will be used as the target during training.

[0090] FIG. 4 shows an example 400 of the training of the diffusion neural network 110 and the projection neural network 130.

[0091] As shown in FIG. 4, the system receives a training image 410 and a text description 420 tya tennis player’).

[0092] The system uses the training image 410 to generate face data 430, i.e., by first cropping the training image 410 to include only a region depicting the face of the tennis player and then processing the cropped training image using a face embedding neural network. The system then projects the face data 430 to generate the face token vectors and processes the text description 420 using the text encoder neural network to generate the text token vectors.

[0093] The system also combines the training image 410 with sampled noise to generate a noisy image 440. e.g., in accordance with a randomly sampled time index for training on the training image 410.

[0094] The system then processes the noisy image 440, the face token vectors, and the text token vectors using the diffusion neural network 110 to generate a denoising output that defines an estimate of the target image 410.

[0095] The system then trains the projection neural network 130 and the diffusion neural network 110 using a mean squared error (MSE) loss between (i) the estimate of the target image 410 and the target image 410 or (ii) the denoising output and the sampled noise, depending on the format of the denoising output.

[0096] When the diffusion neural network 410 operates in the latent space, the system can instead perform the operations depicted in the example 400 on latent representations of the target image 410 and sampled noise in the latent space. That is, the system can generate a latent representation of the target image 410 using the encoder neural network and perform the operations depicted using the latent representation.

[0097] This specification uses the term “configured7’ in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0098] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0099] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be. or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0100] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0101] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0102] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0103] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only- memory or a random access memory- or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry-. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0104] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way' of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0105] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0106] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.

[0107] Machine learning models can be implemented and deployed using a machine learning framework, .e.g.. a TensorFlow framework or a Jax framework.

[0108] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0109] The computing system can include clients and servers. A client and server are generally remote from each other and ty pically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a chent-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0110] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order show n or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0111] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving a target text description that describes a target image to be generated; processing the target text description using a text encoder neural network to generate one or more text token vectors that represent the target text description in a token embedding space; obtaining face data representing properties of a face to be depicted in the target image; processing the face data using a projection neural network to generate one or more face token vectors that represent the properties of the face to be depicted in the target image in the token embedding space; initializing an estimate of the target image; updating the estimate of the target image at each of a plurality of update iterations, comprising, at each update iteration: generating a denoising output for the estimate of the image, comprising processing a denoising input comprising the estimate of the target image, the text token vectors, and the face token vectors using a diffusion neural network to generate an initial denoising output for the target image; and determining an update to the estimate of the target image based on the denoising output for the estimate of the target image; and generating the target image from the estimate of the target image after being updated at a last update iteration of the plurality of update iterations.

2. The method of claim 1, wherein initializing an estimate of the target image comprises initializing the estimate using noise sampled from a noise distribution.

3. The method of claim 1 or claim 2, wherein the target image is equal to the estimate of the target image after being updated at the last update iteration of the plurality of update iterations.

4. The method of claim 1 or claim 2, wherein the estimate of the target image is a representation in a latent space.

5. The method of claim 4, wherein generating the target image from the estimate of the target image after being updated at a last update iteration of the plurality of update iterations comprises: processing the estimate of the target image after being updated at a last update iteration of the plurality of update iterations using a decoder neural network to generate the target image.

6. The method of any preceding claim, wherein generating a denoising output further comprises: processing a second diffusion input that includes the estimate of the target image and does not include the text token vectors or the face token vectors using the diffusion neural network to generate an initial unconditioned denoising output for the initial image; and combining the initial denoising output for the target image and the initial unconditioned denoising output for the target image in accordance with a guidance weight for the training iteration to generate the denoising output.

7. The method of any preceding claim, wherein processing a denoising input comprising the estimate of the target image, the text token vectors, and the face token vectors using a diffusion neural network to generate an initial denoising output for the target image comprises: generating a combined set of token vectors by combining the text token vectors and the face token vectors; and processing a third diffusion input that comprises the estimate of the target image and the combined set of token vectors using the diffusion neural network to generate a jointly conditioned denoising output.

8. The method of claim 7, wherein the jointly conditioned denoising output is the initial denoising output.

9. The method of claim 7, wherein processing a denoising input comprising the estimate of the target image, the text token vectors, and the face token vectors using a diffusion neural network to generate an initial denoising output for the target image comprises: processing a fourth diffusion input that comprises the estimate of the target image and the text token vectors but does not include the face token vectors using the diffusion neural network to generate a text conditioned denoising output; processing a fifth diffusion input that comprises the estimate of the target image and the face token vectors but does not include the text token vectors using the diffusion neural network to generate a face conditioned denoising output; combining the face conditioned denoising output and the text conditioned denoising output in accordance with a first weight value to generate a separately conditioned denoising output; and combining the jointly conditioned denoising output and the separately conditioned denoising output in accordance with a second weight value to generate the initial denoising output.

10. The method of claim 9, wherein the target text description and data defining the face data are received as input from a user, and wherein the method further comprises: prior to updating the estimate of the target image at any of the plurality of update iterations, receiving a user input specifying the first weight value, the second weight value, or both the first weight value and the second value.

11. The method of any one of claims 1-6, wherein processing a denoising input comprising the estimate of the target image, the text token vectors, and the face token vectors using a diffusion neural network to generate an initial denoising output for the target image comprises: processing a fourth diffusion input that comprises the estimate of the target image and the text token vectors but does not include the face token vectors using the diffusion neural network to generate a text conditioned denoising output; processing a fifth diffusion input that comprises the estimate of the target image and the face token vectors but does not include the text token vectors using the diffusion neural network to generate a face conditioned denoising output; and combining the face conditioned denoising output and the text conditioned denoising output in accordance with a first weight value to generate the initial denoising output.

12. The method of claim 11, wherein the target text description and data defining the face data are received as input from a user, and wherein the method further comprises: prior to updating the estimate of the target image at any of the plurality of update iterations, receiving a user input specifying the first weight value.

13. The method of any preceding claim, wherein obtaining face data representing properties of a face to be depicted in the target image comprises: obtaining an image of an initial face; and processing the image using a face embedding neural network to generate an embedding of the face, wherein the embedding is the face data that represents the properties of the face.

14. The method of any one of claims 1-12, wherein obtaining face data representing properties of a face to be depicted in the target image comprises: obtaining a plurality of images, each image being an image of a respective face; and processing each image using a face embedding neural network to generate a respective embedding of the respective face in each image; and combining the respective embeddings to generate a combined embedding, wherein the combined embedding is the face data that represents the properties of the face.

15. The method of claim 14, wherein combining the respective embeddings comprises computing an interpolation between the respective embeddings.

16. The method of claim 15, wherein the target text description and data defining the face data are specified in an input received from a user, and wherein the method further comprises: prior to computing the interpolation, receiving a user input specifying weights for each of the respective embeddings in the interpolation.

17. The method of any one of claims 14-16, wherein one or more of the plurality of images are received as input from a user and wherein another of the plurality of images has been generated using the diffusion neural network conditioned on face tokens representing the respective faces in the one or more images that are received as input from the user.

18. The method of any one of claims 13-17, wherein obtaining each image comprises: receiving a corresponding larger image; processing the corresponding larger image using a face detection neural network to identify a region of the larger image that depicts a face; and cropping the larger image based on the region of the larger image that depicts the face to generate the image.

19. The method of any preceding claim, wherein the target text description and data specifying the face data are received as input from a user of a user device.

20. The method of claim 19. further comprising: providing the target image for presentation to the user on the user device.

21. The method of any preceding claim, wherein the diffusion neural network and the projection neural network have been trained jointly to optimize a diffusion objective on a training data set that comprises a set of training images, a respective text description of each of the training images, and respective face data for each of the images that represents a face depicted in the training image.

22. The method of claim 21 , wherein the text encoder neural network is pre-trained and held frozen during the j oint training.

23. The method of claim 21 or claim 22. wherein the diffusion neural network has been pre-trained without the projection neural network and without being conditioned on face tokens prior to the joint training.

24. The method of any preceding claim, wherein each diffusion input further comprises a timestep that identifies the update iteration.

25. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1 -24.

26. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-24.

Images