Combining text of any style with images

Abstract

A method, system, and apparatus comprising a computer program encoded on a computer storage medium for generating an output image using a text-to-image generative model from a text prompt. The text-to-image generative model comprises one or more sets of first parameters shared across different styles. The text-to-image generative model also comprises one or more sets of second parameters specific to a particular style.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 467,281, filed May 17, 2023. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated herein by reference.

Background Art

[0002] This specification relates to the processing of images using neural networks.

[0003] A machine learning model receives an input and generates an output, for example, a predicted output, based on the received input. Some machine learning models are parametric models that generate an output based on the received input and the values of the model's parameters.

[0004] Some machine learning models are deep models that use multiple layers of the model to generate an output for the received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, and each layer applies a non - linear transformation to the received input to generate an output.

Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that implements a text - to - image generation model configured to generate a text - conditioned image.

[0006] In some cases, a text prompt specifies a particular style for an image, and the text-to-image generation model processes the text prompt to produce an image that has the specific style specified by the text prompt. In other cases, a text prompt specifies a particular object (e.g., a particular person, a particular animal, a particular car, a particular boat, etc.) or a particular instance of an object that should be displayed in the image, and the text-to-image generation model processes the text prompt to produce an image that depicts the particular object / instance specified by the text prompt. In other cases, a text prompt specifies both a particular style for an image and a particular object / instance that should be displayed in the image, and the text-to-image generation model processes the text prompt to produce an image that (i) has the particular style and (ii) depicts the particular object / instance.

[0007] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. The systems described herein implement text-to-image generative models that are easily generalizable across a variety of image styles. The systems described enable users to adapt the text-to-image generative model to a particular style by providing only a few, or in some cases just one, reference images of that style. When augmented with one or more style-specific adapters inserted into each of one or more layers of the model, the text-to-image generative model can be effectively and efficiently fine-tuned to fit a particular style, as it only needs to learn a small fraction of the model parameters specific to the style of images the model generates. Consequently, the adaptation process is computationally significantly less demanding than the pre-training process. Thus, unlike the pre-training process, which typically must be run on distributed computing systems such as data centers, the adaptation process can be run on much less powerful hardware, to name just a few, such as mobile phones or other wearable devices, desktop or laptop computers, or other internet-enabled devices installed in the user's home. The text-to-image generation systems described herein offer better separation of content and style, and are therefore better able to generate images with specific objects and / or styles specified by user prompts. In general, improved text-to-image generation systems are provided because the system has a better ability to generate images in accordance with instructions provided by the user in user prompts.

[0008] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0009] [Figure 1A]This document presents an exemplary image generation system that includes a text-to-image generation model. [Figure 1B] This figure shows an exemplary architecture for performing inference using a text-to-image generative model. [Figure 1C] This figure shows an exemplary architecture for fine-tuning a text-to-image generative model. [Figure 2] This is an illustrative process flowchart for generating an output image using a text-to-image generative model. [Figure 3] This is a flowchart of a substep of one step in the process shown in Figure 2. [Figure 4] This is a flowchart of another exemplary process for generating an output image by using a text-to-image generative model. [Figure 5] This is a flowchart of a substep of one step in the process shown in Figure 4. [Modes for carrying out the invention]

[0010] Similar reference numbers and names in various drawings refer to the same elements.

[0011] Figure 1A shows an exemplary image generation system 100 that includes a text-to-image generation model 120. The image generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

[0012] The image generation system 100 receives a text prompt 102 from the system's user and generates an output image 104 based on the text prompt 102 by using a text-to-image generation model 120. The text prompt 102 specifies a particular style for the output image 104 and, if applicable, specifies a particular object (e.g., a particular person, a particular animal, a particular car, a particular boat, a particular house, etc.) that should be displayed in the output image 104. Optionally, the text prompt 102 may further include a negative prompt specifying content, such as objects, that should not be displayed in the output image 104.

[0013] The output image 104 is the image described by the text prompt 102. That is, the output image 104 has a specific style and, in some cases, depicts a specific object specified by the text prompt 102.

[0014] As used herein, “image” may mean a digital image such as a two-dimensional or three-dimensional image, or a sequence of frames of video. An image may have multiple pixels, and each pixel may have multiple values.

[0015] As used herein, an image “style” specifies a particular set of visual characteristics, including color, texture, and brightness, that are represented by an image with an image style, such as color arrangement, light and dark transitions, shading, and texture features. Different images with different image styles are usually visually distinguishable from one another. For example, photographs, cartoons, caricatures, oil paintings, sketches, watercolors, planar illustrations, and 3D renderings may be considered different image styles. An image style may not include the geometry of objects within the image.

[0016] In Figure 1A, the image generation system 100 receives the text prompt 102, "3D rendering style dog." The image generation system 100 then uses the text prompt 102 and the text-to-image generation model 120 to generate an output image 104 depicting a dog in a 3D rendering style. In this example, the text prompt 102 specifies both a specific style for the image ("3D rendering style") and a specific object to be displayed in the image ("dog").

[0017] For example, the image generation system 100 can provide an output image 104 for presentation to the user on the user's computer, for example, as a response to a user who submitted a text prompt 102. As another example, the image generation system 100 can store the output image 104 in a repository for later use.

[0018] Figure 1B shows an exemplary architecture 100b for performing inference using a text-to-image generative model 120. In addition to the text-to-image generative model 120, architecture 100b also includes a text encoder neural network 130, and optionally an image encoder neural network and an image decoder neural network. The components shown in Figure 1B can be implemented by a system, for example, an image generation system 100, which is implemented as a computer program on one or more computers in one or more locations.

[0019] The text-to-image generation model 120 can be configured to map the text prompt 102 to the output image 104 in any of a variety of ways. In some embodiments, the text-to-image generation model 120 can be configured to perform the mapping by, for example, executing an inverse diffusion process in the pixel space, i.e., by gradually removing noise from a noisy version of the output image to obtain the output image, as described in Chitwan Saharia, et al., "Photorealistic text-to-image diffusion models with deep language understanding" (Advances in Neural Information Processing Systems, 35:36479-36494, 2022).

[0020] In these embodiments, at each of a plurality of inverse diffusion steps throughout the inverse diffusion process, the text-to-image generation model 120 generates an updated intermediate representation 122 of the output image based on either an intermediate representation 112 of the output image generated at the time of a previous inverse diffusion step or (if the inverse diffusion step is the first inverse diffusion step of the inverse diffusion process) a predetermined initial intermediate representation of the output image.

[0021] The updated intermediate representation 122 can be generated directly, for example, if the model output of the text-to-image generation model 120 includes the updated intermediate representation, or indirectly, for example, if the model output includes a noise term calculated by the model of the inverse diffusion step and the updated intermediate representation can then be generated by removing at least a portion of the noise from the intermediate representation 112 according to the noise term. The noise term is an estimated value of the noise calculated by the model and added to the output image to reach the noisy version of the output image.

[0022] In some other embodiments, the text-to-image generation model 120 can be configured to do this, for example, by performing reverse diffusion processing in a latent space that is lower dimensional than the pixel space. During this reverse diffusion process, for example, as described in Aditya Ramesh, et al., "Hierarchical text-conditional image generation with clip latents" (arXiv preprint arXiv:2204.06125), Robin Rombach, et al., "High-resolution image synthesis with latent diffusion model" (Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022), noise is gradually removed from a noisy version of the latent image to provide the latent image. The values of the pixels in the latent image include learned latent values rather than color values.

[0023] In these embodiments, at each of the plurality of reverse diffusion steps throughout the reverse diffusion process, the text-to-image generation model 120 generates a updated intermediate representation 122 of the latent image, i.e., directly or indirectly, based on either an intermediate representation 112 of the latent image generated at the time of a previous reverse diffusion step or (if the reverse diffusion step is the first reverse diffusion step of the reverse diffusion process) a predetermined initial intermediate representation of the output image.

[0024] Furthermore, in these embodiments, the text-to-image generation model 120 can be associated with an image encoder neural network that encodes an image into the latent space and a decoder neural network that receives the latent image and decodes the latent image to generate an output image. For example, the image encoder neural network and the image decoder neural network can be jointly trained for the purpose of image reconstruction, e.g., for VAE purposes, VQ-GAN purposes, or VQ-VAE purposes.

[0025] In yet another embodiment, the text-to-image generation model 120 may be configured to do this by iteratively predicting masked tokens through a decoding process in a discrete token space, as described, for example, in Huiwen Chang, et al. "Muse: Text-to-image generation via masked generative transformers" (arXiv preprint arXiv:2301.00704, 2023).

[0026] In these embodiments, at each of the multiple decoding steps in the decoding process, the text-to-image generation model 120 generates an updated intermediate representation 122 of the output image, which includes multiple discrete visual tokens selected from a given vocabulary of discrete visual tokens, based on the intermediate representation 112 of the output image generated at the time of the previous decoding step.

[0027] The updated intermediate representation 122 can be generated by gradually replacing the mask tokens previously included in the intermediate representation 112 with discrete visual tokens selected from a given vocabulary of discrete visual tokens. Specifically, the model output can define one or more score distributions across a given vocabulary of discrete visual tokens for each of several spatially distinct locations in the output image, and the system can use one or more score distributions to determine one or more discrete visual tokens corresponding to those spatially distinct locations. Once selected, the selected discrete visual tokens replace the mask tokens, i.e., are placed in the positions of the mask tokens in the intermediate representation.

[0028] Multiple spatially distinct locations may contain subsets (possibly overlapping subsets) of the output image, which can be spatially displaced relative to each other. For example, each location may correspond to an image patch of the output image. An image patch of the output image is a subset of pixels in the output image. Typically, each image patch contains multiple consecutive pixels of the output image.

[0029] In some embodiments, the updated intermediate representation 122 in the final step can correspond to the output image 104, or, as described above, the output image 104 can be derived from the updated intermediate representation 122 through appropriate processing.

[0030] The text encoder neural network 130 is configured to process the text prompt 102 and generate one or more text embeddings based on the text prompt 102. As used herein, “embedding” is a sequence of vectors of one or more numerical values, e.g., floating-point values ​​or other values, where each vector has a predetermined number of dimensions.

[0031] For example, if the text prompt 102 contains a description of a specific style that the output image should have, the text encoder neural network 130 can process the text prompt 102 to generate contextual text embeddings that correspond to the description of that specific style.

[0032] As another example, if the text prompt 102 includes both a description of a specific object to be displayed in the output image and a description of a specific style the output image should have, the text encoder neural network 130 can process the text prompt 102 to generate two text embeddings: an object text embedding corresponding to the description of the specific object and a context text embedding corresponding to the description of the specific style. In some embodiments, the text prompt 102 can be made more easily identifiable as a style by adhering to a specific format.

[0033] The text encoder neural network 130 includes any suitable number (e.g., 2, 5, or 10) of any suitable type of neural network layers (e.g., embedding layers, fully connected layers, etc.) and can be connected in any suitable configuration (e.g., as a directed graph of layers). In embodiments, the text encoder neural network 130 can be configured as, for example, a feedforward neural network such as a transformer neural network, or a recurrent neural network such as a long short-term memory neural network.

[0034] The text-to-image generative model 120 can be configured to have any suitable neural network architecture that, according to the parameters of the text-to-image generative model 120, processes a model input including an intermediate representation 112 and one or more text embeddings, and generates an updated intermediate representation 122 as a model output.

[0035] In particular, the parameters of the text-to-image generative model 120 include one or more sets of first parameters 142 and multiple sets of second parameters 144. One or more sets of first parameters 142 are shared across different styles. On the other hand, one or more sets of second parameters 144a to n included in the multiple sets of second parameters 144 are each specific to a different style.

[0036] For example, Figure 1B shows that multiple sets of second parameters 144 include (i) one or more sets of second parameters 144a specific to style 1, (ii) one or more sets of second parameters 144b specific to style 2, ..., and (iii) one or more sets of second parameters 144n specific to style N, where N is any integer greater than or equal to 1.

[0037] To generate output images with different styles, the text-to-image generation model 120 uses the same set of one or more first parameters 142, but uses different sets of one or more second parameters 144a to n. Therefore, an output image with one style is generated by a different combination of parameters of the text-to-image generation model 120 than an output image with a different style.

[0038] In other words, some of the parameters of the text-to-image generative model 120 (i.e., a set of one or more first parameters 142) are used to generate output images with multiple different styles, while other parts of the parameters of the text-to-image generative model 120 (e.g., a set of one or more second parameters 144a) are used only to generate output images with a specific style corresponding to those parameters (e.g., an output image of style 1 corresponding to a set of one or more second parameters 144a).

[0039] In Figure 1B, for example, to facilitate the generation of output images with different styles, the text-to-image generation model 120 maintains multiple sets of second parameters 144 corresponding to different styles 1 to N. Then, depending on the style specified by the received text prompt 102, the image generation system 100 selects one or more sets of second parameters, for example, one or more sets of second parameters 144a, from the multiple sets of second parameters 144 that correspond to the specified style, and uses this in inference calculations using the text-to-image generation model 120, i.e., together with one or more sets of first parameters 142.

[0040] For a particular style, before using the text-to-image generation model 120 to generate an output image having that particular style, the image generation system 100 or another training system fine-tunes the text-to-image generation model 120 to fit the model to that particular style by learning updated values ​​of one or more sets of second parameters 144a~n that correspond to that particular style.

[0041] In some embodiments, the text-to-image generative model 120 is pre-trained on a set of training images by, for example, the image generation system 100 or another training system, based on the optimization of a denoising score matching objective, for example using squared error loss (if configured to perform a despreading process), or based on the optimization of cross-entropy loss (if configured to perform a decoding process), and learns pre-trained values ​​for one or more sets of first parameters 142 of the text-to-image generative model 120.

[0042] One or more sets of first parameters 142 of the text-to-image generative model 120 can be learned independently of multiple sets of second parameters 144 corresponding to different styles 1 to N. That is, multiple sets of second parameters 144a to n were not used to process training images during the pre-training of the text-to-image generative model 120, and therefore were not learned along with one or more sets of first parameters 142 during pre-training.

[0043] Figure 1C shows an exemplary architecture 100c for fine-tuning a text-to-image generation model 120 with fine-tuning data 150. The components shown in Figure 1C can be implemented by a system, for example, an image generation system 100, which is implemented as a computer program on one or more computers in one or more locations.

[0044] The fine-tuning data 150 includes one or more image-text pairs 152. Each image-text pair 152 includes (i) an image having a specific style, and (ii) a text prompt that includes both the content of the image, e.g., a description of the object to be displayed in the image, and a description of the specific style of the image. For example, the specific style can be the same as the style specified in the text prompt 102 in Figure 1A. In particular, due to the way the text-to-image generation model 120 adapts to a specific style, the fine-tuning data 150 may include a small number, or possibly just one, of image-text pairs 152, such as 1 to 10.

[0045] The fine-tuning data can be obtained in one of several ways. For example, the image generation system 100 can receive the fine-tuning data as an upload from a user of the system, for example, the same user who provided the text prompt 102, via a data communication network, for example, using an application programming interface (API) made available by the system. As another example, the image generation system 100 can receive input from a user specifying which data already held by the system or another system accessible by the system should be used to fine-tune the text-to-image generation model.

[0046] The exact architecture of the text-to-image generative model 120 may vary depending on the embodiment, but generally, the text-to-image generative model 120 includes multiple neural network layers, and the parameters of the text-to-image generative model 120 include the parameters of each layer of the text-to-image generative model 120, e.g., weights and, optionally, biases.

[0047] At a high level, one or more sets of first parameters 142 can include the parameters of each of several layers of the text-to-image generative model 120. Each of these layer parameters is pre-trained and remains fixed to the pre-trained values ​​during the fine-tuning process. Multiple sets of second parameters 144 can include the parameters of several other layers of the text-to-image generative model 120. Each of these other layer parameters will be learned as part of the fine-tuning process.

[0048] As shown in Figure 1C, in some embodiments, the text-to-image generation model 120 includes L transformer blocks 140, for example, as part of a masked generational transformer architecture, a diffuse transformer (DiT) architecture, or a general-purpose visual transformer (UViT) architecture, where L is any integer greater than or equal to 1. Each transformer block 140 includes one or more attention layers, e.g., one or more self-attention layers, one or more cross-attention layers, or both, and an adapter (i.e., one or more additional neural network layers for adapting to a particular style). Each transformer block 140 is configured to process a block input derived from an intermediate representation and generate a block output based on the application of an attention mechanism.

[0049] When there are multiple transformer blocks, they can be stacked, that is, arranged in order such that the output of any transformer block except the last one becomes an input to another transformer block. For example, the intermediate representation 112 may be received by the first transformer block in a sequence of L transformer blocks, and the updated intermediate representation 122 may be the output of the last transformer block in the sequence of L transformer blocks, or generated from that output.

[0050] In the example in Figure 1C, each transformer block 140 includes a self-attention layer ("Self-Attn") followed by a cross-attention layer ("Cross-Attn"). The self-attention layer is configured to process the input to the transformer block 140 by applying a self-attention mechanism to generate the output of the self-attention layer. The cross-attention layer is configured to process (i) the output of the self-attention layer and (ii) one or more text embeddings generated by the text encoder neural network 130 based on the text prompt 102 by applying a cross-attention mechanism to generate the output of the cross-attention layer.

[0051] Next, the adapter ("A") includes one or more adapter layers that process the output of the cross-attention layer to produce the adapter's output. In some embodiments, the one or more adapter layers may include one or more fully connected layers. For example, the one or more adapter layers may include two feedforward layers, each of which applies a linear transformation and is separated by an activation layer that applies a nonlinear activation function, such as a ReLU activation function or a GeLU activation function. The adapter may additionally or alternatively include other types of neural network layers, such as convolutional layers, attention layers, and pooling layers.

[0052] In particular, adapters are lightweight; that is, they have a very limited number of parameters compared to attention layers (or other model components). For example, an adapter can contain an order of magnitude fewer parameters than the self-attention layer or cross-attention layer included in transformer block 140.

[0053] In the example in Figure 1C, each transformer block 140 also includes a residual connection layer that combines the output of the last cross-attention layer with the output of the adapter to produce the output of the transformer block 140. In some other examples where each transformer block 140 does not include such a residual connection layer, the output of the last cross-attention layer can be used as the output of the transformer block 140. In yet another example where each transformer block 140 includes additional components, e.g., a multilayer perceptron layer(s), each transformer block 140 may include additional adapters, e.g., additional adapters having a similar architecture placed after the multilayer perceptron layer(s). The output of the transformer block 140 can then be produced, for example, by combining the output of the additional adapter(s) with the output of the multilayer perceptron layer(s).

[0054] In these embodiments, the set of one or more first parameters 142 shared across different styles includes the parameters of one or more attention layers contained in each of the L transformer blocks 140. In the example of Figure 1C, the set of one or more first parameters 142 therefore includes the parameters of the self-attention layer ("Self-Attn") and the cross-attention layer ("Cross-Attn") contained in each of the L transformer blocks 140.

[0055] A set of one or more second parameters specific to a particular style, for example, a set of one or more second parameters 144a, includes the parameters of one or more adapter layers included in each adapter contained in each of the L transformer blocks 140. For example, a set of one or more second parameters 144a could include the parameters of a feedforward layer included in an adapter ("A") in each of the L transformer blocks 140.

[0056] The parameters of the attention layer within each transformer block are shared across many styles, but the parameters of the layer within each adapter are specific to a particular style. In other words, different adapters are used to generate images of different styles (although the same attention layer is used throughout).

[0057] In some embodiments, the text-to-image generative model 120 includes, for example, a U-net architecture, a plurality of convolutional blocks, e.g., ResNet blocks, each containing one or more convolutional layers, and one or more adapters, e.g., adapters for each of the plurality of convolutional blocks. The adapters can be inserted at any suitable position within the plurality of convolutional blocks. Each adapter can contain any suitable number (e.g., one, two, or five) of any suitable type of neural network layers (e.g., convolutional layers, fully connected layers, etc.).

[0058] In these embodiments, one or more sets of first parameters 142 shared across different styles include parameters for multiple convolutional blocks. One or more sets of second parameters specific to a particular style, for example, one or more sets of second parameters 144a, include parameters for neural network layers included in each of one or more adapters.

[0059] For example, if the text-to-image generative model 120 is configured to generate an output image by iteratively predicting masked tokens, the image generation system 100 can fine-tune the text-to-image generative model 120 based on the fine-tuning data 150 by optimizing the cross-entropy fine-tuning objective function as follows, and learn the values ​​of one or more sets of second parameters specific to a particular style.

number

number

[0060] Specifically, the cross-entropy fine-tuning objective function in this example is the fine-tuning data D. tr For each image-text pair included in the dataset, we evaluate the difference between (i) the target discrete visual token generated by an image encoder neural network from the image included in the image-text pair, and (ii) the predicted discrete visual token generated using a text-to-image generative model based on the text included in the image-text pair and the masked discrete visual token.

[0061] In detail, during the fine-tuning process, the image generation system 100 updates the values ​​of one or more sets of second parameters while fixing the values ​​of one or more sets of first parameters 142 that are shared between different styles (e.g., pre-trained values). In the example in Figure 1C, the image generation system 100 can use the fine-tuning data 150 to generate updated values ​​for the parameters of the neural network layers included in the adapter of the text-to-image generation model 120, fixing and holding the parameter values ​​of the self-attention layer and cross-attention layer to pre-trained values.

[0062] The image generation system 100 can incorporate any number of techniques to improve the speed, effectiveness, or both of the fine-tuning process. For example, the system can use weight sharing when updating the parameters of the neural network layers included in the adapters of the text-to-image generation model 120. That is, at any point during the fine-tuning process, the parameters of the neural network layer included in one of the adapters have the same values ​​as the parameters of the neural network layer included in the other of the adapters included in the text-to-image generation model 120. Assuming that weight sharing is applied in the example in Figure 1C, the total number of trainable parameters during the fine-tuning process is reduced (approximately) by the number of transformer blocks L.

[0063] Once the fine-tuning is complete, the image generation system 100 can deploy the fine-tuned text-to-image generation model 120 and use the model to generate a new image with a specific style according to a text prompt submitted to the system by the user. Alternatively or additionally, the image generation system 100 can provide data specifying the fine-tuning text-to-image generation model 120, for example, data specifying updated values ​​for a set of one or more second parameters, and, optionally, an architecture of adapters included in the text-to-image generation model 120 for performing a conditional image generation task on another system, i.e., for deployment to generate a new image with a specific style.

[0064] Figure 2 is a flowchart of an exemplary process 200 for generating an output image by using a text-to-image generation model. For convenience, process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, an image generation system, e.g., the image generation system 100 depicted in Figure 1B, can be appropriately programmed in accordance with this specification and perform process 200.

[0065] A text-to-image generative model includes one or more sets of first parameters that are shared across different styles. A text-to-image generative model also includes one or more sets of second parameters specific to a particular style.

[0066] In the example in Figure 2, the text-to-image generative model is configured to generate an output image by iteratively predicting masked tokens. In this embodiment, the text-to-image generative model may be configured as the Muse text-to-image model (as referenced above) or another masked generative transformer model.

[0067] In some embodiments, the text-to-image generation model includes a plurality of transformer blocks. Each transformer block includes an adapter comprising (i) one or more attention layers, e.g., a self-attention layer, a cross-attention layer, or both, and (ii) one or more additional neural network layers.

[0068] In those embodiments, one or more first sets of parameters may include parameters for one or more attention layers included in each of the multiple transformer blocks, and one or more second sets of parameters may include parameters for one or more neural network layers included in each of the adapters of the multiple transformer blocks.

[0069] To generate the output image, the system repeatedly performs iterations of process 200 using a text-to-image generative model, updating the intermediate representation of the output image at each of the multiple decoding steps throughout the decoding process. That is, process 200 can be performed at each of the multiple decoding steps throughout the decoding process. By repeatedly performing iterations of process 200, the system can generate the output image.

[0070] Before the first iteration of process 200, i.e., before the first decoding step, the system receives a text prompt and then processes the text prompt using a text encoder neural network to generate contextual text embeddings based on the text prompt. The text prompt specifies a particular style that the output image should have. Optionally, the text prompt may also include a negative prompt specifying content that should not be displayed in the output image, such as objects.

[0071] The system obtains an intermediate representation of the output image from the decoding step (step 202). If the decoding step is the first decoding step in the decoding process, the intermediate representation is the initial intermediate representation. In subsequent decoding steps, the intermediate representation is the updated intermediate representation generated in the previous decoding step.

[0072] The initial intermediate representation and the updated intermediate representation have the same dimensions but different values. That is, the initial intermediate representation contains multiple variables, and the updated intermediate representation contains the same number of variables, but the values ​​of these variables are generally different.

[0073] An initial intermediate representation, which has the same dimensions as the updated intermediate representation, can consist entirely of "mask" tokens. A mask token is an intermediate representation token that means the corresponding token in the final intermediate representation has not yet been selected.

[0074] A "token" can represent an individual data element from a discrete (e.g., finite) set of data elements. For example, a token may be one of the discrete visual tokens in a vocabulary corresponding to a codebook, or it may represent one. For this purpose, a token can be in the form of an ordered set of numbers. For example, each token may contain one or more numbers. Examples of such codebooks and the discrete visual tokens corresponding to them are described in Huiwen Chang, et al. "Muse: Text-to-image generation via masked generative transformers" (arXiv:2301.00704, 2023) and Huiwen Chang, et al. "MaskGIT: Masked generative image transformer" (arXiv:2202.04200, 2022).

[0075] The system processes the intermediate representation and contextual text embedding of the output image from the decoding step using both one or more first sets of parameters and one or more second sets of parameters, according to the text-to-image generative model, and generates the first model output from the decoding step (step 204). That is, the system uses the intermediate representation and contextual text embedding to perform a first forward propagation through one or more attention layers and adapters contained in each of the multiple transformer blocks contained in the text-to-image generative model.

[0076] The first model output may include a set of logits for each of several spatially distinct locations in an image, corresponding to an index in a given codebook. A codebook can define a mapping between discrete visual tokens in a given vocabulary of discrete visual tokens corresponding to a given codebook and their corresponding indices. Each set of logits may include a logit score for each of several indices in a given codebook. For example, the system may generate a set of logits for each image patch in the output image.

[0077] The system processes the intermediate representation and contextual text embeddings of the output image from the decoding step using one or more first sets of parameters and without using one or more second sets of parameters, using the text-to-image generative model, and generates a second model output from the decoding step (step 206). That is, the system uses the intermediate representation and contextual text embeddings to perform a second forward pass through one or more attention layers contained in each of the multiple transformer blocks contained in the text-to-image generative model. In particular, the adapter is not used to process the intermediate representation (or data derived from the intermediate representation) in the second forward pass.

[0078] Similar to the first model output, the second model output may contain a set of logits for each of several spatially distinct locations in the image, corresponding to a given codebook index. However, because different sets of parameters of the text-to-image generation model are used to produce the first and second model outputs, the sets of logits included in the second model output are generally different from the sets of logits included in the first model output.

[0079] If in step 204 or step 206 one or more attention layers include one or more cross-attention layers, each cross-attention layer may apply the cross-attention mechanism to the output of a preceding layer, e.g., the output of a self-attention layer or another cross-attention layer, and contextual text embeddings to generate the output of the cross-attention layer.

[0080] The system generates an updated intermediate representation of the output image from the decoding step based on the first and second model outputs from the decoding step (step 208). The generation of the updated intermediate representation is described below with reference to Figure 3, which is a flowchart of the substep of step 208 of the process in Figure 2.

[0081] The system determines the difference between the first model output and the second model output of the decoding step (step 302). Such a difference may include the difference between the set of logits included in the first model output and the set of logits included in the second model output for each of several spatially distinct locations in the image.

[0082] The system generates, at least partially, a set of logits for each of several spatially distinct locations in the output image, corresponding to the index in a given codebook (step 304).

[0083] For example, for any position in an image, the set of logits can be calculated as follows:

number

number

number

[0084] For each of several spatially distinct locations in the output image, the system samples one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to a given codebook, according to its respective set of logits (step 306). For example, the system can determine an index by greedily selecting the best scoring index from each set of logits computed in step 304, or by sampling, for example, by using nuclear sampling or another sampling technique, and then use the discrete visual tokens that map to the determined index as the corresponding sampled discrete visual tokens.

[0085] In each decoding step, the system typically samples one or more discrete visual tokens for each of at least one but not all of several spatially distinct locations in the output image. In some embodiments, in each decoding step, the system may select discrete visual tokens for each of a corresponding number of image patches that have the highest logit score among all the image patches contained in the output image (e.g., measured by the highest logit score of any token in the vocabulary). The number of corresponding image patches in each decoding step may be determined according to a predetermined schedule defined across multiple decoding steps in the decoding process, such as a cosine schedule, exponential schedule, squared schedule, or linear schedule.

[0086] In some embodiments, the system can sample one or more discrete visual tokens for each spatially distinct location in the output image, independently and in parallel with the sampling of one or more discrete visual tokens for other spatially distinct locations. That is, multiple discrete visual tokens can be sampled simultaneously for different locations in the output image.

[0087] The system generates an updated intermediate representation by replacing mask tokens present at several spatially distinct locations within the output image with sampled discrete visual tokens (step 308). Thus, the updated intermediate representation contains one or more sampled discrete visual tokens instead of the mask tokens previously included in the intermediate representation. In this way, at each decoding step, the system generates a new updated intermediate representation with fewer mask tokens.

[0088] Returning to Figure 2, if the decoding step is not the last decoding step in the decoding process, the system can perform the next iteration of process 200, and the updated intermediate representation will be used as the intermediate representation in the next iteration of process 200. Alternatively, if the decoding step is the last decoding step, the system can perform further processing on the updated intermediate representation to generate the output image.

[0089] For example, the system can use a decoder neural network to process the updated intermediate representation to generate a decoder output, which can then be used as the output image. The decoder neural network may be configured through training to map spatially distinct, location-based discrete visual tokens in the output image to one or more pixel values ​​for their respective locations. In another example, the updated intermediate representation in the final step corresponds to the output image.

[0090] Figure 4 is a flowchart of another exemplary process 400 for generating an output image by using a text-to-image generation model. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, an image generation system, e.g., image generation system 100 depicted in Figure 1B, can be appropriately programmed according to this specification to perform process 400. Process 400 differs from process 200 in that it extends the system's conditional image generation capabilities to enable the generation of new images that not only have a specific style but also depict specific content, such as specified instances of specific objects.

[0091] A text-to-image generative model includes one or more first sets of parameters shared across different styles. A text-to-image generative model includes one or more second sets of parameters specific to a particular style. A text-to-image generative model further includes one or more third sets of parameters specific to a particular object or instance of an object. In the example in Figure 4, the text-to-image generative model is configured to produce an output image by iteratively predicting masked tokens. In embodiments, the text-to-image generative model may be configured as a Muse text-to-image model (as referenced above) or another masked generative transformer model.

[0092] In some embodiments, the text-to-image generation model includes multiple transformer blocks. Each transformer block includes (i) one or more attention layers, e.g., a self-attention layer, a cross-attention layer, or both, and (ii) two adapters (a first adapter and a second adapter), each containing one or more neural networks. The two adapters are, for example, one adapter for fitting a particular style and the other for fitting a particular object / instance.

[0093] In those embodiments, one or more sets of first parameters may include parameters for one or more attention layers included in each of the multiple transformer blocks, one or more sets of second parameters may include parameters for one or more neural network layers included in the first adapter of each of the multiple transformer blocks, and one or more sets of third parameters may include parameters for one or more neural network layers included in the second adapter of each of the multiple transformer blocks.

[0094] To generate the output image, the system repeatedly performs iterations of process 400 using a text-to-image generative model, updating the intermediate representation of the output image at each of the multiple decoding steps throughout the decoding process. That is, process 400 can be performed at each of the multiple decoding steps throughout the decoding process. By repeatedly performing iterations of process 400, the system can generate the output image.

[0095] Prior to the first iteration of process 400, i.e., before the first decoding step, the system receives a text prompt specifying a particular style the output image should have, as well as specific objects / instances to be depicted in the output image. Optionally, the text prompt may further include a negative prompt specifying content, such as objects, that should not be displayed in the output image.

[0096] Next, the system processes at least a portion of the text prompt, for example, the portion containing a description of a particular style, using a text encoder neural network to generate contextual text embeddings. The system also processes at least a portion of the text prompt, for example, the portion containing a description of a particular style / instance, using a text encoder neural network to generate object text embeddings.

[0097] The system obtains an intermediate representation of the output image from the decoding step (step 402). If the decoding step is the first decoding step in the decoding process, the intermediate representation is the initial intermediate representation. In subsequent decoding steps, the intermediate representation is the updated intermediate representation generated in the previous decoding step. The initial intermediate representation has the same dimensions as the updated intermediate representation and consists entirely of "mask" tokens.

[0098] The system processes the intermediate representation and contextual text embedding of the output image of the decoding step using both one or more first sets of parameters and one or more second sets of parameters, according to the text-to-image generative model, and generates the first model output of the decoding step (step 404). That is, the system uses the intermediate representation and contextual text embedding to perform a first forward propagation through one or more attention layers and a first adapter, each of the multiple transformer blocks included in the text-to-image generative model.

[0099] The first model output may include a set of logits for each of several spatially distinct locations in the image, corresponding to an index in a given codebook (which may be the same codebook mentioned above, see Figure 2). For example, the system can generate a set of logits for each image patch in the output image.

[0100] The system processes the intermediate representation and contextual text embeddings of the output image from the decoding step using one or more first sets of parameters, without using one or more second sets of parameters or one or more third sets of parameters, by the text-to-image generative model, and generates a second model output from the decoding step (step 406). That is, the system uses the intermediate representation and contextual text embeddings to perform a second forward pass through one or more attention layers contained in each of the multiple transformer blocks contained in the text-to-image generative model. In particular, the first and second adapters are not used to process the intermediate representation (or data derived from the intermediate representation) in the second forward pass.

[0101] The second model output may include a set of logits for each of several spatially distinct locations in the image, corresponding to a given codebook index. Each set of logits included in the second model output is typically different from each set of logits included in the first model output.

[0102] The system processes the intermediate representation and object text embeddings of the output image from the decoding step using both one or more first sets of parameters and one or more third sets of parameters, according to the text-to-image generative model, and generates the third model output from the decoding step (step 408). That is, the system uses the intermediate representation and object text embeddings to perform a third forward propagation through one or more attention layers and a second adapter, each of the multiple transformer blocks included in the text-to-image generative model.

[0103] The third model output may include a set of logits for each of several spatially distinct locations in the image, corresponding to an index in a given codebook. Each set of logits included in the third model output is typically different from each set of logits included in the second model output, and also different from each set of logits included in the first model output.

[0104] The system processes the intermediate representation and object text embeddings of the output image from the decoding step using one or more first sets of parameters, without using one or more second sets of parameters or one or more third sets of parameters, by the text-to-image generative model, and generates a fourth model output from the decoding step (step 410). That is, the system uses the intermediate representation and object text embeddings to perform a fourth forward pass through one or more attention layers contained in each of the multiple transformer blocks contained in the text-to-image generative model. In particular, the first and second adapters are not used to process the intermediate representation (or data derived from the intermediate representation) in the fourth forward pass.

[0105] The fourth model output may include a set of logits for each of several spatially distinct locations in the image, corresponding to an index in a given codebook. Each set of logits included in the fourth model output is typically different from each set of logits included in the third model output, as well as each set of logits included in the second model output, and furthermore, each set of logits included in the first model output.

[0106] If, in step 404, step 406, step 408, or step 410, one or more attention layers include one or more cross-attention layers, each cross-attention layer may apply the cross-attention mechanism to the output of a preceding layer, e.g., the output of a self-attention layer or another cross-attention layer, and contextual text embeddings (or object text embeddings) to generate the output of the cross-attention layer.

[0107] The system generates an updated intermediate representation of the output image from the decoding step (step 412) based on the first, second, third, and fourth model outputs from the decoding step. The generation of the updated intermediate representation is described below with reference to Figure 5, which is a flowchart of the substep of step 412 of the process in Figure 4.

[0108] The system determines a first difference between the first model output and the second model output of the decoding step (step 502).

[0109] The system generates, at least partially based on the first difference, a set of first logits for each of several spatially distinct locations in the output image, corresponding to an index in a given codebook (step 504). For example, the system can generate a set of first logits for each image patch in the output image.

[0110] For example, for any position in the output image, the first set of logits can be calculated as follows:

number

number

number

[0111] The system determines the second difference between the third model output and the fourth model output of the decoding step (step 506).

[0112] The system generates, at least partially based on the second difference, a set of second logits for each of several spatially distinct locations in the image, corresponding to the index in a given codebook (step 508). For example, the system can generate a set of second logits for each image patch in the output image.

[0113] For example, for any position in an image, the second set of logits can be calculated as follows:

number

number

number

[0114] The system generates, from the respective first and second sets of logits, each of the combined sets of logits for each of several spatially distinct locations in the output image, corresponding to the index in a given codebook (step 510). For example, for any location in the output image, each combined set of logits can be calculated as follows:

number

[0115] For at least some of several spatially distinct locations in the output image, the system samples one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to a given codebook, according to each set of combined logits (step 512). For example, the system can determine an index by greedily selecting the best scoring index from each set of combined logits computed in step 510, or by sampling, for example, by using nuclear sampling or another sampling technique, and then use the discrete visual tokens that map to the determined index as the corresponding sampled discrete visual tokens.

[0116] The system generates an updated intermediate representation by replacing mask tokens present at several spatially distinct locations within the output image with sampled discrete visual tokens (step 514). Thus, the updated intermediate representation contains one or more sampled discrete visual tokens instead of the mask tokens previously included in the intermediate representation. In this way, at each decoding step, the system generates a new updated intermediate representation with fewer mask tokens.

[0117] Returning to Figure 4, if the decoding step is not the last decoding step in the decoding process, the system can perform the next iteration of process 400, and the updated intermediate representation will be used as the intermediate representation in the next iteration of process 400. Alternatively, if the decoding step is the last decoding step, the system performs further processing on the updated intermediate representation to generate the output image.

[0118] For example, the system can use a decoder neural network to process the updated intermediate representation to generate a decoder output, which can then be used as the output image. The decoder neural network may be configured through training to map spatially distinct, location-based discrete visual tokens in the image to one or more pixel values ​​for their respective locations. In another example, the updated intermediate representation in the final decoding step corresponds to the output image.

[0119] The examples herein generally describe the process of generating an output image by performing a decoding process using a text-to-image generative model having a transformer architecture. However, the same technique is applicable to other suitable text-to-image generative models, such as convolutional neural networks having a U-Net architecture, and also to other conditional image generation processes that generate an output image by iteratively updating an initial intermediate representation of the output image, such as performing a dediffusion process in pixel space or latent space.

[0120] This specification uses the term “configured” in relation to systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action.

[0121] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, and include structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions, that is, as one or more computer programs encoded in a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable memory board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions may be encoded into artificially generated propagated signals, such as mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0122] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0123] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required to, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or it can be located in one place or distributed across multiple locations and interconnected by a data communication network to run on multiple computers.

[0124] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0125] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0126] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0127] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0128] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0129] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving response messages from the user.

[0130] Data processing equipment for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the general and numerical computation portions (i.e., inference, workloads) of machine learning training or production.

[0131] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the JAX framework.

[0132] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application that enables a user to interact with embodiments of the subject matter described herein, or in a computing system that includes one or more such backend, middleware, or frontend components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0133] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0134] While this specification includes details of many specific embodiments, these should not be interpreted as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0135] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0136] This specification also provides the subject matter of the following numbered clauses:

[0137] Clause 1. A method implemented in a computer, Receiving a text prompt to specify the image style, The text prompt is processed using a text encoder neural network to generate contextual text embeddings, The process includes generating an output image by using a text-to-image generative model, wherein the text-to-image generative model includes one or more sets of first parameters shared between different styles, and one or more sets of second parameters specific to the specified style, and the generation is performed in each of a plurality of time steps. Obtaining an updated intermediate representation of the output image for the aforementioned time step, The text-to-image generation model processes the intermediate representation and contextual text embedding of the output image for the time step so as to use both the one or more first sets of parameters and the one or more second sets of parameters to generate the first model output for the time step, The text-to-image generation model processes the intermediate representation and contextual text embedding of the output image for the time step so as to generate a second model output for the time step using the one or more first set of parameters, without using the one or more second set of parameters, A method comprising generating an updated intermediate representation of the output image of a time step based on the first and second model outputs of the time step.

[0138] Clause 2. The method according to Clause 1, wherein the text-to-image generation model comprises a plurality of transformer layers, each configured to process the input token sequence and intermediate representations to generate an updated token sequence, based on the application of an attention mechanism.

[0139] Clause 3. The method according to Clause 2, wherein each transformer layer includes (i) one or more attention layers having parameters included in the first set of parameters shared between different styles, and (ii) an adapter having parameters included in the second set of parameters specific to the style.

[0140] Clause 4. The method according to Clause 3, wherein the one or more attention layers include one or more self-attention layers, one or more cross-attention layers, or both.

[0141] Clause 5. The method according to any one of Clauses 1 to 4, wherein the total number of the one or more sets of first parameters is greater than the total number of the one or more sets of second parameters.

[0142] Clause 6. Generating the updated intermediate representation of the output image for the time step is: The difference between the first model output and the second model output at the aforementioned time step is determined, Based at least partially on the difference, generate a set of logits for each of the multiple spatially distinct locations in the image, corresponding to the index in a given codebook. Sampling one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to the given codebook, according to each set of logits, The method according to any one of the claims 1 to 5, comprising generating the updated intermediate representation which includes the one or more sampled discrete visual tokens in place of the mask visual tokens at at least some of the plurality of spatially distinct locations in the image.

[0143] Clause 7. Sampling the sequence of discrete visual tokens is: The method according to Clause 6, comprising sampling discrete visual tokens for each spatially distinct location in the image, in parallel with other discrete visual tokens.

[0144] Clause 8. Generating the output image is: The method according to any one of the claims 1 to 7, comprising processing the updated intermediate representation of the output image generated in the last time step, using a decoder neural network to map the discrete visual tokens to one or more pixel values ​​of the spatially distinct locations in the image.

[0145] Clause 9. The method according to any one of Clauses 1 to 8, further comprising fine-tuning the text-to-image generation model based on fine-tuning data comprising one or more image-text pairs before generating the output image, each image-text pair comprising an image and a text prompt comprising both a description of the content of the image and a description of the style of the image, and the fine-tuning comprising learning updated values ​​of one or more second parameters while fixing and holding fixed pre-trained values ​​of one or more first parameters.

[0146] The method according to Clause 10. The method according to Clause 9, wherein fine-tuning the text-to-image generative model includes training the text-to-image generative model on the fine-tuning data to optimize a cross-entropy loss function that evaluates the difference between (i) target discrete visual tokens generated by the image-to-image encoder contained in the image-to-text pair and (ii) predicted discrete visual tokens generated by the text-to-image generative model based on the text contained in the image-to-text pair, for each image-text pair.

[0147] Article 11. A method that is performed by a computer, Receiving a text prompt that includes both a description of a specific object in an image and a description of a specific style, The text prompt is processed using the text encoder neural network to generate contextual text embeddings. Using the text encoder neural network, the description of the particular object is processed to generate an object text embedding. The process includes generating an output image by using a text-to-image generative model, wherein the text-to-image generative model includes one or more sets of first parameters shared between different styles, one or more sets of second parameters specific to the specified style, and one or more sets of third parameters specific to the particular object, and the generation is performed in each of a plurality of time steps. Obtaining an updated intermediate representation of the output image for the aforementioned time step, The text-to-image generation model processes the intermediate representation and contextual text embedding of the output image for the time step to generate a first model output for the time step using both the one or more first sets of parameters and the one or more second sets of parameters. The text-to-image generation model processes the intermediate representation and contextual text embedding of the output image of the time step so as to generate a second model output of the time step using one or more sets of first parameters, without using any of the one or more sets of second parameters or one or more sets of third parameters. The text-to-image generation model processes the intermediate representation and object text embedding of the output image for the time step to generate a third model output for the time step using both the one or more first sets of parameters and the one or more third sets of parameters, The text-to-image generation model processes the intermediate representation and object text embedding of the output image of the time step so as to generate a fourth model output of the time step using one or more sets of first parameters, without using any of the one or more sets of second parameters or one or more sets of third parameters, A method comprising generating an updated intermediate representation of the output image for a time step based on the first, second, third, and fourth model outputs for the time step.

[0148] Clause 12. Generating the updated intermediate representation of the output image for the time step is: The first difference between the first model output and the second model output at the aforementioned time step is determined, To generate, at least partially based on the first difference, a set of first logits for each of several spatially distinct locations in the image, corresponding to the index of a given codebook, Determining the second difference between the third model output and the fourth model output of the aforementioned time step, To generate, at least partially based on the second difference, a set of second logits for each of the plurality of spatially distinct locations in the image, for the index of the given codebook, From each of the first and second sets of logits, a set of combined logits for each of the plurality of spatially distinct locations in the image, corresponding to the index in the given codebook, Sampling one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to the given codebook, according to each of the aforementioned combined logit sets, The method according to clause 11, comprising generating the updated intermediate representation which includes the one or more sampled discrete visual tokens in place of the mask visual tokens at at least some of the plurality of spatially distinct locations in the image.

[0149] Clause 13. A system comprising one or more computers and one or more storage devices for storing instructions, wherein, when executed by the one or more computers, the instructions are operable to cause the one or more computers to perform the operation of each of the methods described in any of the preceding clauses.

[0150] 14. A computer storage medium on which, when executed by one or more computers, instructions are encoded causing the one or more computers to perform the operations of each of the preceding clauses.

[0151] Specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require to be performed in a specific or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer implementation method, Receiving a text prompt to specify the image style, Processing the text prompt to generate contextual text embeddings using a text encoder neural network, By using a text-to-image generative model, the output image is generated, Includes, The text-to-image generation model includes one or more sets of first parameters shared between different styles and one or more sets of second parameters specific to the specified style, wherein the generation is performed in each of a plurality of steps. Obtaining an intermediate representation of the output image in the above step, The text-to-image generation model processes the intermediate representation and contextual text embedding of the output image of the step so as to generate the first model output of the step using both the one or more first sets of parameters and the one or more second sets of parameters, The text-to-image generation model processes the intermediate representation of the output image and the contextual text embedding of the step so as to generate the second model output of the step using the one or more first sets of parameters, without using the one or more second sets of parameters, The process includes generating an updated intermediate representation of the output image of the step based on the first and second model outputs of the step, method.

2. The method according to claim 1, wherein the text-to-image generation model includes a plurality of transformer blocks, each configured to process a block input derived from the intermediate representation and generate a block output, based on the application of an attention mechanism.

3. The method according to claim 2, wherein each transformer block includes (i) one or more attention layers having parameters included in the first set of parameters shared between different styles, and (ii) an adapter having parameters included in the second set of parameters specific to the style.

4. The method according to claim 3, wherein the one or more attention layers include one or more self-attention layers, one or more cross-attention layers, or both.

5. The method according to any one of claims 1 to 4, wherein the total number of the one or more sets of first parameters is greater than the total number of the one or more sets of second parameters.

6. The process of generating the updated intermediate representation of the output image in the above step is: The step of determining the difference between the first model output and the second model output, Based at least partially on the difference, generate a set of logits for each of the multiple spatially distinct locations in the image, corresponding to the index in a given codebook. Sampling one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to the given codebook, according to each set of logits, The method according to any one of claims 1 to 5, comprising generating the updated intermediate representation which includes the one or more sampled discrete visual tokens instead of mask tokens at at least some of the plurality of spatially distinct locations in the image.

7. Sampling the sequence of discrete visual tokens is The method according to claim 6, comprising sampling discrete visual tokens for each spatially distinct location in the image, in parallel with other discrete visual tokens.

8. The method according to any one of claims 1 to 7, wherein generating the output image comprises processing the updated intermediate representation of the output image generated in the last step using a decoder neural network to map the discrete visual tokens to one or more pixel values ​​of the spatially distinct locations in the image.

9. The method according to any one of claims 1 to 8, further comprising fine-tuning the text-to-image generation model based on fine-tuning data comprising one or more image-text pairs before generating the output image, each image-text pair comprising an image and a text prompt comprising both a description of the content of the image and a description of the style of the image, wherein the fine-tuning comprises learning updated values ​​of one or more second sets of parameters while fixing and holding fixed pre-trained values ​​of one or more first sets of parameters.

10. The method according to claim 9, wherein fine-tuning the text-to-image generative model includes training the text-to-image generative model on the fine-tuning data to optimize a cross-entropy loss function that evaluates the difference between (i) target discrete visual tokens generated by an image encoder from the images contained in the image-text pair and (ii) predicted discrete visual tokens generated by the text-to-image generative model based on the texts contained in the image-text pair, for each image-text pair.

11. A computer implementation method, Receiving a text prompt that includes both a description of a specific object in an image and a description of a specific style, Using the text encoder neural network, process the description of a particular style to generate contextual text embeddings, Using the text encoder neural network, the description of the particular object is processed to generate an object text embedding. The process includes generating an output image using a text-to-image generative model, wherein the text-to-image generative model includes one or more sets of first parameters shared between different styles, one or more sets of second parameters specific to a particular style, and one or more sets of third parameters specific to a particular object, and the generation process involves, in each of a plurality of steps, Obtaining an intermediate representation of the output image in the above step, The intermediate representation and contextual text embedding of the output image of the step are processed so that the text-to-image generation model uses both the one or more first sets of parameters and the one or more second sets of parameters to generate the first model output of the step. The text-to-image generation model processes the intermediate representation of the output image and the contextual text embedding of the step to generate the second model output of the step using the one or more first sets of parameters and without using any of the one or more second sets of parameters or the one or more third sets of parameters, The intermediate representation and object text embedding of the output image of the step are processed so that the text-to-image generation model uses both the set of one or more first parameters and the set of one or more third parameters to generate the third model output of the step, The text-to-image generation model processes the intermediate representation and object text embedding of the output image of the step in such a way that it generates the fourth model output of the step using the one or more first sets of parameters, without using any of the one or more second sets of parameters or the one or more third sets of parameters. This includes generating an updated intermediate representation of the output image of the step based on the first, second, third, and fourth model outputs of the step, method.

12. The process of generating the updated intermediate representation of the output image in the above step is: The first step involves determining the first difference between the first model output and the second model output, To generate, at least partially based on the first difference, a set of first logits for each of several spatially distinct locations in the image, corresponding to the index in a given codebook, The second difference between the third model output and the fourth model output of the above step is determined, To generate, at least partially based on the second difference, a set of second logits for each of the plurality of spatially distinct locations in the image, corresponding to the index in the given codebook, From the respective sets of first and second logits, a set of combined logits for each of the multiple spatially distinct locations in the image, corresponding to the index in the given codebook, Sampling one or more discrete visual tokens from a given vocabulary of discrete visual tokens corresponding to the given codebook, according to each of the aforementioned combined logit sets, The method according to claim 11, comprising generating the updated intermediate representation which includes the one or more sampled discrete visual tokens instead of mask visual tokens in at least some of the plurality of spatially distinct locations in the image.

13. A system comprising one or more computers and one or more storage devices for storing instructions, wherein, when executed by the one or more computers, the instructions are operable to cause the one or more computers to perform an operation according to any one of claims 1 to 12.

14. A computer storage medium having, when executed by one or more computers, an encoded instruction causing the one or more computers to perform an operation according to any one of claims 1 to 12.

Images