Diffusion plug-in for conditioned text-to-image generation

Abstract

Methods and systems for performing image generation are disclosed herein. The method can include receiving, by a processor, a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network and receiving, by the processor, a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image. The method can also include applying, by the processor, the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network, performing, by the processor, image generation with the diffusion model based on the text prompt and the multiscale feature vector, and outputting, by the processor, a generated image from the diffusion model.

Description

DIFFUSION PLUG-IN FOR CONDITIONED TEXT-TO-IMAGE GENERATIONPRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application 63 / 509,685 having a filing date of June 22, 2023, which is incorporated by reference herein.FIELD

[0002] The present disclosure relates generally to text-to-image generation. More particularly, the present disclosure relates to an on-device diffusion plug-in for diffusion models that allows for conditioned text-to-image generation.BACKGROUND

[0003] Diffusion models can be used for text-to-image generation, and can generate images of high quality, creativity, and inference performance. Image generation with diffusion models can be performed as an iterative denoising process. At each step, a diffusion model, such as a U-Net model, gradually removes noise from the image. A text embedding can be connected to the U-Net via cross-attention layers. The text prompts (and the embeddings generated from text prompts) can condition the diffusion model to perform text-guided image generation. However, some information is difficult to represent via text prompts alone, such as position and pose of a target subject of the generated image. Furthermore, generative models require a large amount of computing resources to operate, which can cause the generative models to be difficult to implement on personal computing devices such as smartphones.SUMMARY

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] One example aspect of the present disclosure is directed to a computer- implemented method for performing image generation. The method can include receiving, by a processor, a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network and receiving, by the processor, a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image. The method can also include applying, by the processor, the multiscale feature vectorto one or more levels of an encoder of the encoder-decoder network, performing, by the processor, image generation with the diffusion model based on the text prompt and the multiscale feature vector, and outputting, by the processor, a generated image from the diffusion model.

[0006] Another example aspect of the present disclosure is directed to A computing system for performing image generation. The computing system can include one or more processors and one or more non-transitory, computer readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations can include receiving a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network and receiving a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image. The operations can also include applying the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network, performing image generation with the diffusion model based on the text prompt and the multiscale feature vector, and outputting a generated image from the diffusion model.

[0007] A further example embodiment of the present disclosure is directed to one or more non-transitory, computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations can include receiving a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network and receiving a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image. The operations can also include applying the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network, performing image generation with the diffusion model based on the text prompt and the multiscale feature vector, and outputting a generated image from the diffusion model.

[0008] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitor ' computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0011] Figure 1 A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[0012] Figure IB depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0013] Figure 1C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0014] Figure 2 depicts a block diagram of an example image generation model according to example embodiments of the present disclosure.

[0015] Figure 3 depicts a flow chart diagram of an example method to perform image generation according to example embodiments of the present disclosure.

[0016] Figure 4 depicts generated images based on a condition image according to example embodiments of the present disclosure.

[0017] Figure 5 depicts generated images based on a condition image according to example embodiments of the present disclosure.

[0018] Figure 6 depicts generated images based on a condition image according to example embodiments of the present disclosure.

[0019] Figure 7 depicts generated images based on a condition image according to example embodiments of the present disclosure.

[0020] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.DETAILED DESCRIPTIONOverview

[0021] Generally, the present disclosure is directed to portable models for performing additional image conditioning of a denoising diffusion model. For example, aspects of the present disclosure are directed to a plug-in network configured to receive and process a conditioning input to generate a conditioning vector for conditioning a pre-trained denoising diffusion model. The plug-in network can be trained separately from the pre-trained denoising diffusion model and can be configured to plug in to the pre-trained denoising diffusion model. According to another aspect of the present disclosure, the plug-in network can be a portable network that is stored and run separately from the pre-trained denoising diffusion model. For example, the plug-in network can be stored and run '‘on-device” (e.g..on a user’s device) while the diffusion model may be stored and run at a separate device (e.g., a server device). Thus, example aspects of the present disclosure are directed to an on-device diffusion plug-in for conditioned text-to-image generation.

[0022] Diffusion models can be trained by first performing a forward diffusion process. An image is received and noise is iteratively added to the image until an image of pure noise is generated. Then, the diffusion model can perform reverse diffusion, which takes the image of pure noise as input and attempts to "‘re-create” the input image by removing noise from the image of pure noise. For example, an iterative denoising process can be performed. At each step of the iterative denoising process, a denoising diffusion model can remove noise from the current iteration of the image, attempting to recreate the original input image. The diffusion model can then be trained, or modified, based on how close the attempted recreation of the input image is to the actual input image. Over a training period, the diffusion model can process a series of training inputs to better leam the image recreation process through forward and reverse diffusion processes.

[0023] After the diffusion model is trained, the diffusion model can be used for image generation. During image generation, input conditions are applied to the model, which in turn uses random noise to generate an image based on the image conditions. In other words, no forw ard diffusion is performed to transform an image into pure noise. The diffusion model instead takes an input of random noise and, based on the input conditions, generates an image from the random noise input using reverse diffusion with the trained diffusion model.

[0024] This image generation process can be an iterative denoising process. At each step, the diffusion model can remove noise from the input random noise and, after a number of iterations, output a generated image.

[0025] In some embodiments, the diffusion model can include a U-Net architecture. The U-Net architecture is a deep learning architecture that includes two sections: an encoding section and a decoding section. The encoding section can contain encoding layers that process input data, capture contextual information from the input data, and reduce the spatial resolution of the input. The encoding section can identify relevant features in an input and perform convolutional operations that reduce the spatial resolution of feature maps associated with the relevant features. The encoding section can be similar to feed forw ard layers in convolutional neural networks.

[0026] In contrast, the decoding section can contain decoder layers that decode the data encoded by the encoding section. The decoding section can also use “skip” connections that provide information directly from previous encoding layers to decoding layers withoutthat information undergoing encoding. This allows information to be propagated from earlier layers to deeper decoding layers by helping to preserve spatial information lost during encoding. The decoding section can upscale the feature maps and ‘locate” the encoded features while maintaining spatial resolution of the input.

[0027] It is desirable that users be able to add additional controls to the generative process of the diffusion model using input conditions. Input conditions can be any additional data provided to the diffusion model as input for image generation, such as input text, input segmentation masks, input images, and the like.

[0028] Input conditions can be encoded into vectors using an encoding network. The encoding network maps the input data from the input condition into a vector representation, which contains the meaning behind the input condition. This vector representation of the input condition can then be applied to one or more layers of the diffusion model as an additional weight or parameter for use in image generation. For example, in some embodiments, the vector representation of the input condition can be applied to one or more cross-attention layers of the diffusion model

[0029] Cross-attention layers allow for a model to mix two different embedding sequences, even if the embedding sequences are of different modalities (e.g., image and text modalities). Therefore, cross-attention allows for different modalities to be combined in one model. This allows the additional modality (e.g., the text prompt) to be used as guidance for image generation.

[0030] To aid in creating more accurate images based on text prompts, example embodiments of the present disclosure introduce a diffusion plug-in, or separate network that can be utilized in conjunction with pre-trained base diffusion models and in addition to a text encoder that encodes the text prompt as an input condition to the diffusion model. The plugin network can be used with any suitable base diffusion model (e.g., models that include a U- Net architecture) and can be trained from scratch, not taking pretrained weights from the base diffusion model. In other words, the plug-in network can be an independent encoding network that takes in input data (additional text, images, audio, video, and the like) and encodes the input data into a vector representation of the input data, which in turn can be applied to the base diffusion model as an additional input condition for image generation.

[0031] Because the plug-in network runs outside of the base diffusion model and requires much less processing than the base diffusion model, it can be run on mobile devices and other computing devices with limited processing power without unduly burdening the processing of the computing devices. Additionally, because the plug-in network runs outsideof the base diffusion model, there is negligible additional cost added to base diffusion model inference, as the generated vector representation of the input data is applied to the base diffusion model as an additional input condition during inference.

[0032] The plug-in network can receive and process a conditioning input to generate a conditioning vector. As one example, the plug-in network can receive and process one or more condition images, or images associated with particular limitations, guidance, or constraints for the final generated images. These condition images can include edge detected images, images illustrating rough outlines and / or key points of facial features or poses of a body, depth images, semantic segmentation maps, or other conditions that a user may wish to place on a diffusion model that receives a text prompt.

[0033] For example, a user may wish to prompt the model to generate “a man standing with his arms outstretched.” A condition image for this prompt can be a wire-frame model (e.g., a stick figure) standing with arms outstretched. The plug-in network can utilize an encoder, depth-wise convolution, and / or inverted bottlenecks to extract multiscale features from the condition image in a vector representation of the multiscale features. These multiscale features can then be provided as the vector representation to an encoder portion of the denoising diffusion model (e.g., which may be structured as a U-Net) at various levels using, for example, cross-attention layers that combine the vector representation of the features with iterative versions of the processed random noise. The multiscale features can therefore be used by the denoising diffusion model as extra conditioning information for performing image generation, such as generating an image of a man standing in a pose matching the input wire-frame model. In other embodiments, other conditions can be added to images, such as directing the placement of facial features and facial landmarks in a generated image of a face, providing landmark locations for garments or other items in an image, and the like.

[0034] For example, in another embodiment, a user can request that the diffusion model perform inpainting, or modification of an image, such as replacing a person standing next to a chair with a generated image of the same person sitting on the chair. In such an embodiment, in addition to a text prompt of "generate a person sitting on a chair” or similar, other conditioning inputs can include the original image of the user standing next to the chair and / or a wire frame of a stick figure sitting on a chair. Both of these additional conditioning inputs can be encoded and applied to the diffusion model, which can result in a generated image of the same person and the same chair but with the person now sitting on the chair instead of standing next to the chair. The additional conditioning inputs can provide thewireframe for how the diffusion model orients the person in the image, while the original image can provide the diffusion model with values to make the person and the chair look identical or near-identical to the original, thus avoiding generating an image of a new person sitting in a new chair.

[0035] The use of the plug-in network provides a number of advantages over existing diffusion models and even other conditional plug-ins. First, example implementations of the proposed plug-in network are lightweight, containing only a few million parameters, instead of existing models containing hundreds of millions of parameters. The lightweight plug-in network allows for operation on-device without unduly burdening processing on the device. Inference time by the diffusion model is not affected by the inclusion of the plug-in network and the use of the plug-in network proposed herein actually leads to improved inference time by the base diffusion model in comparison to other existing models because of the lightweight nature of the plug-in netw ork. Additionally, allowing the plug-in netw ork to be contained wholly on the device enables better security and privacy while being a scalable solution for conditional image generation usable by a variety of different denoising diffusion models on the device.

[0036] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.Example Devices and Systems

[0037] Figure 1 A depicts a block diagram of an example computing system 100 that performs image generation according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0038] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0039] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., andcombinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0040] In some implementations, the user computing device 102 can store or include one or more image generation models 120. For example, the image generation models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example image generation models 120 are discussed with reference to Figure 2.

[0041] In some implementations, the one or more image generation models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single image generation model 120 (e g., to perform parallel image generation across multiple instances of image generation).

[0042] More particularly, the model 120 can generate new images after being trained on training images. The model 120 can receive a text prompt, such as “man sitting and reading,” and generate from random noise a wholly artificial image of a man sitting in a chair and reading a book. This can be performed using a U-Net encoder-decoder network, which is a U-shaped encoder decoder network architecture that consists of encoder blocks and decoder blocks that are connected via a bridge. The encoder network performs downsampling to halve the spatial dimensions and double the number of filters at each encoder block.Likewise, the decoder network doubles the spatial dimension and half the number of feature channels. The encoder acts as a feature extract and leams an abstract representation of the input image through a sequence of the encoder blocks, which perform convolutions and applies activation functions to the output of the convolution. This output then can act as a skip connection for a corresponding decoder block. Pooling or strided convolution then can be performed, where spatial dimensions of feature maps are reduced.

[0043] The skip connections provide additional information to the decoder to generate better semantic features. The decoder network takes the abstract representation and performs upsampling to generate the novel image.

[0044] Additionally or alternatively, one or more image generation models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the image generation models 140 can be implemented by the server computing system 130 as a portion of a web service (e.g., a image generation service). Thus, one or more models 120 can be stored and implemented at the user computing device 102 and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0045] The user computing device 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g.. a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0046] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM. ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0047] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0048] As described above, the server computing system 130 can store or otherwise include one or more image generation models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage anattention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example models 140 are discussed with reference to Figure 2.

[0049] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0050] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a pl ural i ty of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0051] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0052] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability7of the models being trained.

[0053] In particular, the model trainer 160 can train the image generation models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, input training images that are then "destroyed" by the model being trained byadding noise to the training images and then attempting to “recover” the data by reversing this noising process.

[0054] A plug-in network can also be trained by the model trainer 160 as one of the image generation models 120 and / or 140. In some embodiments, the plug-in network can be trained independently of the diffusion model. Training the plug-in network can include inputting a training condition image into the plug-in network and performing, by the processor, image generation with the diffusion model using the multiscale feature vector generated by the plug-in network based on the training condition image. Advantageously, image generation is performed by the diffusion model with weights of the diffusion model frozen in order to independently train the weights of the plug-in network without inheriting any weights or bias from the diffusion model. This also enables the plug-in network to be trained independent of any specific type of diffusion model and therefore be able to be adaptable to any number of diffusion models.

[0055] In some examples, the training condition image can be generated by performing one or more edge detection algorithms, keypoint detection algorithms, or the like on an original training image. Training the plug-in network can also include performing a comparison of (1) an output image from image generation conditioned with the training condition image and (2) the original training image using a loss function. Based on the loss function (e.g., how similar the generated output image from the diffusion model is to the original training image), one or more weights of the plug-in network can be updated.

[0056] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0057] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 1 0 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0058] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereofand can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0059] The machine-learned models described in this specification may be used in a variety of tasks, applications, and / or use cases.

[0060] In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine- learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

[0061] In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine- learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine- learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or naturallanguage output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

[0062] Figure 1 A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.

[0063] Figure IB depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0064] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a brow ser application, etc.

[0065] As illustrated in Figure IB, each application can communicate with a number of other components of the computing device, such as. for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0066] Figure 1C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0067] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a brow ser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0068] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 1C. a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0069] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository' of data for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).Example Model Arrangements

[0070] Figure 2 depicts a block diagram of an example image generation model 200 according to example embodiments of the present disclosure. In some implementations, the image generation model 200 includes a diffusion model 205 and a plug-in network 210.

[0071] The diffusion model 205 can be a model designed for generating new images, such as image 215, based on a random noise sample 220. For example, the diffusion model 205 can be a denoising diffusion probabilistic model, a score-based generative model, a denoising diffusion implicit model, a latent diffusion model, a flow-based diffusion model, and the like.

[0072] Generally, the diffusion model 205 can receive a text prompt 225 input by a user and can generates a text encoding, or a vector representation of the text prompt 225, using a text encoder 230. The text encoding can then be provided to a U-Net encoder-decoder 235, which also receives the random noise sample 220 as input. In some embodiments, the random noise sample 220 can be an image representation of Gaussian noise, or an image that has a probability density function equal to or similar to the normal distribution.

[0073] The U-Net encoder-decoder 235 can generate an image, such as image 215 based on the text encoding and the random noise sample 220, by performing iterative denoising on the random noise sample 220 with the text encoding being connected to one or more encoding layers 240 and / or one or more decoding layers 245 of the U-Net encoder-decoder235 via cross-attention layers. In one embodiment, the encoder-decoder 235 can include a fully convolutional neural network that includes the one or more encoding layers 240 and the one or more decoding layers 245 arranged in the U-Net architecture.

[0074] The U-Net encoder-decoder 235 can be a deep learning architecture that includes two sections: the one or more encoding layers 240 and the one or more decoding layers 245. The one or more encoding layers 240 can process input data, capture contextual information from the input data, and reduce the spatial resolution of the input. The one or more encoding layers 240 can identify relevant features in an input and perform convolutional operations that reduce the spatial resolution of feature maps associated with the relevant features. The one or more encoding layers 240 can be similar to feedforward layers in convolutional neural networks. The one or more encoding layers 240 can include convolutional layers and pooling layers. A convolution operation (without stride) results in a same size output image as input image, but strided convolution results in a reduction in size of an input image. For example, a 3x3 convolution filter with a stride value of 1 performed on a 4x4 pixel image will result in a 4x4 pixel output image, but a 3x3 convolution with a stride value of 2 can convert an image of size 4x4 to size 2x2. Thus, performing convolutions on generated feature maps (using a dot product between the convolution filter and the input) with striding allows the one or more encoding layers 240 to reduce the size of the input image (in this case, the Gaussian noise image). The convolution filter can be of any size and each entry in the filter can be assigned a particular weight for calculating a dot product with the input.

[0075] The convolutional layers of the one or more encoding layers 240 can perform the convolutional operations and provide output to other convolutional layers or to pooling layers. The pooling layers can perform various dimensionality reduction actions, which reduces the number of parameters in the input. Similar to convolutional layers, the pooling operation can include sweeping a filter across an entire input. However, the pooling filter does not have any weights. Instead, an aggregation function can be applied to values in an input field to select values to send to an output array. Types of pooling can include max pooling (selecting a max value from across a portion of the input) or average pooling (calculating an average of values across a portion of the input).

[0076] In contrast, the one or more decoding layers 245 can decode the data encoded by the one or more encoding layers 240. The one or more decoding layers 245 can also use "skip" connections that provide information directly from previous encoding layers of the one or more encoding layers 240 to decoding layers of the one or more decoding layers 245without that information undergoing encoding. This allows information to be propagated from earlier encoding layers to deeper decoding layers by helping to preserve spatial information lost during encoding. The one or more decoding layers 245 can upscale the feature maps and “locate” the encoded features while maintaining spatial resolution of the input.

[0077] The one or more decoding layers 245 can perform upsampling, which restores the spatial dimensions of the fully encoded input from a final, fully -connected layer of the one or more encoding layers. To perform upsampling, “unpooling”can be used. Unpooling can involve interpolation of values from an encoded input using various interpolation methods, such as nearest-neighbor unpooling, max unpooling, bilinear interpolation, bicubi interpolation, and Lanczos Interpolation.

[0078] In other embodiments, a transpose convolution can be performed to upsample an encoded input. A transpose convolution is the reverse of a convolution operation: values of the input can be multiplied by weights in a convolution filter to produce an upsampled image. The filter for this transposed convolution filter can be learned during training of the image generation model 200.

[0079] The cross-attention layers can be used to emphasize or de-emphasize certain features in the encoder-decoder 235 based on the text encoding. Cross-attention layers allow for a model to mix two different embedding sequences of a same dimension, even if the embedding sequences are of different modalities (e.g., image and text modalities). Therefore, cross-attention allows for different modalities to be combined in one model. This allows the additional modality (e.g., the text prompt) to be used as guidance for image generation.

[0080] In cross-attention, two asymmetric embedding sequences having a same dimension can be combined by calculating a key and a value from a first sequence of the sequences and calculating queries from the second sequence. An attention matrix is calculated and values from the first sequence are applied to the attention matrix. The resulting output sequence has the dimension and length of the second sequence.

[0081] The image generation model 200 also includes the plug-in network 210. In some embodiments, the plug-in network 210 is “plugable.” or able to be used as an input into a variety of base diffusion models, so long as the base diffusion model utilizes a U-Net architecture. Because of the lightweight nature of the plug-in network 210 (on the order of a few million parameters), the plug-in network 210 can also be “portable,” or run outside of the base diffusion model, even on mobile devices, without providing extra processing burden on the base diffusion model inference operations.

[0082] The plug-in network 210 can receive as an input a condition image 250. An encoder of the plug-in network 210 can utilize depth- wise convolution and inverted bottlenecks to extract one or more multiscale feature vectors from the condition image 250. Much like the text embedding, the plug-in network 210 can be connected to one or more encoding layers 240 of the encoder-decoder 235 via cross-attention layers and can be used to provide a further condition for the denoising process.

[0083] The condition image 250 can be an image received from a separate machine- learned model that provides a “simplified” image as a base for creating a condition for image generation. Examples of condition images and resulting generated images can be found in Figures 4-7 (described in more detail below). Other examples of condition images can include one or more facial landmark maps, one or more wire-frames of poses, one or more edge detection maps, one or more feature maps or heat maps, one or more depth maps, and the like. The encoder of the plug-in network 210 takes the condition image 250 as an input and extracts features from the condition image 250 to generate the multiscale feature vector.

[0084] The extracted multiscale feature vector is provided by the plug-in network 210 to various layers of the one or more encoding layers 240 of the encoder-decoder 235. It is not necessary to provide the extracted multiscale feature vector to the one or more decoding layers 245 because the one or more encoding layers 240 are down-sampling the input image (the random noise sample 220) to obtain a representation of the input image and any conditions. The one or more decoder layers 245 take the representation of the input image and reconstruct an image, such as image 215 by up-sampling, and the representation already includes the condition from the multiscale feature vector.

[0085] Similar to the above description of the inclusion of the text prompt using crossattention layers in the denoising process, the multiscale vector representing the condition input 250 can be combined with various stages of the encoding process using cross-attention to better control the encoding process.

[0086] The features from the multiscale feature vector are injected into each diffusion iteration. In other words, the plug-in network is only run once for each image generation process, saving computing resources by requiring less iterations of the encoding of the multiscale feature vector to be run. The same multiscale feature vector that is generated by the plug-in network can be provided to different encoding layers at various levels of the one or more encoding layers 240, enabling controlled generation even at early diffusion steps. While a large number of iterations for diffusion may not be necessary, more iterations canimprove image alignment with the text prompt and can generate images with higher levels of detail.

[0087] Therefore, based on both the input text prompt 225 and the condition image 250, the diffusion model 205 can generate the image 215.Example Methods

[0088] Figure 3 depicts a flow chart diagram of an example method to perform according to example embodiments of the present disclosure. Although Figure 3 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method 300 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0089] At 302, a computing system can receive a text prompt for generating an image using a diffusion model. For example, a user can provide the computing system with a text prompt reciting "a man with his arms outstretched" via one of various input mechanisms, such as audio input, keyboard input, and the like. The text prompt can describe the desired output image generated by the diffusion network at various levels of specificity and detail. For example, the prompt can be as simple as “a man standing” or more complex, such as “a man standing, holding a basketball, wearing sneakers, and bending down to pet a dog.” Other various levels of complexity of the text prompt can be contemplated.

[0090] In another example, the text prompt can be “a pencil drawing of a woman,” such as the generated image 215 illustrated in Fig. 2.

[0091] In some embodiments, the diffusion model can include an encoder-decoder network, which can in turn include one or more encoding layers and one or more decoding layers. The one or more encoding layers can be used to perform down-sampling on one or more inputs to generate a representation of the inputs, such as a random noise input for image generation and one or more conditional inputs for controlling image generation. The one or more decoding layers can perform up-sampling on the representation of the inputs to generate an image, and can also utilize skip connections to use unencoded information to generate the image. In some embodiments, the encoder-decoder network can have a U-Net architecture.

[0092] At 304, the computing system receives a multiscale feature vector from a plug-in network. In some embodiments, the multiscale feature vector models a condition image that is input into the plug-in network. The plug-in network can be an encoder network thatextracts features from the condition image using depth-wise convolution and outputs the multiscale feature vector, which represents the features extracted from the condition image.

[0093] The features of the condition image help to control the generation of the final image, such as providing a pose for an object or person in the output image, controlling the placement of one or more features of the image (e.g., facial features, outlines for objects, and the like), and other “conditions” to which the output image must conform. In some embodiments, the condition image can be received as an output from another machine- learned model (as described below with regards to Figures 4-7 below).

[0094] For example, a condition image can include a wire-frame of a stick figure representation of a person with arms outstretched. This condition image can be encoded to represent the features of the outstretched arms in a multiscale feature vector. The encoded features can be used, as described below, to provide positional references for outstretched arms in the generated image, as well as posture information about the person being generated (e.g., standing upright, slouched, sitting down, lying down. etc.).

[0095] In another example, the condition image can include a facial landmark map, such as the example condition image 250 illustrated in Fig. 2. In the example shown in Fig. 2, the condition image 250 illustrates various locations of landmark features of a human face, such as eye placement, eyebrow placement, mouth placement, and face shape. These features can be encoded into the multiscale feature vector and provided to the diffusion model.

[0096] At 306, the computing system applies the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network. The multiscale feature vector is applied to the encoding, or dow n-sampling, portion of the encoder-decoder network to incorporate the features of the condition image into the diffusion process as a condition or control how the image is generated by the diffusion model.

[0097] In one example, the multiscale feature vector is applied to the various levels of the encoder using one or more cross-attention layers. Cross-attention layers allow for a model to mix two different embedding sequences of a same dimension, even if the embedding sequences are of different modalities (e.g., image and text modalities). Therefore, cross-attention allows for different modalities to be combined in one model. This allows the additional modality (e.g., the text prompt) to be used as guidance for image generation.Values of the various encoding layers and the multiscale feature vector can be combined and processed during encoding using cross-attention.

[0098] At 308, the computing system performs image generation with the diffusion model based on the text prompt and the multiscale feature vector. The text prompt isembedded as a text embedding and the diffusion model uses a random noise sample for input into the encoder-decoder model. The text prompt is provided to each encoding layer and each decoding layer as a condition and the multiscale feature vector is provided to one or more of the encoding layers as a conditional input using cross-attention. The diffusion model then produces, using the random noise sample and the conditions, a generated image.

[0099] For example, as shown in Fig. 2, the condition image 250 can illustrate facial features and placement of said facial features as a conditional input. A vector representation of the condition image 250 can be generated and provided to the one or more encoding layers 240 of the encoder-decoder 235 as an additional condition along with a vector representation of the text prompt 225. which can be "a pencil drawing of a w oman.’' The vector representation of the condition image 250 and the vector representation of the text prompt 225 can be combined with an encoded representation of the random noise sample 220 to create an encoding vector that has the combined information of the random noise sample 220, the vector representation of the text prompt 225, and the vector representation of the condition image 250. The one or more decoding layers 245 can then decode the encoding vector, along with using convolution with skip connections, to generate the final output image 215, w hich is a drawing of a woman that appears as if drawn using a pencil, having facial features located as shown in the condition image 250.

[0100] At 310, the computing system outputs generated image from the diffusion model. For example, the generated image can be output for display on a user interface, output attached as a file via one or more communication methods such as email or SMS messaging, output as a saved file in a database, and the like.Example Generated Images

[0101] Figures 4-7 illustrate various condition images input into a plug-in network along with text prompts for generating illustrations.

[0102] Figure 4 illustrates generated images in which edge detection has been performed on a background (condition image 400) or one or more objects (second condition image 405) to produce condition images. These condition images are processed by a diffusion model along with associated prompts (first prompt 410 and second prompt 415) to generate resulting output images (possible output images 420). The edge detection in the condition image 400 and the second condition 405 indicates general placement of features within a desired image, and can greatly control the placement of objects in the possible output images 420.

[0103] Figure 5 illustrates images in which a face mesh condition image 500 has been used by two different examples of diffusion networks to generate images (ControlNet images 505 and plug-in images 510) based on second prompt 520. Similar to the example shown in Fig. 2, a face mesh condition image 500 can indicate what features to include for generation of an image of a face, as well as the location, size, and relative position of those features to one another.

[0104] Figure 6 illustrates images in which wire frame condition images 600 and 605 are used along with prompts 610, 615, 620, and 625 to generate resulting images 630. Similar to the face mesh condition image 500, the wire frame condition images 600 and 605, when encoded, provide information for the posture and position of objects (such as the human frame), for image generation, even if the prompts are unrelated (e.g., a "‘wooden man, holding flowers” being very different from “statute of liberty, holding flowers”), thus resulting in similar looking pictures with different text prompts. A similar example is shown in “Van Gogh, writing books” and “a man, pencil sketch, reading,” where the condition image 605 can lead to similar posture and placement of objects in images despite the very different prompts.

[0105] Figure 7 illustrates images 700 and 705 that can be generated using a depth condition image 710. The depth condition image 710 is a depth image, which is a visual representation computed from the real w orld providing a two and a half dimensional recreation of a scene, where distances to various surfaces in the scene from the point of capture is represented by a luminance gradient. As shown in the condition image 71 , the depth to each object in the scene is shown by lighter areas in representing closer objects. Based on a prompt, such as “a sunny bedroom,” and the condition image 710, the system can generate resulting images 700 and 705, both of which capture the indicated depth (bed located at the position noted to be closer, etc.).

[0106] Fig. 8 illustrates an iterative generation process 800 according to example aspects of the present disclosure. In some embodiments, an image generation model 805 can output each iterative step or certain selected steps of the iterative generation process 800 for display to the user.

[0107] A condition input 810 can be input into a plug-in network 815, such as plug-in netw orks described above with reference to Fig. 2. In the embodiment illustrated in Fig. 8, the condition input 810 is a facial landmark map illustrating the relative placement and size of facial features. The plug-in network 815 can process the condition input 810 to generate amultiscale feature vector that represents the features of the condition input 810 and provide the multiscale feature vector to the imagine generation model 805.

[0108] The multiscale feature vector can be applied to each iteration of diffusion. In other words, the same multiscale feature vector can be provided via a cross-attention layer to one or more layers of the image generation model 805 at each iteration, so that the same multiscale feature vector is used as a control during each iteration. Using the same control during each iteration of diffusion helps to "guide" the image generation model 805 to the best possible image generation based on the condition input 810, so that controlled generation of images can begin to appear even in early diffusion steps. As more iterations are performed, the resulting output image may become more aligned with the text prompt and may also include more details as the initial iterations are refined.

[0109] Additionally, using the same multiscale feature vector at each iteration saves computational costs, as the multiscale feature vector only needs to be generated once, instead of regenerating the multiscale feature vector or a new vector at each iteration step.

[0110] The image generation model 805 can also receive a prompt 820, such as “a girl, pencil sketch.” The prompt 820 represents the desired final output of the image generation model 805. The prompt 820 can be encoded as a feature vector that represents the prompt 820 and provided to the image generation model 805 for the iterative generation process 800.

[0111] At iteration zero (“t=0”), the image generation model 805 can receive an input image 825. which can be an image of random noise that is modeled after the Gaussian distribution. At iteration one f‘t=l”), the image generation model 805 can generate a first iteration image. As shown, because the multiscale feature vector representing the condition input 810 has been included in the iterative generation process 800, features in the first iteration image can resemble the condition input 810, even at early iterations.

[0112] At iteration 2 C‘t=2”), the image generation model 805 can use the first iteration image as an input and generate a second iteration image. The second iteration image illustrates the image generation model 805 refining the features of the first iteration image while maintaining the same control inputs, such as the feature vector representing the prompt 820 and the multiscale feature vector representing the condition input 810, to maintain consistency across iterations.

[0113] This process can be repeated any number of times (e.g., see iteration five (“t=5”) and iteration fifty (“t=50”)) to further refine the features of the image and bring the image more into alignment with the prompt 820. For example, early iterations may generate images that are closely aligned with the condition input 810 but are not as closely aligned with theprompt 820, such as having the facial features indicated by the condition input 810 in the correct locations and relative sizes, but only being a "blurry’' image of a girl that does not look like a pencil sketch. As the iterations refine the image, the resulting output of each iteration can more align with the prompt 820, resulting in an output image that looks much more like a pencil sketch of a girl.Additional Disclosure

[0114] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0115] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for performing image generation, the method comprising: receiving, by a processor, a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network; receiving, by the processor, a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image; applying, by the processor, the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network; performing, by the processor, image generation with the diffusion model based on the text prompt and the multiscale feature vector; and outputting, by the processor, a generated image from the diffusion model.

2. The computer-implemented method of claim 1, wherein the encoder-decoder network is a U-Net encoder-decoder network.

3. The computer-implemented method of claim 2, wherein the one or more levels of the encoder are down-sampling layers of the U-Net encoder-decoder network.

4. The computer-implemented method of claim 1, wherein the plug-in network is an encoder network utilizing depth-wise convolution to generate the multiscale feature vector.

5. The computer-implemented method of claim 1, wherein the condition image is received by the plug-in network from a machine-learned model configured to output at least one of an edge-detection condition image, a face mesh condition image, and a wire-frame condition image.

6. The computer-implemented method of claim 1, wherein the plug-in network is trained independently of the diffusion model.

7. The computer-implemented method of claim 6, wherein training the plug-in network comprises: inputting, by the processor, a training condition image into the plug-in network, the training condition image generated from an original training image; performing, by the processor, image generation with the diffusion model using the multiscale feature vector generated by the plug-in network based on the training condition image, wherein image generation is performed by the diffusion model with weights of the diffusion model frozen; performing, by the processor, a comparison of an output image from image generation with the original training image using a loss function; and updating, by the processor, the plug-in network based on the comparison using the loss function.

8. A computing system for performing image generation, the computing system comprising: one or more processors; and one or more non-transitory, computer readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder network; receiving a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image: applying the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network; performing image generation with the diffusion model based on the text prompt and the multiscale feature vector; and outputting a generated image from the diffusion model.

9. The computing system of claim 8, wherein the encoder-decoder network is a U-Net encoder-decoder network.

10. The computing system of claim 9, wherein the one or more levels of the encoder are down-sampling layers of the U-Net encoder-decoder network.

11. The computing system of claim 8, w herein the plug-in network is an encoder network utilizing depth-wise convolution to generate the multiscale feature vector.

12. The computing system of claim 8. wherein the condition image is received by the plugin netw ork from a machine-learned model configured to output at least one of an edgedetection condition image, a face mesh condition image, and a wire-frame condition image.

13. The computing system of claim 8. wherein the plug-in network is trained independently of the diffusion model.

14. The computing system of claim 13, wherein training the plug-in netw ork comprises: inputting a training condition image into the plug-in network, the training condition image generated from an original training image; performing image generation with the diffusion model using the multiscale feature vector generated by the plug-in network based on the training condition image, wherein image generation is performed by the diffusion model with weights of the diffusion model frozen; performing a comparison of an output image from image generation with the original training image using a loss function; and updating the plug-in network based on the comparison using the loss function.

15. One or more non-transitory, computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving a text prompt for generating an image using a diffusion model, the diffusion model including an encoder-decoder netw ork;receiving a multiscale feature vector from a plug-in network, the multiscale feature vector modeling a condition image; applying the multiscale feature vector to one or more levels of an encoder of the encoder-decoder network; performing image generation with the diffusion model based on the text prompt and the multiscale feature vector; and outputting a generated image from the diffusion model.

16. The one or more non-transitory, computer-readable media of claim 15, wherein the encoder-decoder network is a U-Net encoder-decoder network.

17. The one or more non-transitory, computer-readable media of claim 16, wherein the one or more levels of the encoder are down-sampling layers of the U-Net encoder-decoder network.

18. The one or more non-transitory, computer-readable media of claim 15, wherein the plug-in netw ork is an encoder.

19. The one or more non-transitory, computer-readable media of claim 15, wherein the plug-in network is trained independently of the diffusion model.

20. A computer system comprising: one or more non-transitory computer-readable media that collectively store a plug-in network configured to receive and process a conditioning input to generate a conditioning vector for conditioning a pre-trained denoising diffusion model; wherein the plug-in network is trained separately from the pre-trained denoising diffusion model and is configured to plug in to the pre-trained denoising diffusion model; and wherein the plug-in network comprises a portable network that is stored and run separately from the pre-trained denoising diffusion model.

Images