System and method for generating subject-driven images

Abstract

Embodiments described herein provide a system and method for subject-driven image generation. In at least one embodiment, the system receives, via a data interface, an image containing a subject, a text description of the subject in the image, and text prompts related to different depictions of the subject. The system encodes the image into an image feature vector via an image encoder. The system encodes the text description in the text feature vector via a text encoder. The system generates a vector representation of the subject based on the image feature vector and the text feature vector using a multimodal encoder. The system generates an output image based on the input, which is a combination of the text prompts and the vector representation, using a neural network-based image generation model.

Description

[Technical Field]

[0001] [Cross reference] This application claims priority under U.S. Provisional Application No. 63 / 500,767, filed on 8 May 2023, under Section 119 of the U.S. Patent Act and under the Patent Cooperation Treaty (PCT), and also claims priority under U.S. Non-Provisional Application No. 18 / 498,768, filed on 31 October 2023, which are expressly incorporated herein by reference in their entirety.

[0002] This application relates to U.S. Nonprovisional Application No. 18 / 160,664, filed on 27 January 2023, which is concurrently pending and jointly owned, and is a nonprovisional application of U.S. Provisional Application No. 63 / 424,413, filed on 10 November 2022, under which priority is claimed under 35 U.S.C. 119. Both of these applications are expressly incorporated herein by reference in their entirety.

[0003] [Technical field] The embodiments generally relate to machine learning systems for image generation, and more specifically to systems and methods for subject-driven image generation. [Background technology]

[0004] Machine learning systems are widely used in image generation tasks. For example, text-to-image generation models generate output images based on an input text prompt, such as "vase in a snowy forest." Existing models can generate images of a specific subject (e.g., "vase") in different contexts or different forms. However, existing generative models require a large number (e.g., hundreds or thousands) of tedious fine-tuning steps for each new subject, which hinders the efficient scaling of these approaches to a wide range of subjects. Therefore, there is a need for systems and methods for subject-driven image generation. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Zhang et al., Adding conditional control to text-to-image diffusion models, arXiv:2302.05543, 2023 [Non-Patent Document 2] Kuznetsove et al., The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale, International Journal of Computer Vision, 128(7):1956-1981, 2020 [Non-Patent Document 3] Loshchilov and Hutter, Decoupled weight decay regularization, In International Conference on Learning Representations, 2017 [Non-Patent Document 4] Ruiz et al., arXiv:2208.12242, 2022 [Non-Patent Document 5] Gal et al., An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv: 2208.01618, 2022 [Non-Patent Document 6] Chen et al., Re-imagen: Retrieval-augmented text-to-image generator, arXiv:2209.14491, 2022 [Non-Patent Document 7] Ruiz et al., Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, arXiv: 2208.12242, 2022

Brief Description of the Drawings

[0006] [Figure 1] A schematic diagram showing a subject-driven image model framework according to some embodiments.

[0007] [Figure 2] A schematic diagram showing a training framework for a subject-driven image model according to some embodiments.

[0008] [Figure 3] A diagram showing an exemplary subject-driven generated image according to some embodiments.

[0009] [Figure 4A] A schematic diagram showing a computing device implementing the subject-driven image model framework described in FIGS. 1 and 2 according to some embodiments. [Figure 4B] A schematic diagram showing a neural network structure according to some embodiments.

[0010] [Figure 5] A simplified block diagram of a networked system suitable for implementing the subject-driven image model framework described in FIGS. 1 and 2 and other embodiments described herein.

[0011] [Figure 6] A schematic diagram showing an exemplary training framework of a denoising diffusion model for generating or editing an image given a conditioning input such as a text prompt according to some embodiments.

[0012] [Figure 7] This is an exemplary logic flow diagram illustrating a method for generating subject-driven images based on the framework shown in Figures 1 and 2, according to several embodiments.

[0013] [Figure 8] This chart illustrates the exemplary performance of different embodiments described herein. [Figure 9] This chart illustrates the exemplary performance of different embodiments described herein. [Figure 10] This chart illustrates the exemplary performance of different embodiments described herein. [Figure 11] This chart illustrates the exemplary performance of different embodiments described herein. [Modes for carrying out the invention]

[0014] The embodiments of this disclosure and their advantages will be best understood by referring to the detailed description below. Similar reference numerals are used to identify similar elements shown in one or more drawings, and it should be understood that the illustrations in the drawings are for illustrative purposes only and not to limit the embodiments of this disclosure.

[0015] [Detailed explanation] As used herein, the term “Network” may include any hardware or software-based framework, including any artificial intelligence network or system, a neural network or system, and / or any training or learning model implemented on or with it.

[0016] As used herein, the term “module” may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0017] [overview] Machine learning systems are widely used in image generation tasks. For example, text-to-image generation models generate output images based on an input text prompt, such as "vase in a snowy forest." Existing models can generate images of a specific subject (e.g., "vase") in different contexts or different forms. However, existing generative models require a large number (e.g., hundreds or thousands) of tedious fine-tuning steps for each new subject, which hinders the efficient scaling of these approaches to a wide range of subjects.

[0018] Considering the need for a system and method for subject-driven image generation, embodiments described herein provide a subject-driven image generation model that generates an accurate image depicting a rendition of a given subject using one or more subject images. The subject-driven image generation model can be built on top of a general-purpose base image generation model, such as a denoising diffusion model that generates images based on input prompts. Information about the subject can be provided to the base image generation model by generating input prompts containing subject representations based on one or more input subject images.

[0019] Training a subject-driven image generation model can be carried out in several stages. In a first pre-training stage, a multimodal encoder may be trained to generate latent representations of input images and associated text inputs. Specifically, this can be done as visual-language (multimodal) representation learning of a multimodal encoder (i.e., Q-Former) as described in U.S. Patent Application No. 18 / 160,664, incorporated herein by reference. In this pre-training stage, visual-language representation learning compels the multimodal encoder to learn the visual representations most relevant to the input text.

[0020] In the second pre-training stage, subject representation learning, the multimodal encoder is trained on subject representations. The goal of this stage is for the model to learn to represent a subject from an input image without representing other details of the input image that are not related to the subject (e.g., background). To achieve this, training input / output pairs of images containing the subject in different contexts may be used. A subject-driven image generation model may be provided with an input image containing the subject and prompts to generate an output image. The output image may be compared to a ground truth subject image in a different context to generate a loss used to update the model's parameters via backpropagation. In some embodiments, the image input / output pairs may be created by performing background substitution on an existing image. The subject representation learning stage is not specific to a particular subject and is performed using images of various subjects.

[0021] After the second pre-training phase (subject representation learning phase), zero-shot image generation can be performed using one or more subject images without additional fine-tuning of the subject-driven image generation model. However, better performance can be achieved, in some situations, by using an additional subject-specific fine-tuning phase.

[0022] The fine-tuning stage may be performed similarly to the subject representation learning stage, but for a specific subject. For example, the user may provide one or more images of a subject (e.g., a dog). The parameters of the subject-driven image generation model may be updated based on a loss objective of comparing the images generated based on the input images with a ground truth image. The ground truth image for the fine-tuning stage may be the same as the input image. In other words, the subject-driven image generation model may be trained to replicate the input image. In some embodiments, a background-replaced version of the input image is used as the ground truth output. Some parameters of the subject-driven image generation model may be frozen to prevent overfitting.

[0023] During inference, given a subject image and a text description of the subject, the multimodal encoder generates a multimodal subject representation. This subject representation is combined with a text prompt and provided to a general-purpose image generation model that generates an image of the subject based on the text prompt.

[0024] The embodiments described herein offer several advantages. For example, various existing image generation models can be used in conjunction with the methods described herein, because the input prompts of various models can be easily replaced with prompts that include subject representations without changing the parameters of the base image generation model itself. This can reduce the amount of training / fine-tuning required to create the final image generation model. By separating subject representations, multiple output images that are not linked to other aspects of the conditioning image can be generated based on a single subject. Compared to other methods, high-quality zero-shot subject-driven image generation is possible, and therefore, less computation and / or memory resources are required to generate the final image. The fine-tuning methods described herein are also more efficient than other methods, requiring fewer fine-tuning steps. Thus, neural network techniques in image generation are improved.

[0025] Figure 1 is a simplified diagram illustrating a subject-driven image model framework in several embodiments. The subject-driven image model 130 includes a base image model 122 and a multimodal encoder 108 that assists in generating input prompts having subject representations for the image model 122. The image model 122 may be, for example, a denoising diffusion model as described in Figure 6. Other image models may be used for the image model 122 if they generate images based on input prompts. The subject-driven image model 130 takes in an input subject image 102, subject text 112, and a text prompt 118, and generates an output image 124 based on these inputs. For example, the input subject image may be an image of a backpack on a wire fence, the subject text may be "backpack", and the text prompt may be "in the Grand Canyon". Using these exemplary inputs, the subject-driven image model 130 generates an image of the backpack in the input image, but in the Grand Canyon.

[0026] The input subject image 102 can be encoded into an image feature vector by an image encoder 104. The image encoder 104 may be a pre-trained image encoder that extracts general-purpose image features. The subject text 112 can be encoded into a text feature vector by a text encoder 106. The image feature vector and the text feature vector can be input to a multimodal encoder 108. The multimodal encoder 108 may be a query transformer ("Q-Former"), such as the one described in U.S. Patent Application No. 18 / 160,664, which is incorporated herein by reference. The multimodal encoder 108 may take a query 110 as input. The query 110 may be a randomly initialized vector that can be tuned as part of the training process. The multimodal encoder 108 generates a vector representation of the subject (e.g., a subject embedding) by using the subject text 112 to focus on the most important parts of the input subject image 102 (i.e., the subject). In some embodiments, a feedforward neural network further updates the vector representation of the subject to provide a subject embedding 116.

[0027] The subject embedding 116 and the text prompt 118 can be combined and input to the text encoder 120 to generate a prompt for the image model 122. The image model 122 can then generate an output image 124 based on the prompt. The subject text 112 may be combined with the subject embedding 116 and the text prompt 118. In some embodiments, the subject embedding 116, the text prompt 118, and the subject text 112 can be combined by using a prompt template. The prompt template may be, for example, "[text prompt], [subject text] is [subject embedding]". For example, if the text prompt is "Grand Canyon backpack" and the subject text is "backpack", the combined prompt is concatenated with the subject embedding 116 as "Grand Canyon backpack, backpack is".

[0028] In some embodiments, multiple input subject images 102 may be used to generate a single output image 124. The multimodal encoder 108 may encode each subject image 102 with subject text 112 to generate a corresponding subject embedding. Each subject embedding may be combined (e.g., by averaging) to generate a combined subject embedding 116. By using multiple images of the same subject as input subject images 102, the resulting averaged subject embedding may more completely separate the subject from the image and remove more non-subject information from the subject embedding 116.

[0029] The training of the subject-driven image model 130 can be carried out in several stages. In a first pre-training stage, the multimodal encoder 108 can be trained to generate latent representations of input images and associated text inputs. Specifically, this can be done as visual-language (multimodal) representation learning of a multimodal encoder (i.e., Q-Former) as described herein by reference. In this pre-training stage, visual-language representation learning causes the multimodal encoder to learn the visual representations most relevant to the input text. The second pre-training stage will be described with reference to Figure 2.

[0030] Figure 2 is a simplified diagram illustrating a training framework for a subject-driven image model according to several embodiments. Specifically, Figure 2 shows a second pre-training stage that may be performed after the first pre-training stage described in Figure 1. In the second pre-training stage, subject representation learning, the multimodal encoder 108 is trained on subject representations. The feedforward 114, query 110, text encoder 120, and / or image model 122 may be trained together with the multimodal encoder 108. The objective of this stage is for the model to learn to represent a subject from the input subject image 102 without representing other details of the input subject image 102 that are irrelevant to the subject (e.g., background). To achieve this, training ground truth input / output pairs of images containing subjects in different contexts may be used.

[0031] To reduce the effort required when collecting multiple images of each subject in different contexts, the input subject image 102 can be automatically modified by a background replacement module 202 that replaces the background. The original input subject image 102 can be used as a ground truth image that the subject-driven image model 130 attempts to replicate. By using the modified image 204 as the input image to the subject-driven image model 130 and the original input subject image 102 as a ground truth comparison for loss calculation 206, this allows the original caption of the source image to be used as a text prompt 118, and the input subject image 102 can have any random background without requiring an accurate text description of the background.

[0032] In some embodiments, the background replacement module 202 receives an input subject image 102 and subject text 112 associated with the input subject image 102. The input subject image 102 and subject text 112 can be input to a text-prompted segmentation model. A trimap can be generated by the segmentation model, which maps portions of the input subject image 102 into foreground, background, and low-confidence regions. Given the trimap, the background replacement module 202 can extract the foreground (i.e., the subject) and place it on a random background image via alpha blending.

[0033] A subject-driven image generation model can generate an output image 124 given an input subject image 102 containing a subject, subject text 112, and a text prompt 118. The output image can be compared to a ground truth subject image (e.g., a modified image 204) by a loss calculation 206. The loss calculated by the loss calculation 206 can be used to update the parameters of the subject-driven image model 130 via backpropagation 208. In some embodiments, backpropagation 208 can update the parameters of a multimodal encoder 108, a query 110, a text encoder 120, and / or an image model 122. The loss calculation 206 may include, for example, a cross-entropy loss function. The subject representation learning stage is not specific to a particular subject and is performed using images of various subjects. During this training stage, multiple input subject images 102 can be used for each output image 124 by averaging the encoded subject representations together, as shown in Figure 1. In some embodiments, to help maintain the original text-to-image generation capability, a certain percentage of the time (e.g., 15%) of subject embeddings 116 are randomly dropped from the combined prompts.

[0034] After the second pre-training phase (subject representation learning phase), zero-shot image generation can be performed using one or more subject images without additional fine-tuning of the subject-driven image generation model. However, better performance can be achieved, in some situations, by using an additional subject-specific fine-tuning phase.

[0035] The fine-tuning stage can be performed in a similar manner to the subject representation learning stage, but for a specific subject. For example, the user may provide one or more input subject images 102 (e.g., dogs). The parameters of the subject-driven image generation model 130 can be updated based on a loss calculation 206 that compares the images generated based on the input images with ground truth images. The ground truth images for the fine-tuning stage can be the same as the input images, without background substitution. Background substitution can be used as in the subject representation learning stage, but sufficient performance can be achieved without background substitution, saving the additional inference time required to generate background substitution. Without background substitution, the subject-driven image generation model is effectively trained to replicate the input images. Only a predetermined number of fine-tuning steps are performed to prevent overfitting.

[0036] In some embodiments, backpropagation 208 during subject-specific fine-tuning may update the parameters of the multimodal encoder 108, query 110, text encoder 120, and / or image model 122. In some embodiments, the text encoder 120 may be trained during the subject representation learning phase and frozen during the fine-tuning phase to prevent overfitting to a particular subject. During fine-tuning, the image encoder 104, text encoder 106, query 110, feedforward 114, and / or multimodal encoder 108 may be frozen (i.e., their parameters are not changed). Once these parameters are frozen, a single subject embedding 116 (based on a single image or averaged over multiple images) is generated once and can be cached for reuse during fine-tuning without requiring other forward passes. This may enable faster fine-tuning.

[0037] As described above, these methods can be used with various image models 122. For example, ControlNet described in Non-Patent Literature 1. Using ControlNet, simultaneous structure-controlled and subject-controlled generation is possible. A conditioning image that can provide the structure of the output image can be provided, while the input subject image 102 provides subjects that can be included in the final output image. In this way, the subject-driven image model 130 takes into account input structural conditions from the conditioning image, such as edge maps and depth maps, in addition to the subject queue.

[0038] In another example, the subject-driven image model 130 may be integrated with an image editing model that edits the original image using subject-specific visual elements. To edit the image, the subject may be identified for replacement in the original image (e.g., "dog"). A cross-attention map from the original generation is then used while a new attention map is generated for the inserted subject embedding. Denoising latents are blended at each step based on the extracted editing mask; that is, the latents of the pre-edited region are from the original generation, and the latents of the edited region are from the subject-driven generation. In this way, the edited image can be generated using subject-specific visual elements while also preserving the unedited region.

[0039] Figure 3 shows exemplary subject-driven generated images according to several embodiments. The exemplary input subject image 102 shows a backpack resembling a dog's face. The exemplary input subject image 102 is shown against a background that includes a wire mesh fence and other features. Exemplary output images 124a and 124b show the backpack in its "cubic shape" and the backpack "in the Grand Canyon," respectively. For example, output image 124a may be generated by the subject-driven image model 130 using the input subject image 102 with subject text 112 "backpack" and text prompt 118 "cubic shape." In another example, output image 124b may be generated by the subject-driven image model 130 using the input subject image 102 with subject text 112 "backpack" and text prompt 118 "in the Grand Canyon."

[0040] [Computer and network environment] Figure 4A is a simplified diagram showing a computing device 400 implementing the subject-driven image model framework described in Figures 1 and 2, according to several embodiments. As shown in Figure 4A, the computing device 400 includes a processor 410 coupled to memory 420. The operation of the computing device 400 is controlled by the processor 410. Although the computing device 400 is shown to have only one processor 410, it is understood that the processor 410 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within the computing device 400. The computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0041] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during the operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0042] The processor 410 and / or memory 420 can be located in any suitable physical arrangement. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, the processor 410 and / or memory 420 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located in one or more data centers and / or cloud computing facilities.

[0043] In some examples, memory 420 may include a non-temporary, tangible, machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 410), can cause one or more processors to perform the methods further described herein. For example, as illustrated, memory 420 includes instructions for a subject-driven image generation module 430 which may be used to implement and / or emulate a system and model and / or to implement any of the methods further described herein. The subject-driven image generation module 430 may receive input 440 such as input training data (e.g., an image, a subject caption, and / or an image with a replaced background) via a data interface 415 and generate an output 450 which may be a generated image.

[0044] The data interface 415 may include a communication interface and a user interface (such as a voice input interface or a graphical user interface). For example, the computing device 400 may receive input 440 (such as a training dataset) from a networked database via the communication interface. Alternatively, the computing device 400 may receive input 440, such as an input subject image, from a user via the user interface.

[0045] In some embodiments, the subject-driven image generation module 430 is configured to generate an image depicting a subject based on one or more input subject images and text prompts. The subject-driven image generation module 430 may further include a multimodal representation learning submodule 431. The multimodal representation learning submodule 431 may be configured to train a multimodal encoder (e.g., multimodal encoder 108) to generate a vector representation of an input image based on associated text, as shown in Figure 1. The subject-driven image generation module 430 may further include a subject representation learning submodule 432. The subject representation learning submodule 432 may be configured to further train a multimodal encoder (e.g., multimodal encoder 108) to generate an output vector representation of a subject based on an input subject image (e.g., input subject image 102) and subject text (e.g., subject text 112), as shown in Figure 1. The subject-driven image generation module 430 may further include a fine-tuning submodule 433. The fine-tuning submodule 433 may be configured to fine-tune the parameters of the subject-driven image model based on a specific subject represented in one or more subject images, as shown in Figure 1. The subject-driven image generation module 430 may further include an inference submodule 434. The inference submodule 434 may be configured to generate an output image based on an input subject image, subject text, and a text prompt, as shown in Figure 1.

[0046] Some examples of computing devices, such as computing device 400, may include non-temporary, tangible, machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 410), causes one or more processors to perform the process of the Method. Some common forms of machine-readable media that may contain the process of the Method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0047] Figure 4B is a simplified diagram showing a neural network structure that implements the subject-driven image generation module 430 described in Figure 4A, according to several embodiments. In some embodiments, one or more of the subject-driven image generation module 430 and / or its submodules 431-434 may be implemented at least partially via the artificial neural network structure shown in Figure 4B. The neural network includes a computing system built on a collection of connected units or nodes referred to as neurons (e.g., 444, 445, 446). Neurons are often connected by edges, and tunable weights (e.g., 451, 452) are often associated with the edges. Neurons are often aggregated into layers so that different layers can perform different transformations on their respective inputs and output the transformed input data to the next layer.

[0048] For example, a neural network architecture may include an input layer 441, one or more hidden layers 442, and an output layer 443. Each layer may contain multiple neurons, and neurons between layers are interconnected according to a specific topology of the neural network topology. The input layer 441 receives input data, such as an input subject image (for example, 440 in Figure 4A). The number of nodes (neurons) in the input layer 441 may be determined by the number of dimensions of the input data (for example, the length of the vector representing the image). Each node in the input layer represents a feature or attribute of the input.

[0049] The hidden layer 442 is an intermediate layer between the input and output layers of the neural network. Two hidden layers 442 are shown in Figure 4B for illustrative purposes only, and it should be noted that any number of hidden layers can be used in the neural network structure. The hidden layer 442 can extract and transform input data through a series of weighted computations and activation functions.

[0050] For example, as illustrated in Figure 4A, the subject-driven image generation module 430 receives input subject images and image generation prompts 440, and, based on the prompts, transforms the inputs into output images 450 of the generated depictions of the subjects. To perform the transformation, each neuron receives the input signals, performs a weighted sum of the inputs according to the weights assigned to each connection (e.g., 451, 452), and then applies the activation function associated with each neuron (e.g., 461, 462, etc.) to the result. The output of the activation function is passed to the next layer of neurons or serves as the final output of the network. The activation function may be the same or different across different layers. Exemplary activation functions, but are not limited to, include sigmoid, hyperbolic tangent, normalized linear unit (ReLU), leaky ReLU, and softmax. In this way, after several hidden layers, the input data received in the input layer 441 is transformed into significantly different values ​​exhibiting data characteristics corresponding to the task the neural network structure is designed to perform.

[0051] The output layer 443 is the final layer of the neural network structure. It generates the network's output or predictions based on calculations performed in the preceding layers (e.g., 441, 442). The number of nodes in the output layer depends on the nature of the task being addressed. For example, in a binary classification problem, the output layer may consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer may have multiple nodes, each representing the probability of belonging to a particular class.

[0052] Therefore, one or more of the subject-driven image generation module 430 and / or its submodules 431-434 may include a transformational neural network structure of layers of neurons and weights and activation functions that describe the nonlinear transformations in each neuron. Such a neural network structure is often implemented on one or more hardware processors 410, such as a graphics processing unit (GPU). An exemplary neural network may be a diffusion model U-Net, for example.

[0053] In one embodiment, the subject-driven image generation module 430 and its submodules 431-434 may be implemented by hardware, software, and / or a combination thereof. For example, the subject-driven image generation module 430 and its submodules 431-434 may include, but are not limited to, specific neural network structures implemented and run on various hardware platforms 460, such as dedicated AI accelerators like CPUs (Central Processing Units), GPUs (Graphics Processing Units), FPGAs (Field-Programmable Gate Arrays), Application-Specific Integrated Circuits (ASICs), TPUs (Tensor Processing Units), and dedicated hardware accelerators specifically designed for the neural network computations described herein. Exemplary specific hardware for the neural network structures may include, but are not limited to, the Google Edge TPU, Deep Learning Accelerator (DLA), and NVIDIA AI-focused GPU. The hardware 460 used to implement the neural network structure is specifically configured based on factors such as the complexity of the neural network, the scale of the task (e.g., training time, input data scale, training dataset size, etc.), and the desired performance.

[0054] In one embodiment, a neural network-based subject-driven image generation module 430 and one or more of its submodules 431-434 may be trained by iteratively updating the underlying parameters of the neural network (e.g., weights 451, 452, bias parameters and / or coefficients in activation functions 461, 462 associated with neurons) based on a loss function. For example, during forward propagation, training data such as subject images, subject descriptions, image generation prompts, and subject images with the background replaced is fed to the neural network. The data flows through layers 441, 442 of the network, each layer performing calculations based on its weights, biases, and activation functions until output layer 443 generates the network output 450. In some embodiments, output layer 443 generates an intermediate output on which the network output 450 is based.

[0055] The output generated by the output layer 443 is compared to the expected output from the training data (e.g., a "ground truth," such as a corresponding subject image with a replaced background) to compute a loss function that measures the discrepancy between the predicted and expected outputs. For example, the loss function could be a cross-entropy loss. Given the loss, the negative gradient of the loss function is computed individually for each weight in each layer. Such negative gradients are computed iteratively in reverse, layer by layer, from the last layer 443 of the neural network to the input layer 441. These gradients quantify the sensitivity of the network's output to changes in parameters. The chain rule is applied to efficiently compute these gradients by propagating them in reverse from the output layer 443 to the input layer 441.

[0056] The parameters of the neural network are updated backward from the last layer to the input layer (backpropagation) based on the calculated negative gradient using an optimization algorithm to minimize the loss. Backpropagation from the last layer 443 to the input layer 441 may be performed for several training samples in several iterative training epochs. In this way, the parameters of the neural network can be gradually updated in a direction that results in less or minimized loss, indicating that the neural network has been trained to produce predicted output values ​​closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching the maximum number of epochs or achieving satisfactory performance on validation data. In this regard, the trained network can be used to make predictions about new, unseen data, such as generating images about new subjects.

[0057] Neural network parameters can be trained over multiple stages. For example, initial training (e.g., pre-training) may be performed on one set of training data, and then additional training stages (e.g., fine-tuning) may be performed using different sets of training data. In some embodiments, all or some of the parameters of one or more neural network models used together may be frozen so that the "frozen" parameters are not updated during their training stages. This may allow, for example, a smaller subset of parameters to be trained without the computational cost of updating all of the parameters.

[0058] Therefore, the training process transforms the neural network into an "updated" trained neural network with updated parameters such as weights, activation functions, and biases. The trained neural network thus improves neural network techniques in image generation.

[0059] Figure 5 is a simplified block diagram of a networked system 500 suitable for implementing the subject-driven image model framework described in Figures 1-2 and other embodiments described herein. In one embodiment, the system 500 includes a user device 510 that can be operated by a user 540, data vendor servers 545, 570, and 580, a server 530, and other forms of devices, servers, and / or software components that operate to perform various methodologies according to the embodiments described. Exemplary devices and servers may include devices, standalone, and enterprise-class servers, which may be similar to the computing device 400 described in Figure 4A, and run an OS such as MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. The devices and / or servers shown in Figure 5 may be deployed in other ways, and it can be understood that the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by more or fewer devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0060] The user device 510, data vendor servers 545, 570, and 580, and server 530 can communicate with each other via network 560. The user device 510 can be utilized by a user 540 (e.g., a driver, system administrator, etc.) to access various functions available to the user device 510, which may include processes and / or applications associated with server 530 for receiving output data anomaly reports.

[0061] The user device 510, the data vendor server 545, and the server 530 may each include one or more processors, memory, and other suitable components for executing instructions such as program code and / or data stored on one or more computer-readable media, and for implementing the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as memory or data storage devices, which are internal to and / or external to various components of the system 500 and / or accessible via the network 560.

[0062] The user device 510 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 545 and / or server 530. For example, in one embodiment, the user device 510 may be implemented as other types of computing devices that can transmit and / or receive data, such as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, an embedded communication device, and / or an iPad® from APPLE®. Although only one communication device is shown, multiple communication devices may function similarly.

[0063] The user device 510 in Figure 5 includes a user interface (UI) application 512 and / or other applications 516, which may correspond to executable processes, procedures, and / or applications having associated hardware. For example, the user device 510 may receive a message from the server 530 indicating a generated image and display the message via the UI application 512. In other embodiments, the user device 510 may include additional or different modules having dedicated hardware and / or software, as needed.

[0064] In various embodiments, the user device 510 includes other applications 516, which may be desired in certain embodiments to provide functionality to the user device 510. For example, other applications 516 may include security applications for implementing client-side security features, program client applications for interfacing with appropriate application programming interfaces (APIs) over the network 560, or other types of applications. Other applications 516 may also include communication applications such as email, text, voice, social networking, and IM applications, which enable the user to send and receive emails, phone calls, texts, and other notifications over the network 560. For example, other applications 516 may be email or instant messaging applications that receive predictive result messages from the server 530. Other applications 516 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 516 may include processor-executable asset management software programs, including a graphical user interface (GUI) configured to provide an interface to the user 540 for viewing generated images.

[0065] The user device 510 may further include a database 518 stored in the temporary and / or non-temporary memory of the user device 510, the database 220 which stores various applications and data that can be used during the execution of various modules of the user device 510. The database 518 may store a user profile associated with user 540, predictions previously viewed or saved by user 540, historical data received from server 530, and / or similar. In some embodiments, the database 518 may be local to the user device 510. However, in other embodiments, the database 518 may be external to the user device 510 and accessible by the user device 510, including a cloud storage system and / or database accessible via network 560.

[0066] The user device 510 includes at least one network interface component 517 adapted to communicate with the data vendor server 545 and / or server 530. In various embodiments, the network interface component 517 may include a variety of other types of wired and / or wireless network communication devices, including DSL (e.g., digital subscriber line) modems, PSTN (public switched telephone network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth®, and short-range wireless communication devices.

[0067] The data vendor server 545 may correspond to a server that hosts the database 519 and provides the server 530 with a training dataset containing training images and prompts. The database 519 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0068] The data vendor server 545 includes at least one network interface component 526 adapted to communicate with user devices 510 and / or server 530. In various embodiments, the network interface component 526 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth®, and near-field communication devices. For example, in one embodiment, the data vendor server 545 may transmit asset information from the database 519 to server 530 via the network interface 526.

[0069] The server 530 may house the subject-driven image generation module 430 and its submodules as shown in Figure 4A. In some implementations, the subject-driven image generation module 430 may receive data from the database 519 in the data vendor server 545 via the network 560 to generate images. The generated images may be transmitted via the network 560 to the user device 510 for review by the user 540.

[0070] The database 532 may be stored in the temporary and / or non-temporary memory of the server 530. In one embodiment, the database 532 may store data obtained from the data vendor server 545. In one embodiment, the database 532 may store parameters of the subject-driven image generation module 430. In one embodiment, the database 532 may store previously generated images and corresponding input feature vectors.

[0071] In some embodiments, the database 532 may be local to the server 530. However, in other embodiments, the database 532 may be external to the server 530 and accessible by the server 530, which includes a cloud storage system and / or database accessible via the network 560.

[0072] Server 530 includes at least one network interface component 533 adapted to communicate with user devices 510 and / or data vendor servers 545, 570, or 580 via network 560. In various embodiments, the network interface component 533 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., digital subscriber line) modems, PSTN (public switched telephone network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0073] Network 560 can be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 560 may include the Internet or one or more intranets, a fixed telephone network, a wireless network, and / or other suitable types of networks. Thus, network 560 can correspond to small-scale communication networks, such as private or local area networks, or large-scale networks, such as wide area networks or the Internet, accessible by various components of system 500.

[0074] Figure 6 is a simplified diagram showing an exemplary training framework 600 for a denoising diffusion model for generating or editing images given a conditioning input, such as a text prompt. In some embodiments, the image model 122 is a diffusion model including a denoising model 612, which may be a U-Net, and is trained or pre-trained according to the training framework 600. In some embodiments, the image model 122 is pre-trained as described below with respect to Figure 6 and can be further trained / fine-tuned according to the frameworks shown in Figures 1-2, which may be trained together with additional models. In one embodiment, the denoising diffusion model is trained to generate an image (e.g., output 616) based on user input (e.g., a text prompt in the conditioning input 610). In inference, the denoising diffusion model receives a text prompt describing the image content, and may start with a random noise vector as a seed vector, and the denoising model progressively removes "noise" from the seed vector so that it is conditioned by the user input (e.g., the text prompt) so that the resulting image can gradually align with the user input. As shown in Figure 1, the conditioning input can be a combination of text prompts 118 and subject embeddings 116. Completely removing noise in a single step is computationally impossible. For this reason, the denoising model is trained to remove small amounts of noise, and the denoising step is repeated iteratively over multiple iterations (e.g., 50 iterations) until the image is finally sharp.

[0075] Framework 600 illustrates how such a diffusion model may be trained to generate a prompt image by gradually removing noise from a seed vector. The upper part of the illustrated framework 600, including the encoder 604 and noise ε608 steps, may only be used during the training process, and not at inference, as described below. The training dataset may contain a variety of images, which may be associated with information such as the caption of each image in the training dataset, which does not necessarily require annotation but can be used as a conditioning input 610. The training images may be used as input 602. Encoder 604 may encode input 602 into a latent representation (e.g., a vector) that represents the image.

[0076] In some embodiments, the diffusion model may be trained using pixel-level data directly. In other embodiments, the diffusion model may be trained on a reduced version of the image. However, generally, the image is trained on a denoising model ε θ An encoder 604 of some form is desirable, such as being in a format that is more easily consumed by 612. The rest of the explanation of framework 600 assumes that encoder 604 generates a latent vector representation of input 602.

[0077] The latent vector representation z0606a represents the first encoded latent representation of input 602. Noise ε608 is added to representation z0606a to generate representation z1606b. Then, noise ε608 is added to representation z1606b to generate a noisier representation. This process is called noise latent representation z T The process is repeated T times (for example, 50 iterations) until 606t is obtained. The random noise ε608 added in each iteration can be a random sample from a probability distribution such as a Gaussian distribution. The amount of noise ε608 added in each iteration (i.e., the variance) may be constant or may change over the iterations. The amount of noise ε608 added may depend on other factors such as image size or resolution.

[0078] This process of incrementally adding noise to the latent representation effectively generates the training data used when training the diffusion noise removal model 612, as described below. As shown, the noise removal model ε θ 612 is repeatedly used to reverse the process of adding noise to the latents, i.e., perform reverse diffusion, from z' T 618t to z'0618a. The noise removal model ε θ 612 can be a neural network-based model with learnable parameters. The input to the noise removal model ε θ 612 can include a noisy latent representation (e.g., the latent representation z T 606t with noise added), and a text prompt describing the desired content of the output image, e.g., a conditioning input 610 such as "a hand holding the Earth". As shown, the noisy latent representation can be gradually supplied to the noise removal model 612 iteratively and progressively, e.g., from z' T 618t to z'0618a, to gradually remove noise from the latent representation vector based on the conditioning input 610.

[0079] Ideally, the progressive outputs z' θ of the repeated noise removal model 612ε T from 618t to z'0618a can be an incrementally denoised version of the input latent representation z' T 618t conditioned by the conditioning input 610. The latent image representation generated using the noise removal model ε θ 612 can be decoded using the decoder 614 to provide an output 616 that is the denoised image.

[0080] In one embodiment, the output image 616 is then compared to the input training image 602 to calculate the loss for updating the denoising model 612 via backpropagation. In another embodiment, the latent representation 602a of the input 606 may be compared to the denoising latent representation 618a to calculate the loss for training. In another embodiment, the loss objective is to remove the noise actually added (e.g., by noise ε608) from the denoising model ε θ The noise can be calculated by comparing it with the noise predicted by 612. Denoising model ε θ 612 can be trained based on this loss objective (for example, the denoising model ε θ The 612 parameters can be updated to minimize the loss by gradient descent using backpropagation. Note that this is based on the denoising model ε. θ During the training process of 612, it is not necessarily required that an actual denoised image be generated (e.g., the output 616 of decoder 614), because the loss is based on each intermediate noise estimate, not necessarily the final image.

[0081] In one embodiment, the conditioning input 610 may include a description of the input image 602, and in this way the noise reduction model ε θ 612 learns to reproduce the described image. Alternatively (or additionally), the conditioning input 610 may include a text prompt, a conditioning image, an attention map, or other conditioning inputs. These inputs are then processed by the denoising model ε θ Before being used by 612, it may be encoded in some way. For example, the conditioning image may be encoded using an encoder similar to encoder 604. The conditioning input 610 may also include a time step that can be used to provide the model with a general estimate of how much noise remains in the image, and the time step may be incremented (or decremented) with each iteration.

[0082] In some embodiments, the noise reduction model εθ 612 can be implemented via a structure called "U-Net". The U-Net structure may include a series of convolutional and pooling layers that generate progressively lower resolution multi-channel feature maps. Each pooling layer and one or more associated convolutional layers can be considered an encoder. Following the convolutional and pooling layers (i.e., encoders), there may be a series of upsampling and convolutional layers that generate progressively higher resolution multi-channel feature maps. Each upsampling layer and one or more associated convolutional layers can be considered a decoder. The U-Net may also include skip connections, where the output of each encoder layer is coupled to the corresponding decoder layer, skipping intermediate encoder / decoder layers. The skip connections allow for information about the precise location of features extracted by the convolutional (encoder) layers. The convolutional kernels of the convolutional layers and the upsampling functions of the upsampling layers can be learned during the training process. Conditioning inputs (e.g., images or natural language prompts) can be used to condition the functionality of the U-Net. For example, the conditioning input may be encoded, and cross-attention may be applied between the encoded conditioning input and the feature map in the encoder / decoder layer.

[0083] Noise reduction model ε θ The direct output of 612 (for example, when implemented as U-Net) can be an estimate of the noise present in the input latent representation, or more generally, the noise distribution. In this sense, the direct output is due to the noise, rather than the latent representation of the image. However, this estimated noise may be used to generate an incrementally denoised image representation, which is done using the denoising model ε θ This could also be the input to the next iteration of 612.

[0084] During inference, the denoising model ε θ612 can be used to denoise a latent expression given a conditioning input 610. Noisy latent expression z T Instead of 606t, the input to the denoising model sequence can be a randomly generated vector used as a seed. By providing different random starting seeds, different images can be generated. The resulting denoised latent representation after T denoising model steps can be decoded by a decoder (e.g., decoder 614) to produce an output 616 of a denoised image. For example, the conditioning input may include an image description, and the output 616 may be an image aligned with that description.

[0085] Noise reduction model ε θ While 612 is presented as the same model used iteratively, it should be noted that models that are clearly distinguishable at different steps of the process may be used. Furthermore, the “denoising diffusion model” is a single denoising model ε θ 612. Multiple noise reduction models ε θ 612 chains, and / or a single denoising model ε θ It should be noted that this may refer to repeated use of 612. The “denoising spread model” may also include related features such as the decoder 614, any preprocessing applied to the conditioning input 610, etc. This framework 600 for training and inference of the denoising spread model may be further modified to provide improved results and / or additional functionality, for example, as in the embodiments described herein.

[0086] [Example Workflow] Figure 7 is an exemplary logic flow diagram illustrating a subject-driven image generation method 700 based on the framework shown in Figures 1 and 2, according to some embodiments described herein. One or more of the processes of Method 700 may be implemented in the form of executable code stored on a non-temporary, tangible, machine-readable medium, which, when executed by one or more processors, can cause one or more processors to perform one or more of the processes. In some embodiments, Method 700 corresponds to the operation of a subject-driven image generation module 430 (e.g., Figures 4A and 5) that performs inference and / or training of a subject-driven image generation model.

[0087] As illustrated, Method 700 includes several enumerated steps, but embodiments of Method 700 may include additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0088] In step 701, the system (e.g., computing device 400 or server 530) receives, via a data interface (e.g., data interface 415 or network interface 533), a subject image containing the subject (e.g., input subject image 102), a text description of the subject in the image (e.g., subject text 112), and text prompts related to different depictions of the subject (e.g., text prompt 118).

[0089] In step 702, the system encodes the subject image into an image feature vector via an image encoder (for example, image encoder 104).

[0090] In step 703, the system encodes the text description into a text feature vector via a text encoder (for example, text encoder 106).

[0091] In step 704, the system generates a vector representation of the subject (e.g., subject embedding 116) based on image feature vectors and text feature vectors using a multimodal encoder (e.g., multimodal encoder 108). In some embodiments, the system generates multiple vector representations of the subject based on multiple image feature vectors using the multimodal encoder, and the vector representation is the average of the multiple vector representations. The average of the multiple vector representations can be cached so that it can be reused to generate images based on different text prompts for the same subject. In some embodiments, the vector representation also passes through a feedforward model (e.g., feedforward 114), which may be a multilayer perceptron (e.g., as shown in Figure 4B). In some embodiments, the feedforward 114 consists of two linear layers.

[0092] In step 705, the system generates an output image (e.g., output image 124) based on an input combining a text prompt and a vector representation, using a neural network-based image generation model (e.g., image model 122). In some embodiments, the text prompt and vector representation are combined by being input to a text encoder (e.g., text encoder 120) which may be part of the image generation model. In some embodiments, the combined text prompt and vector representation are used as a conditioning prompt for a denoising diffusion model. In some embodiments, the denoising diffusion model also takes the conditioning image as input used to guide the generation of the output image. The conditioning image may be received via a data interface.

[0093] In step 706, the system trains parameters associated with at least one model based on the output image. In some embodiments, training parameters involves training a multimodal encoder, a text encoder for subject text and / or a text encoder for text prompts, and a neural network-based image generation model together, based on a comparison of the output image and a modified image in which the subject is on a different background than the background in the subject image. In some embodiments, generating a vector representation is further based on a plurality of query vectors (e.g., query 110), and training involves updating the plurality of query vectors. The query 110 may interact with subject text 112 via a self-attention layer and with image features of the input subject image 102 via a cross-attention layer. In some embodiments, training parameters involves training a neural network-based image generation model based on a comparison of the output image and the subject image. In some embodiments, the parameters of the text encoder are frozen while the neural network-based image generation model is being trained.

[0094] [Example result] Figures 8 to 11 provide charts illustrating exemplary performance of different embodiments described herein. For multimodal representation learning, the experiment followed BLIP-2 and pre-trained the model with 129M image-text pairs, including 115M image-text pairs. As previously stated, the experiment learned subject representations using 16 queries. For subject representation learning, the experiment used a subset of 292K images from OpenImage(Newport)-V6, as described in Non-Patent Literature 2. Each of the OpenImage-V6 images used in the experiment contains a prominent subject. The experiment also removed images with subjects related to humans. Captions were generated as text prompts using BLIP-2 OPT6.7B. Subject inputs were synthesized using 59K background images from the web. Stable Diffusion v1-5 was used as the underlying Diffusion model. As described in Non-Patent Literature 3, the AdamW optimizer was used with a constant learning rate of 2e-6 for 500K steps and a total batch size of 16. The subject images for the experiment are from the DreamBench dataset, as described in Non-Patent Document 4.

[0095] The baseline model for comparison included Textual Inversion, as described in Non-Patent Document 5. Another baseline model for comparison was Re-Imagen, as described in Non-Patent Document 6. Another baseline model for comparison was DreamBooth, as described in Non-Patent Document 7. The metrics used in the experiment included DINO, CLIP-I, and CLIP-T scores, as described in Non-Patent Document 7.

[0096] Figure 8 provides a chart illustrating exemplary performance of at least one embodiment described herein. In particular, Figure 8 shows a quantitative comparison for DreamBench. Mean metrics and differences are shown over 10 experimental runs using different sets of random seeds in zero-shot (ZS) and fine-tuning (FT) setups. DINO and CLIP-I scores measure subject alignment, and CLIP-T measures image-text alignment. Four images were generated for each text prompt, resulting in a total of 3,000 images for all subjects. Generation was repeated using 10 fixed sets of random seeds, and the mean scores were reported. The overall results are consistent with the qualitative finding that BLIP-Diffusion outperforms Textual Inversion and Re-Imagen, while demonstrating comparable performance to DreamBooth, with less fine-tuning effort required. In particular, zero-shot generation is better than the fine-tuned Textual Inversion results. Furthermore, subject-specific metrics are shown, observing that fine-tuning significantly improves subject alignment. On the other hand, fine-tuning also improves image-text alignment on average. When fine-tuning harms image-text alignment, it is because the model overfits to the target input and therefore produces results regardless of the text prompt. This is particularly problematic when the provided subject images have limited visual diversity.

[0097] Figure 9 provides a chart illustrating the exemplary performance of at least one embodiment described herein. Specifically, Figure 9 shows the alignment metrics (DINO and CLIP-T) of the sample subject in zero-shot and fine-tuning setups.

[0098] Figure 10 provides a chart illustrating exemplary performance of at least one embodiment described herein. Specifically, Figure 10 shows an ablation study relating to an embodiment described herein. The ablation study was performed using 250K subject representation learning steps. Figure 10 shows the zero-shot evaluation results. The findings are as follows: First, it is important to perform multimodal representation learning to bridge the representation gap between subject embeddings and text prompt embeddings. Second, freezing the text encoder of the diffusion model worsens the interaction between subject embeddings and text embeddings. This leads to generation that copies the subject input and disregards the text prompt. Despite yielding a higher subject alignment score, it does not allow for text control and misrepresents the task of text-to-image generation. Third, providing subject text to the multimodal encoder helps inject a class-specific visual prior distribution, thereby leading to a moderate improvement in the metric. Fourth, pre-training with random subject embedding dropping helps to better preserve the generation capability of the diffusion model and therefore benefits the results.

[0099] Figure 11 provides a chart illustrating exemplary performance of at least one embodiment described herein. Specifically, Figure 11 shows DINO and CLIP-T scores for a varying number of pre-training steps. This chart shows that both image-text alignment and subject alignment improve with increasing pre-training steps for subject representation learning.

[0100] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and the claims. In some examples, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0101] This description provides specific details illustrating several embodiments consistent with the present disclosure. Numerous specific details are provided to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be implemented without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not limiting. Those skilled in the art can implement other elements not specifically described herein but within the scope and spirit of the disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment can be incorporated into other embodiments unless otherwise noted, or if one or more features would render the embodiment non-functional.

[0102] While exemplary embodiments have been shown and described, extensive modifications, alterations, and substitutions are intended in the foregoing disclosure, and in some cases, some features of the embodiments may be adopted without corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the following claims, which should be interpreted to broadly coincide with the scope of the embodiments disclosed herein.

Claims

1. A method for generating a subject-driven image, wherein the method is: The steps include receiving a subject image including the subject, a text description of the subject in the subject image, and text prompts related to different depictions of the subject via a data interface. The steps include encoding the subject image into an image feature vector via an image encoder, The steps include encoding the text description into a text feature vector via a text encoder, A multimodal encoder generates a vector representation of the subject based on the image feature vector and the text feature vector, A method comprising the steps of generating an output image based on an input combining a text prompt and a vector representation, using a neural network-based image generation model.

2. The method according to claim 1, further comprising the step of training the multimodal encoder, the text encoder, and the neural network-based image generation model together based on a comparison of the output image with a modified image in which the subject is placed on a background different from the background in the subject image.

3. The step of generating the aforementioned vector representation is based on a plurality of query vectors, The training step includes updating the plurality of query vectors, The method according to claim 2.

4. The step further includes training the neural network-based image generation model based on a comparison between the output image and the subject image, The method according to claim 1.

5. The further step includes keeping the parameters of the text encoder frozen while training the neural network-based image generation model. The method according to claim 4.

6. The multimodal encoder further includes the step of generating multiple vector representations of the subject based on multiple image feature vectors, The aforementioned vector representation is the average of the plurality of vector representations, The method according to claim 1.

7. The process further includes receiving conditioning images via the aforementioned data interface, The step of generating the output image is further based on the conditioning image. The method according to claim 1.

8. A system for generating subject-driven images, wherein the system is A neural network-based image generation model and memory for storing multiple processor-executable instructions, A data interface that receives a subject image including the subject, a text description of the subject in the subject image, and text prompts related to different depictions of the subject, The system includes one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory and perform an operation, wherein the operation is Encoding the subject image into an image feature vector via an image encoder, Encoding the text description into a text feature vector via a text encoder, A multimodal encoder generates a vector representation of the subject based on the image feature vector and the text feature vector, A system comprising generating an output image based on an input combining the text prompt and the vector representation, using the neural network-based image generation model.

9. The aforementioned operation is, The system according to claim 8, further comprising training the multimodal encoder, the text encoder, and the neural network-based image generation model together based on a comparison of the output image with a modified image in which the subject is placed on a background different from the background in the subject image.

10. The generation of the aforementioned vector representation is based on multiple query vectors, The training described above includes updating the plurality of query vectors. The system according to claim 9.

11. The aforementioned operation is, The system according to claim 8, further comprising training the neural network-based image generation model based on a comparison of the output image with the subject image.

12. The aforementioned operation is, The system according to claim 11, further comprising keeping the parameters of the text encoder frozen while training the neural network-based image generation model.

13. The aforementioned operation is, The multimodal encoder further includes the step of generating multiple vector representations of the subject based on multiple image feature vectors, The system according to claim 8, wherein the vector representation is the average of the plurality of vector representations.

14. The aforementioned operation is, The data interface further includes receiving conditioning images, The system according to claim 8, wherein generating the output image is further based on the conditioning image.

15. A non-temporary, machine-readable medium containing a plurality of machine-executable instructions, wherein, when the plurality of machine-executable instructions are executed by one or more processors, the one or more processors The data interface receives a subject image including the subject, a text description of the subject in the subject image, and text prompts related to different depictions of the subject. Encoding the subject image into an image feature vector via an image encoder, Encoding the text description into a text feature vector via a text encoder, A multimodal encoder generates a vector representation of the subject based on the image feature vector and the text feature vector, A non-temporary, machine-readable medium comprising generating an output image based on an input combining a text prompt and a vector representation, using a neural network-based image generation model.

16. The aforementioned operation is, A non-transient machine-readable medium according to claim 15, further comprising training the multimodal encoder, the text encoder, and the neural network-based image generation model together based on a comparison of the output image with a modified image in which the subject is placed on a background different from the background in the subject image.

17. The generation of the aforementioned vector representation is based on multiple query vectors, The training described above includes updating the plurality of query vectors. A non-temporary machine-readable medium according to claim 16.

18. The aforementioned operation is, A non-transient machine-readable medium according to claim 15, further comprising training the neural network-based image generation model based on a comparison of the output image with the subject image.

19. The aforementioned operation is, A non-transient machine-readable medium according to claim 18, further comprising keeping the parameters of the text encoder frozen while training the neural network-based image generation model.

20. The aforementioned operation is, The multimodal encoder further includes generating multiple vector representations of the subject based on multiple image feature vectors, The non-transient machine-readable medium according to claim 15, further comprising the vector representation being the average of the plurality of vector representations.

Images