Personalized text-to-image generation method and system based on text embedding learning

Abstract

The invention relates to the technical field of computer vision and deep learning, and provides a personalized text-to-image generation method and system based on text embedding learning. The method comprises the steps that a plurality of target concept images are acquired, and each target concept image is associated with a target concept vocabulary item; constructing a first cue word and a second cue word about the target concept lexical item, and respectively processing the first cue word and the second cue word through a text encoder to obtain a first output embedding and a second output embedding; inputting the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein optimization training is performed on the diffusion model based on minimizing the similarity between the first output embedding and the second output embedding and minimizing the attention map similarity between the first cue word and the second cue word; and obtaining a cue word containing the target concept, embedding and inputting the cue word and the target concept text into the trained diffusion model, and generating a personalized image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technologies, and more specifically, to a personalized text-to-image generation method and system based on text embedding learning. Background Art

[0002] Text-to-image (T2I) personalization refers to the use of text-to-image generative models to create unique, personalized images, primarily by adapting a pre-trained diffusion model to generate images of new scenes or styles that incorporate user-provided concepts. By providing a few image examples of a new concept, users can leverage personalization techniques to create unique images based on new prompts that incorporate that concept. These techniques generally fall into two categories: the first involves reverse engineering the new concept into text embeddings, while the second involves fine-tuning the diffusion model to learn the new concept. The goal of personalization techniques is to produce high-quality images that accurately represent the user-provided concept while maintaining identity preservation and text consistency.

[0003] However, balancing the trade-off between identity preservation and textual consistency in the personalization process of diffusion models remains a core challenge. Many methods have explored this trade-off by analyzing the embeddings or attention maps of new concepts. For example, latent diffusion models are trained using textual inversion and cross-attention regularization to learn target concept attention maps, target concept features, and target concept details. This overcomes the flaw in DreamBooth, which neglects learned specific concepts, resulting in insufficient relevance between generated images and target concepts. However, when optimizing the input embeddings of new concepts using techniques such as textual inversion, the output embeddings of these contextual words can change significantly. This change is because the overfitted embeddings of the new concepts interact unfavorably with the contextual words within the CLIP text encoder, resulting in incorrect output embeddings. This mismatch leads to improper distribution of contextual words in the cross-attention map, which in turn causes the generated images to fail to accurately reflect the content of the new concepts or to contain errors and missing details. Summary of the Invention

[0004] In order to overcome the defect of improper distribution of contextual vocabulary in the cross-attention graph described in the above-mentioned prior art, which leads to the inability of the generated image to accurately reflect the content of new concepts, the present invention provides a personalized text-to-image generation method and system based on text embedding learning.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A personalized text-to-image generation method based on text embedding learning includes the following steps:

[0007] Acquire a plurality of target concept images, each of the target concept images being associated with a target concept vocabulary item;

[0008] Constructing a first prompt word and a second prompt word for the target concept vocabulary item, and passing the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding respectively;

[0009] Inputting the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity between the attention graphs of the first prompt word and the second prompt word;

[0010] Obtain a prompt word containing a target concept, embed the prompt word and the target concept text into the trained diffusion model, and generate a personalized image.

[0011] Furthermore, the present invention also proposes a personalized text-to-image generation system based on text embedding learning, which applies the personalized text-to-image generation method proposed in the present invention. The system includes:

[0012] An acquisition module, configured to acquire a plurality of target concept images, each of which is associated with a target concept vocabulary item;

[0013] An encoding module, configured to construct a first prompt word and a second prompt word for the target concept vocabulary item, and pass the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding respectively;

[0014] an optimization module, configured to input the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity between the attention graphs of the first prompt word and the second prompt word;

[0015] The generation module is used to embed the input prompt words containing the target concept and the optimized target concept text into the trained diffusion model to generate a personalized image.

[0016] Furthermore, the present invention also proposes a device comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor performs the steps of the personalized text-to-image generation method as described in the present invention.

[0017] Furthermore, the present invention also proposes a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the personalized text-to-image generation method as described in the present invention.

[0018] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: the present invention aims to use several target concept images to learn an input embedding for a new concept so that it is compatible with and seamlessly integrated with existing tokens; the present invention improves the text embedding learning of new concepts by adjusting the output embedding of context words, imposes similarity constraints on the output embedding of context words, and introduces an additional regularization term to impose similarity constraints by minimizing the similarity of attention maps to enhance identity preservation and text consistency, avoid improper distribution of context words in cross-attention maps, and thus ensure that the generated personalized images can accurately reflect the content of the new concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is a flowchart of a personalized text-to-image generation method according to an embodiment of the present invention.

[0020] Figure 2 The figure is a schematic diagram showing a comparison of personalized image generation results according to an embodiment of the present invention.

[0021] Figure 3 FIG. 1 is another schematic diagram showing a comparison of personalized image generation results according to an embodiment of the present invention.

[0022] Figure 4 FIG. 1 is an architecture diagram of a personalized text-to-image generation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0024] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0026] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] This embodiment proposes a personalized text-to-image generation method based on text embedding learning, such as Figure 1 FIG. 1 is a flow chart of the personalized text-to-image generation method of this embodiment.

[0029] The personalized text-to-image generation method based on text embedding learning proposed in this embodiment includes the following steps:

[0030] S1. Acquire a plurality of target concept images, each of which is associated with a target concept vocabulary item (token);

[0031] S2. Construct a first prompt word and a second prompt word for the target concept vocabulary item, and pass the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding respectively;

[0032] S3. Inputting the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity between the attention graphs of the first prompt word and the second prompt word;

[0033] S4. Obtain prompt words containing the target concept, embed the prompt words containing the target concept and the target concept text into the trained diffusion model, and generate a personalized image.

[0034] This embodiment aims to learn an input embedding for a new concept using several example images, making it compatible with existing tokens and seamlessly integrated. This embodiment improves the text embedding learning of new concepts by adjusting the output embeddings of the contextual vocabulary surrounding the new concept, because the quality of these output embeddings depends largely on their interaction with the properly embedded new concept.

[0035] Specifically, this embodiment constructs a second prompt word as a reference prompt for a first prompt word containing a new concept. Then, during the optimization training process, a similarity constraint is imposed on the output embeddings of the context vocabulary between the first prompt word and the second prompt word. That is, identity preservation and text consistency are enhanced by minimizing the similarity between the first output embedding and the second output embedding.

[0036] Furthermore, considering that improper output embedding of contextual tags may lead to errors in the attention map in the cross-attention layer, this embodiment introduces an additional regularization term to impose a similarity constraint by minimizing the similarity of the attention maps between the first prompt word and the second prompt word, thereby avoiding improper distribution of contextual vocabulary in the cross-attention map, thereby ensuring that the generated personalized image can accurately reflect the content of the new concept.

[0037] In an optional embodiment, in step S2, the step of constructing the first prompt word and the second prompt word about the target concept vocabulary item includes: randomly selecting a prompt template from a preset regularized prompt set, filling the prompt template with the target concept vocabulary item and the supercategory vocabulary item of the target concept, respectively, to obtain the first prompt word and the second prompt word.

[0038] It should be noted that this embodiment constructs a regularized prompt set designed to cover a wide range of prompt words. For each randomly selected regularized prompt, a pair of prompts is generated: one is a training prompt containing the new concept, and the other is a reference prompt that replaces the new concept vocabulary with vocabulary from the supercategory, resulting in the first prompt word and the second prompt word.

[0039] Furthermore, this embodiment avoids directly imposing constraints between new concepts and their superclass words, which may lead to a significant decrease in identity preservation. Usually, there are large differences between new concepts and their superclass words.

[0040] For example, assuming that the target concept vocabulary item [S*] is "cat", a prompt template "{} in the jungle" is randomly selected from the preset regularized prompt set and filled with the new concept tag and its supercategory tag respectively, thereby generating the first prompt word "[cat] in the jungle" and the second prompt word "[dog] in the jungle".

[0041] Unlike fixed training prompts, such as a photo of [S*], which must be paired with a training image, this embodiment allows the use of arbitrary prompts because this regularization can be applied to the output embeddings and attention maps without generating images. This embodiment constructs first and second prompt words to impose similarity constraints during model optimization training, preventing overfitting of input embeddings for new concepts from significantly changing the output embeddings of contextual tags, thereby avoiding incorrect allocation of attention maps in the cross-attention layer.

[0042] In an optional embodiment, in step S3, the target concept image is input into the diffusion model for optimization training, including the following steps:

[0043] The constraint on the output embedding is constructed by minimizing the average cosine distance between the first output embedding and the second output embedding; its expression is:

[0044]

[0045] Where n is the prompt length, k is the position index of the target concept vocabulary item and the supercategory vocabulary item; E(v i ) represents the token sequence v of the first prompt word i The first output embedding obtained after inputting the text encoder;

[0046] E(v′ i ) represents the token sequence v′ of the second prompt word i The second output embedding obtained after inputting the text encoder.

[0047] Furthermore, when the target concept image is input into the diffusion model for optimization training in step S3, the following steps are also included:

[0048] Input the first output embedding and the second output embedding into a U-Net containing N cross attention layers respectively to obtain a first attention map and a second attention map;

[0049] By minimizing the squared difference between the means of the attention maps and minimizing the similarity of the attention maps between the first prompt word and the second prompt word, an attention map regularization constraint is constructed; its expression is:

[0050]

[0051] in, represents the first attention map output by 1 to N cross-attention layers, represents the second attention map output by 1 to N cross-attention layers; μ(·) represents the mean across all attention maps.

[0052] Further optionally, the U-Net includes 16 different cross-attention layers; the first output embedding and the second output embedding are respectively input into the U-Net to obtain 16 first attention maps and 16 second attention maps.

[0053] For example, the first output is embedded in W(v i ) and the second output embedding E(v′) i ) are input into 16 different cross attention layers in U-Net, and the attention maps are obtained respectively. and

[0054] This embodiment takes into account that improper output embedding of contextual tags will cause errors in the attention map in the cross-attention layer. An additional regularization term is introduced to impose a similarity constraint between the attention maps corresponding to the first output embedding and the second output embedding to enhance identity preservation and text consistency.

[0055] Furthermore, in an optional embodiment, the objective function of the diffusion model is expressed as:

[0056]

[0057] Among them, α and β are weight parameters used to balance the importance of different loss items; v * The target concept text embedding obtained after each round of optimization training; L diffusion is the diffusion model objective function; z is the potential representation, which is obtained by transforming the input image x through the encoder ε(·); y is the prompt word; ∈ is the random noise sampled from the standard normal distribution; t is the time step; ∈ θ (·) represents the denoising network, z t is the noise latent variable, c θ (y) represents condition information.

[0058] In this embodiment, during the training process of the diffusion model, based on the output embedding similarity constraint and the attention graph similarity constraint, the objective function of the diffusion model is minimized at a given specific time step t and the condition vector c. θ (y) case, the potential code z from the noise t Recover the error of the original latent code z. The purpose of this step is to enable the diffusion model to learn new concepts using the above objective function.

[0059] For example, this embodiment performs optimization training based on textual inversion (TI). During the optimization training process, only the text embedding v of the new token is updated. * (embedding), all other parts of the model remain unchanged, including the pre-trained language model (such as CLIP's text encoder) will not be updated. At the same time, the optimization process satisfies the output embedding similarity constraint L emb and attention graph similarity constraint L attn , ensuring that the input embedding of new concepts is learned correctly, enhancing identity preservation and text consistency.

[0060] Furthermore, in an optional embodiment, the step of inputting the target concept image into the diffusion model for optimization training further includes the following steps:

[0061] In each step of the optimization process, the norm of the target concept text embedding optimized by the diffusion model is set to the norm before optimization; its expression is:

[0062]

[0063] Among them, v * represents the target concept text embedding obtained after each round of optimization training; s represents the s-th optimization step.

[0064] It should be noted that during the optimization process, the scale of text embeddings of new concepts tends to become very large, which will cause the text alignment performance to degrade significantly.

[0065] This embodiment rescales the norm of the text embedding during the optimization process. Specifically, in each step of the optimization process, the updated embedding v * The norm is set to the embedding v of the previous round of optimization * The norm of .

[0066] This embodiment applies the above rescaling strategy only in the middle stage of the optimization to avoid information loss caused by rescaling the embedding at the beginning or end stage, thereby avoiding degradation of the identity preservation effect.

[0067] After completing the training and learning of the target concept, the user can provide a new prompt containing the new concept token S*, and the model will use the optimized embedding v * Generate the corresponding image.

[0068] For example, Figure 2 、 3 The figure shows personalized images generated using the personalized text-to-image generation method proposed in this embodiment, as well as personalized images generated using existing baseline methods, including TI+DB, NeTI, ViCo, and OFT. Compared to the topic selection and baseline methods, the personalized text-to-image generation method of this embodiment demonstrates superior performance in text alignment and identity preservation. This embodiment can more accurately understand textual cues while maintaining the original concept identity, thereby generating higher-quality, expected images.

[0069] Example 2

[0070] This embodiment applies the personalized text-to-image generation method proposed in Example 1 and proposes a personalized text-to-image generation system based on text embedding learning. Figure 4 , which is an architecture diagram of the personalized text-to-image generation system of this embodiment.

[0071] The personalized text-to-image generation system proposed in this embodiment includes:

[0072] An acquisition module, configured to acquire a plurality of target concept images, each of which is associated with a target concept vocabulary item;

[0073] An encoding module, configured to construct a first prompt word and a second prompt word for the target concept vocabulary item, and pass the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding respectively;

[0074] an optimization module, configured to input the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity between the attention graphs of the first prompt word and the second prompt word;

[0075] The generation module is used to embed the input prompt words containing the target concept and the optimized target concept text into the trained diffusion model to generate a personalized image.

[0076] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described again here.

[0077] Example 3

[0078] This embodiment proposes a computer device including a memory and a processor, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs the steps of the personalized text-to-image generation method proposed in Example 1.

[0079] Example 4

[0080] This embodiment provides a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the personalized text-to-image generation method provided in Embodiment 1.

[0081] Exemplarily, the storage medium includes but is not limited to a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0082] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.

[0083] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is configured to execute all or part of the steps of the personalized text-to-image generation method described in Example 1.

[0084] Each embodiment of the present invention is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely exemplary. The modules described as separate components may or may not be physically separated. When implementing the scheme of the present invention, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment.

[0085] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A personalized text-to-image generation method based on text embedding learning, characterized in that: The following steps are involved: Acquire a plurality of target concept images, each of the target concept images being associated with a target concept vocabulary item; Constructing a first prompt word and a second prompt word about the target concept vocabulary item, and respectively passing the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding; Inputting the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity of the attention graph between the first prompt word and the second prompt word; Obtain a prompt word containing a target concept, embed the prompt word and the target concept text into the trained diffusion model, and generate a personalized image.

2. The personalized text-to-image generation method according to claim 1, characterized in that: The construction of the first prompt word and the second prompt word about the target concept vocabulary item includes the following steps: randomly selecting a prompt template from a preset regularized prompt set, filling the prompt template with the target concept vocabulary item and the super-category vocabulary item of the target concept respectively, and obtaining the first prompt word and the second prompt word.

3. The personalized text-to-image generation method according to claim 2, characterized in that: The step of inputting the target concept image into a diffusion model for optimization training comprises the following steps: The constraint on the output embedding is constructed by minimizing the average cosine distance between the first output embedding and the second output embedding; its expression is: Where n is the prompt length, k is the position index of the target concept vocabulary item and the super-category vocabulary item; E(v i ) represents the token sequence v of the first prompt word i The first output embedding obtained after inputting the text encoder; E(v′ i ) represents the token sequence v′ of the second prompt word i The second output embedding after feeding the text encoder.

4. The personalized text-to-image generation method according to claim 3, characterized in that: The step of inputting the target concept image into the diffusion model for optimization training also includes the following steps: Input the first output embedding and the second output embedding into a U-Net including N cross attention layers respectively to obtain a first attention map and a second attention map; By minimizing the squared difference between the means of the attention maps and minimizing the similarity of the attention maps between the first prompt word and the second prompt word, an attention map regularization constraint is constructed; its expression is: in, represents the first attention map output by 1 to N cross-attention layers, represents the second attention map output by 1 to N cross-attention layers; μ(·) represents the mean across all attention maps.

5. The personalized text-to-image generation method according to claim 4, characterized in that: The U-Net contains 16 different cross-attention layers.

6. The personalized text-to-image generation method according to claim 4, characterized in that: The objective function of the diffusion model is expressed as: Among them, α and β are weight parameters used to balance the importance of different loss terms; v * The target concept text embedding obtained after each round of optimization training; L diffusion is the diffusion model objective function; z is the potential representation, which is obtained by transforming the input image x through the encoder ε(·); y is the prompt word; ∈ is the random noise sampled from the standard normal distribution; t is the time step; ∈ θ (·) represents the denoising network, z t is the noise latent variable, c θ (y) represents condition information.

7. The personalized text-to-image generation method according to any one of claims 1 to 6, characterized in that: The step of inputting the target concept image into the diffusion model for optimization training also includes the following steps: In each step of the optimization process, the norm of the target concept text embedding optimized by the diffusion model is set to the norm before optimization; its expression is: Among them, v * represents the target concept text embedding obtained after each round of optimization training; s represents the sth optimization step.

8. A personalized text-to-image generation system based on text embedding learning, applying the personalized text-to-image generation method according to any one of claims 1 to 7, characterized in that: include: A collection module, used for acquiring a plurality of target concept images, each of which is associated with a target concept vocabulary item; An encoding module, configured to construct a first prompt word and a second prompt word about the target concept vocabulary item, and pass the first prompt word and the second prompt word through a text encoder to obtain a first output embedding and a second output embedding respectively; an optimization module, configured to input the target concept image into a diffusion model for optimization training to obtain an optimized target concept text embedding; wherein the diffusion model is optimized and trained based on minimizing the similarity between the first output embedding and the second output embedding, and minimizing the similarity between the first prompt word and the second prompt word in the attention graph; The generation module is used to embed the input prompt words containing the target concept and the optimized target concept text into the trained diffusion model to generate a personalized image.

9. A device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor is caused to perform the steps of the personalized text-to-image generation method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the personalized text-to-image generation method according to any one of claims 1 to 7 are implemented.

Images