A personalized text-to-image generation method and system based on diffusion model
By adopting a three-stage training diffusion model method in personalized image generation, combining text inverse mapping and DreamBooth technology, the problem of insufficient diversity and correlation in the generated image in the existing technology is solved, and higher visual fidelity is achieved and editability.
Patent Information
- Application Number
- CN202410690538.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-05-30
AI Technical Summary
Existing text inverse mapping and DreamBooth technologies have their own shortcomings in personalized image generation. Text inverse mapping ignores other elements in text prompts, resulting in insufficient diversity of generated images, while DreamBooth ignores specific concepts learned, resulting in correlation between generated images and target concepts. insufficient.
Using a three-stage training method based on diffusion model, first update the text embedding of the target concept through text inverse mapping and cross-attention regularization, then fine-tune U-Net with DreamBooth to learn the characteristics of the target concept, and in the third stage, self-attention Layer fine-tuning refines the detailed features of the target concept, combined with cross-attention regularization to keep the attention map close to the rough descriptive element.
A better balance between personalization and diversity is achieved, the correlation between generated images and target concepts is improved, and high visual fidelity and editability are maintained.
Smart Images
Figure CN118429485B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and image processing technology, and more specifically, to a method and system for generating personalized text to image based on a diffusion model. Background Art
[0002] Text-to-Image (T2I) personalization refers to the use of text-to-image generation models to create unique and personalized images. Thanks to the rapid progress of text-to-image models, especially diffusion models, personalized concept images can be synthesized with high quality and text can be flexibly controlled. Current personalized image generation technologies are mainly divided into two categories: textual inversion and DreamBooth. Among them, textual inversion usually refers to arranging the characters, words or sentences in a text in reverse order. In the field of personalized image generation, textual inversion personalizes the pre-trained diffusion model by encoding the target concept into the text embedding. DreamBooth technology learns the target concept by fine-tuning the pre-trained diffusion model. Specifically, DreamBooth first uses a diffusion model that has been pre-trained on a large-scale dataset, and then customizes the training of the model by introducing a small number of pictures of specific individuals or objects, so that the model learns to recognize and generate specific features related to these specific pictures. After personalized training, users can guide the model to generate specific images through text prompts.
[0003] The current text inverse mapping technology can effectively learn and generate images related to specific concepts or objects, but when it focuses too much on the learned concepts, it often ignores other important elements in the text prompts, and the attention map is mainly concentrated on the learned concepts, which may cause the generated images to be insufficient in diversity and richness; while DreamBooth can maintain a high diversity of images when generating images related to specific concepts, will not over-focus on specific concepts, and can better handle multiple elements in text prompts. However, it will ignore the learned specific concepts to a certain extent, resulting in insufficient correlation between the generated images and the target concepts. Summary of the invention
[0004] In order to overcome the defect of the above-mentioned prior art that the generated image has insufficient correlation with the target concept due to neglect of the specific concepts learned, the present invention provides a personalized text-to-image generation method and system based on a diffusion model.
[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0006] A personalized text-to-image generation method based on a diffusion model comprises the following steps:
[0007] Acquire a target concept image and a target concept prompt word; the target concept prompt word includes a V word and a rough description word;
[0008] Based on the target concept image and the target concept prompt word, a latent diffusion model (LDM) is trained using text inverse mapping and cross attention regularization, and only the text embedding of the target concept is updated during the training process to obtain a first diffusion model that completes the learning of the target concept attention map, and an updated target concept text embedding;
[0009] The first diffusion model is trained using DreamBooth. During the training process, only the U-Net of the model is fine-tuned, and the learning of the attention map of the target concept is improved using cross-attention regularization to obtain a second diffusion model that completes the feature learning of the target concept.
[0010] Based on the second diffusion model, the detailed features of the target concept are learned by fine-tuning the self-attention layer of U-Net, and the learning of the attention map of the target concept is improved by using cross-attention regularization, so as to obtain a third diffusion model that completes the learning of the target concept details.
[0011] A personalized prompt word is obtained, and a personalized concept image is generated by using the target concept text embedding and the third diffusion model.
[0012] Furthermore, the present invention also proposes a personalized text-to-image generation system based on a diffusion model, and applies the personalized text-to-image generation method based on a diffusion model proposed in the present invention. The system includes:
[0013] An input module, used to obtain target concept images and target concept prompt words, as well as personalized prompt words;
[0014] A first training module is used to train a potential diffusion model based on the target concept image and the target concept prompt word by using text inverse mapping and cross attention regularization, and only update the text embedding of the target concept during the training process to obtain a first diffusion model that completes the learning of the target concept attention map, and an updated target concept text embedding;
[0015] The second training module is used to train the first diffusion model using DreamBooth, and only fine-tune the U-Net of the model during the training process, and use cross-attention regularization to improve the learning of the attention map of the target concept, so as to obtain a second diffusion model that completes the feature learning of the target concept;
[0016] A third training module is used to learn the detailed features of the target concept by fine-tuning the self-attention layer of the U-Net based on the second diffusion model, and continue to use cross-attention regularization to improve the learning of the attention map of the target concept, so as to obtain a third diffusion model that completes the learning of the details of the target concept;
[0017] An image generation module is equipped with a third diffusion model trained by the third training module, and is used to generate a personalized concept image based on the personalized prompt word by using the target concept text embedding and the third diffusion model.
[0018] Furthermore, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the personalized text-to-image generation method based on the diffusion model proposed in the present invention.
[0019] Furthermore, the present invention also proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement the steps of the personalized text-to-image generation method based on a diffusion model proposed by the present invention.
[0020] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: the present invention selects a latent diffusion model that combines the concepts of a diffusion model and a latent space as a personalized text-to-image generation model, and trains the model from three stages to respectively complete the learning of a target concept attention map, target concept features, and target concept details, so as to overcome the defect in DreamBooth that the generated image has insufficient correlation with the target concept due to the neglect of learned specific concepts, and achieve a better balance between personalization and diversity; by introducing cross-attention regularization, the learned attention map is kept close to a coarse description word meta-tag, reducing the impact of the fine-tuning process on the completed learned attention map, and achieving high visual fidelity and editability. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The flowchart of the personalized text-to-image generation method based on the diffusion model of the present invention.
[0022] Figure 2 Set of conceptual images for cat shaped toys target.
[0023] Figure 3 Generate picture sets for personalization based on the concept of a cat-shaped toy target.
[0024] Figure 4 A schematic diagram showing the comparison of personalized generation results when the input is a cat-shaped toy image.
[0025] Figure 5A schematic diagram showing the comparison of personalized generation results when the input is an alarm clock image.
[0026] Figure 6 A schematic diagram showing the comparison of personalized generation results when the input is a teapot image.
[0027] Figure 7 A schematic diagram showing the comparison of personalized generation results when the input is an elephant sculpture image.
[0028] Figure 8 A schematic diagram showing the comparison of personalized generation results when the input is a mug image.
[0029] Fig. 9 Schematic diagram for comparing personalized generation results when the input is a bird-shaped stone statue image.
[0030] Fig.10 This is an architecture diagram of the personalized text-to-image generation system based on the diffusion model of the present invention. DETAILED DESCRIPTION
[0031] The drawings are only used for illustrative purposes and are not to be construed as limiting the present invention. It is understandable to those skilled in the art that some well-known descriptions may be omitted in the drawings.
[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0033] Example 1
[0034] This embodiment proposes a personalized text-to-image generation method based on a diffusion model. Figure 1 , which is a flow chart of the personalized text-to-image generation method based on the diffusion model of this embodiment.
[0035] The personalized text-to-image generation method based on the diffusion model proposed in this embodiment includes the following steps:
[0036] S1, obtaining a target concept image and a target concept prompt word; the target concept prompt word includes a V word and a rough description word;
[0037] S2, based on the target concept image and the target concept prompt word, the potential diffusion model is trained using text inverse mapping and cross attention regularization, and only the text embedding of the target concept is updated during the training process to obtain a first diffusion model that completes the target concept attention map learning, and an updated target concept text embedding;
[0038] S3, based on the target concept image and the target concept prompt word, using DreamBooth to train the first diffusion model, during the training process, only fine-tuning the U-Net of the model, and using cross-attention regularization to improve the learning of the attention map of the target concept, to obtain a second diffusion model that completes the learning of the target concept features;
[0039] S4, based on the target concept image and the target concept prompt word, and based on the second diffusion model, by fine-tuning the self-attention layer of U-Net to learn the detailed features of the target concept, and using cross-attention regularization to improve the learning of the attention map of the target concept, to obtain a third diffusion model that completes the learning of the target concept details;
[0040] S5. Obtain personalized prompt words, and generate personalized concept images using the target concept text embedding and the third diffusion model.
[0041] In the personalized text-to-image generation method proposed in this embodiment, a latent space diffusion model that combines the concepts of diffusion model and latent space is selected as the personalized text-to-image generation model, and the model is trained in three stages.
[0042] Among them, in the first stage of training, only the text embedding of the target concept is updated, and the text inverse mapping method is used to learn a suitable attention map for the target concept to solve the defect of DreamBooth that the generated image has insufficient correlation with the target concept due to the neglect of learning specific concepts; at the same time, cross-attention regularization is introduced to keep the learned attention map close to the coarse description word meta-tag.
[0043] In the second stage of training, the target concept features are learned based on the text embedding learned in the first stage. At the same time, by fixing other parameters of the model, only the U-Net of the model is fine-tuned to learn the specific features of the target concept while avoiding damage to the attention map learned in the first stage of training.
[0044] During the third stage of training, considering that the self-attention layer plays a key role in understanding the spatial context, in order to further refine the details of the target concept, we chose to fine-tune only the self-attention layer of the U-Net in the model, while avoiding damage to the learned attention map.
[0045] As an exemplary illustration, this embodiment uses the Stable Diffusion 2.1 pre-trained model as the initial potential diffusion model. Its basic architecture mainly includes: variational autoencoder (VAE), U-Net architecture, text encoder, diffusion process and noise conditions. Among them, VAE is used to convert the image into a potential representation; the U-Net architecture is a fully convolutional network with symmetrical encoder and decoder parts, which are used to add and remove noise during the diffusion process, and also includes a self-attention layer; the text encoder is used to convert the text description into a latent space representation, which is generally a Transformer-based model to capture the semantic information in the text and convert it into a vector representation; the diffusion process is a process of gradually adding noise to the image (forward diffusion) and recovering the image from the noise (backward diffusion); the noise condition is used to generate images with different styles and qualities.
[0046] As an exemplary explanation, the target concept prompt word in this embodiment is set to "a photo of [V] [rough description word]", which includes V word and rough description word.
[0047] In an optional embodiment, the process of training the potential diffusion model includes the following steps:
[0048] Inputting the target concept image and the target concept prompt word into an initial latent diffusion model, wherein the text embedding corresponding to the V word is set to be trainable, and other parameters of the latent diffusion model are set to be non-trainable;
[0049] Using text inverse mapping, the latent diffusion model is trained for a step based on the training objective function of the latent diffusion model and cross attention regularization to obtain a first diffusion model.
[0050] During the training process of this stage, the text inverse mapping technique is used to effectively learn the attention map, and the cross-attention regularization is introduced to keep the learned attention map close to the coarse description word.
[0051] Further, in an optional embodiment, the process of training the first diffusion model includes the following steps:
[0052] Inputting the target concept image and the target concept prompt word into the first diffusion model, wherein the text embedding corresponding to the V word is set to be non-trainable, the text encoder in the first diffusion model is set to be non-trainable, and the parameters of the U-Net layer in the first diffusion model are set to be trainable;
[0053] Using DreamBooth, the first diffusion model is trained for b steps based on the training objective function of the latent diffusion model and cross-attention regularization to obtain the second diffusion model.
[0054] In the training process of this stage, based on DreamBooth technology, the diffusion model is fine-tuned using the training objective function of the potential diffusion model to learn the target concept features. In order to avoid destroying the learned attention map, the target concept prompt words and the parameters of the text encoder are fixed during the fine-tuning process of DreamBooth to prevent changes to the learned attention map.
[0055] Further, in an optional embodiment, the process of training the second diffusion model includes the following steps:
[0056] Inputting the target concept image and the target concept prompt word into the second diffusion model, wherein the text embedding corresponding to the V word is set to be non-trainable, the text encoder in the first diffusion model is set to be non-trainable, and the parameters of the self-attention layer in the second diffusion model are set to be trainable;
[0057] Using DreamBooth, the second diffusion model is trained for c steps based on the training objective function of the latent diffusion model and cross-attention regularization to obtain the third diffusion model.
[0058] During this phase of training, the diffusion model is fine-tuned using the training objective function of the latent diffusion model, but only the self-attention layer of the diffusion model is updated, which has a significant effect in reproducing the subtle details of the target concept.
[0059] In an optional embodiment, the training objective function of the potential diffusion model is expressed as:
[0060]
[0061] Where z is the potential representation, which is obtained by transforming the input image x through the encoder E; y is the target concept prompt word; ε is the random noise sampled from the standard normal distribution; t is the preset time step; ε θ (·) represents the denoising network; z t is the noisy latent code, and c(y) is the conditional vector.
[0062] In the potential diffusion model of this embodiment, given a specific time step t and conditional vector c(y), the denoising network ε θ The task is to extract the noisy latent code z t Restore the original latent code z0.
[0063] Further, in an optional embodiment, when training the model based on cross-attention regularization, the following steps are included: in each cross-attention layer in the potential diffusion model, the L2 distance between the V word and the attention map of the coarse description word is minimized; its expression is:
[0064]
[0065] Among them, M V represents the attention map of V word, M coarse An attention map that roughly describes a word. represents the L2 norm; λ is used to control the weight of the regularization loss.
[0066] Furthermore, in an optional embodiment, during the training of the potential diffusion model, the number of training steps a is set to 30-50; during the training of the first diffusion model, the number of training steps b is set to 700-800; during the training of the second diffusion model, the number of training steps c is set to 150-200.
[0067] This embodiment optimizes the number of training steps in each training stage to effectively learn the attention map.
[0068] In a specific implementation process, Figure 2 The cat-shaped toy shown in the figure is used as the target concept for personalized text-to-image generation based on the diffusion model. The Stable Diffusion 2.1 pre-trained model is used as the initial latent diffusion model, and the target concept prompt word is set to "a photo of [V] cat toy", where [V] represents a given cat-shaped toy. The target concept image and the target concept prompt word are input into the initial latent diffusion model to perform three-stage training, respectively learning the attention map of the cat-shaped toy, the concept features of the cat-shaped toy, and the concept details of the cat-shaped toy.
[0069] Specifically, during the first stage of training, the text embedding corresponding to the [V] word is set to be trainable, and the other parameters of the potential diffusion model are set to be non-trainable. The loss of formula (1) is used, plus the regularization loss of formula (2). The model is trained for 30 steps to obtain the first diffusion model and the updated target concept text embedding.
[0070] In the second stage of training, the text embedding corresponding to the [V] word is set to non-trainable, the text encoder of the first diffusion model is set to non-trainable, and the parameters of the U-Net in the model are set to trainable. The loss of formula (1) is used, plus the regularization loss of formula (2). The model is trained for 800 steps to obtain the second diffusion model.
[0071] In the third stage of training, the text embedding corresponding to the [V] word is set to non-trainable, the text encoder of the second diffusion model is set to non-trainable, the parameters of the self-attention layer of U-Net are set to trainable, and other parameters are set to non-trainable. Using the loss of formula (1) and the regularization loss of formula (2), the model is trained for 200 steps to obtain the third diffusion model, which is the potential diffusion model that has completed the training.
[0072] Save the text embedding corresponding to the cat-shaped toy and the trained latent diffusion model. The saved text embedding and model can be used to generate new cat-shaped toy images. In the personalized image generation process, obtain the personalized prompt word "Manga drawing of [V] cat toy" and use the saved text embedding and model to perform the process of sampling new images to generate the following Figure 3 The personalized generated image shown.
[0073] In another specific implementation process, the method described in this embodiment is applied to Figures 4 to 9 The image test set shown is used to evaluate the effect of the method. The method described in this embodiment is compared with DreamBooth and NeTI (Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, AH, Chechik, G., & Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv: 2208.01618). Figures 4 to 9 It can be seen that the image generated by the method described in this embodiment can more accurately retain the conceptual features of the input image and can be more consistent with the prompt word.
[0074] Example 2
[0075] This embodiment proposes a personalized text-to-image generation system based on a diffusion model, and applies the personalized text-to-image generation method described in Embodiment 1. Fig.10 , which is an architecture diagram of the personalized text-to-image generation system based on the diffusion model of this embodiment.
[0076] The personalized text-to-image generation system based on the diffusion model proposed in this embodiment includes:
[0077] An input module, used to obtain target concept images and target concept prompt words, as well as personalized prompt words;
[0078] A first training module is used to train a potential diffusion model based on the target concept image and the target concept prompt word by using text inverse mapping and cross attention regularization, and only update the text embedding of the target concept during the training process to obtain a first diffusion model that completes the learning of the target concept attention map, and an updated target concept text embedding;
[0079] The second training module is used to train the first diffusion model using DreamBooth, and only fine-tune the U-Net of the model during the training process, and use cross-attention regularization to improve the learning of the attention map of the target concept, so as to obtain a second diffusion model that completes the feature learning of the target concept;
[0080] A third training module is used to learn the detailed features of the target concept by fine-tuning the self-attention layer of the U-Net based on the second diffusion model, and to improve the learning of the attention map of the target concept by using cross-attention regularization, so as to obtain a third diffusion model that completes the learning of the details of the target concept;
[0081] An image generation module is equipped with a third diffusion model trained by the third training module, and is used to generate a personalized concept image based on the personalized prompt word by using the target concept text embedding and the third diffusion model.
[0082] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.
[0083] Example 3
[0084] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the personalized text-to-image generation method proposed in Embodiment 1.
[0085] Example 4
[0086] This embodiment provides a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement the steps of the personalized text-to-image generation method provided in Embodiment 1.
[0087] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0088] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.
[0089] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the personalized text-to-image generation method described in Example 1.
[0090] The terms in the drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;
[0091] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A personalized text-to-image generation method based on a diffusion model, characterized in that: The following steps are involved: Acquire a target concept image and a target concept prompt word; the target concept prompt word includes a V word and a rough description word; Inputting the target concept image and the target concept prompt word into an initial latent diffusion model, training the latent diffusion model using text inverse mapping and based on the training objective function and cross attention regularization of the latent diffusion model, updating only the text embedding of the target concept during the training process, and obtaining a first diffusion model that completes the learning of the target concept attention map, and an updated target concept text embedding; Inputting the target concept image and the target concept prompt word into the first diffusion model, training the first diffusion model using DreamBooth, fine-tuning only the U-Net of the model during the training process, and improving the learning of the attention map of the target concept based on the training objective function of the potential diffusion model and cross-attention regularization, thereby obtaining a second diffusion model that completes the learning of the target concept features; Inputting the target concept image and the target concept prompt word into the second diffusion model, learning the detailed features of the target concept by fine-tuning the self-attention layer of the U-Net, and improving the learning of the attention map of the target concept based on the training objective function of the potential diffusion model and the cross-attention regularization, so as to obtain a third diffusion model that completes the learning of the target concept details; A personalized prompt word is obtained, and a personalized concept image is generated by using the target concept text embedding and the third diffusion model.
2. The personalized text-to-image generation method based on diffusion model according to claim 1, characterized in that: The process of training the latent diffusion model includes the following steps: Inputting the target concept image and the target concept prompt word into an initial latent diffusion model, wherein the text embedding corresponding to the V word is set to be trainable, and other parameters of the latent diffusion model are set to be non-trainable; Using the text inverse mapping, the latent diffusion model is trained based on the training objective function of the latent diffusion model and cross attention regularization. a Step 1, and the first diffusion model is obtained.
3. The personalized text-to-image generation method based on diffusion model according to claim 1, characterized in that: The process of training the first diffusion model includes the following steps: Inputting the target concept image and the target concept prompt word into the first diffusion model, wherein the text embedding corresponding to the V word is set to be non-trainable, the text encoder in the first diffusion model is set to be non-trainable, and the parameters of the U-Net layer in the first diffusion model are set to be trainable; Using DreamBooth, the first diffusion model is trained based on the training objective function of the latent diffusion model and cross attention regularization b Step 2, and the second diffusion model is obtained.
4. The personalized text-to-image generation method based on diffusion model according to claim 1, characterized in that: The process of training the second diffusion model includes the following steps: Inputting the target concept image and the target concept prompt word into the second diffusion model, wherein the text embedding corresponding to the V word is set to be non-trainable, the text encoder in the first diffusion model is set to be non-trainable, and the parameters of the self-attention layer in the second diffusion model are set to be trainable; Using DreamBooth, the second diffusion model is trained based on the training objective function of the latent diffusion model and cross attention regularization c Step 2, and the third diffusion model is obtained.
5. The personalized text-to-image generation method based on a diffusion model according to any one of claims 1 to 4, characterized in that: The training objective function of the potential diffusion model is expressed as: in, is the potential representation, which is composed of the input image x Via encoder Convert to get; y Provide a cue word for the target concept; is random noise sampled from a standard normal distribution, t is the preset time step; represents the denoising network; is the noise-added latent code, is the conditional vector.
6. The personalized text-to-image generation method based on diffusion model according to claim 5, characterized in that: When training the model based on cross-attention regularization, the following steps are included: in each cross-attention layer in the potential diffusion model, the L2 distance between the V word and the attention map of the coarse description word is minimized; its expression is: in, represents the attention map of V-word, An attention map that roughly describes a word. represents the L2 norm; Used to control the weight of this regularization term.
7. The personalized text-to-image generation method based on a diffusion model according to any one of claims 1 to 4, characterized in that: When the latent diffusion model is trained using text inverse mapping and based on the training objective function and cross-attention regularization of the latent diffusion model, the number of training steps is set to 30~50; when the first diffusion model is trained using DreamBooth, the number of training steps is set to 700~800; when the second diffusion model is trained by fine-tuning the self-attention layer of U-Net to learn the detailed features of the target concept, the number of training steps is set to 150~200.
8. A personalized text-to-image generation system based on a diffusion model, applying the personalized text-to-image generation method based on a diffusion model according to any one of claims 1 to 7, characterized in that: include: An input module, used to obtain target concept images and target concept prompt words, as well as personalized prompt words; A first training module is used to train a potential diffusion model based on the target concept image and the target concept prompt word by using text inverse mapping and cross attention regularization, and only update the text embedding of the target concept during the training process to obtain a first diffusion model that completes the learning of the target concept attention map, and an updated target concept text embedding; The second training module is used to train the first diffusion model using DreamBooth, and only fine-tune the U-Net of the model during the training process, and use cross-attention regularization to improve the learning of the attention map of the target concept, so as to obtain a second diffusion model that completes the feature learning of the target concept; A third training module is used to learn the detailed features of the target concept by fine-tuning the self-attention layer of the U-Net based on the second diffusion model, and to improve the learning of the attention map of the target concept by using cross-attention regularization, so as to obtain a third diffusion model that completes the learning of the details of the target concept; An image generation module is equipped with a third diffusion model trained by the third training module, and is used to generate a personalized concept image based on the personalized prompt word by using the target concept text embedding and the third diffusion model.
9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor performs the steps of the personalized text-to-image generation method based on the diffusion model as described in any one of claims 1 to 7.
10. A storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the personalized text-to-image generation method based on a diffusion model as described in any one of claims 1 to 7 are implemented.