Universal style figure image customization generation method based on diffusion model

Abstract

The invention discloses a universal style figure image customization generation method based on a diffusion model. The generation process of the method is divided into two continuous stages: semantic scene construction and concept feature injection; in the first stage, features of potential images can be extracted from images of all concepts at the same time by referring to perceptual self-attention (RSA), so that coarse-grained overall semantic understanding is obtained, and an initial semantic layout is established; and in the second stage, firstly, generating positions of all concepts in the potential image are accurately positioned based on an attention semantic segmentation method, then, the potential image is divided into a plurality of semantic groups by region grouping mixed attention RBA, and each group queries fine-grained features from the corresponding reference concept so as to ensure accurate attribute alignment and feature injection. The invention also adopts a weight mask strategy to ensure that the model pays more attention to the concept features of reference. The method has remarkable advantages in the aspects of human-centered image generation and multi-concept portrait customization.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal image customization and generation, and in particular to a method for customizing and generating a character image of a general style based on a diffusion model. Background Art

[0002] The goal of personalized character image generation is to generate images of a specific individual in a specified scene, style, and action. Current methods are mainly divided into two categories: fine-tuning-based personalization and zero-shot personalization.

[0003] Fine-tuning-based customization. Fine-tuning-based methods require additional fine-tuning of the model based on reference images of a specific subject at test time. The pioneering work Dreambooth[1] fine-tunes a pre-trained diffusion model using a large number of reference images to bind a unique identifier to the identity features of a given subject. The contemporary work TextualInversion[2] converts subject images into simple learnable text embeddings to encode the subject's identity. Subsequently, subsequent works such as NeTI[3] and XTI[4] introduced implicit time-aware representations and layer-by-layer learnable embeddings to achieve better performance. In addition, Tuning-Encoder[5] generates a set of initial latent codes by using a pre-trained encoder and optimizes these codes through fewer fine-tuning iterations to better preserve the subject's identity. Although these methods are effective, they are inefficient at test time due to the large amount of time and computational resources required for fine-tuning.

[0004] Zero-shot personalization. Zero-shot methods attempt to use a single image to generate customized images through a single forward denoising process, significantly accelerating the personalization process. For example, ELITE[6] and InstantBooth[7] achieve this goal by using a global mapping network to encode the reference image into word embeddings and injecting local features of the reference image into a cross-attention layer through a local mapping network. Fastcomposer[8] and PhotoMaker[9] obtain identity-centric embedding features by fine-tuning the image encoder and fusing the embeddings of category words and images. Face-Diffuser

[10] reveals the problems of training imbalance and quality compromise in these methods and addresses them by proposing a novel collaborative generation method. InstantID

[11] proposes an IdentityNet that guides generation by combining facial features and pose conditions with textual prompts. FlashFace

[12] encodes the reference image into a series of feature maps to preserve more details. Although these methods have achieved impressive results, they only focus on single-concept portrait personalization and cannot handle complex scenes involving multiple concepts.

[0005] In addition, recent studies have focused on multi-concept personalized customization generation. Custom Diffusion

[13] achieves multi-concept customized generation through closed-form constrained optimization. Perfusion

[14] proposes a dynamic rank-1 update strategy to ensure the visual fidelity of the concept. FreeCustom

[15] proposes a multi-reference self-attention mechanism that allows latent images to interact with input concepts. ClassDiffusion

[16] explicitly adjusts the concept space by utilizing semantic preservation loss to maintain semantic consistency when learning new concepts. Although these methods are effective in dealing with general objects with coarse-grained textures, they still have shortcomings when dealing with portrait customization due to the subtle semantics of portraits and the higher details and fidelity requirements for facial identity.

[0006] As mentioned above, existing methods for personalized generation of character images usually require tedious training: either fine-tuning the pre-trained model through a small number of reference images, or retraining the pre-trained model on a large-scale dataset. However, these training-based methods are prone to overfitting problems, making them unable to generate personalized characters of different styles. In addition, existing methods for personalized generation of character images are unable to perform multi-concept customized generation of characters, and existing multi-concept personalized customization generation methods are only targeted at general objects with coarse-grained textures, and are still insufficient for customized generation of portraits. Summary of the invention

[0007] In order to solve the above problems of the prior art, the present invention proposes a method for customizing and generating images of characters of a universal style, which can perform high-fidelity custom generation of single-concept or multi-concept character images for characters of any style without the need for additional training.

[0008] The present invention introduces a coarse-to-fine forward control generation process, which includes two consecutive stages: semantic layout construction and concept feature injection. During the generation process, the technical solution gradually extracts visual features from the input reference concept image to guide the generation, thereby realizing high-fidelity personalized generation of character images; its core lies in the Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms proposed by the present invention. The technical solution of the present invention is specifically introduced as follows.

[0009] The present invention provides a method for customizing and generating a general style character image based on a diffusion model. The method realizes the generation of a single-concept or multi-concept custom image of a character of any style based on a dual-path guidance of a reference path and a custom generation path. In the custom generation path, a forward control generation process from coarse to fine is realized through a reference-perceived self-attention RSA and a regional grouping hybrid attention RBA mechanism. The generation process is divided into the following two stages:

[0010] Phase 1: Semantic scene construction

[0011] RSA enables the latent image to query features from all the concept’s reference images simultaneously, adaptively integrating coarse-grained semantic information to extract comprehensive semantic understanding from the reference images in order to establish an initial semantic layout;

[0012] The second stage of concept feature injection

[0013] First, the latent semantic map of each concept is calculated based on the attention map to accurately locate the generation positions of all concepts in the latent image; then, RBA divides the latent image into multiple semantic groups, and enables each group to query fine-grained features from its corresponding reference concept image to ensure accurate attribute alignment and feature injection; finally, the features of each reference concept image are accurately injected into the corresponding position to ensure that the generated concept is highly similar to the reference image.

[0014] In the present invention, in the reference path, the pre-trained diffusion model Stable Diffusion is used as the basic model, and the U-Net∈ θ As a denoiser, visual features are gradually extracted from the input reference concept image to guide generation; in the customized generation path, the self-attention layer in the first stage U-Net is replaced by a reference-aware self-attention RSA module, and in the second stage it is replaced by a region-grouped hybrid attention RBA module.

[0015] In the present invention, in the reference path, for each reference concept image, the forward diffusion process is first applied to calculate the noise latent variable z of the reference image T '; At each time step t, the noise latent variable z T ' and the corresponding text prompt are input to the denoiser; in the self-attention layer l and time step t, the key feature K of the i-th concept is extracted i，l，t Sum value feature V i，l，t , to guide image generation for a custom build path.

[0016] In the present invention, in the first stage, in each self-attention layer l and time step t, the latent variable z t The query, key, and value features are Q l,t , K l,t and Vl,t , l and t are omitted for brevity, and the key and value features are concatenated with the N key features and value features obtained from the reference path, respectively, to obtain and Introduce the segmentation masks of the reference concept and concatenate them with the all-1 matrix to obtain M = [1, M 1 , M 2 , ..., M N ], thereby correcting the model's attention; finally, RSA is expressed as:

[0017]

[0018] Where ⊙ represents the Hadamard product, Q is the query feature obtained from the latent image features through different linear mappings, and d represents the dimension of Q.

[0019] In the present invention, in the second stage, the steps of calculating the potential semantic graph of each concept based on the attention graph are as follows: 1) Generate the attention graph

[0020] In the attention layer l and time step t, the spatial features of the latent image are Or linearly project the text embedding e to calculate the self-attention map and cross attention map

[0021]

[0022] Among them, Q * (·) and K * (·) is a linear projection of dimension d;

[0023] 2) Semantic Segmentation Based on Cross-Attention

[0024] The latent variable z t Split into [m 1 , m 2 , ..., m K ] The set of masked regions, where K represents the number of text tags, m i ∈{0, 1} represents the text tag P i The potential semantic regions are as follows:

[0025] First, all the cross attention maps Upsample to the same size, then average and renormalize them in the spatial dimension to obtain the final attention map C t , which estimates the assignment of image patch s to text label P i Finally, an argmax operation is applied over the text token dimension to determine the activation for each image patch:

[0026]

[0027] By combining the image patch set {s:i s =i} is set to 1, and the other elements are set to 0 to calculate m i ;

[0028] The number of semantic masks of reference concepts required for feature injection is N, so masks of concept-specific tags are retained and the rest of the masks are merged into a new mask m 0 To represent the background area, we finally get the latent semantic mask M = [m 0 , m 1 , m 2 , ..., m N ];

[0029] 3) Segmentation completion based on self-attention

[0030] Refine the criss-cross attention map by multiplying the self-attention map with the corresponding criss-cross attention map:

[0031]

[0032] Then, all the cross attention maps obtained in the spatial dimension The average and renormalization are performed, and the result is input into formula (1) to calculate a more refined latent semantic mask.

[0033] In the present invention, in the second stage, RBA divides the potential image into multiple conceptual semantic groups, and enables each group to query fine-grained features from its corresponding reference image and the specific method of feature injection is as follows:

[0034] At each self-attention layer l and time step t, the latent variable z is sampled from a Gaussian distribution t The query, key, and value features are Q, K, and V, respectively. For brevity, l and t are omitted here. Next, RBA divides Q into multiple groups [q 0 ,q 1 ,q 2 , ..., q N ].

[0035] For each group q i , where i>0, concatenate K and V with the key and value features in their corresponding reference paths to obtain and For the feature group q corresponding to the background 0 ,have and

[0036] Then, the attention output x of each region is calculated separately i as follows:

[0037]

[0038] in

[0039] Finally, each output pixel group is placed at its corresponding position and mixed based on the latent semantic mask M to obtain the final output.

[0040] In the present invention, in the first and second stages, RSA and RBA are segmentation masks M of each reference concept image. i Introducing the scaling factor w i , to enhance the model’s attention to the features of each reference concept image.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention discloses the first high-fidelity personalized image generation technology solution for universal style characters, which does not require additional training and supports single / multi-concept customization; the coarse-to-fine two-stage control generation process proposed in the present invention, combined with our reference-aware self-attention and region-grouping hybrid attention mechanism, ensures accurate attribute alignment and feature injection for each generated concept; the present invention proposes a reference-aware self-attention mechanism, which enables the generated latent image to query coarse-grained features from the reference images of all concepts at the same time, extracts comprehensive semantic understanding, and promotes the initial semantic layout construction of the generated image; the present invention proposes an attention-based semantic segmentation method in a diffusion model, which can accurately locate the potential generation position of each generated concept; the present invention proposes a region-grouping hybrid attention mechanism, which accurately injects the fine-grained features of the reference concept image into its corresponding potential generation area on the generated image, ensuring that the generated concept and the reference image are highly similar; a large number of experiments show that the method of the present invention has significant advantages in human-centric image generation and multi-concept portrait customization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 : Model architecture diagram of the present invention.

[0044] Figure 2 : Segmentation examples of latent semantic masks.

[0045] Figure 3 : Explore the effect of weighted masking strategy under different weight settings.

[0046] Figure 4 :Comparison with existing technical solutions Figure 1 .

[0047] Figure 5 :Comparison with existing technical solutions Figure 2 .

[0048] Figure 6 :The effect of the present invention is shown Figure 1 .

[0049] Figure 7 :The effect of the present invention is shown Figure 2 .

[0050] Figure 8 :The effect of the present invention is shown Figure 3 .

[0051] Fig. 9 : Visual analysis of attention maps in RSA and RBA. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0053] The present invention aims to propose a method for generating customized images of characters of a universal style, which is specifically as follows.

[0054] 1. Method Overview

[0055] The goal of the present invention is to achieve high-fidelity character personalized generation, and to achieve single-concept or multi-concept customized image generation of multi-style characters without training. To this end, the present invention proposes a coarse-to-fine controlled generation process, combining the reference-perceived self-attention (RSA) and regional grouping hybrid attention (RBA) mechanisms proposed in the present invention. Specifically, the sampling process of the present technical solution is divided into two consecutive stages, with a total of T denoising steps.

[0056] In the first stage, the present invention uses RSA to gradually extract the overall semantic understanding from the reference images of all concepts to promote the initial semantic scene construction. This process gradually extracts a comprehensive semantic understanding from the reference images. Once the overall scene is initially established, it enters the second stage to refine the features of the generated concepts:

[0057] In the second stage, the present invention first generates a latent semantic map in each step to accurately locate the generation position of all concepts. Then, RBA accurately injects the features of each reference concept image into the corresponding position to ensure that the generated concept is highly similar to the reference image. In each step, the present invention proposes a semantic segmentation method that does not require training to identify the potential generation area of ​​all concepts. Specifically, the present invention first normalizes the cross-attention map obtained in the generation process, assigns each image block to its corresponding concept semantics, and uses the self-attention map to improve and supplement the semantic area. Subsequently, according to the obtained latent semantic map, RBA divides the latent image into multiple semantic groups, and enables each group to query fine-grained features from its respective reference concept image. This process effectively ensures the precise alignment of attributes of each concept and meticulous feature injection. In order to guide the model to pay more attention to the given concept features and eliminate irrelevant information in the reference image, the present invention implements a weighted mask strategy in the generation process.

[0058] 1. Self-attention layer in the original diffusion model

[0059] The present invention adopts the pre-trained diffusion model Stable Diffusion (SD) v2.1

[17] as the basic model of the present invention, and uses the U-Net∈ θ As a denoiser. The original SD U-Net consists of 16 layers, each of which includes a residual block, self-attention module and cross-attention module. The self-attention (SA) layer in U-Net is crucial for the establishment of image layout and refinement of features, and its mathematical expression is:

[0060]

[0061] Among them, Q, K, and V are query, key, and value features obtained from the latent image features through different linear mappings, with dimension d.

[0062] 2. Customized generation of dual boot

[0063] like Figure 1 As shown, the overall generation process includes two paths: a reference path and a customized generation path. The following will describe these two paths in detail.

[0064] Reference path: For each reference concept image in the reference path, we first apply the forward diffusion process

[18] to calculate the noise latent variable z of the reference image. T '. In each time step t, the present invention converts z T ' and the corresponding text prompt is input to U-Net∈ θ In the self-attention layer l and time step t, the present invention extracts the key and value features K of the i-th concepti，l，t and V i，l，t , to guide image generation for a custom build path.

[0065] Custom generation path: This path first samples the latent variable z from the Gaussian distribution N(0,I) T Next, the present invention modifies the original self-attention module of SD U-Net and expands it into RSA / RBA of the present invention. In each step t, the present invention converts the latent variable z t And the target text prompt P is input into the modified U-Net In the example, K is obtained from the reference path. i，l，t and V i，l，t The features of each reference concept image are extracted so that the generated concept is highly similar to its reference image. The final denoising result of this path is the final generated personalized image.

[0066] 2. Control Generation Process from Coarse to Fine

[0067] In the customized generation path, the present invention proposes a coarse-to-fine controlled generation process to generate images: coarse-grained semantic scene construction is performed in the first αT steps, and fine-grained concept feature injection is performed in the subsequent T(1-α) steps. In the latter stage, the present invention proposes an attention-based semantic segmentation method to lay the foundation for accurate feature injection. The specific details of each stage will be introduced in detail below.

[0068] 1. Semantic scene construction

[0069] Reference-aware self-attention (RSA): RSA enables the latent image to query the reference image features of all concepts simultaneously to integrate coarse-grained semantic information for initial semantic scene construction. Specifically, in each self-attention layer l and time step t, from z t The query, key, and value features obtained are Q l,t , K l,t and V l,t The present invention converts K l,t and V l,t

[0070] and concatenate with the N key and value features obtained from the reference path (N is the number of reference concept images,) to obtain and For brevity, l, t are omitted here. However, the goal of the present invention is to query features only from the concept region in the reference image, because irrelevant background information in the reference image may distract the model and reduce its effectiveness. To solve this problem, the present invention introduces the segmentation masks of the reference concepts and concatenates them with the all-1 matrix to obtain M = [1, M 1 , M2 , ..., M N ], thereby correcting the model's attention. Finally, RSA can be expressed as:

[0071]

[0072] Where ⊙ represents the Hadamard product. This process ensures that the generated image can effectively interact with the overall semantic information in the reference image while filtering out irrelevant noise.

[0073] 2. Latent Semantic Mask Generation

[0074] In order to ensure accurate feature injection for each generated concept at the pixel level, the present invention needs to obtain the potential generated area of ​​each concept at each step. The attention layer in U-Net contains rich semantic information, which can be used to effectively identify semantic units. Therefore, the present invention proposes to calculate the potential semantic map of each concept based on the attention map, which includes two consecutive steps: semantic segmentation based on cross attention and segmentation completion based on self-attention.

[0075] Generate attention map:

[0076] In the attention layer l and time step t, the present invention uses the spatial features of the latent image Or linearly project the text embedding e to calculate the self-attention map and cross attention map

[0077]

[0078] Among them, Q * (·) and K * (·) is a linear projection of dimension d.

[0079] Semantic Segmentation Based on Cross-Attention:

[0080] The cross attention map contains the image patch s and the text label P i Therefore, in In each row of Represents s and P i The relationship between them is closer. Based on this, the present invention will t Split into [m 1 , m 2 , ..., m K ] The set of masked regions, where K represents the number of text tags, m i ∈{0, 1} represents the label P i Latent semantic regions.

[0081] Specifically, the present invention firstly transforms all cross attention maps Upsample to the same size, then average and renormalize them in the spatial dimension to obtain the final attention map C t This figure estimates the assignment of image patch s to text label P i Finally, an argmax operation is applied over the text token dimension to determine the activation for each image patch:

[0082]

[0083] Following this idea, the present invention transforms the image block set {s:i s =i} is set to 1, and the other elements are set to 0 to calculate m i It should be noted that the present invention only requires the semantic masks of N reference concepts for feature injection. Therefore, the present invention retains the masks of concept-specific tags and merges the remaining masks into a new mask m 0 To represent the background area, we finally get M = [m 0 , m 1 , m 2 , ..., m N ].

[0084] Figure 2 The first row of shows an example of the above semantic segmentation results. These semantic maps are effective in identifying the approximate location of the generated concepts. However, they often show unclear boundaries and may contain internal gaps, resulting in unsatisfactory results. To address this problem, the present invention uses self-attention maps to further refine and complete the latent semantic map, which will be described in detail in the following section.

[0085] Self-attention based segmentation completion:

[0086] Self-Attention Map The correlation between image patches is estimated, so the present invention refines the incomplete activation areas in the cross-attention map by transferring semantic information between image patches. This approach is similar to feature transfer in spectral domain graph convolution, because the self-attention map can be regarded as a transfer matrix, where each element is non-negative and the sum of each row is 1. Therefore, we refine the cross-attention map by multiplying the self-attention map with the corresponding cross-attention map: Based on this, the present invention further obtains all the cross attention maps in the spatial dimension. The average and renormalization are performed, and the result is input into (1) to calculate a more refined latent semantic mask.

[0087] Figure 2The last row of demonstrates the effect of refinement, where the generated segmentation map clearly indicates the boundaries of the object and the internal holes are significantly reduced.

[0088] 3. Concept feature injection

[0089] Region Grouped Hybrid Attention (RBA): According to the latent semantic mask M, RBA divides the latent image into multiple concept semantic groups and makes each generated concept query features from its corresponding reference image. Specifically, at each self-attention layer l and time step t, the features from z t The query, key, and value features are Q, K, and V, respectively. For brevity, l and t are omitted here. Next, RBA divides Q into multiple groups according to M [q 0 ,q 1 ,q 2 , ..., q N ].

[0090] For each group q i (where i>0), we concatenate K and V with the key and value features in their corresponding reference paths to obtain and For the feature group q corresponding to the background 0 , we have and

[0091] Based on this, we calculate the attention output x for each region separately i as follows:

[0092]

[0093] in

[0094] Finally, we place each output pixel group into its corresponding position for blending based on M to obtain the final output. This process effectively ensures accurate attribute alignment and feature injection for each generated concept.

[0095] 4. Weighted Masking Strategy

[0096] Although the semantic segmentation mask of each concept introduced helps to avoid the influence of irrelevant features in the reference image, the existing model still has difficulty in accurately capturing the unique attributes of the target concept, especially in preserving the information of the person's identity. To solve this problem, the present invention generates a segmentation mask M for each reference concept image in the RSA and RBA processes. i Introducing the scaling factor w i , which aims to enhance the model's attention to the desired conceptual features. The effectiveness of this strategy is Figure 3 Explored in.

[0097] Example 1

[0098] The present invention is compared with the latest existing public technology solutions PhotoMaker[9], FlashFace

[12] , Fastcomposer[8] and InstantID

[11] in terms of single-concept character image customization generation, and with ClassDiffusion

[13] , FreeCustom

[15] , CustomDiffusion

[16] and Prefusion

[14] in terms of multi-concept character image customization generation. The comparison results are shown in Figure 2. Figure 4 and Figure 5 shown.

[0099] More effect diagrams of the present invention are in Figure 6 , Figure 7 and Figure 8 Displayed in.

[0100] In order to further verify the effectiveness of the proposed method, the present invention first visualizes the correlation diagram between the latent image and all reference concept images based on the attention map generated by RSA in the first stage of the customized generation path, where the areas with the greatest correlation between the latent image and the reference concept image are represented by the same color. The visualization results are shown in Figure 2. Fig. 9 The present invention also visualizes the attention graph between certain image blocks on the generated image in the second stage of the customized generation path and their corresponding reference concept images in RBA. The visualization result is shown as follows: Fig. 9 As shown on the right, it can be seen from the visualization results that the character image customized generation process proposed by the present invention enables the pixels in the potential generation area of ​​each concept to accurately extract features from its corresponding reference image, thereby ensuring that the generated concept is similar to the reference concept.

[0101] References

[0102] [1]Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation. In CVPR, pages 22500-22510, 2023.

[0103] [2]Rinon Gal,Yuval Alaluf,Yuval Atzmon,Or Patashnik,Amit H Bermano,GalChechik,and Daniel Cohen-Or.An image is worth one word:Personalizing text-to-image generation using textual inversion.In ICLR,2023.

[0104] [3]Yuval Alaluf,Elad Richardson,Gal Metzer,and Daniel Cohen-Or.Aneural space-time representation for text-to-image personalization.ACMTransactions on Graphics,42(6):1-10,2023.

[0105] [4]Andrey Voynov,Qinghao Chu,Daniel Cohen-Or,and Kfir Aberman.p+:Extended textual conditioning in text-to-image generation.arXiv preprintarXiv:2303.09522,2023.

[0106] [5]Moab Arar,Rinon Gal,Yuval Atzmon,Gal Chechik,Daniel Cohen-Or,ArielShamir,and Amit H.Bermano.Domain-agnostic tuning-encoder for fastpersonalization of text-to-image models.In SIGGRAPH Asia,pages 1-10,2023.

[0107] [6]Yuxiang Wei,Yabo Zhang,Zhilong Ji,Jinfeng Bai,Lei Zhang,andWangmeng Zuo.Elite:Encoding visual concepts into textual embeddings forcustomized text-to-image generation.arXiv preprint arXiv:2302.13848,2023.

[0108] [7]Jing Shi,Wei Xiong,Zhe Lin,and Hyun Joon Jung.Instantbooth:Personalized text-to-image generation without test-time finetuning.arXivpreprint arXiv:2304.03411,2023.

[0109] [8]Guangxuan Xiao,Tianwei Yin,William T Freeman, Durand,and SongHan.Fastcomposer:Tuning-free multi-subject image generation with localizedattention.arXiv preprint arXiv:2305.10431,2023.

[0110] [9]Zhen Li,Mingdeng Cao,Xintao Wang,Zhongang Qi,Ming-Ming Cheng,andYing Shan.Photomaker:Customizing realistic human photos via stacked idembedding.In CVPR,pages8640-8650,2024.

[0111]

[10] Yibin Wang,Weizhong Zhang,Jianwei Zheng,and Cheng Jin.High-fidelity person-centric subject-to-image synthesis.In CVPR,pages 7675-7684,2024.

[0112]

[11] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024.

[0113]

[12] Shilong Zhang, Lianghua Huang, Xi Chen, Yifei Zhang, Zhi-Fan Wu, Yutong Feng, Wei Wang, Yujun Shen, Yu Liu, and Ping Luo. Flashface: Human image personalization with high-fidelity identity preservation. arXiv preprint arXiv:2403.17008, 2024.

[0114]

[13] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shecht-man, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, pages 1931-1941, 2023.

[0115]

[14] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rankone editing for text-to-image personalization. In ACM SIGGRAPH, pages 1-11, 2023.

[0116]

[15] Ganggui Ding,Canyu Zhao,Wen Wang,Zhen Yang,Zide Liu,Hao Chen,andChunhua Shen.Freecustom:Tuning-free customized image generation for multi-concept composition.In CVPR,pages 9089-9098,2024.

[0117]

[16] Jiannan Huang,Jun Hao Liew,Hanshu Yan,Yuyang Yin,Yao Zhao,andYunchao Wei.Classdiffusion:More aligned personalization tuning with explicitclass guidance.arXiv preprint arXiv:2405.17532,2024.

[0118]

[17] Robin Rombach,Andreas Blattmann,Dominik Lorenz,Patrick Esser,and Ommer.High-resolution imagesynthesis with latent diffusion models.InCVPR,pages 10684–10695,2022.

[0119]

[18] Ho,J.,Jain,A.,Abbeel,P.:Denoising diffusion probabilisticmodels.NeurIPS 33,6840–6851,2020.

Claims

1. A method for customizing and generating a general style character image based on a diffusion model, characterized in that: This method realizes the generation of single-concept or multi-concept customized images of characters of any style based on the dual-path guidance of the reference path and the customized generation path. In the customized generation path, the forward control generation from coarse to fine is realized through the reference-perceived self-attention RSA and the regional grouping hybrid attention RBA mechanism. The generation process is divided into the following two stages: Phase 1: Semantic scene construction RSA enables the latent image to query features from all the concept’s reference images simultaneously, adaptively integrating coarse-grained semantic information to extract comprehensive semantic understanding from the reference images in order to establish an initial semantic layout; The second stage of concept feature injection First, the latent semantic map of each concept is calculated based on the attention map to accurately locate the generation positions of all concepts in the latent image; then, RBA divides the latent image into multiple concept semantic groups, and enables each group to query fine-grained features from its corresponding reference concept image to ensure accurate attribute alignment and feature injection; finally, the features of each reference concept image are accurately injected into the corresponding position to ensure that the generated concept is highly similar to the reference image.

2. The method for customizing and generating a universal style character image based on a diffusion model according to claim 1, characterized in that: In the reference path, the pre-trained diffusion model Stable Diffusion is used as the basic model, and the U-Net∈ θ As a denoiser, visual features are gradually extracted from the input reference concept image to guide generation; in the customized generation path, the self-attention layer in the first stage U-Net is replaced by a reference-aware self-attention RSA module, and in the second stage it is replaced by a region-grouped hybrid attention RBA module.

3. The method for customizing and generating a universal style character image based on a diffusion model according to claim 1, characterized in that: In the reference path, for each reference concept image, the forward diffusion process is first applied to calculate the noise latent variable z of the reference image T '; At each time step t, the noise latent variable z T ' and the corresponding text prompt are input to the denoiser; in the self-attention layer l and time step t, the key feature K of the i-th concept is extracted i,l,t Sum value feature V i,l,t , to guide image generation for a custom build path.

4. The method for customizing and generating a universal style character image based on a diffusion model according to claim 1, characterized in that: In the first stage, at each self-attention layer l and time step t, the latent variable z t The query, key, and value features are Q l,t , K l,t and V l,t , l, t are omitted for brevity, and the key and value features are concatenated with the N key features and value features obtained from the reference path, respectively, to obtain and Introduce the segmentation masks of the reference concept and concatenate them with the all-1 matrix to obtain M = [1, M1, M2, ..., M N ], thereby correcting the model's attention; finally, RSA is expressed as: Where ⊙ represents the Hadamard product and d represents the dimension of Q.

5. The method for customizing and generating a universal style character image based on a diffusion model according to claim 1, characterized in that: In the second stage, the steps to calculate the latent semantic graph of each concept based on the attention map are as follows: 1) Generate attention map In the attention layer l and time step t, the spatial features of the latent image are Or linearly project the text embedding e to calculate the self-attention map and cross attention map Among them, Q * (·) and K * (·) is a linear projection of dimension d; 2) Semantic Segmentation Based on Cross-Attention The latent variable z t Split into [m1,m2,...,m K ] The set of masked regions, where K represents the number of text tags, m i ∈{0,1} represents the text tag P i The potential semantic regions are as follows: First, all the cross attention maps Upsample to the same size, then average and renormalize them in the spatial dimension to obtain the final attention map C t , which estimates the assignment of image patch s to text label P i Finally, an argmax operation is applied over the text token dimension to determine the activation for each image patch: By combining the image patch set {s:i s =i} is set to 1, and the other elements are set to 0 to calculate m i ; The number of semantic masks of the reference concepts required for feature injection is N, so the masks of concept-specific markers are retained and the remaining masks are merged into a new mask m0 to represent the background area, and finally the latent semantic mask M = [m0,m1,m2,...,m N ]; 3) Segmentation completion based on self-attention Refine the criss-cross attention map by multiplying the self-attention map with the corresponding criss-cross attention map: Then, all the cross attention maps obtained in the spatial dimension The average and renormalization are performed, and the result is input into formula (1) to calculate a more refined latent semantic mask M.

6. The method for customizing and generating a universal style character image based on a diffusion model according to claim 5, characterized in that: In the second stage, RBA divides the potential image into multiple conceptual semantic groups, so that each group can query fine-grained features from its corresponding reference image and the specific method of feature injection is as follows: At each self-attention layer l and time step t, the latent variable z t The query feature, key feature, and value feature are Q l,t , K l,t and V l,t , l, t are omitted for brevity. RBA divides Q into multiple groups [q0, q1, q2, ..., q N ]. For each group q i , where i>0, concatenate K and V with the key and value features in their corresponding reference paths to obtain and For the feature group q0 corresponding to the background, we have and Then, the attention output x of each region is calculated separately i as follows: in Finally, each output pixel group is placed at the position of its corresponding concept generation based on the latent semantic mask M and mixed to obtain the final output.

7. The method for customizing and generating a universal style character image based on a diffusion model according to claim 1, characterized in that: In the first and second stages, RSA and RBA are segmentation masks M for each reference concept image. i Introducing the scaling factor w i , to enhance the model’s attention to the features of each reference concept image.

Images