A subject-driven personalized generation method based on decoupled mask prompt attention fine-tuning

Abstract

The application discloses a subject-driven personalized generation method based on decoupled mask prompt attention fine-tuning, and belongs to the field of deep learning, computer vision and artificial intelligence generated content. A mask prompt decoupling module is designed to decompose the unified text prompt into a text prompt containing only the subject and a text prompt containing only the context. A subject attention focusing module and a context attention adjusting module are introduced, and independent subject feature extraction paths and context semantic adaptation paths are established, respectively. The subject identity learning and the context scene modeling are explicitly separated, and the double constraint loss guided by the mask is used for joint optimization to further prevent feature coupling. Finally, the algorithm effectively preserves the fine appearance features of the subject, significantly enhances the adaptability of the model to novel context instructions, and effectively improves the quality of personalized generation and the flexibility of context editing.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, computer vision, and artificial intelligence generated content (AIGC). It designs a subject-driven customized generation algorithm based on decoupled masking cue attention fine-tuning. Background Technology

[0002] Text-to-image generation is a fundamental topic and research hotspot in the field of computer vision and AI-generated content (AIGC). Its main task is to generate high-quality, creative images based on input text descriptions. In recent years, driven by advanced technologies such as large-scale image-text datasets and diffusion models, generative AI has made groundbreaking progress. Building on this foundation, subject-driven customized generation, as an emerging and important application direction, aims to utilize a small number of reference images containing a specific subject, injecting the visual concept of that subject into a pre-trained model to generate personalized images that retain the subject's identity features while conforming to the new text description. This has significant application value in personalized art creation, e-commerce advertising, and commercial design, attracting widespread attention.

[0003] Currently, effectively preserving the subject's identity while achieving flexible scene editing is a key factor in the success of personalized generation. Existing subject-driven generation methods mainly adopt a fine-tuning paradigm based on unified text prompts. These methods (such as DreamBooth and Custom Diffusion) typically construct a text prompt containing a unique identifier and category name (e.g., "A photo of a V* dog") to train the model. However, this method forcibly couples the subject's identity information with the accompanying contextual background (such as grass, indoor background, etc.) in the reference image into the same prompt representation during fine-tuning. This strong "identity-background" coupling leads to two main problems: firstly, the model is prone to overfitting to the background of the training samples, making it difficult to change the background through textual instructions during inference; secondly, there is a dilemma between subject consistency and contextual flexibility during inference, i.e., the ability to generate novel scenes is often sacrificed in order to preserve subject details, or key features of the subject are lost in order to adapt to new scenes. In addition, directly fine-tuning all or part of the parameters on a limited number of samples can easily lead to catastrophic forgetting, causing the model to lose its original prior knowledge.

[0004] To address the aforementioned issues, this invention proposes a novel subject-driven personalized generation algorithm based on decoupled masked cue attention fine-tuning. By designing a masked cue decoupling module, a unified text cue is decomposed into text cue containing only the subject and text cue containing only the context. Furthermore, a subject attention focusing module and a context attention adjustment module are introduced, establishing independent subject feature extraction paths and context semantic adaptation paths, respectively. Subject identity learning and context scene modeling are explicitly separated, and joint optimization is performed using a masked-guided dual-constraint loss to further prevent feature coupling. Finally, this algorithm effectively preserves the subject's fine appearance features while significantly enhancing the model's adaptability to novel contextual instructions, effectively improving the quality of personalized generation and the flexibility of context editing. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a novel subject-driven personalized generation algorithm based on decoupled masked cue attention fine-tuning. By designing a masked cue decoupling module, a unified text cue is decomposed into text cue containing only the subject and text cue containing only the context. Furthermore, a subject attention focusing module and a context attention adjustment module are introduced, establishing independent subject feature extraction paths and context semantic adaptation paths, respectively. Subject identity learning and context scene modeling are explicitly separated, and joint optimization is performed using a masked-guided dual-constraint loss to further prevent feature coupling. Finally, this algorithm effectively preserves the subject's fine appearance features while significantly enhancing the model's adaptability to novel contextual instructions, effectively improving the quality of personalized generation and the flexibility of context editing.

[0006] To achieve the above objectives, the present invention adopts the following specific technical solution: a subject-driven personalized generation method based on decoupling mask cue attention fine-tuning, the method comprising:

[0007] Step 1: Mask hint decoupling;

[0008] The data to be processed is a segment of length. Original text prompt sequence ,in Representing the Each token; defines the index set of the main key tokens. And employing a structure-preserving masking strategy, the original text prompt sequence is preserved. Decomposed into two complementary text prompt sequences and : A text prompt containing only the main body. Provides a text hint with only context;

[0009] Hint sequence The aim is to extract main features by masking contextual information; for each token in the sequence Its generation rules are as follows:

[0010] ;

[0011] in Represents the mask token. A set of indices representing the key tokens of the entity;

[0012] Hint sequence The aim is to capture environmental scene information by masking the identity of the subject; for each token in the sequence Its generation rules are as follows:

[0013] ;

[0014] Then, the original text prompt , and Parallel input to a pre-trained text encoder In the process, corresponding text embedding features are generated respectively:

[0015] ;

[0016] The corresponding embedded features , , , This indicates the feature channel dimension of the text prompt;

[0017] Step 2: Focusing attention;

[0018] Given the visual features of the reference image extracted by the pre-trained image encoder and query latent features from pre-trained generative models. ;in, and These represent the height and width of the image features, respectively. Indicates its channel dimension, The channel dimension for querying latent features;

[0019] Calculate the cross-attention between query latent features and reference image features within a specific subject space region. ,in, The channel dimension after projection;

[0020] Cross-attention spectrum Visual cross attention to reference image Dynamic weighting is performed to obtain the final pure subject-related latent features. :

[0021] ;

[0022] in This represents an element-wise multiplication operation;

[0023] Step 3: Contextual attention adjustment;

[0024] Step 3.1: Utilize and Projecting the text prompt features containing only context, we obtain the context key vectors. and context value vector :

[0025] Step 3.2: Define an inverted mask Its operating logic is the opposite of that of a subject mask: it forces attention to non-subject areas while masking the subject area; its specific definition is:

[0026] ;

[0027] Step 3.3: Combine query latent features Calculate context attention features constrained by the inverted mask. This mask acts as an additive bias term on the attention score, preventing the leakage of main features into the background region.

[0028] ;

[0029] Step 4: Mask-guided double-constraint loss optimization;

[0030] Step 4.1: A composite objective function based on spatial masking was constructed; first, the subject consistency loss was defined. Using the subject space mask of the reference image Gradient backpropagation is strictly limited to the main region; set For the diffusion model at time step The latent characteristics of noise addition, For the sampled Gaussian noise, For the network's noise prediction output, then this loss... for:

[0031] ;

[0032] in, This represents the mathematical expectation calculation operation, specifically, for the initial image sample. Gaussian noise variables and diffusion time step The probability distribution is sampled, and the average value of the corresponding loss function is calculated. By minimizing... Let the model only in the mask By fitting the noise distribution within the covered pixel area, accurate reconstruction of the main features can be achieved;

[0033] Step 4.2: To orthogonalize the representations of the subject and background in the feature space, a context separation loss is introduced. ;

[0034] ;

[0035] Step 4.3: Joint Optimization Objective Function of the Overall Network A threshold-based regularization method is adopted, by introducing a balance coefficient. and interval threshold To dynamically adjust the weights of the two losses:

[0036] ;

[0037] During training, if the calculated separation Less than the preset threshold If the gradient generated by this term is positive, it will penalize the model to accelerate decoupling; otherwise, the optimization process will be mainly driven by... leading;

[0038] Step 5: Generate personalized images using adaptive feature fusion;

[0039] prompts from the original text The standard attention path is used to calculate and obtain global attention features. Based on this, the subject-related latent features extracted in step 2 will be... and the contextual attention features obtained in step 3.3 Considered as a refined enhancement of the global features, the above features are weighted and fused according to the following formula:

[0040] ;

[0041] in, This represents the final feature representation after fusion; For a learnable adaptive mixture parameter; during training, It can automatically adjust the intensity of subject feature fusion based on gradient feedback, thereby dynamically finding the best balance between preserving the subject identity of the reference image and adapting to the context of novel text description;

[0042] Finally, the adaptive fusion features obtained from the above calculations are... This serves as a key condition to guide the inverse denoising process of the diffusion model, in order to obtain the final latent feature representation. Subsequently, the pre-trained decoding module was used. The latent features are mapped back to the pixel space using the following decoding formula:

[0043]

[0044] in, This represents the final generated image. Indicates the final latent feature, This indicates a pre-trained decoding module.

[0045] Furthermore, in step 2, visual cross-attention The calculation method is as follows:

[0046] Introducing a reference subject space mask To limit key spatial regions in the reference image; wherein, the query vector is calculated using the projection matrix. Key vector Sum value vector ;in, , , These represent the projection matrices of the query vector, key vector, and value vector, respectively.

[0047] The formula for calculating visual cross attention is as follows:

[0048] ;

[0049] in, This represents the transpose of the key vector. The feature dimension of the key vector. The calculation method is as follows: Main area:

[0050] ;

[0051] in, This represents the spatial location index of the feature map. This represents the set of main regions.

[0052] Furthermore, in step 2, the cross-attention spectrum The calculation method is as follows:

[0053] Calculate query vector Text prompts containing only the main body Cross-attention spectrum Projection matrix Project text features into key vectors Cross-attention spectrum The calculation formula is:

[0054] ;

[0055] in, The key vector projection matrix represents the embedding features of the main text. This indicates the embedding features of the main text.

[0056] Furthermore, in step 3.1, the context key vector The calculation method is as follows:

[0057] ;

[0058] in, and These represent the context key vector projection matrix and the context value vector projection matrix, respectively.

[0059] The proposed subject-driven personalized generation algorithm based on decoupled mask cue attention fine-tuning explicitly separates subject identity learning from contextual scene modeling through a cleverly designed mask cue decoupling module. Combined with a subject attention focusing module and a contextual attention adjustment module, this algorithm can not only reconstruct the subject in the reference image with high fidelity but also flexibly place the subject in diverse contextual situations, effectively solving the overfitting and conceptual coupling problems common in existing technologies. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the overall process of the subject-driven personalized generation algorithm for decoupling masking prompts and attention fine-tuning in this invention. Detailed Implementation

[0061] This invention proposes a subject-driven personalized generation algorithm based on decoupled masking cue attention fine-tuning. It aims to solve the problems of existing subject-driven generation technologies that usually rely on uniform text cues for fine-tuning, which leads to an inherent aliasing and coupling between the subject identity and the context background. This makes it difficult to effectively balance the subject fidelity and background context flexibility of the generated image, i.e., the generated image is inconsistent with the reference image and the generated image is difficult to adapt to novel context scenes.

[0062] This invention is implemented based on the PyTorch deep learning platform and the Diffusers library, and specifically includes the following steps:

[0063] Step 1: Select the pre-trained IP-Adapter and IP-Adapter-Plus in conjunction with the SDXL model as the basic text-to-image diffusion generation model. Simultaneously, use OpenCLIP ViT-bigG / 14 as the image encoder to extract image features.

[0064] Step 2: Conduct subject-driven personalized generation experiments using DreamBench as the benchmark dataset. This dataset contains 30 specific subjects (including 21 object subjects and 9 creature / pet objects, such as backpacks, plush toys, dogs, cats, etc.) and 25 corresponding prompts. The prompt set covers various scenarios such as resetting context, modifying attributes, and adding accessories.

[0065] Step 3: The text hint is processed into three parallel sequences using the mask hint decoupling module: original text hint Text prompts containing only the main body and text hints with only context All text prompts are embedded using a pre-trained text encoder with frozen parameters to extract their corresponding features. , and ;

[0066] Mask prompt decoupling module. This module first inputs a segment of length... Original text prompt sequence ,in Representing the Each token. To address the conceptual coupling between subject identity and contextual information in existing technologies, this module defines an index set of subject key tokens. It employs a structure-preserving masking strategy to mask the original text prompts. Decomposed into two complementary text prompt sequences and One part consists of text prompts containing only the main content. This hint aims to extract key features by masking contextual information. For each token in the sequence... Its generation rules are as follows:

[0067]

[0068] in The first part represents the mask token. For example, the original text hint "a photo of a V* robot" is transformed into "[M][M][M][M] V* robot". The second part is the text hint containing only context. This prompt aims to capture environmental context information by masking the identity of the main user. For each token in the sequence... Its generation rules are as follows:

[0069]

[0070] For example, the above text prompt is transformed into "a photo of a [M] [M]".

[0071] Then, the original text prompt Text prompts containing only the main body and text hints with only context Parallel input to a pre-trained text encoder In the process, corresponding text embedding features are generated respectively:

[0072]

[0073] The obtained embedding features , , This provides the basic input for subsequent feature decoupling learning based on the attention mechanism.

[0074] Step 4: Integrate the proposed subject attention focusing module and context attention adjustment module into the original cross-attention layer of the SDXL model to construct a dual-branch feature interaction path that independently processes the subject and context. During the fine-tuning stage, to accurately inject new concepts while maximizing the preservation of the original model's general generative capabilities, a parameter-efficient fine-tuning strategy was adopted. Specifically, the backbone U-Net parameters of the SDXL and SDXL_plus models, as well as all weights of the OpenCLIPViT-bigG / 14 image encoder, were frozen. The fine-tuning process was performed only on specific parameters in the text cross-attention layer, namely, only on the projection matrix responsible for mapping reference image features and mapped text features, and the adaptive parameters used for dynamically balancing feature fusion. Open gradient backpropagation.

[0075] The subject attention focusing module aims to establish a direct visual correspondence between the generated image and the appearance of the reference subject at the latent space level, and to dynamically weight the data using semantic information. Specifically, given the subject-only text cue features obtained in step one... Visual features of the reference image extracted by the pre-trained image encoder and query latent features from pre-trained generative models. .in, Indicates the length of the text sequence. This indicates the feature channel dimension of the text prompt. and These represent the height and width of the image features, respectively. Indicates its channel dimension, To query the channel dimension of latent features.

[0076] The context attention conditioning module aims to address the problem of background overfitting in subject-driven generation tasks, while fully leveraging the prior ability of pre-trained diffusion models to generate diverse scenes. To achieve this, the module establishes a context adaptation path independent of subject features, directly utilizing the context-only cue features generated in step one. As a semantic guidance signal. Because... By explicitly masking the subject's identity information and retaining only the contextual scene description, the model can avoid being constrained by the original background of the reference image and focus on constructing entirely new environmental semantics based on textual instructions.

[0077] Step 5: Optimize using the AdamW optimizer, setting the learning rate to 1e-5, weight decay to 0.01, and batch size to 2. The total training steps are 250.

[0078] Step 6: Calculate the masked guided double-constraint loss to update the guiding parameters. Coefficients used to balance subject consistency and context separation. Set to 0.1, interval threshold Set to 0.5. By minimizing the total loss. This is used to update the trainable parameters in the cross-attention layer. The overall network's joint optimization objective function... A threshold-based regularization method is adopted, by introducing a balance coefficient. and interval threshold To dynamically adjust the weights of the two losses:

[0079]

[0080] During training, if the calculated separation Less than the preset threshold If the gradient generated by this term is positive, it will penalize the model to accelerate decoupling; otherwise, the optimization process will be mainly driven by... This dynamic adjustment mechanism provides effective optimization guidance for the model, achieving an optimal balance between subject fidelity and context decoupling.

[0081] Step 7: During the inference phase, the DDIM sampler is used, with a sampling step count of 50 and a classifier-free guidance ratio of 5.0. Images are generated from diverse text prompts containing subject identifiers. To verify the implementation's effectiveness, five images are generated for each text prompt to ensure evaluation stability. The generation results are quantitatively evaluated using three metrics: CLIP-T, CLIP-I, and DINO, along with their average values.

[0082] The main protection point of this invention lies in proposing a decoupled masked cue attention fine-tuning framework. The text cue is decomposed by a masked cue decoupler, and the subject attention focusing module and the context attention adjustment module are combined to process the subject and context information respectively. At the same time, a masked guided dual-constraint loss function is designed to optimize the model feature decoupling, thereby achieving an effective balance between subject fidelity and context flexibility in the subject-driven generation task.

Claims

1. A subject-driven personalized generation method based on decoupled masking cue attention fine-tuning, the method comprising: Step 1: Mask hint decoupling; The data to be processed is a segment of length. Original text prompt sequence ,in Representing the Each token; defines the index set of the main key tokens. And employing a structure-preserving masking strategy, the original text prompt sequence is preserved. Decomposed into two complementary text prompt sequences and : A text prompt containing only the main body. Provides a text hint with only context; Hint sequence The aim is to extract subject features by masking contextual information; for Each Token in the sequence Its generation rules are as follows: ； in Represents the mask token. A set of indices representing the key tokens of the entity; Hint sequence The aim is to capture environmental scene information by masking the identity of the main entity; for each token in the sequence Its generation rules are as follows: ； Then, the original text prompt , and Parallel input to a pre-trained text encoder In the process, corresponding text embedding features are generated respectively: ； The corresponding embedded features , , , This indicates the feature channel dimension of the text prompt; Step 2: Focusing attention; Given the visual features of the reference image extracted by the pre-trained image encoder and query latent features from pre-trained generative models. ;in, and These represent the height and width of the image features, respectively. Indicates its channel dimension, The channel dimension for querying latent features; Calculate the cross-attention between query latent features and reference image features within a specific subject space region. ,in, The channel dimension after projection; Cross-attention spectrum Visual cross attention to reference image Dynamic weighting is performed to obtain the final pure subject-related latent features. : ； in This represents an element-wise multiplication operation; Step 3: Contextual attention adjustment; Step 3.1: Utilize and Projecting the text prompt features containing only context, we obtain the context key vectors. and context value vector : Step 3.2: Define an inverted mask Its operating logic is the opposite of that of a subject mask: it forces attention to non-subject areas while masking the subject area; its specific definition is: ； Step 3.3: Combine query latent features Calculate context attention features constrained by the inverted mask. This mask acts as an additive bias term on the attention score, preventing the leakage of main features into the background region. ； Step 4: Mask-guided double-constraint loss optimization; Step 4.1: A composite objective function based on spatial masking was constructed; first, the subject consistency loss was defined. Using the subject space mask of the reference image Gradient backpropagation is strictly limited to the main region; set For the diffusion model at time step The latent characteristics of noise addition, For the sampled Gaussian noise, For the network's noise prediction output, then this loss... for: ； in, This represents the mathematical expectation calculation operation, specifically, for the initial image sample. Gaussian noise variables and diffusion time step The probability distribution is sampled, and the average value of the corresponding loss function is calculated. By minimizing... Let the model only in the mask By fitting the noise distribution within the covered pixel area, accurate reconstruction of the main features can be achieved; Step 4.2: To orthogonalize the representations of the subject and background in the feature space, a context separation loss is introduced. ; ； Step 4.3: Joint Optimization Objective Function of the Overall Network A threshold-based regularization method is adopted, by introducing a balance coefficient. and interval threshold To dynamically adjust the weights of the two losses: ； During training, if the calculated separation degree Less than the preset threshold If the gradient generated by this term is positive, it will penalize the model to accelerate decoupling; otherwise, the optimization process will be mainly driven by... leading; Step 5: Generate personalized images using adaptive feature fusion; prompts from the original text The standard attention path is used to calculate and obtain global attention features. Based on this, the subject-related latent features extracted in step 2 will be... and the contextual attention features obtained in step 3.3 Considered as a refined enhancement of the global features, the above features are weighted and fused according to the following formula: ； in, This represents the final feature representation after fusion; For a learnable adaptive mixture parameter; during training, It can automatically adjust the intensity of subject feature fusion based on gradient feedback, thereby dynamically finding the best balance between preserving the subject identity of the reference image and adapting to the context of novel text description; Finally, the adaptive fusion features obtained from the above calculations are... This serves as a key condition to guide the inverse denoising process of the diffusion model, in order to obtain the final latent feature representation. Subsequently, the pre-trained decoding module was used. The latent features are mapped back to the pixel space using the following decoding formula: ； in, This represents the final generated image. Indicates the final latent feature, This indicates a pre-trained decoding module.

2. The subject-driven personalized generation method based on decoupling mask cue attention fine-tuning as described in claim 1, characterized in that, Visual cross attention in step 2 The calculation method is as follows: Introducing a reference subject space mask To limit key spatial regions in the reference image; wherein, the query vector is calculated using the projection matrix. Key vector Sum value vector ;in, , , These represent the projection matrices of the query vector, key vector, and value vector, respectively. The formula for calculating visual cross attention is as follows: ； in, This represents the transpose of the key vector. The feature dimension of the key vector. The calculation method is as follows: Main area: ； in, This represents the spatial location index of the feature map. This represents the set of main regions.

3. The subject-driven personalized generation method based on decoupling mask cue attention fine-tuning as described in claim 1, characterized in that, Cross-attention spectrum in step 2 The calculation method is as follows: Calculate query vector Text prompts containing only the main body Cross-attention spectrum Projection matrix Project text features into key vectors Cross-attention spectrum The calculation formula is: ； in, The key vector projection matrix represents the embedding features of the main text. This indicates the embedding features of the main text.

4. The subject-driven personalized generation method based on decoupling mask cue attention fine-tuning as described in claim 1, characterized in that, Context key vector in step 3.1 The calculation method is as follows: ； in, and These represent the context key vector projection matrix and the context value vector projection matrix, respectively.

Images