Personalized image generation method and system based on maximum difference anchoring and mask attention guidance

Abstract

The invention discloses a personalized image generation method and system based on maximum difference anchoring and mask attention guidance. In order to solve the problems of concept excessive generalization and background interference distortion of an existing text-to-image diffusion model in a user-defined concept and general concept combined scene, an end-to-end dynamic constraint framework is constructed. The method comprises the following steps: firstly, generating a candidate set related to a user concept through a multi-modal large model; a plurality of anchor point concepts with the maximum semantic difference are selected through K-means clustering; secondly, designing a double-stage training mechanism, wherein in the first stage, attention loss guided by an SAM mask is adopted to suppress background interference, and in the second stage, maximum differentiation anchor point constraint loss is introduced to optimize embedded distribution; and finally, a high-quality image fusing the custom concept and the general concept is generated. According to the method provided by the invention, unique features of a user-specified concept can be better reserved in the generated image, meanwhile, the deficiency or distortion of a general concept is avoided, and the consistency and the structural accuracy of the generated image are improved.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer vision, and in particular to an improved text inversion method and system that integrates Maximal Divergent Anchoring (MDA), Background Leakage Suppression (BLS), and a two-stage training paradigm. Background Technology

[0002] In recent years, diffusion models (DM) have made breakthrough progress in the field of text-based images. Systems such as Stable Diffusion, DALL-E 2, and Imagen are able to generate high-fidelity, high-resolution images based on open text prompts. However, when users want to insert personalized concepts such as "the cat in this picture," existing technologies still face challenges: pseudo-word embeddings e[v] obtained by directly training with Textual Inversion (TI) are prone to deviating from the core distribution area of ​​the general concept "cat." Existing improved schemes, such as Semantic Inversion (SI), introduce fixed anchor points... (like The position of e[v] is explicitly constrained, and its optimization objective can be formalized as:

[0003]

[0004] However, the drawback of this constraint is that it excessively compresses the optimization space, forcibly locking e[v] within a certain range. Optimization within a range with an increasingly smaller center and radius limits the model's ability to find the optimal point representing user-specific features (such as "[v]cat"); and simply relying on fixed anchor points to constrain pseudoword embeddings cannot completely solve the problem of personalized concepts being interfered with by background information during the model optimization process.

[0005] The method described in this invention uses a pre-trained diffusion model (such as Stable Diffusion) as a foundation. By introducing a semantic clustering-based strategy to maximize difference anchoring, it anchors user-provided personalized concepts (such as a specific pet cat) to the core semantic space distribution region of a general concept (such as "cat"), while maximizing the semantic difference between anchor points. This significantly suppresses the loss of personalized features caused by excessive anchor constraints. Furthermore, this invention innovatively designs a background leakage suppression mechanism based on attention masks. By guiding the model to focus on the target concept region, it effectively blocks the interference of background-irrelevant information (such as wall textures and furniture outlines) on the encoding of custom concepts, significantly reducing the probability of deformation or missing general concepts (such as "computer" and "backpack") in the generated image.

[0006] The system described in this invention supports plug-and-play deployment, requiring only 3-5 user-provided reference images to complete concept embedding. The training time is reduced by more than 80% compared to the traditional DreamBooth method, the FID index of the generated images is improved by 4.3%, the text-image alignment is improved by 5.2%, the personalized concept retention is improved by 3.1%, and real-time inference is achieved on consumer-grade GPUs (such as RTX 3090Ti). Summary of the Invention

[0007] The purpose of this invention is to address the two core bottlenecks of existing text-to-image (T2I) generation methods in personalized scenarios: concept over-generalization and background interference distortion. This invention proposes a personalized image generation method and system based on maximum difference anchoring and mask-guided attention. This method can improve the fidelity, consistency, and semantic expressiveness of the personalized subject in the generated image without modifying the original diffusion model structure. Specifically, existing methods such as semantic inversion (SI), while constraining the distribution area of ​​pseudo-word embeddings through fixed anchor point concepts, are prone to loss of personalized features and background interference. This invention, however, significantly improves the quality and consistency of the generated image through dynamic anchor point selection and background mask attention suppression mechanisms.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] According to a first aspect of this specification, a personalized image generation method based on maximum difference anchoring and mask-guided attention is provided, the method comprising the following steps:

[0010] S1 receives a set of custom concept images provided by the user to represent the user's unique visual concept;

[0011] S2, input the image samples provided by the user and the general visual concept text in the general concept set into the multimodal large model, sort the output concept relevance confidence, and construct a candidate concept set;

[0012] S3. Use a text encoder to convert each concept in the candidate concept set into an embedding vector to construct a high-dimensional semantic representation set that can be used for clustering; use the K-means algorithm to initialize the cluster centers, and apply a convergence threshold during the iteration process by setting the K value and using Euclidean distance as a metric to ensure the stability and interpretability of each clustering result; select the concept closest to the cluster center in each cluster as the anchor point to construct the anchor point set.

[0013] S4 uses the SAM model to segment the target region of the user image and generate a pixel-level aligned binary mask.

[0014] S5, based on the pre-trained latent space structure of Stable Diffusion, implements a two-stage progressive optimization strategy for pseudo-word embeddings. It achieves decoupling optimization of semantic anchoring and background suppression through staged regularization switching, specifically:

[0015] The first stage focuses on eliminating background interference in the encoding of personalized concept features; in this stage, the cross attention map of the second layer of the U-Net model is extracted, and a background leakage suppression loss function is constructed.

[0016] The second stage focuses on consolidating semantic anchoring constraints to prevent pseudo-word embeddings from drifting out of the core region; in this stage, the maximum difference anchor point loss is constructed to optimize the embedding distribution.

[0017] S6 embeds the finally converged pseudo-words into the Stable Diffusion text encoder as conditional input to participate in the diffusion process, generating image content that simultaneously integrates user-defined concepts and general concepts.

[0018] Furthermore, the multimodal large model has an end-to-end Transformer architecture. Its visual encoder divides the input image into patches and encodes them into image features, while the text encoder encodes general visual concept text into word vector features. The two are semantically fused and matched through a multi-layer cross-modal attention mechanism. Finally, a probability distribution of candidate semantic words is generated through a softmax normalization operation, which is used to select the concept most relevant to the semantics of the user image to form a candidate concept set.

[0019] Furthermore, in S3, the K-means initialization strategy optimizes the initial value by maximizing the minimum distance between the initial cluster centers; assuming the data set to be clustered is... N is the total number of data points to be clustered, and the preset number of cluster centers is K. The initialization process first randomly selects a data point as the first cluster center. Then, when determining the kth cluster center, first calculate the value of each unselected data point. To the current set of cluster centers shortest distance Then, among all the unselected data points, the point corresponding to the maximum value of these shortest distances is selected as the new cluster center. This method guarantees the maximum spacing between initial cluster centers, promoting the diversity and coverage of center distribution.

[0020] Furthermore, in S3, K-means clustering uses K=3 and is paired with a K-means initialization strategy, which can improve the global divergence of anchor point distribution and avoid the concept shrinkage problem caused by pseudo-word embeddings being too close to a certain semantic region. By establishing an anchor region composed of three points in the semantic space, a soft constraint mechanism is provided for pseudo-word embeddings to balance individual expression and semantic alignment. By maximizing the semantic differences between anchor points, pseudo-word embeddings can maintain personalized features while avoiding deviation from the core distribution area of ​​general concepts.

[0021] Furthermore, in S4, the SAM model's segmentation process employs a pre-trained ViT-H image encoder as its core architecture, achieving global context modeling through cascaded multi-head self-attention modules. During the prompting and guidance phase, user-provided point / box prompts are encoded as location embeddings, interacting with image features through cross-attention. The final output binary mask M is obtained through pixel-by-pixel binary classification, maintaining a strict spatial correspondence with the input image, and its boundary error is controlled within a certain range. Within the pixel range.

[0022] Furthermore, in S5, the stage switching threshold n is determined based on the following convergence criterion:

[0023]

[0024]

[0025] in, This is the gradient convergence threshold. These are the background leakage suppression loss function values ​​at the initial time and the nth iteration, respectively. The empirical attenuation coefficient represents the background leakage suppression loss. The moving average curve satisfies:

[0026]

[0027] in, For the length of the window, This is the saliency threshold, at which point a smooth transition from background suppression to semantic anchoring is achieved.

[0028] Furthermore, the total loss function in the first stage of S5 for:

[0029]

[0030] in, To rebuild the losses, To mitigate the loss caused by background leakage, These are the weight coefficients of the corresponding loss function; The formula is as follows:

[0031]

[0032] Where A is the cross attention map of the second layer of the U-Net model, M is the binary mask, and ⊙ represents element-wise product.

[0033] Furthermore, the total loss function in the second stage of S5 for:

[0034]

[0035] in, To rebuild the losses, For the maximum difference anchor point loss, These are the weight coefficients of the corresponding loss function; The formula is as follows:

[0036]

[0037] in, For pseudoword embedding vectors, Let be the embedding vector of the k-th anchor point; To relax the threshold, pseudoword embeddings are allowed to be freely optimized within the semantic region enclosed by anchor points, avoiding excessive contraction that would limit expressive power.

[0038] Furthermore, in S5, the cross-attention map A is directly extracted from the second-layer cross-attention module of the Stable Diffusion U-Net decoder. Its spatial structure is consistent with the latent variables of the current time step, and its resolution is 64×64. It is the most sensitive position for concept alignment and is suitable as a key layer for applying attention constraints. The attention map of this layer can effectively capture the semantic information of the target subject while suppressing the interference of the background region.

[0039] According to a second aspect of this specification, a personalized image generation system based on maximum difference anchoring and mask-guided attention is provided. This system includes a memory, a processor, and an executable computer program stored in the memory. When the processor runs the program, it can implement the method described in the first aspect. The system uses the PyTorch framework to construct the core computational flow, ensuring the modularity, scalability, and efficient training characteristics of the model. Simultaneously, by calling the Diffusers library provided by HuggingFace, the system can conveniently load the weights of the Stable Diffusion pre-trained model, achieving refined optimization of personalized pseudoword embeddings without requiring full parameter fine-tuning. The system supports pipelined scheduling of the complete process of mask generation, anchor point concept mining, and two-stage training.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention establishes an anchor point triangle region in the embedding space through a maximum difference anchoring mechanism, effectively constraining the distribution range of pseudo-word embeddings, preventing semantic drift, and improving the clarity and preservation of personalized expressions. Experiments show that this method can better preserve the unique features of user-specified concepts in the generated images, while avoiding the loss or distortion of general concepts.

[0042] 2. This invention addresses the interference of background regions on pseudoword learning by introducing a mask-guided attention suppression mechanism, thereby improving the consistency and structural accuracy of generated images. User studies show that images generated by this method account for 83.4% of "Good" or "Excellent" ratings, significantly outperforming existing methods.

[0043] 3. The method of this invention is fully compatible with mainstream Stable Diffusion frameworks, achieving optimization goals only through the addition of lightweight modules. It possesses good deployment flexibility, training efficiency, and engineering adaptability, making it suitable for various practical scenarios such as personalized image generation, content creation, and brand image customization. Furthermore, there is still room for improvement in local detail optimization and complex scene generation; future development could further enhance the generation quality by introducing local alignment loss. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1This is a schematic diagram illustrating the challenges of personalized image generation provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the overall framework provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the differences in the pseudo-word concept optimization process provided in the embodiments of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the mask-guided attention concentration effect provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the generation result of combining multiple personalized and general concepts in an embodiment of the present invention. Detailed Implementation

[0050] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0052] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0053] The personalized image generation method based on maximum difference anchoring and mask-guided attention provided in this embodiment includes the following steps:

[0054] S1 receives a user-provided set of custom concept images. This image set typically contains 3 to 5 images reflecting the target's personalized characteristics, with a consistent visual style and content composition, used to represent the user's unique visual concepts. This step is the starting point of the entire personalized image generation process, and its core purpose is to extract stable and recognizable user concepts from a very small number of samples, providing an accurate basic representation for subsequent pseudoword embedding optimization;

[0055] S2 utilizes the GPT-4o multimodal large model for efficient cross-modal joint analysis. Specifically, user-uploaded image samples and 80 common visual concept texts from the COCO dataset are simultaneously input into the visual-language model GPT-4o. Leveraging its sophisticated cross-modal attention mechanism, GPT-4o can deeply explore the latent semantic connections between different regions in the image and various textual concepts, outputting a set of the top 20 candidate concepts ranked by their association confidence with the input concepts. This process not only improves the relevance of candidate semantics, but also avoids the biases and ambiguities caused by traditional rule matching, laying a high-quality candidate space for the subsequent determination of semantic anchors;

[0056] S3, using CLIP-ViT / L-14 text encoder to Each concept is converted into a 768-dimensional embedding vector, thus constructing a high-dimensional semantic representation set that can be used for clustering. To improve the representativeness of the initial clustering points, the K-means algorithm is used to initialize the cluster centers, which is more effective than the traditional method of manually setting anchor points in avoiding excessively small semantic differences among the selected anchor points. By setting K=3 and using Euclidean distance as the metric, a strict convergence threshold is applied during the iteration process. This ensures the stability and interpretability of each clustering result. Finally, the concept closest to the cluster center in each cluster is selected as the final anchor point to form the set. This strategy ensures that pseudoword embeddings can maintain sufficient diversity and flexibility within the core regions of a general concept distribution, thereby avoiding the problem of individual feature degradation caused by a single semantic subdomain;

[0057] S4 introduces the Segment Anything Model (SAM) to achieve accurate target region segmentation of user images. It generates a pixel-level aligned binary mask M with zero samples, without requiring fine-tuning for specific objects. In this mask, the target region is labeled as 1, and the remaining background region is labeled as 0. This mask information plays a crucial guiding role in the subsequent attention mechanism, helping the model focus its attention on the target region during training, avoiding semantic interference from background regions on pseudoword embeddings, and improving the conceptual consistency and clarity of the final generated image.

[0058] S5, based on the pre-trained latent space structure of Stable Diffusion v1.5, implements a two-stage progressive optimization strategy for the pseudoword embedding 'e'. This strategy aims to decouple semantic anchoring and background suppression through staged regularization switching.

[0059] The first stage (≤4000 iterations) focuses on eliminating background interference in the encoding of personalized concept features. In this stage, the cross-attention map A (preserving latent variables) of the second layer of a U-Net model such as Stable Diffusion v1.5 is extracted. Consistent (Spatial dimension), combined with the binary mask of the target concept, calculate the background leakage suppression loss function, as follows:

[0060]

[0061] Here, ⊙ represents the element-wise product (Hadamard product). This loss function forces attention to focus on the target region, suppressing the influence of background interference on pseudo-word embedding encoding and solving the problem of distorted or unnatural combinations of general concepts caused by background information leakage. The mask is generated by the interactive segmentation tool SAM to ensure accurate coverage of the target region. In the early stages of training, the mask boundaries are allowed to be appropriately blurred to enhance the model's robustness to slight localization errors; as training progresses, the mask accuracy requirements are gradually tightened, forcing the model to learn more accurate feature representations. This mechanism effectively solves the "embedding drift" phenomenon caused by background regions or irrelevant objects interfering with pseudo-word embedding encoding in traditional methods, allowing the generated general concepts (such as computer, chair) to blend naturally with custom concepts, avoiding ambiguity or distortion. By setting... Corresponding weight coefficients The model is guided to focus its attention on the foreground region labeled by the mask, reducing false interference from background-irrelevant regions. This effectively alleviates the pollution of pseudo-word encoding by background information and improves the structural consistency and semantic clarity of the generated image. Furthermore, it incorporates reconstruction loss... ,by By assigning them higher weights, the optimization direction balances reconstruction accuracy and semantic consistency, achieving alignment with the general semantic space while preserving the expression of individual features. The total loss function in the first stage... for:

[0062]

[0063] The second phase (iterations > 4000) focuses on consolidating semantic anchoring constraints to prevent pseudo-word embeddings from drifting outside the core region. Specifically, a maximum difference anchor loss is introduced, with the following loss function:

[0064]

[0065] in, For pseudoword embedding vectors, Let be the embedding vector of the k-th anchor point; To relax the threshold, pseudo-word embeddings are allowed to optimize freely within the semantic region enclosed by the anchor points (K≥2; if K=3, then the three anchor points form a semantic triangle), avoiding excessive contraction that limits expressive power. In practical applications, it is usually set to... Experiments verified that this value can retain sufficient feature learning space while ensuring that pseudo-word embeddings do not deviate from the core distribution region. This design solves the problem in traditional semantic inversion (SI) methods where the optimization process of pseudo-word embeddings is limited to a small region due to fixed anchor concept constraints, enabling the generated image to more accurately retain the uniqueness of the user-specified concept, and combining this with reconstruction loss. ,by By assigning them higher weights, the optimization direction balances reconstruction accuracy and semantic consistency, achieving alignment with the general semantic space while preserving the expression of individual features. The total loss function in the second stage... for:

[0066]

[0067] in, For loss function Weighting coefficients;

[0068] S6 embeds the pseudowords obtained at the final convergence. The proposed method injects text into a Stable Diffusion-based text encoder as a conditional input to participate in the diffusion process, thereby generating image content that simultaneously integrates user-defined and general concepts. The generated images demonstrate significant improvements across several key metrics: a 4.3% reduction in Frechet Inception Distance (FID), a 5.2% improvement in CLIP-based text alignment, and a 3.1% improvement in image alignment. These improvements significantly address issues such as missing concepts, feature distortion, and background inconsistency found in traditional methods, demonstrating the significant advantages of the proposed method in generating personalized and general concept combinations.

[0069] Furthermore, the visual inference interface of GPT-4o described in S2 is built on a complete end-to-end multimodal architecture. In processing input images and text concepts, this architecture first uses a visual encoder (typically a VisionTransformer, ViT structure) to divide the user input image into multiple patches of equal size, denoted as... Each patch, after linear projection and positional encoding, forms a corresponding feature vector. Simultaneously, the text encoder will process each concept word in the COCO concept list. Convert to fixed-length word vector representation , where d represents the feature dimension.

[0070] Next, the semantic similarity between image patch features and text word vectors is calculated using a multi-layer cross-modal attention mechanism. Specifically, a cross-modal attention weight matrix is ​​defined. The expressions for the matrix elements are as follows:

[0071]

[0072] in, Indicates the first The patch and the The correlation between textual concept words This represents the vector dot product operation. and Let be a trainable linear projection matrix. The scaling factor ensures the stability of the dot product values, and the softmax function is used for normalization to obtain the probability distribution. This matrix... It expresses the correlation between each image patch and each text concept word.

[0073] Finally, based on the matrix The maximum or average confidence score of each text concept across all patches is calculated, and the top 20 concepts with the highest confidence scores are selected as the candidate concept set. This candidate concept set is highly representative, thus providing strong semantic support for subsequent anchor clustering and semantic parsing.

[0074] Furthermore, the K-means initialization strategy described in S3 optimizes the initial values ​​by maximizing the minimum distance between the initial cluster centers, significantly reducing cluster degradation caused by improper initial point selection. Specifically, assuming the data set to be clustered is... , The total number of data points to be clustered; the preset number of cluster centers is K. The initialization process first randomly selects a data point as the first cluster center. Subsequently, when determining the k-th cluster center, the calculation is first performed for each unselected data point. To the existing central set Shortest distance:

[0075]

[0076] Next, among all the unselected data points, the point corresponding to the maximum value of these shortest distances is selected as the new cluster center:

[0077]

[0078] This method ensures the maximum spacing between initial cluster centers, promoting diversity and coverage of center distribution. In the experiment, when K=3 was set, the three selected initial centers not only maximized the semantic differences between anchor points, but also effectively covered the semantic space boundary of the custom pseudoword embedding, thus ensuring the complementarity of anchor point semantics.

[0079] In subsequent clustering iterations, the traditional K-means update step is used, but a strict center drift error control condition is added to prevent the center points from jumping due to local perturbations. Specifically, the control condition is: if the maximum movement distance of all cluster centers in two consecutive iterations is less than or equal to the maximum movement distance of the center points, then the cluster center drift error is minimized.

[0080]

[0081] Then, clustering is determined to be convergent, where the convergence threshold is... Set as t represents the t-th iteration. This convergence criterion effectively avoids unstable jumps caused by noise or small perturbations, thus ensuring the stability of the anchoring structure and the reproducibility of the results during training.

[0082] Furthermore, the SAM segmentation process described in S4 uses a pre-trained ViT-H image encoder as its core architecture, and its multi-level feature extraction process can be formalized as follows:

[0083]

[0084] in, For the input image, Indicates the first of ViT-H Layer Transformer blocks, where L is the total number of Transformer block layers in ViT-H. For the corresponding layer feature map, Let C be the downsampling rate, and C be the number of samples taken. The number of output channels for each layer. This architecture achieves global context modeling through cascaded MSA (Multi-Head Self-Attention) modules, and the formula for calculating the elements of its attention weight matrix is ​​as follows:

[0085]

[0086] in, These represent the query vector and key vector, respectively, where d is the feature dimension and N is the total number of patches. T represents the transpose operation. This global attention mechanism enables the model to establish pixel-level long-range dependencies, maintaining semantic consistency of segmentation boundaries even for unseen object categories.

[0087] During the prompting and guidance phase, user-provided point / box prompts are encoded as location embeddings. U represents the number of prompts, which interact with image features through cross-attention:

[0088]

[0089] in, , , The features are those after linear projection. It is a linear projection matrix. This refers to the relevance after interaction. This mechanism enables the model to accurately locate the target region based on sparse user input.

[0090] The final output binary mask The following was obtained through pixel-by-pixel binary classification:

[0091]

[0092] in, coordinates The mask value; It is a lightweight mask decoder. For the Sigmoid function, This is an empirical threshold (default value is 0.5). This mask maintains a strict spatial correspondence with the input image, and its boundary error is controlled within... Within a pixel range, this provides a precise attention constraint region for subsequent pseudoword embedding training. Specifically, during the diffusion model training phase, this mask guides attention to the target region through background leakage suppression loss:

[0093]

[0094] Here, ⊙ represents element-wise product (Hadamard product).

[0095] Furthermore, the optimal parameter for the stage switching threshold n=4000 in S5 was obtained through rigorous verification using multiple sets of control experiments. This threshold was determined based on the following convergence criterion:

[0096]

[0097]

[0098] in, This is the gradient convergence threshold. The initial background leakage suppression loss function value. This represents the background leakage suppression loss function value at the nth iteration. The moving average curve of the background leakage suppression loss, where the empirical attenuation coefficient is , satisfies:

[0099]

[0100] in, Indicates the expectation. For the length of the window, Set a significance threshold (e.g., set the window length). Significance threshold At this point, the model completes a smooth transition from background suppression to semantic anchoring.

[0101] In terms of loss function design, a multi-objective weighted fusion strategy is adopted:

[0102]

[0103]

[0104] The specific implementation of each component is as follows:

[0105] (1) The reconstruction loss adopts a hybrid form of perceptual loss and pixel loss:

[0106]

[0107] in, The balancing coefficients are used, and VGG is a pre-trained feature extractor. To reconstruct the generated image, It is a real image;

[0108] (2) Background leakage suppression loss is achieved through attention masking:

[0109]

[0110] Here, ⊙ represents element-wise product (Hadamard product). Weighting coefficients are set... This guides the model to focus its attention on the foreground region labeled by the mask, reducing false interference from background-irrelevant regions, thereby effectively mitigating the pollution problem of background information on pseudo-word encoding and improving the structural consistency and semantic clarity of the generated image.

[0111] (3) The maximum differential anchor point loss includes two constraints:

[0112]

[0113] in, To relax the threshold, pseudoword embeddings are allowed to be freely optimized within the semantic region enclosed by anchor points, avoiding excessive contraction that would limit expressive power.

[0114] Furthermore, the cross-attention map A described in S5 is directly extracted from the second-layer cross-attention module of the U-Net decoder in Stable Diffusion v1.5. The computation process of this module can be formally represented as follows:

[0115]

[0116] in, as latent variables The linear projection; C is the number of output channels. Embedding text The mapping, The scaling factor determines the resolution of the output feature map. Strict alignment with the latent space. The selection of this level is based on the following quantitative indicators: alignment with the current latent variables. Maintaining a strict one-to-one correspondence in the spatial dimension, this layer's feature space has a spatial resolution of 64×64, making it the most sensitive location for concept alignment and suitable as a key layer for applying attention constraints. The attention map in this layer can effectively capture the semantic information of the target subject while suppressing interference from background regions. A mask-based background leakage suppression loss is introduced at this layer to guide attention to the target region.

[0117]

[0118] This mechanism can accurately suppress interference from irrelevant background regions during pseudoword embedding training, thereby improving the semantic coordination between general and personalized concepts and significantly improving the structural consistency and regional clarity in the generated images.

[0119] This invention also provides a personalized image generation system based on maximum difference anchoring and mask-guided attention, including a memory, a processor, and an executable computer program stored in the memory. When the processor runs the program, it can implement the aforementioned personalized image generation method based on maximum difference anchoring and mask-guided attention. The system uses the PyTorch framework to construct the core computational process, ensuring the modularity, scalability, and efficient training characteristics of the model. Simultaneously, by calling the Diffusers library provided by HuggingFace, the system can easily load the weights of the StableDiffusion v1.5 pre-trained model, achieving refined optimization of personalized pseudoword embedding without requiring full parameter fine-tuning. Furthermore, the system supports pipelined scheduling of the entire process, including mask generation, anchor point concept mining, and two-stage training, greatly improving the efficiency and practicality of personalized image generation tasks. This system, through its modular design, supports efficient training and deployment, and is suitable for various practical application scenarios such as personalized content creation and brand image customization.

[0120] In one implementation, the personalized image generation method includes the following steps: First, acquiring a personalized concept image and a general concept set provided by the user; second, generating candidate anchor concepts related to the user's concept using GPT-4o, and selecting the three anchor concepts with the greatest semantic differences using the K-means clustering algorithm; next, constructing a dynamic constraint framework including a background leakage suppression mechanism, and optimizing pseudo-word embeddings through a mask-guided attention mechanism; finally, using a two-stage training paradigm to progressively optimize the model's capabilities, achieving a natural fusion of personalized and general concepts. In specific implementation, the personalized concept image provided by the user must contain a clear target object, and the background should be as simple as possible to reduce interference; the general concept set is selected from over 80 categories in the COCO dataset to ensure broad semantic coverage. Using the personalized concept provided by the user and the COCO general concept set as input, GPT-4o generates 20 of the most relevant candidate anchor concepts. These candidate anchor concepts need to be manually verified to remove obviously irrelevant concepts, ensuring the accuracy of subsequent clustering analysis. In the K-means clustering process, the CLIP model is used to extract the semantic embedding vectors of these candidate anchor concepts. Euclidean distance is used as the similarity measure, and the candidate concepts are recursively divided into three clusters (K=3). The cluster centers are iteratively optimized until convergence, and finally, the concept closest to the cluster center in each cluster is selected as the anchor concept. These three anchor concepts exhibit maximum divergence in the semantic space, providing multi-directional constraints for pseudo-word embeddings, keeping them within the core distribution area of ​​general concepts, while retaining sufficient optimization freedom to learn the unique features of user concepts. The core idea of ​​this process comes from the principle of maximizing inter-cluster differences in the K-means algorithm. By dividing anchor concepts into semantic clusters and using spatial distance to select representative cluster centers as anchor concepts, the generalization problem caused by using semantically similar anchor concepts to constrain custom concepts is effectively alleviated. Experiments show that when K=3, the model achieves optimal performance on Image-Align, Text-Align, and FID metrics.

[0121] In one implementation, the two-stage training paradigm is divided into a background suppression stage and a semantic constraint stage, based on a dynamic regularization strategy. The first stage (number of iterations) , Jointly optimize image reconstruction loss Background leakage suppression loss Prioritize learning personalized conceptual features and suppress background interference; the second stage (number of iterations) Joint optimization Maximum difference anchor point loss This ensures that pseudoword embeddings are located within the core distribution region of the common concept. The total loss function is defined as:

[0122]

[0123] The weighting coefficient is set as follows: , , This ensures the stability of the optimization process and the accuracy of target focusing; The stage switching threshold was experimentally verified to significantly distinguish the target tasks of the two optimization stages. Through this dynamic adjustment of the optimization objective, the model gradually achieves a synergistic improvement in background interference suppression and semantic distribution alignment, forming a hierarchical regularized learning framework. During training, the Adam optimizer was used, with an initial learning rate of 0.01, which decayed to 0.9 times the original rate every 1000 iterations; the batch size was set to 16 to adapt to memory limitations and maintain training stability; the total number of training iterations was 6000. A linear warm-start strategy was introduced during stage switching, gradually increasing the learning rate. The weights are adjusted to avoid training oscillations caused by sudden changes in loss weights. This phased training strategy decouples the optimization objectives of background suppression and semantic consistency constraints, achieving more stable model optimization through gradual capability enhancement.

[0124] The personalized image generation method provided by this invention significantly improves the quality of generated images through the synergistic optimization of Anchor Concept Constraint (MDA) that maximizes semantic differences and Background Leakage Suppression (BLS). Quantitative experiments show that this method outperforms mainstream methods (such as Semantic Inversion, ELITE, and Hiper) in core metrics such as FID (4.3% improvement), Text-Align (5.2% improvement), and Image-Align (3.1% improvement). Compared to baseline methods, this method, when generating combined concepts such as "a [v] cat and computer" and "a [v] dog and chair," accurately preserves the uniqueness of user-specified concepts (such as the specific appearance of a cat / dog) while effectively avoiding the absence, distortion, or spatial artifacts at the junction of general concepts (such as computer and chair) and custom concepts (such as blurred keyboards and deformed cup handles). This solves the common problems of concept generalization (generated cats losing their individuality) and distortion of general concepts caused by background leakage in methods such as SI. User reviews further confirm that the average percentage of images generated by this method receiving "Good" or "Excellent" ratings reached 83.4%, significantly improving user satisfaction. Experiments show that when generating complex scenes, this method improves the naturalness of background-foreground blending by 37.5% and detail fidelity by 28.6%, significantly outperforming existing mainstream technologies. These results validate the effectiveness of MDA-Mask in addressing the two major technical bottlenecks of "conceptual gaps" and "conceptual generalization" in personalized image generation, providing a more reliable solution for personalized image generation.

[0125] Figure 1 This is a schematic diagram illustrating the challenges of personalized image generation provided in this embodiment of the invention. When the SI method based on the fixed anchor concept is used to generate a combination of special and general concepts, the model has insufficient representation of special concepts and causes distortion and inconsistency in the generation of general concepts.

[0126] Figure 2 This is a schematic diagram of the overall framework provided in an embodiment of the present invention. First, by inputting an image and a general concept file to the GPT-4o user, 20 general concepts strongly related to the input concept are obtained. Cluster analysis is then used to obtain three anchor concepts. Next, during training, Attention Maps obtained from the U-Net network are processed to obtain attention maps corresponding to the custom concepts. Finally, background leakage suppression loss, image reconstruction loss, and maximum difference anchor point loss are calculated to optimize the pseudo-word embedding.

[0127] Figure 3 This is a schematic diagram illustrating the differences in the pseudo-word concept optimization process provided in this embodiment of the invention. The method on the left (SI) is constrained by fixed anchor concepts that are close to each other in the semantic space. The optimization process of the pseudo-word concept is limited to a single direction and is ultimately restricted to a small range, which is not conducive to the model finding the optimal point for specific feature representation. The method provided in this embodiment (right) finds the anchor concept with the greatest semantic difference in the semantic space to constrain the pseudo-word concept. The degree of freedom of the pseudo-word concept optimization process is enhanced, which is conducive to the model finding the optimal point for specific feature representation.

[0128] Figure 4 This is a schematic diagram of the mask-guided attention concentration effect provided in the embodiment of the present invention; the specific testing method in this embodiment is as follows: given a specific concept image, the trained pseudo-word concept is integrated into the embedding vector space of the model, and the general concept is input into the diffusion model to obtain the test results based on the personalized image generation method proposed in this embodiment.

[0129] Figure 5 This is a schematic diagram illustrating the generation results of multiple personalized and general concepts in an embodiment of the present invention. The method proposed in this invention highlights the differences between the generation results output by different methods.

[0130] The results show that the method proposed in this invention is more competitive compared with other advanced methods.

[0131] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A personalized image generation method based on maximum difference anchoring and mask guided attention, characterized in that, include: S1 receives a set of custom concept images provided by the user to represent the user's unique visual concept; S2, input the image samples provided by the user and the general visual concept text in the general concept set into the multimodal large model, sort the output concept relevance confidence, and construct a candidate concept set; S3. Use a text encoder to convert each concept in the candidate concept set into an embedding vector to construct a high-dimensional semantic representation set that can be used for clustering; use the K-means algorithm to initialize the cluster centers, and apply a convergence threshold during the iteration process by setting the K value and using Euclidean distance as a metric to ensure the stability and interpretability of each clustering result; select the concept closest to the cluster center in each cluster as the anchor point to construct the anchor point set. S4 uses the SAM model to segment the target region of the user image and generate a pixel-level aligned binary mask. S5, based on the pre-trained latent space structure of Stable Diffusion, implements a two-stage progressive optimization strategy for pseudo-word embeddings. It achieves decoupling optimization of semantic anchoring and background suppression through staged regularization switching, specifically: The first stage focuses on eliminating background interference in the encoding of personalized concept features; in this stage, the cross attention map of the second layer of the U-Net model is extracted, and a background leakage suppression loss function is constructed. The second stage focuses on consolidating semantic anchoring constraints to prevent pseudo-word embeddings from drifting out of the core region; in this stage, the maximum difference anchor point loss is constructed to optimize the embedding distribution. S6 embeds the finally converged pseudo-words into the Stable Diffusion text encoder as conditional input to participate in the diffusion process, generating image content that simultaneously integrates user-defined concepts and general concepts.

2. The method of claim 1, wherein, The multimodal large model has an end-to-end Transformer architecture. Its visual encoder divides the input image into patches and encodes them into image features, while the text encoder encodes general visual concept text into word vector features. The two are semantically fused and matched through a multi-layer cross-modal attention mechanism. Finally, a probability distribution of candidate semantic words is generated through a softmax normalization operation, which is used to select the concept most relevant to the semantics of the user image to form a candidate concept set.

3. The method according to claim 1, characterized in that, In S3, the K-means initialization strategy optimizes the initial values ​​by maximizing the minimum distance between the initial cluster centers; assuming the data set to be clustered is... , N The total number of data points to be clustered, with a preset number of cluster centers. K The initialization process first randomly selects a data point as the first cluster center. Subsequently, the number was determined. k When calculating cluster centers, first calculate each unselected data point. To the current set of cluster centers shortest distance Then, among all the unselected data points, the point corresponding to the maximum value of these shortest distances is selected as the new cluster center. This method guarantees the maximum spacing between initial cluster centers, promoting the diversity and coverage of center distribution.

4. The method according to claim 3, characterized in that, In S3, K-means clustering is used K Setting =3 and using the K-means initialization strategy can improve the global divergence of anchor point distribution and avoid the concept shrinkage problem caused by pseudo-word embeddings being too close to a certain semantic region. By establishing a three-point anchoring region in the semantic space, a soft constraint mechanism is provided for pseudo-word embeddings to balance individual expression and semantic alignment. By maximizing the semantic differences between anchor points, pseudo-word embeddings can avoid deviating from the core distribution area of ​​general concepts while maintaining personalized features.

5. The method according to claim 1, characterized in that, In S4, the segmentation process of the SAM model uses a pre-trained ViT-H image encoder as the core architecture, and global context modeling is achieved through cascaded multi-head self-attention modules. In the prompt guidance stage, the point / box prompt provided by the user is encoded into a position embedding, which interacts with the image features through cross-attention; the final output is a binary mask M The mask obtained by pixel-by-pixel binary classification has a strict spatial correspondence with the input image, and the boundary error is controlled within a pixel range.

6. The method according to claim 1, characterized in that, In S5, the phase switching threshold n is determined based on the following convergence criterion: wherein, is a gradient convergence threshold, are background leakage suppression loss function values at initial time and at the n nth iteration, respectively, is an empirical decay coefficient, and the moving average curve of the background leakage suppression loss satisfies: wherein, is the window length, is a saliency threshold, at which a smooth transition from background suppression to semantic anchoring is accomplished.

7. The method according to claim 1, characterized in that, Total loss function for first stage in S5 is: in, To rebuild the losses, To mitigate the loss caused by background leakage, These are the weight coefficients of the corresponding loss function; The formula is as follows: in, A This is the cross-attention map of the second layer of the U-Net model. M This is a binary mask, where ⊙ represents element-wise product.

8. The method according to claim 1, characterized in that, The total loss function in the second stage of S5 for: in, To rebuild the losses, For the maximum difference anchor point loss, These are the weight coefficients of the corresponding loss function; The formula is as follows: in, For pseudoword embedding vectors, For the first k Embedding vectors of anchor points; To relax the threshold, pseudoword embeddings are allowed to be freely optimized within the semantic region enclosed by anchor points, avoiding excessive contraction that would limit expressive power.

9. The method according to claim 7, characterized in that, In S5, the cross-attention graph A It is extracted directly from the second-layer cross-attention module of the StableDiffusion U-Net decoder. Its spatial structure is consistent with the latent variables of the current time step, with a resolution of 64×64. It is the most sensitive position for concept alignment and is suitable as a key layer for applying attention constraints. The attention map of this layer can effectively capture the semantic information of the target subject while suppressing the interference of the background region.

10. A personalized image generation system based on maximum difference anchoring and mask-guided attention, comprising a memory, a processor, and an executable computer program stored in the memory, characterized in that, When the processor runs the program, it can implement the methods described in claims 1 to 9; the system uses the PyTorch framework to build the core computing process, ensuring the modularity, scalability, and efficient training characteristics of the model; at the same time, by calling the Diffusers library provided by HuggingFace, the system can conveniently load the weights of the Stable Diffusion pre-trained model, and achieve refined optimization of personalized pseudoword embedding without the need for full parameter fine-tuning; the system supports pipelined scheduling of the complete process of mask generation, anchor point concept mining, and two-stage training.

Images