Multi-subject personalized image generation method based on conditional diffusion model

Abstract

The invention discloses a multi-subject personalized image generation method based on a conditional diffusion model. The multi-subject personalized image generation method comprises the following nine steps: step 1, setting a diffusion model trunk and related parameters; 2, standardizing the image; step 3, extracting and enhancing main body identity features; step 4, extracting text embedding; step 5, carrying out cross-modal semantic fusion and enhancement; step 6, carrying out adaptive gating weight fusion and projection; step 7, calculating a spatial weight mapping score; 8, performing forward propagation and reverse propagation; step 9, sampling and generating an image; according to the invention, the main body detail reduction capability of the generated image is effectively improved; and in combination with deep cross-modal semantic fusion, the consistency of text description and visual content is enhanced. The method strictly follows the principle that fine adjustment is not needed, on the basis that pre-training model parameters are completely frozen, only a small number of external adaptive modules are optimized, efficient and lightweight multi-subject personalized image generation is achieved, and an advanced and feasible solution is provided for multi-subject image generation in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, and relates to a conditional image generation method, and more particularly to a multi-subject personalized image generation method based on a conditional diffusion model. Background Technology

[0002] Image generation is a crucial research area in computer vision technology, aiming to automatically generate high-quality images that meet user requirements based on user-provided intent information (such as text descriptions, reference images, etc.). In recent years, diffusion models (DMs), a breakthrough technology in this field, have significantly improved generation efficiency and image quality by simulating the noise addition and denoising processes in data distribution. DMs have been widely applied in creative design, advertising animation, virtual reality, and other scenarios, greatly lowering the barrier to entry for visual content creation and promoting the personalized and diversified development of visual expression.

[0003] Against this backdrop, multi-subject personalized image generation further expands the technological boundaries of image generation. Its focus is on generating semantically consistent, clearly defined, and logically arranged customized images based on multiple specific subjects (such as objects, people, or animals) and text descriptions input by the user. The Conditional Diffusion Model plays a crucial role here, guiding the model to maintain the identity features of each subject while coordinating the visual relationships between multiple subjects by introducing text or images as generation conditions. However, this task still faces several technical challenges: how to effectively avoid feature confusion between different subjects while ensuring generation efficiency, improve identity preservation capabilities, and enhance the consistency between the generated image and the text description have become key issues that urgently need to be addressed in current research.

[0004] Current multi-subject personalized image generation methods can be mainly divided into two categories: test-time fine-tuning-based methods and fine-tuning-free methods. Test-time fine-tuning-based methods can achieve extremely high subject fidelity by optimizing the base model parameters for each new subject. However, their main bottleneck lies in low efficiency. The fine-tuning process for new subjects usually consumes a lot of computational resources and time, severely limiting their widespread application in real-world scenarios. In addition, when merging multiple subjects, such methods are prone to subject confusion due to interference between model parameters. To overcome the efficiency bottleneck of fine-tuning methods, fine-tuning-free methods have become a research focus in recent years. These methods typically train a general visual encoder to embed the features of the reference image into the text or cross-attention layers of a pre-trained diffusion model, thereby achieving instantaneous personalized generation. Although these methods have significant advantages in generation efficiency, they also introduce new problems. On the one hand, some methods still require full-weight retraining of the base diffusion model to improve subject fidelity. This is not only computationally burdensome but may also damage the model's original strong prior knowledge, leading to a decrease in generation diversity and text compliance. On the other hand, when generating complex scenes containing multiple subjects, the lack of a clear generation guidance mechanism can easily lead to model attention being distracted, resulting in problems such as loss or confusion of subject features. Although some works have attempted to alleviate this problem by introducing external layout information (such as bounding boxes and segmentation maps) or designing attention constraint loss, this increases the user's input burden and may limit the creativity of the generated results.

[0005] To address the problems of the aforementioned fine-tuning-free methods, this invention discloses a multi-subject personalized image generation method based on a conditional diffusion model. This method follows the principle of no fine-tuning and aims to effectively solve problems such as subject feature confusion and insufficient semantic consistency in existing technologies. First, this method extracts fine-grained features from the input reference object using a subject identity enhancer. Then, the extracted global identity features are fused with text embeddings across modalities and semantically enhanced to form semantically consistent multimodal fusion features. Finally, an adaptive fusion mechanism integrates the identity features and multimodal features, injecting them as conditional information into the U-Net network to guide image generation. Furthermore, to solve the feature confusion problem in the multi-subject generation process, this invention designs a spatial weight mapping mechanism that explicitly constrains the semantic regions of different subjects, effectively resolving the subject feature confusion problem. Summary of the Invention

[0006] This invention aims to solve the above problems and proposes a multi-subject personalized image generation method based on a conditional diffusion model. By using methods such as subject identity enhancement, cross-modal semantic fusion and spatial weight mapping, personalized image generation for multiple reference subjects can be achieved without fine-tuning the parameters of the basic model, effectively overcoming the defects of subject feature confusion and insufficient semantic consistency.

[0007] To achieve the above objectives, a multi-agent personalized image generation method based on a conditional diffusion model is proposed, comprising the following nine steps:

[0008] Step 1. Set the backbone and relevant parameters of the diffusion model;

[0009] StableDiffusionXL (SDXL) U-Net is used as the pre-trained diffusion model backbone. The total diffusion time step T is set to 1000. Furthermore, all pre-trained parameters of SDXL are frozen to maintain its basic generative capabilities. Based on this, a novel, trainable image cross-attention layer is introduced to incorporate external conditional features without modifying the original cross-attention module and its parameters.

[0010] Step 2. Image standardization;

[0011] The input image resolution is uniformly adjusted to 224×224 pixels, and the pixel values ​​are normalized to conform to a normal distribution with a specific mean and standard deviation.

[0012] Step 3. Subject identity feature extraction and enhancement;

[0013] For multiple input reference subject images, instead of relying solely on the CLIP image encoder to extract global semantic embeddings, the DINOv2 image encoder is introduced, employing a multi-scale feature enhancement technique that fuses the two encoders to enhance the capture of local details and structural information. For example... Figure 3 The fine-grained features extracted by DINOv2 are processed by a multi-scale extraction and enhancement module to capture cross-scale visual information from local details to global structure. Subsequently, the global semantic features extracted by the CLIP encoder are cross-attentionally interacted with the fused multi-scale DINOv2 features, so that the global semantics guides the semantic alignment of local features, ultimately resulting in enhanced features for the subject's identity.

[0014] Step 4. Extract the text embedding;

[0015] For the input text prompts, the text embedding is obtained through the CLIP text encoder.

[0016] Step 5. Cross-modal semantic fusion and enhancement;

[0017] Enhance the interaction between embedded text information and global image features. For example... Figure 6This module first applies multi-head self-attention (MSA) to visual features to model the interaction relationships between subjects. Then, multi-head cross-attention (MCA) is used to achieve interactive alignment of text and image features. However, for scenarios with multiple entities and complex relationships, simple cross-attention mechanisms have limited representational capabilities and insufficient text consistency. This method extracts multi-level features from the initially obtained multi-modal features based on a multi-layer Transformer architecture, performing intra-level and cross-level enhancements respectively to address the semantic consistency problem in multi-subject generation.

[0018] Step 6. Adaptive gated weight fusion and projection;

[0019] By employing a gating mechanism, enhanced subject identity features are adaptively fused with multimodal features to obtain features that combine subject identity and textual semantics. Subsequently, the enhanced features are injected into the cross-attention layer of U-Net through a lightweight projection layer, serving as generation conditions to guide image generation.

[0020] Step 7. Calculate the spatial weight mapping score;

[0021] Cross-attention calculation is performed between the injected multimodal features and the latent space features Z to generate a spatial weight score matrix (the value range of matrix elements is [0,1]) for each subject. This matrix quantifies the correlation between the multimodal features and each position in the latent space. This method uses this spatial weight matrix to weight and fuse the features of each subject, thereby generating spatially constrained latent space features to ensure that different subjects are located in the expected regions in the generated image. The formula for calculating the spatial weight score matrix is ​​as follows:

[0022] ;

[0023] is the spatial weight mapping matrix for the i-th subject, with dimensions (image height H × image width W) and element values ​​ranging from [0,1]. It quantifies the correlation between each pixel position in the latent space and the subject (the larger the value, the stronger the correlation). L is the number of tokens for the multimodal features, a fixed value (determined by the output dimensions of the subject identity enhancement module and the cross-modal fusion module, such as a preset value of 4). j: the token index of the multimodal features, ranging from 1 to L, used to traverse the attention score of all multimodal tokens. Consistent with the above definition, the i-th subject's multimodal joint key matrix is ​​defined as follows: Q is the query matrix of latent space features; d is the hidden layer dimension hyperparameter; and Softmax represents the normalization of the attention score between each multimodal token and the latent space, ensuring that the weights of a single token sum to 1. The final spatial weight distribution is obtained by averaging the attention scores of all L tokens, thus smoothing out the differences in contribution between different tokens.

[0024] Step 8. Forward propagation, backward propagation;

[0025] This method optimizes only the newly added trainable parameters (such as the subject identity enhancement module, cross-modal interaction module, and image cross-attention layer), while keeping the parameters of the pre-trained diffusion backbone frozen. During training, the fusion conditional features and spatial weights are calculated via forward propagation and input into U-Net to perform denoising. Through backpropagation, the trainable components are optimized using prediction noise and mean squared error as losses, enabling the model to learn to accurately fuse the enhanced subject identity with textual semantics and generate a consistent image under spatial weight constraints.

[0026] Step 9. Sampling and image generation;

[0027] Based on the DDIM sampler, this method iteratively denoises from Gaussian noise. Each step utilizes conditional features that integrate subject identity and textual semantics, along with spatial weight maps, to guide U-Net, ultimately generating multi-subject images with faithful subject features.

[0028] Advantages of this invention:

[0029] In terms of preserving subject identity, this method effectively improves the ability to restore subject details in generated images through dual-encoder fusion and multi-scale feature enhancement. Furthermore, by combining deep cross-modal semantic fusion, it further strengthens the consistency between text description and visual content. Addressing the common feature confusion problem in multi-subject generation, the proposed spatial weight mapping mechanism can explicitly constrain the semantic regions of each subject, fundamentally avoiding feature interference. This method strictly adheres to the principle of no fine-tuning, optimizing only a small number of external adaptation modules while completely freezing the pre-trained model parameters. This achieves efficient and lightweight multi-subject personalized image generation, providing an advanced and feasible solution for multi-subject image generation in complex scenes. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of the present invention.

[0032] Figure 2 This is a diagram of the overall model architecture of the present invention.

[0033] Figure 3 This is a schematic diagram of the main body identity feature extraction and enhancement module.

[0034] Figure 4 This is a schematic diagram of multi-scale feature enhancement.

[0035] Figure 5 This is a schematic diagram of multi-scale fusion.

[0036] Figure 6 This is a schematic diagram of the cross-modal semantic fusion and enhancement module.

[0037] Figure 7 This is a schematic diagram of multi-level feature cross-enhancement. Detailed Implementation

[0038] The present invention will now be further described with reference to the accompanying drawings.

[0039] See attached document Figure 1 The implementation steps of this invention are as follows:

[0040] Step 1. Set the backbone and relevant parameters of the diffusion model;

[0041] This method uses StableDiffusionXL (SDXL) U-Net as the pre-trained diffusion model backbone, with a total diffusion time step T set to 1000. A linear noise scheduling strategy consistent with the pre-training stage is employed, with an initial noise variance of 0.0001 to maintain stability during the denoising process. During training, all pre-trained parameters of SDXLU-Net are frozen, including the weights and biases of convolutional layers, attention layers, and normalization layers. Building upon this, to incorporate external conditional features, a novel, trainable image cross-attention layer is introduced into all cross-attention modules of U-Net. This layer uses the intermediate latent features of U-Net as the query and the enhanced identity features extracted from the reference image as the key and value, achieving conditional injection through a multi-head cross-attention mechanism.

[0042] Step 2. Image standardization;

[0043] Each training dataset contains multiple reference images, a text description, and a target image. First, all reference and target images are uniformly scaled to a resolution of 224×224 pixels to fit the expected input size of the subsequent CLIP and DINOv2 image encoders. Then, the image pixel values ​​are standardized: they are converted from integers in the range [0,255] to floating-point numbers in the range [0,1], and normalized using the mean and standard deviation of the ImageNet dataset (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]).

[0044] Step 3. Subject identity feature extraction and enhancement;

[0045] For multiple input reference subject images, this method employs a dual encoder fusion strategy of CLIP and DINOv2 to perform fine-grained feature extraction and enhancement. The overall process is shown in the attached figure. Figure 3 As shown: First, fine-grained visual features are extracted using the DINOv2 encoder, and cross-scale information is captured using a multi-scale feature enhancement module; second, the enhanced features are adaptively fused using a feature fusion structure; finally, the fused local features are cross-attentioned with the global semantic features extracted by the CLIP encoder to output a highly enhanced subject feature representation of identity information.

[0046] The multi-scale enhancement module (Mult-ScaleFE) is responsible for extracting and enhancing features at different scales, as shown in the attached diagram. Figure 4 As shown, this module first generates learnable weight parameters α, multiplies the input features by α and 1-α respectively, and then sums them. Next, the weighted features are sequentially processed through average pooling (AvgPool), standard convolution (Conv), and two depthwise separable convolutions (DWConv) to obtain features enhanced at different scales.

[0047] Feature fusion structure (Fusion), such as Figure 5 As shown, a dual-branch parallel architecture is adopted: the upper branch first compresses the number of feature channels through 1×1 convolutions, then generates a spatial attention weight map through layer normalization and a sigmoid activation function, preserving high-frequency details in the original features. The lower branch downsamples the input features and extracts more generalized semantic context information through convolutional layers. Then, the attention weights generated by the upper branch are multiplied element-wise with the context features provided by the lower branch. Finally, the fused features are added to the original input of the module through residual connections, outputting multi-scale enhanced features with rich details.

[0048] Step 4. Extract the text embedding;

[0049] For the input text prompts, a text tokenizer is first used to convert them into a series of discrete tokens. These tokens are then fed into a pre-trained CLIP text encoder. This encoder performs deep semantic modeling on the input token sequence through a multi-layer self-attention mechanism, ultimately outputting a high-dimensional text embedding vector rich in semantic information.

[0050] Step 5. Cross-modal semantic fusion and enhancement;

[0051] Since using only image features containing subject identity to guide image generation can lead to insufficient text consistency, we enhance text consistency by performing cross-modal interaction and enhancement of text embedding information and global image features, as shown in the attached figure. Figure 6 As shown, the process begins with a preliminary interaction between visual features and text embeddings: visual features are first fed into a multi-head self-attention module (MSA) to model the intrinsic interaction relationships between multiple reference subjects, strengthening the structural connections between them. Text features undergo independent mapping processing. Subsequently, both are fed into a multi-head cross-attention module (MCA), where text features serve as the query and visual features as the key and value, achieving text-guided preliminary feature alignment.

[0052] The multimodal features following the initial interaction are input in parallel into three structurally identical Transformer enhancement blocks. Each enhancement block employs a cascaded structure of "Layer Normalization (LN) - Multi-Head Self-Attention (MSA) - Residual Connection" to perform in-depth enhancement on the features within each layer. Then, the features from the three different layers enter a cross-layer enhancement module, as shown in the appendix. Figure 7 As shown, information is first aggregated through feature concatenation and linear transformation; then further fused using layer normalization (LN) and multilayer perceptron (MLP); and finally, deep interaction is achieved through multiple sets of parallel multi-head cross-attention (MCA) modules. Finally, the enhanced features from all levels are fused to output a conditional embedding that combines identity details and semantic consistency. This progressive architecture, from primary interaction to intra-level enhancement and then to cross-level fusion, can significantly improve the semantic alignment quality in complex multi-agent scenarios.

[0053] Step 6. Adaptive gated weight fusion and projection;

[0054] This step effectively fuses enhanced subject identity features (rich in visual details) with cross-modal semantic features (containing textual semantics), and processes them into conditional signals suitable for guiding U-Net generation through a lightweight projection module. For adaptive gating weights, a gating weight vector α between 0 and 1 is first generated. This weight vector determines the proportion of subject identity features and textual semantic features in the final fused features, as shown in the following formula:

[0055] ;

[0056] The fused features need to pass through a lightweight projection module before they can be injected into the cross-attention layer of U-Net, such as... Figure 7 As shown, this projection module consists of a linear layer, a GELU activation function, another linear layer, and a layer normalization component connected in series to form a standard feedforward neural network structure. The output is reshaped into a fixed feature sequence with the number of tokens set to 4, and the dimension will be aligned with the text dimension to facilitate cross-attention calculation.

[0057] Step 7. Calculate the spatial weight mapping score;

[0058] For text-guided image generation, during the forward propagation, each of U-Net's cross-attention layers performs cross-attention calculations: latent space features serve as the query, and text embeddings serve as the key / value pair. As shown in the following formula:

[0059] ;

[0060] in, The Query projection of the latent space Z is obtained by transforming the latent space features through trainable projection weights, and its dimension is (number of latent space tokens × hidden layer dimension d). and Key / Value Projection of Text Embedding

[0061] d is the dimension of the hidden layer, and Softmax is the normalized activation function that maps the attention score to the interval [0,1].

[0062] For image generation with a reference image, a decoupled cross-attention mechanism is often used. This involves introducing a new image cross-attention layer for cross-attention calculation: latent space features serve as the query, and image embeddings serve as the key / value pair. The result is then calculated... As shown in the following formula:

[0063] ;

[0064] in, The Query projection of the latent space Z is obtained by transforming the latent space features through trainable projection weights, and its dimension is (number of latent space tokens × hidden layer dimension d).

[0065] d is the dimension of the hidden layer, and Softmax is the normalized activation function that maps the attention score to the interval [0,1].

[0066] Next, regarding the above... The features are directly added together to obtain the final latent space feature Z of this layer, which is used for subsequent image generation. The formula is as follows:

[0067] ;

[0068] The direct addition method described above often leads to subject confusion. This method introduces a spatial weighting score matrix to guide the weighted fusion of images at corresponding spatial locations, effectively solving this problem. First, a spatial weighting score matrix is ​​calculated using the multimodal features and latent space features Z of each input entity, where... Every A point represents the degree of relevance between this location and the entity, as shown in the following formula:

[0069] ;

[0070] This represents the multimodal joint key matrix of the i-th subject, used for subsequent spatial weight mapping calculation, with a dimension of (number of multimodal feature tokens × d). The multimodal fusion feature vector of the i-th subject is obtained by adaptive gating fusion of subject identity enhancement features and cross-modal semantic fusion features, with a dimension of (number of multimodal feature tokens × original feature dimension). The Key projection weight matrix represents the multimodal features. It consists of trainable parameters with dimensions (original feature dimension × d), and its function is to... Mapping to Dimensionally consistent attention computation space

[0071] ;

[0072] The Query matrix, representing latent space features, is used to calculate the spatial attention score in conjunction with the multimodal joint Key matrix. Its dimension is (number of latent space feature tokens × d). This represents the latent space feature vector in the diffusion model, which is an intermediate output of the U-Net network, with a dimension of (number of latent space feature tokens × original latent space dimension). Let Z be the Query projection weight matrix of the latent space features, and let Z be the trainable parameters with dimensions (original latent space dimension × d), used to map Z to the attention computation space.

[0073] ;

[0074] is the spatial weight mapping matrix for the i-th subject, with dimensions (image height H × image width W) and element values ​​ranging from [0,1]. It quantifies the correlation between each pixel position in the latent space and the subject (the larger the value, the stronger the correlation). L is the number of tokens for the multimodal features, a fixed value (determined by the output dimensions of the subject identity enhancement module and the cross-modal fusion module, such as a preset value of 4). j: the token index of the multimodal features, ranging from 1 to L, used to traverse the attention score of all multimodal tokens. Consistent with the above definition, the i-th subject's multimodal joint key matrix is ​​defined as follows: Q is the query matrix of latent space features; d is the hidden layer dimension hyperparameter; and Softmax represents the normalization of the attention score between each multimodal token and the latent space, ensuring that the weights of a single token sum to 1. The final spatial weight distribution is obtained by averaging the attention scores of all L tokens, thus smoothing out the differences in contribution between different tokens.

[0075] Finally, the image features are fused based on the spatial weight mapping matrix to obtain the final latent space feature Z, as shown in the following formula:

[0076]

[0077] The final latent space feature vector is derived by fusing textual semantics, subject visual features, and spatial weight constraints, with dimensions and... Consistency is the core feature that guides image generation. Consistent with the previous formula definition, this is the intermediate feature vector guided by text semantics. M: The number of input reference subjects, a dynamic parameter (determined by the number of reference images input by the user, such as 2, 3, etc.). i: Subject index, ranging from 1 to M, used to traverse all input subjects. Consistent with the previous formula definition, this is the spatial weight mapping matrix for the i-th subject. HW: The total number of pixels in the image (H=224, W=224), a fixed value (determined by the image normalization step). p: The pixel position index of the spatial weight mapping matrix, ranging from 1 to HW, used to traverse all pixels in the matrix. This represents the global mean of the spatial weight matrix of the i-th subject, used to normalize the weight matrix and avoid fusion imbalance caused by differences in weight scale among different subjects. The element-wise multiplication operation multiplies the normalized spatial weight matrix with the image feature vector of the i-th subject element by element, thereby achieving the constraint that the subject features are activated only in the corresponding spatial region. The image feature vector of the i-th subject.

[0078] Step 8. Forward propagation, backward propagation;

[0079] The entire forward propagation process is shown in the attached figure. Figure 2 As shown. This method only optimizes the newly added trainable parameters (including the subject identity enhancement module, cross-modal interaction module, image cross-attention layer, etc.), while keeping the pre-trained diffusion backbone parameters frozen to maintain generation quality and reduce computational overhead. During training, the forward propagation stage first calculates the fused image conditional features and spatial weight mapping score matrix, and inputs it into U-Net to perform the denoising process. In the backpropagation stage, the mean squared error (MSE) loss function is used for optimization, and the loss formula is as follows:

[0080]

[0081] Where ε is the actual noise. For the predicted noise, Let be the noise latent feature at time step t, and c be the fusion condition feature.

[0082] Step 9. Sample and generate an image;

[0083] The sampling process begins with Gaussian noise and iteratively executes denoising steps. In each denoising step, conditional features incorporating subject identity and textual semantics, along with the generated spatial weight map, guide the U-Net generation process. The spatial weight map ensures that each subject feature is activated within the expected spatial region, effectively avoiding mutual interference between features. Finally, after a complete denoising process (using the DDIM sampler), a multi-subject image that accurately reflects the textual description and preserves the subject features is generated.

[0084] Experiments and Analysis

[0085] 1. Experimental conditions;

[0086] This invention was carried out in the following hardware and software environment: the hardware platform uses an Intel Xeon E5-2678 CPU, an NVIDIA RTX 3090 GPU, and 32GB of memory; the software is based on the Ubuntu 18.04 operating system, using the Python programming language and the PyTorch deep learning framework.

[0087] 2. Experimental data;

[0088] This invention uses the large-scale dataset SA-1B as the training set. Released by Meta, this dataset is specifically designed for general image segmentation tasks and contains over 1 billion mask annotations, covering approximately 11 million images. To balance training efficiency and data quality, 100,000 images were selected as training samples, and the Concept101 dataset was chosen as the benchmark set to evaluate the model's generalization ability. During training, the batch size was set to 16, the AdamW optimizer was used, the initial learning rate was 1e-4, and a total of 150,000 training steps were performed. A text- and image-independent drop-off strategy with a probability of 0.05, as well as a joint drop-off strategy with equal probability, were used during training to support classifier-independent guidance. The DDIM sampler was used during the inference phase to ensure the stability of the generation process.

[0089] 3. Performance comparison;

[0090] To demonstrate the effectiveness of the algorithm, and considering both its popularity and novelty, this invention selects five methods for comparison. The comparison algorithms are from:

[0091] (1) Kumari's experimental results are from the literature Kumari, Nupur, et al. "Multi-concept customization of text-to-image diffusion." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2023.

[0092] (2) Ruiz’s experimental results are from the literature Ruiz, Nataniel, et al. "Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2023.

[0093] (3) Xiao's experimental results are from the literature Xiao, Guangxuan, et al. "Fastcomposer: Tuning-free multi-subject image generation with localized attention." International Journal of Computer Vision 133.3 (2025): 1175-1194.

[0094] (4) Ye's experimental results are from the literature Ye, Hu, et al. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprint arXiv:2308.06721 (2023).

[0095] (5) Zhang's experimental results are from the literature Zhang, Yuxuan, et al. "Ssr-encoder: Encoding selective subject representation for subject-driven generation." Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2024.

[0096] Testing indicators:

[0097] This method uses three evaluation metrics—CLIP-I, CLIP-T, and DINO—to measure the quality of the generated images. CLIP-T assesses the semantic consistency between the image content and the text description by calculating the cosine similarity between the generated image and the input text prompt in the CLIP joint embedding space. DINO assesses the fidelity of the generated image to the identity features of the reference image by comparing the similarity between the generated image and the reference image in the DINO model feature space. CLIP-I assesses the overall visual consistency between the generated image and the reference image by calculating the similarity between the two images in the CLIP image embedding space.

[0098] The experimental comparison results are shown in Table 1:

[0099] Table 1 Experimental results of the method disclosed in this invention

[0100]

[0101] Table 1 shows the experimental results of the five algorithms and the algorithm of this invention on the Concept101 dataset. The method of this invention achieves CLIP-I and DINO metrics of 0.6979 and 0.4177 respectively, outperforming other models and indicating that the generated images and reference images produced by this method have good visual consistency. The method also performs well in the CLIP-T metric of 0.7540, second only to the CustomDiffusion method's 0.7599.

[0102] In summary, this invention proposes a multi-subject personalized image generation method based on a conditional diffusion model. This method significantly improves the preservation of subject details in the generated images through dual-encoder fusion and multi-scale feature enhancement techniques. Furthermore, by combining deep cross-modal semantic fusion, it effectively strengthens the consistency between textual and visual content. Addressing the common feature confusion problem in multi-subject generation, the designed spatial weight mapping mechanism explicitly constrains the semantic regions of each subject, fundamentally solving the feature confusion problem. Experimental results show that this method achieves good results on several key metrics in the Concept101 benchmark test and can generate multi-subject images with high visual quality and clear subject relationships.

Claims

1. A multi-subject personalized image generation method based on a conditional diffusion model, characterized in that, It includes the following 9 steps: Step 1: Set the backbone and relevant parameters of the diffusion model; U-Net from SDXL was used as the backbone of the pre-trained diffusion model; The total diffusion time step T is set to 1000; and all pre-trained parameters of SDXL are frozen to maintain its basic generative capabilities; a trainable image cross-attention layer is introduced to incorporate external conditional features without modifying the original cross-attention module and its parameters. Step 2, image standardization; The resolution of the input image is uniformly adjusted to 224×224 pixels, and the pixel values ​​are normalized to conform to a normal distribution with a specific mean and standard deviation. Step 3, Subject identity feature extraction and enhancement; For multiple input reference subject images, the DINOv2 image encoder is introduced. Multi-scale feature enhancement by fusing dual encoders is used to enhance the capture of local details and structural information. The fine-grained features extracted by DINOv2 are then processed by a multi-scale extraction and enhancement module to capture cross-scale visual information from local details to global structure. Subsequently, the global semantic features extracted by the CLIP encoder are cross-attentionally interacted with the fused multi-scale DINOv2 features, so that the global semantics guides the semantic alignment of local features, and finally the enhanced features of subject identity are obtained. Step 4, extract the text embedding; For the input text prompts, the text embedding is obtained through the CLIP text encoder; Step 5, cross-modal semantic fusion and enhancement; The interaction between embedded text information and global image features is enhanced. First, multi-head self-attention (MSA) is applied to visual features to model the interaction between subjects. Then, multi-head cross-attention (MCA) is used to achieve interactive alignment of text and image features. For the initially obtained multimodal features, multi-level features are extracted based on a multi-level Transformer architecture, and intra-level enhancement and cross-level enhancement are performed respectively to solve the semantic consistency problem in multi-agent generation. Step 6, Adaptive gated weight fusion and projection; By using a gating mechanism, enhanced subject identity features and multimodal features are adaptively fused to obtain features that combine subject identity and textual semantics. Subsequently, the enhanced features are injected into the cross-attention layer of U-Net through a lightweight projection layer as generation conditions to guide image generation. Step 7, Spatial weight mapping score calculation; Cross-attention calculation is performed between the injected multimodal features and the latent space features Z to generate a spatial weight score matrix for each subject. This matrix quantifies the correlation between the multimodal features and each location in the latent space. The spatial weight score matrix weights and fuses the features of each subject. Step 8: Forward propagation and backward propagation; During training, the fusion conditional features and spatial weights are computed via forward propagation and input into U-Net to perform the denoising process; through backpropagation, the trainable components are optimized using prediction noise and mean square error as losses; under spatial weight constraints, the enhanced subject identity and text semantics are accurately fused to generate a consistent image. Step 9: Sampling and image generation; Based on the DDIM sampler, iterative denoising starts from Gaussian noise. At each step, U-Net is guided by conditional features that fuse subject identity and text semantics, as well as spatial weight maps, ultimately generating multi-subject images with preserved subject features.

2. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, In step 1, during training, all pre-trained parameters of SDXLU-Net are frozen, including the weights and biases of convolutional layers, attention layers, and normalization layers. The image cross-attention layer uses the intermediate latent features of U-Net as the query and the enhanced identity features extracted from the reference image as the key and value, and implements conditional injection through a multi-head cross-attention mechanism.

3. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, In step 2, all reference and target images are first scaled to a resolution of 224×224 pixels to fit the expected input size of the subsequent CLIP and DINOv2 image encoders. Then, the image pixel values ​​are standardized: they are converted from integers in the range of [0,255] to floating-point numbers in the range of [0,1], and normalized using the mean and standard deviation of the ImageNet dataset.

4. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, Step 3, subject identity feature extraction and enhancement, includes: For multiple input reference subject images, a strategy of fusing CLIP and DINOv2 dual encoders is adopted to perform fine-grained feature extraction and enhancement. First, fine-grained visual features are extracted by the DINOv2 encoder, and cross-scale information is captured using a multi-scale feature enhancement module. Second, the enhanced features are adaptively fused through a feature fusion structure. Finally, the fused local features are cross-attentioned with the global semantic features extracted by the CLIP encoder to output a subject feature representation with highly enhanced identity information. The multi-scale enhancement module is responsible for extracting and enhancing features at different scales. First, it generates learnable weight parameters α, multiplies the input features by α and 1-α respectively, and then adds them together. Then, the weighted features are subjected to average pooling, standard convolution, and two depthwise separable convolution operations in sequence to obtain features enhanced at different scales. The feature fusion structure Fusion adopts a dual-branch parallel architecture: the upper branch first compresses the number of feature channels through 1×1 convolution, then generates a spatial attention weight map through layer normalization and a sigmoid activation function, preserving high-frequency details in the original features; the lower branch downsamples the input features and extracts more generalized semantic context information through convolutional layers; then, the attention weights generated by the upper branch are multiplied element-wise with the context features provided by the lower branch; finally, the fused features are added to the original input of the module through residual connections, outputting multi-scale enhanced features with rich details.

5. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, Step 4: For the input text prompts, a text segmenter is first used to convert them into a series of discrete tokens. The discrete tokens are then fed into a pre-trained CLIP text encoder. The CLIP text encoder performs deep semantic modeling on the input token sequence through a multi-layer self-attention mechanism, and finally outputs a high-dimensional text embedding vector rich in semantic information.

6. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, In step 5, the visual features and text embeddings first undergo a preliminary interaction: the visual features are first input into the multi-head self-attention module MSA to model the intrinsic interaction relationship between multiple reference subjects and strengthen the structural association between subjects; the text features undergo independent mapping processing; both are then input into the multi-head cross-attention module MCA, where the text features serve as the query and the visual features serve as the key and value, to achieve text-guided preliminary feature alignment. After the initial interaction, the multimodal features are input in parallel into three structurally identical Transformer enhancement blocks. Each enhancement block employs a cascaded structure of layer normalized LN, multi-head self-attention MSA, and residual connections to perform in-depth enhancement of the features within each layer. Then, the features from the three different layers enter the cross-layer enhancement module. First, information is aggregated through feature concatenation and linear transformation. Then, it is further fused through layer normalized LN and multilayer perceptron MLP. Next, in-depth interaction is achieved through multiple sets of parallel multi-head cross-attention MCA modules. Finally, the enhanced features from all layers are fused to output a conditional embedding that combines identity details and semantic consistency.

7. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, Step 6 effectively fuses the enhanced subject identity features with cross-modal semantic features, and processes them into a conditional signal suitable for guiding U-Net generation through a lightweight projection module. For adaptive gating weights, a gating weight vector α between 0 and 1 is first generated. This weight vector determines the proportion of subject identity features and textual semantic features in the final fused features, as shown in the following formula: ； The fused features are injected into the cross-attention layer of U-Net through a lightweight projection module. This projection module consists of a linear layer, a GELU activation function, another linear layer, and a layer normalization component connected in series to form a standard feedforward neural network structure. The output is reshaped into a fixed feature sequence.

8. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, Step 7: For text-guided image generation, during the forward propagation, the cross-attention layers of U-Net perform cross-attention calculations: latent space features are used as the query, and text embeddings are used as the key / value pairs. The calculated results are... As shown in the following formula: ； in, The Query projection of the latent space Z is obtained by transforming the latent space features through trainable projection weights, and its dimension is: number of latent space tokens × hidden layer dimension d; d is the hidden layer dimension, and Softmax is the normalized activation function that maps the attention score to the [0,1] interval; For image generation with a reference image, a decoupled cross-attention mechanism is often used. This involves introducing a new image cross-attention layer for cross-attention calculation: latent space features serve as the query, and image embeddings serve as the key / value pair. The result is then calculated... As shown in the following formula: ； in, The Query projection of the latent space Z is obtained by transforming the latent space features through trainable projection weights, with the dimension being: number of latent space tokens × hidden layer dimension d. d is the hidden layer dimension, and Softmax is the normalized activation function that maps the attention score to the [0,1] interval; right The features are directly added together to obtain the final latent space feature Z of this layer, which is used for subsequent image generation. The formula is as follows: ； A spatial weighting score matrix is ​​introduced to guide the weighted fusion of images at corresponding spatial locations. First, a spatial weighting score matrix is ​​calculated using the multimodal features and latent space features Z of each input entity. Every A point represents the degree of relevance between this location and the entity, as shown in the following formula: ； The multimodal joint key matrix of the i-th subject is used for subsequent spatial weight mapping calculation, and its dimension is: number of multimodal feature tokens × d; The multimodal fusion feature vector of the i-th subject is obtained by adaptive gating fusion of subject identity enhancement features and cross-modal semantic fusion features, with the dimension being: number of multimodal feature tokens × original feature dimension; The Key projection weight matrix is ​​a multimodal feature matrix, which is a trainable parameter with dimensions of: original feature dimension × d. Its function is to... Mapping to A dimensionally consistent attention computation space: ； The Query matrix, representing latent space features, is used to calculate the spatial attention score in conjunction with the multimodal joint Key matrix. Its dimension is: number of latent space feature tokens × d. This represents the latent space feature vector in the diffusion model, which is an intermediate output of the U-Net network. Its dimension is: number of latent space feature tokens × original latent space dimension. is the Query projection weight matrix for latent space features, and is a trainable parameter with the dimension of: original latent space dimension × d, used to map Z to the attention computation space; ； is the spatial weight mapping matrix for the i-th subject, with dimension L and element value range [0,1], quantizing the correlation between each pixel position in the latent space and the subject; L is the number of tokens for the multimodal features; j is the token index of the multimodal features, with a value range from 1 to L, used to traverse the attention score of all multimodal tokens. Let be the multimodal joint key matrix of the i-th subject; Q be the query matrix of latent space features; d be the hidden layer dimension hyperparameter; Softmax represents the normalization of the attention score of each multimodal token and the latent space to ensure that the weight sum of a single token is 1; The final spatial weight distribution is obtained by averaging the attention scores of all L tokens, which smooths out the differences in contribution between different tokens. Finally, the image features are fused based on the spatial weight mapping matrix to obtain the final latent space feature Z, as shown in the following formula: ； The final latent space feature vector is derived by fusing textual semantics, subject visual features, and spatial weight constraints, with dimensions and... Consistency is a core feature that guides image generation; is the intermediate feature vector guided by text semantics; M: the number of reference subjects in the input, which is a dynamic parameter; i: subject index, with a value ranging from 1 to M, used to traverse all input subjects; Consistent with the previous formula definition, the spatial weight mapping matrix of the i-th subject; HW: the total number of pixels in the image, which is a fixed value; p: the pixel position index of the spatial weight mapping matrix, which ranges from 1 to HW and is used to traverse all pixels of the matrix; This represents the global mean of the spatial weight matrix of the i-th subject, used to normalize the weight matrix and avoid fusion imbalance caused by differences in the weight scale of different subjects; The element-wise multiplication operation multiplies the normalized spatial weight matrix with the image feature vector of the i-th subject element by element, thereby constraining the subject features to be activated only in the corresponding spatial region. The image feature vector of the i-th subject.

9. The multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, In step 8 of the training process, the forward propagation phase first calculates the fused image conditional features and spatial weight mapping score matrix, and inputs it into U-Net to perform the denoising process; in the backpropagation phase, the mean squared error (MSE) loss function is used for optimization, and the loss formula is as follows: ； Where ε is the actual noise. For the predicted noise, Let be the noise latent feature at time step t, and c be the fusion condition feature.

10. A multi-subject personalized image generation method based on a conditional diffusion model according to claim 1, characterized in that, The sampling process in step 9 starts with Gaussian noise and iteratively executes the denoising steps. In each denoising step, the conditional features of subject identity and text semantics are fused together with the generated spatial weight map to guide the generation process of U-Net. The spatial weight map ensures that each subject feature is activated in the expected spatial region and avoids mutual interference between features. After denoising, a multi-subject image that conforms to the text description and preserves the subject features is generated.

Images