Multi-concept personalized image generation method and electronic device

Abstract

The application relates to the technical field of artificial intelligence image generation, and specifically discloses a multi-concept personalized image generation method and an electronic device, which comprises the following steps: acquiring a reference image, a subject region mask and an image diffusion model of a pre-trained text of a target subject; constructing a decomposed concept identifier for the target subject, and dividing a text portal of the target subject into category semantic identifiers and subject exclusive identifiers; performing diffusion training on the reference image according to the decomposed concept identifier to obtain a diffusion training main loss; constructing concept focus supervision and background suppression supervision; performing optimization training on the image diffusion model of the pre-trained text to obtain a standardized single-concept module; performing multi-concept fusion training according to a plurality of standardized single-concept modules to obtain a unified multi-concept model; and inputting a combined prompt word into the unified multi-concept model to obtain a personalized image. The multi-concept personalized image generation method provided by the application can improve the generation quality and stability of personalized images.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence image generation technology, and in particular to a multi-concept personalized image generation method and electronic device. Background Technology

[0002] In recent years, diffusion models have made significant progress in text-to-image generation tasks. Latent diffusion models such as Stable Diffusion can generate high-quality images based on natural language prompts and have been widely applied in digital content production, brand visual design, e-commerce displays, interactive creation, and intelligent design assistance. While general text-to-image models can generate image content at the category level, when users expect the model to consistently generate images of a specific person, pet, item, or brand, relying solely on ordinary category terms is insufficient to maintain the target subject's identity characteristics, local texture, color distribution, and cross-scene consistency. Therefore, personalized image generation for specific subjects has become an important technological direction connecting general image generation models with the needs of real-world content production.

[0003] Existing personalized image generation methods mainly include text embedding learning, subject-driven fine-tuning, and low-rank parameter adaptation. Text embedding learning methods typically only update placeholder embeddings, resulting in low training overhead, but they struggle to fully capture the fine-grained appearance information of complex subjects. Subject-driven fine-tuning methods can maintain subject consistency well, but their training and storage costs are high. Low-rank adaptation methods reduce training overhead by adding lightweight low-rank increments to the model weights, facilitating the preservation and reuse of single-concept modules. However, in few-shot personalized training, the target subject often appears simultaneously with a fixed background, shooting environment, lighting conditions, and locally co-occurring elements. Without explicit constraints on parameter writing paths and spatial response ranges, subject appearance, category semantics, and background statistics can easily be co-written into the same adaptation space, leading to impure single-concept module representations and a decreased ability to maintain subjects across scenes.

[0004] Furthermore, real-world applications often require multiple personalized subjects, accessories, products, or environmental elements to coexist stably within the same image. For example, advertising design typically requires characters, products, and scenes to appear together, and interactive creation often demands that several reference concepts be flexibly combined into the same scene. Existing multi-concept personalized generation methods usually integrate multiple single-concept modules into a unified model through a central-end fusion approach. However, multiple concept modules may have potential overlaps in semantic representation space and parameter update paths, and direct fusion can easily lead to problems such as identity mixing, concept crosstalk, local attribute misalignment, object missingness, and unclear subject boundaries.

[0005] Therefore, how to provide a personalized image generation method that simultaneously improves the learning quality of single-concept modules and the stability of multi-concept fusion has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a method and electronic device for generating personalized images with multiple concepts, which solves the problems of impure representation of single-concept modules and poor quality of generated personalized images caused by concept crosstalk in multi-concept fusion in related technologies.

[0007] As a first aspect of the present invention, a multi-concept personalized image generation method is provided, comprising:

[0008] Obtain the reference image of the target subject, the subject region mask, and the image diffusion model of the pre-trained text;

[0009] A decompositional conceptual identifier is constructed for the target entity, and the text entry of the target entity is divided into category semantic identifiers and entity-specific identifiers;

[0010] The diffusion training master loss is obtained by performing diffusion training on the reference image based on the decomposed concept identifier;

[0011] Conceptual focus supervision is constructed based on the main region mask, and background suppression supervision is constructed based on the complementary region of the main region mask;

[0012] The image diffusion model of the pre-trained text is optimized and trained based on the diffusion training master loss, concept focusing supervision and background suppression supervision to obtain a standardized single concept module.

[0013] A unified multi-concept model is obtained by performing multi-concept fusion training based on multiple standardized single-concept modules.

[0014] The combined prompts obtained in real time are input into the unified multi-concept model to obtain personalized images including multiple target subjects.

[0015] Furthermore, the image diffusion model for the pre-trained text includes at least a text encoder and a denoising U-Net, and performs diffusion training on the reference image based on the decomposed concept identifier, including:

[0016] A hierarchical concept embedding is constructed in the text encoder so that the target subject has a corresponding concept embedding representation in different layers of the text encoder. The hierarchical concept embedding includes shallow concept embedding and deep concept embedding. The shallow concept embedding is used to store word-level and local semantic information, and the deep concept embedding is used to store high-level semantic aggregation information.

[0017] A selective parameter update space is constructed in the denoised U-Net, wherein the parameters corresponding to the key branches and value branches in the cross-attention module of the denoised U-Net are retained to participate in low-rank adaptation, and the parameters corresponding to the query branches in the cross-attention module of the denoised U-Net are frozen.

[0018] Furthermore, the diffusion training of the reference image based on the decomposed concept identifier also includes:

[0019] The reference image is subjected to noise processing to obtain noisy latent variables;

[0020] The decomposed concept identifier and the hierarchical concept are embedded and input into the text encoder to obtain text conditional features;

[0021] The text conditional features and the noisy latent variables are input into the denoising U-Net to obtain noise prediction results;

[0022] The diffusion training master loss is calculated based on the difference between the noise prediction result and the actual noise.

[0023] Furthermore, conceptual focus supervision is constructed based on the main region mask, and background suppression supervision is constructed based on the complementary regions of the main region mask, including:

[0024] Extract the attention response map corresponding to the target subject from the cross-attention weight map of the denoised U-Net;

[0025] The attention response maps corresponding to multiple concept identifier locations of the same target subject are averaged, and the attention response maps at different spatial scales are aligned to a uniform resolution and then fused to obtain the subject concept attention map.

[0026] The loss value of the concept focus supervision is calculated based on the difference between the subject concept attention map and the subject region mask;

[0027] The loss value of the background suppression supervision is calculated based on the response intensity of the subject concept attention map on the complementary region of the subject region mask.

[0028] Furthermore, the image diffusion model of the pre-trained text is optimized and trained based on the diffusion training master loss, concept focusing supervision, and background suppression supervision to obtain a standardized single-concept module, including:

[0029] The overall optimization objective is obtained by weighted fusion of the main loss of diffusion training, the loss value of concept focusing supervision, and the loss value of background suppression supervision.

[0030] Based on the overall optimization objective, the low-rank parameters of the key branch and value branch in the hierarchical concept embedding and the cross-attention module of the denoising U-Net are iteratively optimized;

[0031] The standardized single-concept module is obtained by associating the iteratively optimized parameters with the decomposed concept identifier.

[0032] Furthermore, multi-concept fusion training is performed based on multiple standardized single-concept modules, including:

[0033] Semantic representation features and parameter update features are extracted for each standardized single-concept module;

[0034] Calculate the compatibility relationship between standardized single-concept modules based on the semantic representation features and the parameter update features;

[0035] The weighting coefficients for each standardized single-concept module are determined based on the compatibility relationships between them.

[0036] The standardized single-concept modules are weighted and fused according to the weighting coefficients of each standardized single-concept module to obtain the unified multi-concept model.

[0037] Furthermore, the compatibility relationship between standardized single-concept modules is calculated based on the semantic representation features and the parameter update features, including:

[0038] For each standardized single-concept module, semantic representation features are extracted from the hierarchical concept embedding, and parameter update features are extracted from the low-rank parameter increments of the key and value branches in the cross-attention module of the denoised U-Net.

[0039] For any two standardized single-concept modules, calculate the semantic similarity between their semantic representation features and the parameter update similarity between their parameter update features.

[0040] The semantic similarity and the parameter update similarity are weighted and summed to obtain a comprehensive relationship score;

[0041] The module compatibility matrix is ​​obtained after normalizing the comprehensive relationship scores.

[0042] Furthermore, the weighting coefficients for each standardized single-concept module are determined based on the compatibility relationships between them, including:

[0043] Calculate the average conflict intensity between each standardized single-concept module and the other standardized single-concept modules based on the module compatibility relationship matrix, and map the average conflict intensity to a concept-level global scaling factor.

[0044] In the cross-attention module of the denoising U-Net, the key branch and value branch, as well as the text encoder, extract the intra-layer output features of each standardized single-concept module in each layer, and calculate the intra-layer local conflict intensity and map it to the layer local scaling factor.

[0045] The compatibility weighting coefficient of the corresponding standardized single-concept module is obtained based on the concept-level global scaling factor and the hierarchical local scaling factor.

[0046] Further, the standardized single-concept modules are weighted and fused according to the weighting coefficients of each standardized single-concept module to obtain the unified multi-concept model, including:

[0047] Based on the compatibility weighting coefficient, the input features and target output of the corresponding standardized single-concept module in each layer are scaled to obtain weighted sample pairs;

[0048] The weighted sample pairs are input into the gradient fusion solver to reconstruct the key and value branches in the cross-attention module of the denoising U-Net and the unified model parameters of the text encoder layer by layer.

[0049] The reconstructed unified model parameters are written back to the image diffusion model of the pre-trained text, and the unified multi-concept model is obtained by combining the identifier binding configuration of each standardized single-concept module.

[0050] As another aspect of the present invention, an electronic device is provided, comprising:

[0051] Memory, used to store program instructions;

[0052] The processor, communicatively connected to the memory, is used to call and execute the program instructions to implement the multi-concept personalized image generation method as described above.

[0053] The multi-concept personalized image generation method provided by this invention assigns category semantics and subject identity to independent text entry points through decompositional concept labeling. Layered concept embedding preserves word-level and local semantic information at the shallow level, while deeper layers handle high-level semantic aggregation. Selective parameter update space allows subject appearance information to be written into the model along paths directly related to text conditional injection. Concept-focused supervision and background suppression supervision reduce the mixing of background texture and shooting environment information, thereby obtaining standardized single-concept modules that are pure in representation, have clear boundaries, and are easy to reuse. In the multi-concept fusion stage, the compatibility relationships between modules are explicitly estimated, and the fusion intensity is adaptively allocated at different concepts, branches, and levels, thereby improving the subject fidelity, concept boundary clarity, and combined generation stability of the unified multi-concept model. Therefore, this multi-concept personalized image generation method can simultaneously improve the learning quality of single-concept modules and the stability of multi-concept fusion, thus enhancing the quality of personalized image generation. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0055] Figure 1 The flowchart illustrates the multi-concept personalized image generation method provided by this invention.

[0056] Figure 2 This is a flowchart for performing diffusion training on a reference image, as provided by the present invention.

[0057] Figure 3 This is a schematic diagram illustrating the specific implementation architecture of the multi-concept personalized image generation method provided by the present invention.

[0058] Figure 4 The flowcharts for the construction concept focus supervision and construction background suppression supervision provided by this invention are shown.

[0059] Figure 5 This is a flowchart illustrating the optimization training of an image diffusion model for pre-trained text, as provided in this invention.

[0060] Figure 6 The flowchart for multi-concept fusion training provided by this invention.

[0061] Figure 7 This is a schematic diagram of the compatibility-aware multi-concept fusion process provided by the present invention.

[0062] Figure 8 A schematic diagram of the concept response constraints for concept-focused supervision and background suppression supervision provided by this invention.

[0063] Figure 9 This is a structural block diagram of the electronic device provided by the present invention. Detailed Implementation

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0066] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0067] This embodiment provides a method for generating multi-concept personalized images. Figure 1 This is a flowchart of a multi-concept personalized image generation method provided according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0068] S100. Obtain the reference image of the target subject, the subject region mask, and the image diffusion model of the pre-trained text.

[0069] Specifically, a small set of reference images of the target subject, a set of corresponding text descriptions, a set of subject region masks, and a pre-trained image diffusion model of the text are obtained. The pre-trained image diffusion model of the text includes a variational autoencoder, a text encoder, and a denoising U-Net. The variational autoencoder maps the image to a low-dimensional latent space and restores it to an image during final decoding; the text encoder encodes text prompts into text conditional features; and the denoising U-Net performs diffusion denoising in the latent space.

[0070] S200. Construct a decompositional concept identifier for the target subject, and divide the text entry of the target subject into category semantic identifier and subject-specific identifier;

[0071] Specifically, a decompositional conceptual identifier is constructed for the target subject, dividing the text entry into a category semantic identifier and a subject-specific identifier. The category semantic identifier is used to inherit the general prior of the category to which the target subject belongs, and the subject-specific identifier is used to characterize the identity features that distinguish the subject from other samples in the same category.

[0072] S300. Perform diffusion training on the reference image based on the decomposed concept identifier to obtain the diffusion training master loss;

[0073] Specifically, after the reference image is input into a variational autoencoder and mapped to the latent space, noise is added to obtain noisy latent variables. The decomposed concept identifiers and the hierarchical concepts are embedded into an input text encoder to obtain text conditional features. The text conditional features and the noisy latent variables are input into the denoising U-Net to obtain noise prediction results. The diffusion loss is calculated based on the difference between the noise prediction results and the actual noise, and is used as the main loss for diffusion training.

[0074] S400. Construct concept focus supervision based on the main region mask, and construct background suppression supervision based on the complementary region of the main region mask;

[0075] Specifically, concept focus supervision is constructed based on the main region mask to increase the proportion of concept response within the main region, so that the attention of the main concept is concentrated in the main region as much as possible; background suppression supervision is constructed based on the complementary region of the main region mask to reduce the intensity of concept response on the background region, and reduce the number of background textures, shooting environment and co-occurring elements that are jointly written into the single concept module.

[0076] S500. The image diffusion model of the pre-trained text is optimized and trained according to the diffusion training master loss, concept focusing supervision and background suppression supervision to obtain a standardized single concept module.

[0077] Specifically, the joint diffusion training master loss, concept focusing supervision, and background suppression supervision optimize the low-rank parameters of the text encoder, hierarchical concept embedding, and low-rank parameters of the U-Net key-value branch, and output a standardized single-concept module after training.

[0078] S600: Perform multi-concept fusion training based on multiple standardized single-concept modules to obtain a unified multi-concept model;

[0079] Specifically, when it is necessary to generate a multi-concept image, multiple trained standardized single-concept modules are read, features are extracted from the semantic representation and parameter update of each module, and compatibility relationships are calculated. After determining the weighting coefficients, weighted fusion is performed, and unified model parameters are reconstructed layer by layer to obtain a unified multi-concept model.

[0080] S700: Input the combined prompts obtained in real time into the unified multi-concept model to obtain a personalized image including multiple target subjects.

[0081] Specifically, the parameters of each branch obtained by fusion are written back to the image diffusion model of the pre-trained text, and a unified multi-concept model is obtained by combining the multi-concept identifier binding configuration. The combined prompt words are input into the unified multi-concept model to generate a personalized image containing multiple target subjects and maintaining clear concept boundaries.

[0082] Therefore, the multi-concept personalized image generation method provided by this invention assigns category semantics and subject identity to independent text entry points through decompositional concept labeling; it preserves word-level and local semantic information at the shallow level and undertakes high-level semantic aggregation at the deep level through hierarchical concept embedding; it writes subject appearance information into the model along paths directly related to text conditional injection through selective parameter update space; and it reduces the mixing of background texture and shooting environment information through concept focusing supervision and background suppression supervision, thereby obtaining standardized single-concept modules with pure representation, clear boundaries, and easy reusability. In the multi-concept fusion stage, it explicitly estimates the compatibility relationships between modules and adaptively allocates fusion intensity at different concepts, branches, and levels, thereby improving the subject fidelity, concept boundary clarity, and combined generation stability of the multi-concept unified model. Therefore, this multi-concept personalized image generation method can simultaneously improve the learning quality of single-concept modules and the stability of multi-concept fusion, thus improving the quality of personalized image generation.

[0083] In this embodiment of the invention, the image diffusion model of the pre-trained text includes at least a text encoder and a denoising U-Net, and the reference image is trained for diffusion based on the decomposed concept identifier, such as... Figure 2 As shown, it includes:

[0084] S310. Construct hierarchical concept embeddings in the text encoder so that the target subject has corresponding concept embedding representations in different layers of the text encoder. The hierarchical concept embeddings include shallow concept embeddings and deep concept embeddings. The shallow concept embeddings are used to store word-level and local semantic information, and the deep concept embeddings are used to store high-level semantic aggregation information.

[0085] In embodiments of the present invention, such as Figure 3 As shown, in the single-concept training phase, a small number of target subject reference images, text descriptions, and subject region masks are used as input. First, a decomposed concept identifier is constructed for the target subject, so that category semantics and subject identity each have independent entry points.

[0086] It should be understood that hierarchical concept embeddings are constructed on the text encoder side, and corresponding concept embedding representations are initialized in different layers of the text encoder. This allows shallow concept embeddings to preserve word-level and local semantic information, while deep concept embeddings preserve high-level semantic aggregation information. The low-rank adaptation parameters on the text encoder side work together with the hierarchical concept embeddings to maintain the stability of the subject concept conditional expression.

[0087] S320. Construct a selective parameter update space in the denoised U-Net, wherein the parameters corresponding to the key branches and value branches in the cross-attention module of the denoised U-Net are retained to participate in low-rank adaptation, and the parameters corresponding to the query branches in the cross-attention module of the denoised U-Net are frozen.

[0088] In embodiments of the present invention, such as Figure 3 As shown, a selective parameter update space is constructed on the U-Net side. Specifically, in the cross-attention module, image features generate query vectors, and text conditions generate key and value vectors. Updating the query branch directly changes the matching method of visual features, potentially causing significant perturbations to the pre-trained priors; the key branch participates in the construction of the text semantic matching benchmark; and the value branch determines how the matched semantic information is written into the image representation. Therefore, this invention only retains the parameters corresponding to the key and value branches in the cross-attention module for low-rank adaptation, freezing the parameters corresponding to the query branch, allowing the subject's appearance information to be written into the model along a path more directly related to text conditional injection.

[0089] Furthermore, the diffusion training of the reference image based on the decomposed concept identifier also includes:

[0090] 1) Noise is added to the reference image to obtain noisy latent variables;

[0091] 2) Embed the decomposed concept identifier and the hierarchical concept into the text encoder to obtain text conditional features;

[0092] 3) Input the text conditional features and the noisy latent variables into the denoising U-Net to obtain noise prediction results;

[0093] 4) Calculate the diffusion training master loss based on the difference between the noise prediction result and the actual noise.

[0094] Therefore, in the single-concept training iteration of this embodiment of the invention, a diffusion training process is performed on the reference image to construct noisy latent variables, and decomposed concept identifiers and hierarchical concept embeddings are used as conditional inputs to the diffusion model to obtain noise prediction results and calculate the diffusion training master loss.

[0095] In this embodiment of the invention, conceptual focus supervision is constructed based on the main region mask, and background suppression supervision is constructed based on the complementary regions of the main region mask, such as... Figure 4 As shown, it includes:

[0096] S410. Extract the attention response map corresponding to the target subject from the cross-attention weight map of the denoised U-Net;

[0097] Specifically, on the training constraint side, the attention response map corresponding to the target subject is extracted from the cross-attention weight map of the denoised U-Net.

[0098] like Figure 3As shown, the specific process of obtaining the cross-attention weight map is as follows: In the forward inference process of the denoising U-Net, image features are linearly mapped to generate a query vector Q, and text conditional features are linearly mapped to generate a key vector K and a value vector V. This is achieved through QK... T The attention score is calculated and normalized by Softmax to obtain the cross-attention weight map of each layer, from which the attention response map corresponding to the target subject is extracted.

[0099] S420. Average the attention response maps corresponding to multiple concept identifier positions of the same target subject, and then merge the attention response maps of different spatial scales after aligning them to a uniform resolution to obtain the subject concept attention map.

[0100] S430. Calculate the loss value of the concept focus supervision based on the difference between the subject concept attention map and the subject region mask;

[0101] Specifically, the expression for the concept-focused supervision loss is:

[0102] ,

[0103] Where H and W represent the height and width of the subject concept attention map, respectively, and ε represents a preset positive number to prevent the denominator from being zero. This concept focus supervision loss represents the relative proportion of concept responses outside the subject region among all concept responses. When the concept responses corresponding to the target subject are more concentrated within the subject region, Reduce; when the concept response is dispersed outside the main region, Increase. When a training batch includes multiple training samples, the concept-focused supervision loss can be further expressed as:

[0104] ,

[0105] Where B represents the number of samples in the training batch.

[0106] S440. Calculate the background suppression supervision loss value based on the response intensity of the subject concept attention map on the complementary region of the subject region mask.

[0107] Specifically, the background suppression supervision loss corresponding to target subject i can be expressed as:

[0108] ,

[0109] in, Represents the complementary region of the main region mask. A conceptual response indicating a location outside the main area.

[0110] In this embodiment of the invention, based on the subject concept attention map and the subject region mask, a concept focus supervision loss is constructed to increase the proportion of concept responses within the subject region; at the same time, a background suppression supervision loss is constructed to reduce the background responses outside the subject region.

[0111] In this embodiment of the invention, the image diffusion model of the pre-trained text is optimized and trained based on the diffusion training master loss, concept focusing supervision, and background suppression supervision to obtain a standardized single-concept module, such as... Figure 5 As shown, it includes:

[0112] S510. The overall optimization objective is obtained by weighted fusion of the main loss of diffusion training, the loss value of concept focusing supervision, and the loss value of background suppression supervision.

[0113] In this embodiment of the invention, the overall optimization objective of single-concept training is expressed as the formula:

[0114] ,

[0115] in, Indicates the main loss during diffusion training. The concept focuses on loss. This represents background suppression loss. The weighting coefficients represent the concept-focused loss. This represents the weighting coefficients of the background suppression loss.

[0116] S520. Based on the overall optimization objective, iteratively optimize the low-rank parameters of the key branch and value branch in the cross-attention module of the hierarchical concept embedding and the denoising U-Net;

[0117] S530. The standardized single-concept module is obtained by associating the iteratively optimized parameters with the decomposed concept identifier.

[0118] In this embodiment of the invention, the standardized single-concept module includes decomposed concept identification, hierarchical concept embedding, low-rank parameter increments on the text encoder side, and low-rank parameter increments of the key and value branches in the cross-attention module of the denoising U-Net. This module can be used for single-concept generation and can also be used as input for the subsequent multi-concept fusion stage.

[0119] Specifically, a single-concept module can be represented as: ,in, Represents a decomposable set of conceptual identifiers. This indicates that the hierarchical concept is embedded in the set. This represents the increment of the low-rank parameter on the text encoder side. This represents the increment of the low-rank parameters on the U-Net side.

[0120] It should be noted that the expression for updating the weights of the low-rank adaptation layer is:

[0121] ,

[0122] in, Indicates pre-trained weights, and Both represent low-rank matrices.

[0123] In this embodiment of the invention, multi-concept fusion training is performed based on multiple standardized single-concept modules, such as... Figure 6 As shown, it includes:

[0124] S610. Extract semantic representation features and parameter update features for each standardized single-concept module;

[0125] In embodiments of the present invention, such as Figure 7 As shown, in the multi-concept fusion stage, multiple trained standardized single-concept modules are used as input. First, the central unit reads the corresponding module according to the set of concepts to be fused, completes the multi-concept identifier expansion and binding, so that each concept has an independent identifier entry in the unified vocabulary.

[0126] It should be noted that the central end here can be understood as the multi-concept fusion processing end that performs multi-concept module reading, compatibility relationship calculation, weighted sample construction, and unified model parameter reconstruction. It can be deployed in the processor of a server, workstation, or other electronic device, and does not specifically refer to a single network node.

[0127] like Figure 7 As shown, to characterize the compatibility relationships between modules, two types of global features are extracted for each standardized single-concept module: the first type is semantic representation features, extracted from the hierarchical concept embeddings of the standardized single-concept module, reflecting the proximity of different concepts in the text conditional space. The second type is parameter update features, extracted from the low-rank parameter increments of the key and value branches of the standardized single-concept module in the cross-attention module of the denoising U-Net, reflecting the update direction of different concepts on the conditional injection path.

[0128] S620. Calculate the compatibility relationship between standardized single-concept modules based on the semantic representation features and the parameter update features;

[0129] Specifically, the compatibility relationship between standardized single-concept modules is calculated based on the semantic representation features and the parameter update features, including:

[0130] (1) For each standardized single concept module, extract semantic representation features from the hierarchical concept embedding and extract parameter update features from the low-rank parameter increments of the key branch and value branch in the cross attention module of the denoised U-Net.

[0131] (2) For any two standardized single-concept modules, calculate the semantic similarity between their semantic representation features and the parameter update similarity between their parameter update features;

[0132] Specifically, the semantic similarity between any two standardized single-concept modules i and j can be expressed as:

[0133] ,

[0134] Parameter update similarity can be represented as:

[0135] ,

[0136] in, Let L be the norm 2, and ε be a preset positive number used to ensure computational stability. (3) The semantic similarity and the parameter update similarity are weighted and summed to obtain a comprehensive relationship score;

[0137] In this embodiment of the invention, the comprehensive relationship score between any two conceptual modules can be expressed as:

[0138] ,

[0139] in, Indicates semantic representation features, Indicates parameter update features, It represents the balance coefficient between semantic representation features and parameter update features.

[0140] (4) After normalizing the comprehensive relationship score, the module compatibility relationship matrix is ​​obtained.

[0141] In this embodiment of the invention, the formulas for calculating pairwise conflict intensity and concept-level global scaling factor are expressed as follows:

[0142] ,

[0143] in, This indicates the intensity of conflict between concept i and concept j. Indicates the score of the comprehensive relationship. Indicates the global conflict suppression strength. Represents the concept-level global scaling factor for concept i.

[0144] S630. Determine the weighting coefficient of each standardized single-concept module based on the compatibility relationship between the standardized single-concept modules;

[0145] Specifically, the weighting coefficients for each standardized single-concept module are determined based on the compatibility relationships between them, including:

[0146] (1) Calculate the average conflict intensity between each standardized single concept module and the other standardized single concept modules based on the module compatibility relationship matrix, and map the average conflict intensity to a concept-level global scaling factor;

[0147] It should be understood that the higher the average conflict intensity, the easier it is for the concept to compete and interfere with other concepts when they are integrated, and the smaller its concept-level global scaling factor; the lower the average conflict intensity, the better its coexistence basis in the current combination, and the larger its concept-level global scaling factor.

[0148] (2) In the key branch and value branch of the cross attention module of the denoising U-Net and in the text encoder, extract the output features of each standardized single concept module in the layer of the layer, calculate the local conflict intensity in the layer and map it to the local scaling factor of the layer.

[0149] (3) Obtain the compatibility weighting coefficient of the corresponding standardized single concept module based on the concept-level global scaling factor and the hierarchical local scaling factor.

[0150] In this embodiment of the invention, the concept-level global scaling factor is multiplied by the hierarchical local scaling factor to obtain the compatibility weighting coefficient of the corresponding standardized single concept module.

[0151] Specifically, the corresponding concept is in the branch. The Layer compatibility weighting coefficient It can be represented as:

[0152] ,

[0153] in, Represents the conceptual-level global scaling factor. This represents the local scaling factor for the hierarchy.

[0154] To be more specific, , where N represents the number of standardized single-concept modules participating in the fusion, and γ represents the hierarchical local conflict suppression coefficient.

[0155] S640. The standardized single-concept modules are weighted and fused according to the weighting coefficient of each standardized single-concept module to obtain the unified multi-concept model.

[0156] Specifically, the standardized single-concept modules are weighted and fused according to the weighting coefficients of each standardized single-concept module to obtain the unified multi-concept model, including:

[0157] (1) Based on the compatibility weighting coefficient, scale the input features and target output of the corresponding standardized single-concept module in each layer to obtain weighted sample pairs;

[0158] (2) Input the weighted sample pairs into the gradient fusion solver to reconstruct the key branches and value branches in the cross attention module of the denoising U-Net and the unified model parameters of the text encoder layer by layer;

[0159] It should be understood that, such as Figure 8 As shown, during the gradient fusion solution process at the center end, compatibility weighting coefficients are used to scale the input features and target output of the corresponding standardized single-concept module in each layer to obtain weighted sample pairs. By adjusting the input features and target output of the corresponding concept in that layer, the supervision strength of that concept in the current layer can be directly adjusted, making the layer-by-layer parameter reconstruction process of the unified model rely more on the stability constraints provided by low-conflict concepts, while reducing the interference of high-conflict concepts in high-conflict layers. The weighted sample pairs are input to the gradient fusion solver to reconstruct the key and value branches in the cross-attention module of the denoising U-Net and the unified model parameters of the text encoder layer by layer.

[0160] Specifically, no. The fusion solution of the unified parameters of the layers can be expressed as:

[0161] ,

[0162] in, and They represent the first The concept in the first The layer's input features and target output, This represents the compatibility weighting coefficient.

[0163] (3) Write the reconstructed unified model parameters back to the image diffusion model of the pre-trained text, and obtain the unified multi-concept model by combining the identifier binding configuration of each standardized single-concept module.

[0164] In summary, the multi-concept personalized image generation method provided by this invention improves the personalized image generation effect of the diffusion model from two levels: single-concept training and multi-concept fusion. First, by using decomposed concept labeling, hierarchical concept embedding, and low-rank adaptation on the text encoder side, the conditional expression of the subject concept can be stably organized. Through selective updating of the key-value branch on the U-Net side, the coupling between subject appearance, category semantics, and background statistics in the parameter space can be weakened. Through concept focusing supervision and background suppression supervision, the response of the subject region can be enhanced while suppressing background interference, thereby obtaining a standardized single-concept module with pure representation, clear boundaries, and easy reusability. Second, by jointly combining semantic representation and parameter update feature estimation module compatibility, and performing hierarchical weighted fusion in the text encoding branch and the U-Net cross-attention key-value branch, identity mixing, concept crosstalk, and local attribute misalignment in multi-concept unified modeling can be reduced, improving the subject fidelity, concept boundary clarity, and generation stability of multi-concept combined generation, ultimately achieving improved stability and clarity of the personalized image generation results.

[0165] To verify the effectiveness of the method of this invention, experiments were conducted using two personalized image-generated public evaluation datasets: DreamBooth and CustomConcept101. The DreamBooth dataset contains 30 concepts covering 15 categories, including live animal categories such as dogs and cats, as well as various object categories. Each concept provides 4 to 6 reference images and test prompts. The CustomConcept101 dataset contains 101 concepts covering 16 different categories. Each concept provides 3 to 12 reference images and test prompts. Further multi-concept experiments involved compiling 30 category combinations from the DreamBooth dataset and 50 category combinations from the CustomConcept101 dataset, with each combination accompanied by 20 test prompts.

[0166] The experiments used CLIP-T, CLIP-I, and DINO as evaluation metrics. CLIP-T measures the semantic matching between the generated image and the input text prompt, reflecting the model's responsiveness to textual conditions. CLIP-I measures the similarity between the generated image and the reference image in the CLIP image feature space, reflecting the ability to preserve the subject's appearance and local details. DINO measures the semantic similarity between the generated image and the reference image in the DINO visual feature space, focusing more on the ability to preserve the subject's contour structure and high-level semantic features. For the multi-concept generation task, CLIP-I and DINO respectively calculate the similarity between the generated image and the reference image set for each target concept, and then average the results for each target concept.

[0167] Stable Diffusion 1.5 was selected as the pre-trained base diffusion model for the experiment. During the single-concept training phase, the concept focusing supervision coefficient was set to 0.1, and the background suppression supervision coefficient was set to 0.05. On the U-Net side, only the cross-attention key and value branches underwent low-rank updates, and the learning rate for the cross-attention part was set to... The learning rate for the text encoder is set to... The learning rate for the text embedding part is set to The batch size was set to 2, the training steps were 3000, and the input image size was 512×512. In the multi-concept fusion stage, the text encoder optimization steps were set to 500, and the U-Net optimization steps were set to 50. The semantic and parameter update balance coefficient in compatibility-aware fusion was set to 0.5, the global conflict suppression coefficient was set to 0.3, and the hierarchical conflict suppression coefficient was set to 0.15. In the inference stage, the DDIM sampler was used with 50 sampling steps, and the classifier-free guidance coefficient was set to 7.5.

[0168] The specific experimental results are as follows:

[0169] Table 1. Comparative Experiments on Single-Concept Image Generation on the DreamBooth Dataset

[0170] As shown in Table 1, on the DreamBooth dataset, the method of this invention achieves the best results in both DINO and CLIP-I subject-related metrics, reaching 0.554 and 0.760, respectively. Compared with the basic ED-LoRA, DINO improves from 0.523 to 0.554, and CLIP-I improves from 0.748 to 0.760, indicating that the proposed selective parameter update space and concept decoupling constraint can effectively improve the single-concept subject preservation capability.

[0171] Table 2 Comparison Experiments on Single-Concept Image Generation on the CustomConcept101 Dataset

[0172] As shown in Table 2, on the CustomConcept101 dataset with a richer concept distribution, the method of this invention also achieves the best DINO and CLIP-I results, at 0.556 and 0.747, respectively. Compared with ED-LoRA, both DINO and CLIP-I show significant improvements, indicating that this invention can maintain good consistency in the main structure and appearance under complex concept categories.

[0173] Table 3 Selective parameter update space ablation experiment

[0174] Table 3 shows that while updating only the Value branch can write personalized appearance information, it lacks sufficient ability to adjust semantic matching relationships; a complete update of the settings will expand the coupling range of U-Net side parameters. Joint updates of Key and Value cover both conditional matching and semantic writing, achieving a more stable balance between maintaining the subject and text consistency.

[0175] Table 4 Training Constraint Ablation Experiment

[0176] Table 4 shows that after adding concept-focused supervision and background suppression supervision to the key-value joint update, both DINO and CLIP-I are further improved, with KV+Focus+BG achieving the best subject relevance index. This indicates that concept-focused supervision can promote the concentration of responses in the subject region, and background suppression supervision can reduce irrelevant activations in the background region. The combination of the two can improve the purity of single-concept module representation.

[0177] Table 5 Comparative experiments on multi-concept image generation on the DreamBooth dataset

[0178] Table 5 shows that in the DreamBooth multi-concept combination task, the CLIP-I of the method of this invention reaches 0.705, which is a significant improvement compared to 0.689 of Mix-of-Show. DINO and CLIP-T are also better than Mix-of-Show, indicating that compatibility-aware fusion can alleviate module conflicts in multi-concept unified modeling and improve the ability to maintain the appearance of multiple subjects.

[0179] Table 6 Comparative Experiments on Multi-Concept Image Generation on the CustomConcept101 Dataset

[0180] Table 6 shows that in the CustomConcept101 multi-concept composition task, the method of this invention achieves the best results on both DINO and CLIP-I, with scores of 0.402 and 0.680, respectively. This result indicates that in scenarios with more complex concept distributions and richer compositional relationships, compatibility-aware fusion can more effectively reduce structural shifts and appearance mixing caused by concept competition.

[0181] Table 7. Joint Experiments of Single-Concept Optimization Method and Fusion Method

[0182] Table 7 shows that single-concept training optimization and compatibility-aware fusion have a synergistic effect. When Ours Trainer is used in conjunction with Ours Fusion, DINO, CLIP-I, and CLIP-T all show improvements compared to the basic combination of ED-LoRA and Gradient Fusion, indicating that the improvement of single-concept module purity and multi-concept fusion compatibility control can jointly improve the final generation quality.

[0183] Table 8 Progressive ablation experiments of compatibility-aware fusion

[0184] Table 8 shows that as global compatibility estimation, text-level weighting, and key-value (K / V)-level weighting are gradually added, the three indicators generally show an upward trend. This result indicates that the performance improvement of this invention does not come from a single local modification, but is formed by the combined effect of global module relationship modeling, text condition organization control, and condition injection path control.

[0185] Table 9 Ablation experiments on compatibility signal sources

[0186] Table 9 shows that a single source of compatibility signals is insufficient to reliably balance the preservation of the main structure, identity consistency, and text alignment. However, the combined semantic representation signal and parameter update signal achieve better overall performance. This indicates that the potential conflicts in multi-concept fusion stem from both semantic proximity in the text conditional space and overlapping parameter effects along the conditional injection path, necessitating the estimation of module compatibility relationships from both types of information.

[0187] Based on the above experimental results, the method of this invention can improve the subject-related indicators in single-concept tasks, indicating that decoupled parameter updates and spatial response constraints can reduce background information mixing and enhance subject preservation capabilities. In multi-concept tasks, the method of this invention maintains a stable advantage over fusion baselines such as Mix-of-Show on DINO and CLIP-I, indicating that compatibility estimation and hierarchical weighted fusion can reduce identity mixing, concept crosstalk, and local overlay phenomena in the unified modeling process of multi-concepts.

[0188] Therefore, the multi-concept personalized image generation method provided by this invention first constructs a decomposed concept label, hierarchical concept embedding, and selective parameter update space in the single-concept training stage, and obtains standardized single-concept modules by combining concept focusing supervision and background suppression supervision; then, in the multi-concept fusion stage, compatibility estimation is performed by jointly combining semantic representation and parameter update features, and hierarchical weighted fusion is performed in the text encoding branch and the U-Net cross-attention key-value branch to obtain a unified multi-concept generation model. Experimental results show that this invention can improve the subject preservation ability of single concepts and enhance subject fidelity, concept boundary clarity, and combination stability in multi-concept combination generation.

[0189] As another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when loaded and executed by a processor, implements the multi-concept personalized image generation method as described above.

[0190] In this embodiment of the invention, a non-transitory computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can execute the personalized image generation method in any of the above method embodiments. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0191] As another embodiment of the present invention, an electronic device is provided, comprising:

[0192] Memory, used to store program instructions;

[0193] The processor, communicatively connected to the memory, is used to call and execute the program instructions to implement the multi-concept personalized image generation method as described above.

[0194] like Figure 9As shown, the electronic device 10 may include: at least one processor 11, such as a CPU (Central Processing Unit), at least one communication interface 13, a memory 14, and at least one communication bus 12. The communication bus 12 is used to enable communication between these components. The communication interface 13 may include a display screen or a keyboard; optionally, the communication interface 13 may also include a standard wired interface or a wireless interface. The memory 14 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk drive. Optionally, the memory 14 may also be at least one storage device located remotely from the aforementioned processor 11. The memory 14 stores application programs, and the processor 11 calls the program code stored in the memory 14 to execute any of the aforementioned method steps.

[0195] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 12 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0196] The memory 14 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 14 may also include a combination of the above types of memory.

[0197] The processor 11 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0198] The processor 11 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0199] Optionally, memory 14 is also used to store program instructions. Processor 11 can invoke program instructions to implement the present invention. Figure 1 The multi-concept personalized image generation method shown in the embodiments.

[0200] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating multi-concept personalized images, characterized in that, include: Obtain the reference image of the target subject, the subject region mask, and the image diffusion model of the pre-trained text; A decompositional conceptual identifier is constructed for the target entity, and the text entry of the target entity is divided into category semantic identifiers and entity-specific identifiers; The diffusion training master loss is obtained by performing diffusion training on the reference image based on the decomposed concept identifier; Conceptual focus supervision is constructed based on the main region mask, and background suppression supervision is constructed based on the complementary region of the main region mask; The image diffusion model of the pre-trained text is optimized and trained based on the diffusion training master loss, concept focusing supervision and background suppression supervision to obtain a standardized single concept module. A unified multi-concept model is obtained by performing multi-concept fusion training based on multiple standardized single-concept modules. The combined prompts obtained in real time are input into the unified multi-concept model to obtain personalized images including multiple target subjects.

2. The multi-concept personalized image generation method according to claim 1, characterized in that, The image diffusion model for the pre-trained text includes at least a text encoder and a denoising U-Net. Diffusion training is performed on the reference image based on the decomposed concept identifiers, including: A hierarchical concept embedding is constructed in the text encoder so that the target subject has a corresponding concept embedding representation in different layers of the text encoder. The hierarchical concept embedding includes shallow concept embedding and deep concept embedding. The shallow concept embedding is used to store word-level and local semantic information, and the deep concept embedding is used to store high-level semantic aggregation information. A selective parameter update space is constructed in the denoised U-Net, wherein the parameters corresponding to the key branches and value branches in the cross-attention module of the denoised U-Net are retained to participate in low-rank adaptation, and the parameters corresponding to the query branches in the cross-attention module of the denoised U-Net are frozen.

3. The multi-concept personalized image generation method according to claim 2, characterized in that, Diffusion training of the reference image based on the decomposed concept identifier further includes: The reference image is subjected to noise processing to obtain noisy latent variables; The decomposed concept identifier and the hierarchical concept are embedded and input into the text encoder to obtain text conditional features; The text conditional features and the noisy latent variables are input into the denoising U-Net to obtain noise prediction results; The diffusion training master loss is calculated based on the difference between the noise prediction result and the actual noise.

4. The multi-concept personalized image generation method according to claim 2, characterized in that, Conceptual focus supervision is constructed based on the main region mask, and background suppression supervision is constructed based on the complementary regions of the main region mask, including: Extract the attention response map corresponding to the target subject from the cross-attention weight map of the denoised U-Net; The attention response maps corresponding to multiple concept identifier locations of the same target subject are averaged, and the attention response maps at different spatial scales are aligned to a uniform resolution and then fused to obtain the subject concept attention map. The loss value of the concept focus supervision is calculated based on the difference between the subject concept attention map and the subject region mask; The loss value of the background suppression supervision is calculated based on the response intensity of the subject concept attention map on the complementary region of the subject region mask.

5. The multi-concept personalized image generation method according to claim 2, characterized in that, The image diffusion model of the pre-trained text is optimized and trained based on the diffusion training master loss, concept focusing supervision, and background suppression supervision to obtain a standardized single-concept module, including: The overall optimization objective is obtained by weighted fusion of the main loss of diffusion training, the loss value of concept focusing supervision, and the loss value of background suppression supervision. Based on the overall optimization objective, the low-rank parameters of the key branch and value branch in the hierarchical concept embedding and the cross-attention module of the denoising U-Net are iteratively optimized; The standardized single-concept module is obtained by associating the iteratively optimized parameters with the decomposed concept identifier.

6. The multi-concept personalized image generation method according to claim 1, characterized in that, Multi-concept fusion training is performed based on multiple standardized single-concept modules, including: Semantic representation features and parameter update features are extracted for each standardized single-concept module; Calculate the compatibility relationship between standardized single-concept modules based on the semantic representation features and the parameter update features; The weighting coefficients for each standardized single-concept module are determined based on the compatibility relationships between them. The standardized single-concept modules are weighted and fused according to the weighting coefficients of each standardized single-concept module to obtain the unified multi-concept model.

7. The multi-concept personalized image generation method according to claim 6, characterized in that, The compatibility relationship between standardized single-concept modules is calculated based on the semantic representation features and the parameter update features, including: For each standardized single-concept module, semantic representation features are extracted from the hierarchical concept embedding, and parameter update features are extracted from the low-rank parameter increments of the key and value branches in the cross-attention module of the denoised U-Net. For any two standardized single-concept modules, calculate the semantic similarity between their semantic representation features and the parameter update similarity between their parameter update features. The semantic similarity and the parameter update similarity are weighted and summed to obtain a comprehensive relationship score. The module compatibility matrix is ​​obtained after normalizing the comprehensive relationship scores.

8. The multi-concept personalized image generation method according to claim 7, characterized in that, The weighting coefficients for each standardized single-concept module are determined based on the compatibility relationships between them, including: Calculate the average conflict intensity between each standardized single-concept module and the other standardized single-concept modules based on the module compatibility relationship matrix, and map the average conflict intensity to a concept-level global scaling factor. In the cross-attention module of the denoising U-Net, the key branch and value branch, as well as the text encoder, extract the intra-layer output features of each standardized single-concept module in each layer, and calculate the intra-layer local conflict intensity and map it to the layer local scaling factor. The compatibility weighting coefficient of the corresponding standardized single-concept module is obtained based on the concept-level global scaling factor and the hierarchical local scaling factor.

9. The multi-concept personalized image generation method according to claim 8, characterized in that, The standardized single-concept modules are weighted and fused according to the weighting coefficients of each standardized single-concept module to obtain the unified multi-concept model, including: Based on the compatibility weighting coefficient, the input features and target output of the corresponding standardized single-concept module in each layer are scaled to obtain weighted sample pairs; The weighted sample pairs are input into the gradient fusion solver to reconstruct the key and value branches in the cross-attention module of the denoising U-Net and the unified model parameters of the text encoder layer by layer. The reconstructed unified model parameters are written back to the image diffusion model of the pre-trained text, and the unified multi-concept model is obtained by combining the identifier binding configuration of each standardized single-concept module.

10. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, communicatively connected to the memory, is used to invoke and execute the program instructions to implement the multi-concept personalized image generation method as described in any one of claims 1 to 8.

Images