A personalized image generation method and system for explicitly decoupling image objects and styles

Abstract

This invention relates to the field of personalized image generation technology using diffusion models, specifically a personalized image generation method and system that explicitly decouples image objects and styles. The method includes performing hierarchical semantic quantitative analysis on the diffusion model UNet, quantifying the contribution of each logical layer to object construction and style rendering, and selecting object-dominant and style-dominant layers. Only the dominant layer deploys a low-rank adaptation (LoRA) module, employing a two-stage binary gating strategy for phased training: the first stage updates only the object-layer LoRA to learn object features; the second stage freezes the object layer and updates only the style-layer LoRA to learn style features, ultimately obtaining a single LoRA model that simultaneously carries two independent concepts. This invention achieves explicit decoupling of objects and styles, enabling dual-concept generation with a single LoRA, eliminating the need for multiple LoRA combinations. It improves object fidelity and style similarity by 15.4% and 12.1%, respectively, while significantly reducing training and inference overhead, exhibiting excellent generalization and robustness.

Description

Technical Field

[0001] This invention relates to the field of personalized image generation technology using diffusion models, specifically a personalized image generation method and system that explicitly decouples image objects and styles. Background Technology

[0002] With the rapid development of text-to-image diffusion models, personalized image generation has become a core requirement in creative design, content production, and digital entertainment. Users expect to inject specific objects and artistic styles into the generation process using only a few reference images, achieving flexible context reconstruction and style transfer.

[0003] Among the current mainstream personalized generation methods, DreamBooth achieves concept implantation by fine-tuning all parameters of UNet, enabling contextual reconstruction and artistic reproduction of objects. However, full parameter fine-tuning has extremely high computational costs and is prone to catastrophic forgetting. When learning multiple concepts sequentially, later-learned concepts can destroy the features of already learned concepts. CustomDiffusion only fine-tunes the Key and Value matrices of cross-attention, reducing the number of parameters to some extent, but it still modifies the model's native parameters, cannot completely avoid catastrophic forgetting, and has limited feature fusion effects when learning multiple concepts.

[0004] Text embedding-based methods, such as Textual Inversion and P+, encode personalized concepts into text embedding vectors, avoiding modifications to model parameters. However, they suffer from insufficient expressive power, difficulty in accurately capturing fine-grained features of objects, and limited generalization ability. Among them, P+ first proposed the concept hierarchy phenomenon of UNet, but it did not quantitatively verify the semantic functions of each layer, and the hierarchical division relied on experience, failing to achieve accurate attribute decoupling.

[0005] Low-rank adaptation (LoRA) has become the most widely used fine-tuning scheme due to its advantages of efficient parameters, pluggability, and no impact on inference speed. However, existing LoRA personalization methods have significant bottlenecks: a single LoRA can only learn a single concept. When both object preservation and style transfer are required, object LoRAs and style LoRAs must be trained separately and then combined. Multi-LoRA merging methods such as ZipLoRA are prone to feature conflicts, often requiring secondary optimization training to alleviate these conflicts. This not only increases training overhead but may also prevent convergence to the optimal solution. Furthermore, stacking multiple LoRAs increases model size and raises the complexity of deployment and use.

[0006] Furthermore, existing methods generally suffer from conceptual entanglement: when learning object concepts, irrelevant information such as background, lighting, and style from the reference image is coupled into the model, leading to style leakage during generation; when learning styles, the inherent structural features of the object are easily destroyed, causing object distortion. Methods such as MATTE attempt to decouple attributes from the layer and time step dimensions, but their layer division relies on manual experience, resulting in coarse granularity and deviation from the model's true semantic distribution. Moreover, the decoupling effect is evaluated only through qualitative observation, lacking quantitative data support, and thus failing to achieve true explicit decoupling.

[0007] In summary, existing technologies suffer from the following core shortcomings: First, they lack a systematic quantitative analysis of the semantic functions of UNet layers, resulting in insufficient basis for layer division and low decoupling accuracy. Second, a single LoRA cannot simultaneously learn two independent concepts, object and style, leading to conflicts, high overhead, and operational complexity when combining multiple LoRAs. Third, there is severe conceptual entanglement, with style and object features interfering with each other, making it difficult to simultaneously achieve object fidelity and style similarity in the generated images. Therefore, developing a personalized generation method that can explicitly decouple objects and styles within a single LoRA framework has significant technical value and application implications. Summary of the Invention

[0008] To address the problems in existing technologies, this invention provides a personalized image generation method and system that explicitly decouples image objects and styles. By quantitatively revealing the hierarchical semantic differentiation rules of UNet, using hierarchical functions as structural priors, and combining selective LoRA deployment and two-stage binary gating training, explicit decoupling learning of objects and styles is achieved in a single LoRA. This solves problems such as concept entanglement, single LoRA single concept bottleneck, and multi-LoRA merging conflicts in existing technologies, while improving generation quality and reducing training and inference overhead.

[0009] The technical solution adopted by this invention to solve its technical problem is: a personalized image generation method that explicitly decouples image objects and styles, comprising the following steps: S1. Semantic Quantitative Analysis of Diffusion Model UNet Layers: The pre-trained diffusion model UNet network is divided into multiple logical layers according to network depth and attention module distribution; text prompts are constructed for target objects and target styles, and the prompts are input into each logical layer layer by layer, while empty text is input into the remaining layers to generate the corresponding layer-generated images. The object similarity and style similarity are quantitatively evaluated with the standard generated images obtained by inputting prompts into all layers to obtain the object contribution and style contribution of each logical layer, and the object-dominant layer and style-dominant layer are selected. S2. Selective LoRA module construction: Low-rank adaptation LoRA modules are deployed only in the attention modules of the selected object-dominant and style-dominant layers, and LoRA modules are not deployed in other layers; the LoRA modules are connected in parallel to the Query, Key, Value matrix and output mapping matrix of the attention module; S3. Two-stage binary gating decoupled training: The LoRA module is trained in two stages using a binary gating strategy. The first stage is the object learning stage, where an object reference image and object prompts are input, LoRA gradient updates of the object-dominant layer are enabled, and LoRA gradient updates of the style-dominant layer are completely disabled to learn the visual features of the target object. The second stage is the style learning stage, where all LoRA parameters of the object-dominant layer are frozen, a style reference image and style prompts are input, and LoRA gradient updates of only the style-dominant layer are enabled to learn the visual features of the target style. Prior preservation loss is added as a regularization term during training to suppress few-sample overfitting. S4. Single LoRA Inference Generation: Input a combination of prompt words containing object identifiers and style identifiers, call a trained single LoRA model, and generate a personalized image that retains both the target object features and the target style features.

[0010] Specifically, in step S1, dividing the UNet network into multiple logical layers involves merging the basic attention modules in the UNet downsampling block, intermediate block, and upsampling block into 11 logical layers based on spatial resolution and network depth. Each logical layer contains 2 to 10 basic attention modules that are connected in series with self-attention and cross-attention.

[0011] Specifically, in step S1, the quantitative evaluation includes: constructing a weighted multi-dimensional object similarity evaluation system and a style similarity evaluation system; using a multimodal visual large model to score the dimensions of the layer-generated image and the standard generated image; and then weighting and normalizing by Min-Max to obtain the contribution value of each layer; the object similarity system includes three dimensions: morphological similarity, contour similarity, and compositional structure; and the style similarity system includes four dimensions: brushstroke features, color application, composition techniques, and overall effect.

[0012] Specifically, in step S1, the criteria for selecting the object-dominant layer and the style-dominant layer are as follows: the logical layer with significantly higher object similarity than the other layers is selected as the object-dominant layer, and the logical layer with significantly higher style similarity than the other layers is selected as the style-dominant layer; in the SDXL model, the object-dominant layers are the 4th and 6th logical layers, and the style-dominant layer is the 7th logical layer.

[0013] Specifically, in step S2, the rank of the LoRA module is set to 32, and it is connected to the original network weights in parallel. During the inference phase, the LoRA weights and the original weights are calculated together without increasing the additional inference time.

[0014] Specifically, in step S3, the binary gating strategy is implemented by controlling the range of trainable parameters: during the target training phase, only the LoRA parameters of the target layer are included in the optimizer parameter list, and the LoRA parameters of non-target layers do not participate in gradient calculation and update, which is equivalent to the binary value of the scaling factor, with non-target layer scaling=0 and target layer scaling=1.

[0015] Specifically, in step S3, the object prompt word used in the first stage is: The style prompts used in the second phase are: ,in An object-specific identifier. It is a style-specific identifier; no additional scene or attribute description prompts are added during the training process.

[0016] Specifically, in step S3, the overall loss function for training is a weighted sum of the reconstruction loss and the prior preservation loss, expressed as: ,in, This is the latent representation of the reference image after adding noise. For the text conditions corresponding to the reference image, The original Gaussian noise, The noise predicted by the diffusion model, For the noisy latent representation of the prior class image, For prior category text conditions. The weighting coefficients for the prior hold loss.

[0017] Specifically, in step S4, the format of the combined prompt words is as follows: A single LoRA model can output both object features and style features simultaneously, without the need to load and merge multiple LoRA modules.

[0018] A personalized image generation system that explicitly decouples image objects and styles, comprising: The hierarchical semantic analysis module is used to divide the UNet network of the pre-trained diffusion model into logical layers, and quantitatively evaluate the object contribution and style contribution of each logical layer to filter out the object-dominant layer and style-dominant layer. The LoRA configuration module is used to deploy the LoRA module only in the attention modules of the object-dominated and style-dominated layers, and no adaptation structure is added to other layers. The phased training module is used to perform two-stage binary gating training, sequentially completing object feature learning and style feature learning, and suppressing small sample overfitting through prior preservation loss. The inference generation module receives combined prompts and calls a single, trained LoRA model to generate a personalized image of the target.

[0019] The beneficial effects of this invention are: Explicit decoupling significantly improves generation quality: Based on quantitative hierarchical semantic priors and combined with binary gating spatial isolation, complete decoupling of objects and styles is achieved, effectively solving problems such as concept entanglement, style leakage, and object distortion. Experiments show that this method improves object fidelity by 15.4% relative to the baseline and style similarity by 12.1%, while maintaining good text consistency and structural fidelity.

[0020] Single LoRA dual concept, breaking through existing technical bottlenecks: For the first time, explicit decoupling learning of two independent concepts is achieved under a single LoRA framework. There is no need to train and combine multiple LoRAs, avoiding the problems of multiple LoRA merging conflicts and secondary training. The model size is only half that of the multiple LoRA solution, and the operation and deployment are simpler.

[0021] Highly efficient parameters and significantly reduced computational overhead: LoRA is deployed only in a few dominant layers, and the number of trainable parameters is only about 30% of that of the full-layer LoRA scheme, improving training speed by more than 40%. At the same time, there is no extra time spent in the inference stage, balancing training efficiency and inference efficiency.

[0022] High generalization and wide applicability: This method is based on the hierarchical semantic differentiation pattern that is common in diffusion models. It is not only applicable to the SDXL model, but can also be transferred to other diffusion models with hierarchical structures such as Stable Diffusion v1.5, and has broad architectural compatibility.

[0023] Highly scalable: The hierarchical quantitative analysis and selective gating framework of this invention can be extended to decouple more attributes, such as color, layout, and material, providing a new research approach for the refined and personalized generation of multiple attributes. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] Figure 1 Examples of personalized image effects generated by the method of this invention for different scenes are shown; the trained model generates multiple sets of scene images under different text prompts such as "on a wooden table", "in the garden", "on top of a hill" and "wearing sunglasses".

[0026] Figure 2 This is a schematic diagram of the UNet hierarchical structure of the SDXL model; the top is a schematic diagram of the basic attention modules, showing the chained structure of self-attention and cross-attention; the bottom is the hierarchical structure of UNet, where L1~L11 represent 11 logical layers, and the number of basic attention modules contained in each layer is marked, as well as the positional relationship of the corresponding down blocks, mid blocks, and up blocks.

[0027] Figure 3 This is a flowchart illustrating the hierarchical semantic quantitative analysis process of the present invention. It shows the complete process of hierarchical semantic quantitative analysis: input text prompts, input UNet layer by layer to generate corresponding layer generated images imgk, input them together with the standard images imgall obtained from all layers into a multimodal large model, calculate object similarity and style similarity based on evaluation criteria, and complete the quantitative evaluation by traversing all logical layers.

[0028] Figure 4 Qualitative comparison of images generated by each layer of UNet; showing the sequence of layer images generated by each of UNet layers 1 to 11 under multiple prompts, allowing for a direct observation of the differences in object shaping and style rendering between different layers, which helps to qualitatively verify the hierarchical functional differentiation pattern of UNet.

[0029] Figure 5 This is a schematic diagram illustrating the principle of the two-stage binary gating training of the present invention. It shows the principle of the two-stage binary gating training: the first stage is the object training stage, in which only the LoRA parameters of the 4th and 6th layers are fine-tuned, and the LoRA of the 7th layer is disabled due to scaling=0 (indicated by the dashed arrow); the second stage is the style training stage, in which the LoRA of the 4th and 6th layers is frozen, and only the LoRA parameters of the 7th layer are optimized, so as to achieve spatial isolation and decoupling learning between objects and styles. Detailed Implementation

[0030] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0031] like Figures 1-5 As shown, the personalized image generation method for explicitly decoupling image objects and styles according to the present invention includes four core steps: hierarchical semantic quantitative analysis, selective LoRA construction, two-stage gating training, and single LoRA inference, as detailed below: (I) Quantitative analysis of UNet hierarchical semantics using the diffusion model The core of this step is to clarify the semantic functions of each layer of UNet through quantitative methods, providing accurate structural priors for subsequent decoupling. The specific steps are as follows: Logical layer partitioning: Based on network depth and feature space scale, the UNet network of the pre-trained diffusion model is grouped into a single logical layer by grouping basic attention modules that are in the same generation stage and have the same resolution. Taking SDXL as an example, its UNet contains 3 downsampling blocks, 3 upsampling blocks, and 1 intermediate block, totaling 70 basic attention modules, which can be divided into 11 logical layers. Each logical layer contains 2 to 10 basic attention units composed of self-attention and cross-attention concatenation.

[0032] Hierarchical Image Generation: Standard text prompts are constructed for the target object and target style. Hierarchical images are generated using a "single-layer input, empty text for the remaining layers" approach. Specifically, in the cross-attention calculation stage, only the text embedding corresponding to the target prompt is input to the k-th logical layer; all other logical layers are input with text embeddings corresponding to empty strings. This sequentially generates hierarchical images img1~img1 from layers 1 to N. N Simultaneously, input the prompt words into all layers to generate a standard generated image (img). all As a reference standard.

[0033] Multi-dimensional quantitative assessment: Two weighted quantitative evaluation systems are constructed to calculate object similarity and style similarity, respectively. The object similarity evaluation system includes three dimensions: morphological similarity (40%), outline similarity (40%), and compositional structure (20%), measuring the degree of object reproduction from the perspectives of shape, outline, and structure. The style similarity evaluation system includes four dimensions: brushstroke characteristics (30%), color usage (40%), composition techniques (10%), and overall effect (20%), measuring the degree of style matching from the perspectives of brushstroke, color, composition, and atmosphere.

[0034] The hierarchical generated images and standard generated images are input into the multimodal visual large model in pairs. The instruction model scores each item according to the above evaluation criteria, and the weighted sum is used to obtain the similarity score of a single set of images. Through repeated experiments with multiple sets of different prompt words, the scores are normalized by Min-Max and the average value is taken to obtain the final values ​​of object similarity and style similarity for each logical layer.

[0035] Dominant Layer Selection: Based on similarity scores, logical layers with significantly higher object similarity than other layers are selected as object dominant layers, and logical layers with significantly higher style similarity than other layers are selected as style dominant layers. Taking SDXL as an example, layers 4 and 6 have the highest object similarity scores (0.788 and 0.751 respectively), thus being identified as object dominant layers; layer 7 has the highest style similarity score (0.906), thus being identified as style dominant layer.

[0036] This quantitative analysis reveals the layered generation mechanism of UNet, which consists of "basic composition → object generation → style rendering → overall optimization". The bottom layer handles basic features, the middle layer dominates object construction, the upper-middle layer dominates style rendering, and the top layer is responsible for global coordination, providing a reliable structural basis for explicit decoupling.

[0037] (II) Construction of Selective LoRA Module Based on the hierarchical semantic analysis results, a selective deployment strategy is adopted, deploying the LoRA module only in the attention modules of the object-dominated and style-dominated layers, and not deploying LoRA in all other layers.

[0038] In the self-attention and cross-attention sub-modules of each basic attention module, LoRA is connected in parallel to the three weight matrices Query, Key, and Value, as well as the output projection matrix out.0 of the attention module, to ensure complete adaptation of the feature generation capability of this layer. LoRA adopts a low-rank matrix factorization structure with a rank of 32. During training, only the low-rank matrix parameters of LoRA are updated, and the weights of the original model are frozen throughout the process.

[0039] The advantages of this selective deployment are: first, it significantly reduces the number of trainable parameters, thereby reducing training computation overhead and model storage volume; second, it restricts the optimization process to the corresponding semantic subspace, avoiding feature interference introduced by irrelevant layers and improving the purity of decoupling; and third, the parallel structure of LoRA ensures that it can be combined with the original weights for calculation during inference without increasing any inference time.

[0040] (III) Two-stage binary gated decoupling training By employing a phased training and binary gating mechanism, object learning and style learning are completely confined to their respective hierarchical subspaces, achieving explicit decoupling through orthogonality. The specific steps are as follows: Training configuration: The training mode is small sample training, with a small number of object reference images and style reference images as input; the optimizer uses the Adam algorithm with weight decay, the training mode is mixed precision, and a prior preservation loss is added as a regularization term to suppress the overfitting problem of small sample training.

[0041] The overall loss function is expressed as follows: ; The first term is the reconstruction loss, which constrains the model's ability to reconstruct the reference image; the second term is the prior preservation loss, which constrains the model to retain the general prior knowledge from the pre-training stage and avoids overfitting. This is the prior loss weight, usually set to 1.0.

[0042] Phase 1: Object Learning Phase: Using Minimalist Object Hints ( (For each object, a unique identifier is used), and the system is trained using a reference image of the input object. Through a binary gating mechanism, only the LoRA parameters of the object-dominant layer are included in the trainable list of the optimizer; the LoRA parameters of the style-dominant layer do not participate in gradient calculation and updates (equivalent to scaling=0). This stage optimizes only within the object's semantic subspace, accurately learning the core features of the target object, such as its shape, contour, and structure.

[0043] Phase Two: Style Learning Phase: Freeze the LoRA parameters of the object layer trained in Phase One, so that they do not participate in gradient updates at all; use minimalist style prompts. ( (For style-specific identifiers), input style reference images for training. In this stage, only the LoRA parameters of the style-dominant layer are included in the optimizer list, and optimization is performed only within the style semantic subspace, learning features such as brushstrokes, colors, and atmosphere of the target style.

[0044] This invention employs a minimalist prompt word strategy, omitting additional descriptions such as scenarios and attributes during training. This avoids overfitting the model to specific texts and improves its generalization ability across different scenarios. The binary gating mechanism achieves complete spatial isolation between the two learning processes, structurally preventing style learning from damaging object features and avoiding the introduction of irrelevant style information during object learning, thus achieving true explicit decoupling.

[0045] (iv) Single LoRA inference generation After training, a single LoRA model is obtained that simultaneously carries the concepts of object and style. During inference, combined cues are used. By loading this single LoRA model, personalized images that retain both the features of the target object and the features of the target style can be generated. There is no need to train multiple LoRA models or perform LoRA merging operations, which greatly reduces the complexity of use and deployment costs.

[0046] Example 1: Explicit Decoupling of Personalized Image Generation Based on the SDXL Model: This embodiment uses Stable Diffusion XL (SDXL) 1.0 as the basic model to fully implement the explicit decoupling personalized image generation method of the present invention, and verifies the effect through multiple sets of comparative experiments.

[0047] 1. Experimental environment and basic configuration: The hardware environment uses an NVIDIA RTX A6000 graphics card with 48GB of video memory; the software framework is based on the Hugging FaceDiffusers library, the base model uses stabilityai / stable-diffusion-xl-base-1.0, and an improved version of VAE weights madebyollin / sdxl-vae-fp16-fix is ​​used to improve the stability of latent space encoding.

[0048] The basic inference parameters are set as follows: EulerDiscreteScheduler is used as the noise scheduler, the number of sampling steps is 50, the mixing precision is set to fp16 mode, the generated image resolution is 1024×1024, and the unguided coefficient is set to 10.0.

[0049] The basic training parameters were set as follows: the optimizer was AdamW, the weight decay was constant, the batch size was 1, the prior hold loss weight was λ=1.0, the random seed was fixed at 469 to ensure reproducibility, the LoRA module rank was 32, and the basic learning rate was set to 6e-5.

[0050] 2. Quantitative analysis of UNet hierarchical semantics and selection of dominant layer: First, the SDXL UNet is logically divided: the SDXL UNet consists of 3 downsampling blocks, 1 intermediate block, and 3 upsampling blocks, containing a total of 70 basic attention modules. Each module is composed of self-attention and cross-attention concatenated. According to the network depth and feature resolution, all attention modules are grouped into 11 logical layers. The first and second logical layers each contain 4 attention modules, and the fourth, sixth, and seventh deeper logical layers each contain 6 to 10 attention modules, covering the complete generation stage from low resolution to high resolution.

[0051] A hierarchical image generation experiment was then conducted: 100 sets of text prompts containing different objects and artistic styles were selected. For each set of prompts, the text embeddings were sequentially input only into logical layers 1 to 11, while empty text embeddings were input into the remaining layers, generating 11 hierarchically generated images. Simultaneously, prompts were input into all layers, generating one standard generated image. This experiment generated a total of 1200 images, covering various object categories such as objects, animals, and people, as well as various style categories such as oil painting, watercolor, pixel art, and realism.

[0052] After constructing a quantitative evaluation system, the Gemini 2.0 Pro visual model was used to score each pair of layered images and the standard image according to the dimensions of object similarity and style similarity. The scores of 100 experimental groups were normalized using Min-Max and then the arithmetic mean was taken to obtain the final similarity data for each logical layer: Layer 1: Object similarity 0.246, Style similarity 0.239; Layer 2: Object similarity 0.269, Style similarity 0.238; Layer 3: Object similarity 0.085, Style similarity 0.113; Layer 4: Object similarity 0.788, Style similarity 0.361; Layer 5: Object similarity 0.218, Style similarity 0.248; Layer 6: Object similarity 0.751, Style similarity 0.495; Layer 7: Object similarity 0.336, Style similarity 0.906; The object and style similarities for Layers 8 to 11 remained stable at around 0.264.

[0053] Based on the quantification results, the dominant layers were selected as follows: Layers 4 and 6, with significantly higher object similarity than the other layers, are responsible for constructing the core features of objects and are therefore identified as object-dominant layers. Layer 7, with much higher style similarity than the other layers, is responsible for the core function of style rendering and is therefore identified as style-dominant layers. The remaining layers contribute little to both objects and styles, primarily handling basic feature processing and global coordination, and are not considered core semantic layers.

[0054] 3. Selective LoRA Deployment and Two-Phase Gated Training: LoRA modules are deployed only in the attention modules of the 4th, 6th, and 7th logical layers. In the self-attention and cross-attention sub-modules of each attention module, LoRA is connected to the Query, Key, and Value matrices and the output projection matrix out.0, respectively. LoRA is not deployed in the remaining 8 logical layers. According to statistics, the number of trainable parameters is only about 28% of that of deploying LoRA in all layers, which significantly reduces the parameter size.

[0055] The training was conducted in two phases: The first stage is the object learning stage, which selects 3-5 object reference images as training data and uses prompt words. ,in This serves as a unique identifier for the object. Binary gating is implemented through optimizer parameter configuration: only the LoRA parameters of layers 4 and 6 are added to the trainable parameter list, while the LoRA parameters of layer 7 are not added, and their gradients are not calculated during backpropagation, effectively shielding this layer's updates. This stage involves 800 training iterations with a constant learning rate of 6e-5. Prior images of the corresponding class are used to calculate the prior loss, preventing overfitting. After training, the LoRA parameters of layers 4 and 6 have fully learned the core visual features of the target object, including its shape, contour, and structure.

[0056] The second stage is the style learning stage, which selects 3-5 style reference images as training data and uses prompt words. ,in This serves as a unique identifier for the style. The LoRA parameters of layers 4 and 6 are kept completely frozen and not included in the optimizer's update scope; only the LoRA parameters of layer 7 are added to the trainable list for gradient updates. This stage involves 500 training iterations with a constant learning rate of 6e-5, and a prior loss is also incorporated to suppress overfitting. After training, the LoRA of layer 7 fully learns the brushstrokes, colors, atmosphere, and other features of the target style without interfering with the already learned object features.

[0057] 4. Reasoning Tests and Controlled Experiments: During inference, the prompt "A[o]in[s]style" is used to load a single trained LoRA model, which generates a personalized image that simultaneously preserves both the target object and its style. To verify the performance of this method, two sets of control examples are set up for comparison: Comparison with Example 1: DreamBooth full-parameter fine-tuning method. Using the same object and style reference images, a sequential training strategy was employed, training the object concept first and then the style concept, using the default DreamBooth configuration from the Diffusers library. Experimental results show significant catastrophic forgetting during the style training phase, with significant distortion of object features, such as deformed handles of ceramic teapots, altered ear shapes of pet dogs, and distorted facial features of people; quantitative assessment showed an object similarity of 0.685 and a style similarity of 0.772.

[0058] Comparative Example 2: ZipLoRA multi-LoRA combination method. Object LoRAs and style LoRAs are trained separately, and then merged using the ZipLoRA algorithm. Experimental results show that the two LoRAs have obvious feature conflicts, resulting in blurry images, edge artifacts, and style leakage into the background, leading to unnatural fusion effects. Quantitative evaluation shows an object similarity of 0.588 and a style similarity of 0.697.

[0059] The quantitative evaluation results of the method in this embodiment are: object similarity 0.801, style similarity 0.913. Compared with the best baseline Custom Diffusion (object similarity 0.696, style similarity 0.801), the object fidelity is improved by 15.1%, and the style similarity is improved by 14.0%; compared with the average level of all baselines, the object fidelity is improved by 15.4%, and the style similarity is improved by 12.1%. Furthermore, the images generated by this method show no obvious style leakage or object distortion, and can stably generate high-quality images under different scene prompts, demonstrating excellent generalization ability.

[0060] Example 2: Migration verification based on Stable Diffusion v1.5: This embodiment migrates the method of the present invention to the Stable Diffusion v1.5 model to verify the cross-architecture generalization ability of the method and compares it with existing methods.

[0061] 1. Model hierarchy analysis and dominant layer selection: Stable Diffusion v1.5's UNet also adopts a downsampling-intermediate block-upsampling hierarchical structure, containing a total of 16 basic attention modules. Following the same logical layer division principle as Example 1, it is divided into 8 logical layers based on network depth and feature resolution.

[0062] Using the same quantitative analysis process as in Example 1, 50 sets of objects plus style prompts were selected to generate images at each level and a standard image. Similarity scoring and normalization were performed using a multimodal large model. Quantitative results showed that StableDiffusion v1.5 also exhibited significant hierarchical functional differentiation: the middle layer had the highest object similarity, dominating object construction; the upper-middle layers had the highest style similarity, dominating style rendering. Based on the quantitative results, the 3rd and 5th logical layers were selected as object-dominant layers, and the 6th logical layer as style-dominant layer.

[0063] 2. Model training configuration: LoRA modules were deployed only in the selected 3rd, 5th, and 6th logical layers. The deployment locations also covered the Q, K, V matrices and output projection matrix of the attention module. The LoRA rank was set to 32, and the base learning rate was set to 5e-5.

[0064] Training also employs a two-stage binary gating strategy: the first stage involves object learning, updating only the LoRA parameters of layers 3 and 5, iterating for 600 steps, with prompts provided by... The second phase of style learning involves freezing object layer parameters and updating only the LoRA parameters at level 6, iterating for 400 steps. The prompt word is... The training batch size is 1, and a prior hold-out loss is included. The inference-generated image resolution is 512×512, the sampling steps are 30, and the guidance coefficient is 7.5.

[0065] 3. Comparative Experiment and Result Analysis: Four objects and four style reference images were selected and combined into 16 test samples. Three control examples were set up for comparison. Comparative Example A: DreamBooth full parameter fine-tuning. A sequential training strategy was adopted, with the same training data and number of iterations. Experimental results show that after pattern training, object features are severely lost, object contours are significantly deformed, and catastrophic forgetting is prominent; quantitative assessment shows object similarity of 0.695 and pattern similarity of 0.793.

[0066] Comparative Example B: Custom Diffusion method. Only the cross-attention K and V matrices are fine-tuned, and objects and styles are trained sequentially. Experimental results show that the object preservation ability is better than DreamBooth, but the style learning is insufficient, some style features are not effectively transferred, and the fusion effect between objects and styles is awkward; the quantitative evaluation of object similarity is 0.706 and style similarity is 0.821.

[0067] Comparative Example C: ZipLoRA multi-LoRA combination. Object and style LoRAs were trained separately and then merged. Experimental results show significant merging conflicts, resulting in blurred details in the generated images, and some samples exhibiting object structure corruption; quantitative assessment shows object similarity of 0.597 and style similarity of 0.707.

[0068] The quantitative evaluation results of the method in this embodiment are: object similarity 0.794, style similarity 0.899. Compared with the best baseline CustomDiffusion, the object similarity is improved by 12.5% ​​and the style similarity is improved by 9.5%, consistent with the improvement trend of the SDXL model. Qualitative observation shows that the image objects generated by this method have complete structures, clear details, accurate and natural style restoration, and no obvious feature conflicts or leakage problems.

[0069] This embodiment verifies that the method of the present invention does not depend on a specific model architecture. As long as it is a diffusion model with a hierarchical attention structure, the dominant layer can be screened through hierarchical semantic quantitative analysis to achieve explicit decoupling and personalized generation, which has wide applicability and transferability.

[0070] Example 3: Comparative verification of different decoupling strategies: This embodiment compares three different decoupled training strategies on the SDXL model to verify the superiority of the binary strict gating plus selective LoRA scheme adopted in this invention, and also to verify the feasibility and rationality of the scheme. The experiment uses the same object, style dataset and basic training parameters, only changing the LoRA deployment scope and gating strategy.

[0071] Option 1: Full-layer LoRA differential scaling strategy (Comparative Example A) This scheme deploys LoRA modules in all 11 logical layers of UNet, achieving decoupling through differentiated scaling settings. In the object learning phase: scaling of layers 4 and 6 is set to 3.0 to enhance object learning, scaling of layer 7 is set to 0 to disable pattern learning, and scaling of all other auxiliary layers is kept at 1.0 for normal training. In the pattern learning phase: scaling of layer 7 is set to 3.5 to enhance pattern learning, scaling of layers 4 and 6 is set to 0 to protect object features, and scaling of all other auxiliary layers is kept at 1.0.

[0072] Experimental results show that the proposed scheme can achieve a certain degree of object-style separation, but it has significant drawbacks: First, the auxiliary layer participates in training in both stages. In the object learning stage, the auxiliary layer learns some secondary object features, and in the style learning stage, fine-tuning of the auxiliary layer interferes with the learned object features through feature hierarchy, resulting in obvious object feature degradation, such as the disappearance of the handle of a ceramic teapot or changes in the shape of a pet dog's ears. Second, the optimization of the auxiliary layer in both stages creates implicit antagonism, requiring repeated adjustments to the learning rate, iteration count, and scaling coefficient to achieve a fragile balance, resulting in extremely high parameter tuning costs. Third, the full-layer deployment of LoRA leads to a large number of parameters and slow training speed, with training overhead more than three times that of the proposed scheme. Fourth, there is an unpredictable style leakage phenomenon, where background areas may unexpectedly introduce style features. Quantitative evaluation shows that the proposed scheme has an object similarity of 0.723 and a style similarity of 0.845, both significantly lower than the proposed scheme.

[0073] Option 2: Critical Layer LoRA Soft Gating Strategy (Comparative Example B) This scheme deploys LoRA only at layers 4, 6, and 7, consistent with the deployment scope of this invention, but employs a soft gating strategy, i.e., setting extremely low scaling for non-target layers instead of complete masking. Object learning phase: scaling = 1.0 for layers 4 and 6, scaling = 0.1 for layer 7; style learning phase: scaling = 1.0 for layer 7, scaling = 0.1 for layers 4 and 6.

[0074] Experimental results show that the decoupling effect of this scheme is extremely poor: even when the scaling of non-target layers is reduced to 0.1, the minor updates to layers 4 and 6 during the style learning stage still severely damage the object features learned in the first stage, leading to object contour deformation, structural distortion, and generation results that deviate from the target object. Even after repeatedly adjusting the scaling coefficient (e.g., 0.01, 0.05), it is still impossible to balance style learning intensity and object feature preservation; either style learning is insufficient or object distortion is severe. Quantitative evaluation shows that the object similarity of this scheme is only 0.634, and the style similarity is 0.782, even lower than the baseline of full-parameter fine-tuning.

[0075] Option 3: Critical Layer LoRA Binary Strict Gating Strategy (Invention Solution) This scheme deploys LoRA only in layers 4, 6, and 7, using full binary gating: during the object learning phase, scaling is 1 in layers 4 and 6 and 0 in layer 7; during the pattern learning phase, scaling is 0 in layers 4 and 6 and 1 in layer 7, and non-target layers do not participate in gradient updates at all.

[0076] To verify that the object-dominant layer alone can effectively capture object features, this embodiment supplements the single-layer input verification experiment: only text prompts are input to layers 4 and 6, and empty text is input to the remaining layers. Compared with the standard image with full-layer input, the generated image can fully present the core shape, outline and structure of the object, proving that the object features can be effectively encoded by the object-dominant layer alone, and the underlying logic of this solution is feasible.

[0077] Experimental results show that this scheme achieves optimal decoupling: style learning does not destroy object features at all, resulting in high object fidelity; style feature restoration is accurate with no obvious leakage; generated images are naturally blended without any awkward splicing. Simultaneously, this scheme has the fewest parameters, the fastest training speed (only 32% of the training time of the full-layer scheme), and simple parameter tuning without the need for complex coefficient balancing. Quantitative evaluation shows that the object similarity is 0.801 and the style similarity is 0.913, the highest among the three schemes.

[0078] A system comparison of the three schemes demonstrates that only a combination of critical layer deployment and strict binary gating can simultaneously achieve high decoupling, high generation quality, and low computational overhead. The involvement of auxiliary layers introduces implicit adversarial behavior and feature interference; soft gating cannot prevent feature destruction of non-target layers. Only complete spatial isolation can achieve true explicit decoupling. The third strategy adopted in this invention is optimal in terms of decoupling effect, generation quality, computational efficiency, and ease of use, making it the best overall performance solution.

[0079] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A personalized image generation method that explicitly decouples image objects and styles, characterized in that, Includes the following steps: S1. Semantic Quantitative Analysis of Diffusion Model UNet Layers: The pre-trained diffusion model UNet network is divided into multiple logical layers according to network depth and attention module distribution; text prompts are constructed for target objects and target styles, and the prompts are input into each logical layer layer by layer, while empty text is input into the remaining layers to generate the corresponding layer-generated images. The object similarity and style similarity are quantitatively evaluated with the standard generated images obtained by inputting prompts into all layers to obtain the object contribution and style contribution of each logical layer, and the object-dominant layer and style-dominant layer are selected. S2. Selective LoRA module construction: Low-rank adaptation LoRA modules are deployed only in the attention modules of the selected object-dominant and style-dominant layers, and LoRA modules are not deployed in other layers; the LoRA modules are connected in parallel to the Query, Key, Value matrix and output mapping matrix of the attention module; S3. Two-stage binary gating decoupled training: The LoRA module is trained in two stages using a binary gating strategy. The first stage is the object learning stage, where the input object reference image and object prompt words are used. The LoRA gradient update of the object-dominant layer is enabled, while the LoRA gradient update of the style-dominant layer is completely disabled to learn the visual features of the target object. The second stage is the style learning stage, which freezes all LoRA parameters of the object's dominant layer, inputs a style reference image and style cue words, and enables LoRA gradient updates only for the style's dominant layer to learn the visual features of the target style. During training, a prior preservation loss is incorporated as a regularization term to suppress overfitting of small samples. S4. Single LoRA Inference Generation: Input a combination of prompt words containing object identifiers and style identifiers, call a trained single LoRA model, and generate a personalized image that retains both the target object features and the target style features.

2. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S1, dividing the UNet network into multiple logical layers specifically involves: merging the basic attention modules in the UNet downsampling block, intermediate block, and upsampling block into 11 logical layers according to spatial resolution and network depth. Each logical layer contains 2 to 10 basic attention modules that are connected in series with self-attention and cross-attention.

3. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S1, the quantitative evaluation specifically includes: constructing a weighted multi-dimensional object similarity evaluation system and a style similarity evaluation system; using a multimodal visual large model to score the dimensions of the hierarchical generated images and the standard generated images; and then weighting and normalizing by Min-Max to obtain the contribution value of each layer; the object similarity system includes three dimensions: morphological similarity, contour similarity, and compositional structure; and the style similarity system includes four dimensions: brushstroke features, color application, composition techniques, and overall effect.

4. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S1, the criteria for selecting the object-dominant layer and the style-dominant layer are as follows: the logical layer with significantly higher object similarity than the other layers is selected as the object-dominant layer, and the logical layer with significantly higher style similarity than the other layers is selected as the style-dominant layer. In the SDXL model, the object-dominant layers are the 4th and 6th logical layers, and the style-dominant layer is the 7th logical layer.

5. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S2, the rank of the LoRA module is set to 32, and it is connected to the original network weights in parallel. During the inference phase, the LoRA weights are calculated together with the original weights, without increasing the additional inference time.

6. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S3, the binary gating strategy is implemented by controlling the range of trainable parameters: during the target training phase, only the LoRA parameters of the target layer are included in the optimizer parameter list, and the LoRA parameters of non-target layers do not participate in gradient calculation and update, which is equivalent to the binary value of the scaling factor, with non-target layer scaling=0 and target layer scaling=1.

7. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S3, the object prompt word used in the first stage is: The style prompts used in the second phase are: ,in An object-specific identifier. It is a style-specific identifier; no additional scene or attribute description prompts are added during the training process.

8. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S3, the overall loss function for training is a weighted sum of the reconstruction loss and the prior preservation loss, expressed as: ,in, This is the latent representation of the reference image after adding noise. For the text conditions corresponding to the reference image, The original Gaussian noise, The noise predicted by the diffusion model, For the noisy latent representation of the prior category image, For prior category text conditions. The weighting coefficients for the prior preservation loss.

9. The personalized image generation method for explicitly decoupling image objects and styles according to claim 1, characterized in that: In step S4, the format of the combined prompt words is as follows: A single LoRA model can output both object features and style features simultaneously, without the need to load and merge multiple LoRA modules.

10. A personalized image generation system that explicitly decouples image objects and styles, characterized in that, include: The hierarchical semantic analysis module is used to divide the UNet network of the pre-trained diffusion model into logical layers, and quantitatively evaluate the object contribution and style contribution of each logical layer to filter out the object-dominant layer and style-dominant layer. The LoRA configuration module is used to deploy the LoRA module only in the attention modules of the object-dominated and style-dominated layers, and no adaptation structure is added to other layers. The phased training module is used to perform two-stage binary gating training, sequentially completing object feature learning and style feature learning, and suppressing small sample overfitting through prior preservation loss. The inference generation module receives combined prompts and calls a single, trained LoRA model to generate a personalized image of the target.

Images