An Image Stylization Method and System Based on Dual-Parallel Low-Rank Adaptation and Feature Decoupling
Patent Information
- Application Number
- CN202610903292.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-23
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2046-06-23
AI Technical Summary
[0005]本发明旨在克服现有技术中风格元素耦合、控制粒度粗、多模块兼容性差的问题,提供一种基于双并行低秩适配与特征解耦的图像风格化方法及系统
[0039]精细解耦控制:首次在扩散模型框架内,通过并行正交LoRA实现了纹理与颜色风格的显式解耦与独立控制,提供了前所未有的创作灵活性。
Smart Images

Figure CN122434723B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and in particular relates to an image stylization method and system based on dual parallel low-rank adaptation and feature decoupling. Background Technology
[0002] With the rapid development of text-to-image diffusion models, generative image stylization techniques have made significant progress, achieving high-quality style transfer with only a small number of samples. However, most existing diffusion-based stylization methods treat "style" as an indivisible whole concept, introducing style features into the model through a one-time injection. While this method can reproduce the overall style, it cannot independently manipulate and flexibly combine the constituent elements of the style, such as color, texture, and brushstrokes, making it difficult to meet the needs of personalized and customized creative generation.
[0003] For example, existing methods such as StyleDrop and ZipLoRA, while effectively capturing specific styles, only allow users to accept the full visual attributes of that style when applying it. Existing methods struggle to preserve the texture of a reference image while changing its color, or to combine the textures and colors of different images. Furthermore, simply merging multiple separately trained LoRA modules targeting different concepts often leads to feature interference, conceptual confusion, and a decline in generation quality, because the parameter update directions of different LoRA modules may conflict.
[0004] Therefore, the current technology field urgently needs a controllable style transfer method that can decouple image style into independent and controllable visual style elements while maintaining the quality of generation, and flexibly and compatiblely combine these elements into a new style. Summary of the Invention
[0005] This invention aims to overcome the problems of style element coupling, coarse control granularity, and poor multi-module compatibility in existing technologies, and provides an image stylization method and system based on dual-parallel low-rank adaptation and feature decoupling. This method can separate texture and color features from one or more reference images, and independently learn and bind them through two orthogonalized, low-interference LoRA modules trained in parallel, ultimately achieving decoupling and free control of texture and color in the image generation process.
[0006] In one aspect, this invention provides an image stylization method based on dual-parallel low-rank adaptation and feature decoupling, comprising the following steps:
[0007] Step 1: Initialize two independent low-rank adaptation modules to decouple features.
[0008] Two independent low-rank adaptation modules are constructed: a texture LoRA module and a color LoRA module.
[0009] For general diffusion models, Unet image features are used during their inference and training processes. Projected layer Projecting into a query matrix, hinting at embedding Through the projection layer and It is projected into a key matrix and a value matrix.
[0010] Each LoRA should be assigned a different hint. <t>It should only correspond to the projection layer of texture LoRA. ,and <c>Should only be used with the projection layer of color LoRA The resulting feature mapping becomes:
[0011]
[0012] in It is a texture descriptor <t>Embedded, It is a color descriptor <c>Embedded.
[0013] Step 2: Parallel training of importance selection and orthogonalization.
[0014] The texture LoRA module and the color LoRA module are trained in parallel using a style reference image. To ensure compatibility between the two LoRA modules, an importance selection strategy is introduced during training to generate images containing both the corresponding color and texture. During training, the weights updated in the first k rounds of each LoRA module are accumulated. After k rounds (starting from round k+1), only the top h% of parameter blocks with the highest weights in each module are updated. A selection mechanism is introduced so that for the same weighted block, only one LoRA module will select it for updating, ensuring that the sets of parameter blocks selected for updating by the two LoRA modules are different. This strategy avoids losing important information from the outset and significantly improves the compatibility of the two LoRA modules.
[0015] Simultaneously, a learnable coefficient matrix is introduced to gate the output of each LoRA module, further forcing the parameter spaces of the two modules to tend towards orthogonality, minimizing mutual interference, and achieving good compatibility. By introducing these masks, the attention map is updated as follows:
[0016]
[0017] Here, m represents the contribution of the corresponding LoRA column to the merged output, which is used to evaluate LoRA during inference.
[0018] Step 3: Enhanced color sensitivity.
[0019] An additional color consistency loss is designed into the training loss function of the color LoRA module. This loss is defined by calculating the difference in histogram distribution between the generated image and the color reference image in the color space.
[0020] Step 4: Decoupled reasoning generation.
[0021] During the inference phase, the trained texture LoRA and color LoRA modules are simultaneously loaded into the pre-trained diffusion model. Users can independently provide texture and color reference images (which can be the same or different images), activating the corresponding LoRA module through relevant prompts. During the denoising and generation process, the diffusion model integrates the adapted features from the two modules in parallel, ultimately synthesizing a stylized image that possesses both the specified texture and color features.
[0022] During inference, the integration of adaptive features from the two modules is specifically as follows: The weights of each LoRA are calculated after weighting to determine the potential contribution of each LoRA during inference. The outputs of the two branches of the LoRA in each attention layer are weighted based on these contributions. First, the key matrix projection layer and the learnable coefficient matrix in the gating control are multiplied by the dot product in each of the two LoRA modules, and the absolute values are taken to obtain the contribution coefficients of the two LoRA modules. Finally, the difference between the two contribution coefficients is used to obtain the contribution degree. .
[0023] First, take the top k of the contributions, and get :
[0024]
[0025] Where k is the proportion of elements selected, used to activate the weight matrix:
[0026]
[0027]
[0028] Weight threshold
[0029] in This is a matrix of all ones. The final LoRA weights used in the inference are obtained as follows:
[0030]
[0031]
[0032] This yields a value matrix and a key matrix that combine different image textures and colors, which can be used for subsequent attention map calculations and inference generation.
[0033] In another aspect, the present invention provides an image stylization system based on dual-parallel low-rank adaptation and feature decoupling, for implementing the aforementioned image stylization method, comprising the following modules:
[0034] The feature decoupling module is used to build two independent low-rank adaptation modules, namely the texture LoRA module and the color LoRA module, to achieve feature decoupling.
[0035] The parallel training module provides style reference images and performs importance selection and orthogonalization parallel training on two independent low-rank adaptation modules.
[0036] The color sensitivity enhancement module is used to design a color consistency loss in the training loss function of the color LoRA module to enhance color sensitivity.
[0037] The stylization result map output module is used to load the trained texture LoRA module and color LoRA module into the pre-trained diffusion model, perform decoupled inference generation, and obtain the image stylization result map.
[0038] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0039] Fine-grained decoupling control: For the first time within the diffusion model framework, explicit decoupling and independent control of texture and color styles are achieved through parallel orthogonal LoRA, providing unprecedented creative flexibility.
[0040] High compatibility and quality: By employing importance selection and orthogonalization training strategies, the feature interference problem during multi-LoRA merging is effectively solved, ensuring the compatibility of the two decoupled modules during joint inference. The generated images achieve high levels of color accuracy and texture fidelity.
[0041] Training is efficient and flexible: Only one or a few reference images are needed to train two decoupled feature modules at the same time. The computational cost is comparable to training a single LoRA, making it highly efficient.
[0042] High scalability: The method can be seamlessly integrated with existing conditional generation frameworks such as ControlNet, and can be easily extended to complex application scenarios such as real image editing and multimodal control. Attached Figure Description
[0043] Figure 1 This forms the overall framework of the present invention;
[0044] Figure 2 This is a stylized result of the present invention. Detailed Implementation
[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0046] This invention discloses an image stylization method and system based on dual-parallel low-rank adaptation and feature decoupling, aiming to solve the problems of style element coupling, coarse control granularity, and difficulty in achieving independent control of texture and color in existing diffusion model style transfer. The method first constructs two parallel and independent low-rank adaptation modules, respectively used to learn the texture and color features of the target style. By introducing an importance selection strategy, orthogonal and highly important parameter blocks are assigned to the two modules for updating during training, effectively avoiding feature interference between modules. Simultaneously, a color consistency loss based on color histograms is introduced for the color adaptation module to further enhance its sensitivity to color distribution. During the inference phase, the corresponding adaptation modules are activated through independent texture and color cues, and image generation is performed in conjunction with a pre-trained diffusion model, achieving decoupled and flexible control of texture and color. Furthermore, this method can be further combined with conditional control networks to support structure-preserving stylization of real images. This invention effectively overcomes the shortcomings of existing methods in style element decoupling and fine control, providing a more flexible creative tool for image stylization while ensuring generation quality. Experiments show that this method outperforms existing mainstream style transfer methods in terms of color alignment, texture fidelity, and overall aesthetic quality. For example... Figure 1 As shown, the training process uses two different cue words to learn two independent LoRA modules (color LoRA and texture LoRA), and applies orthogonality constraints through a key block selection strategy. This invention discloses an image stylization method based on dual-parallel low-rank adaptation and feature decoupling, specifically including the following steps:
[0047] Step 1: Initialize two independent low-rank adaptation modules to decouple features.
[0048] Two independent low-rank adaptation modules are constructed: a texture LoRA module and a color LoRA module.
[0049] For general diffusion models, Unet image features are used during their inference and training processes. Projected layer Projecting into a query matrix, hinting at embedding Through the projection layer and Projected as a key matrix and a value matrix:
[0050]
[0051] If the same hint is used for two LoRAs, the texture descriptor <t>and color descriptor <c>Both will be subject to cross-attention mechanisms, leading to cross-contamination during training. When used simultaneously... <t>and <c>At that time, the mapping in the cross-attention layer becomes:
[0052]
[0053] This can lead to content LoRA learning some style features, and style LoRA learning some content features, both overfitting to the entire input image. To avoid this cross-contamination, each LoRA should be assigned a different cue. For example, <t>It should only correspond to the projection layer of texture LoRA. ,and <c>Should only be used with the projection layer of color LoRA They are correlated. The resulting feature map becomes:
[0054]
[0055] in It is a texture descriptor <t>Embedded, It is a color descriptor <c>The embedding of LoRA allows the two LoRAs to focus on different concepts, thus effectively decoupling their learning objectives.
[0056] Step 2: Parallel training of importance selection and orthogonalization.
[0057] The texture LoRA module and the color LoRA module are trained in parallel using a style reference image. To ensure compatibility between the two LoRA modules, an importance selection strategy is introduced during training to generate images containing both the corresponding color and texture. This strategy involves accumulating the weights updated within each LoRA module over the first k rounds. After k rounds of training, only the top h% of parameter blocks with the highest weights in each module are updated. Furthermore, a selection mechanism is introduced so that for the same weighted block, only one LoRA module will select it for updating, ensuring that the sets of parameter blocks selected for updating by the two modules are different. This strategy avoids losing important information from the outset and significantly improves the compatibility between the two LoRA modules.
[0058] Simultaneously, a learnable coefficient matrix is introduced to gate the output of each LoRA module, further forcing the parameter spaces of the two modules to tend towards orthogonality, minimizing mutual interference, and achieving good compatibility. By introducing these masks, the attention map is updated as follows:
[0059]
[0060] Here, m represents the contribution of the corresponding LoRA column to the merged output, which is used to evaluate LoRA during inference.
[0061] Step 3: Enhanced color sensitivity.
[0062] Although the learning regions of the two LoRA modules are separated, the diffusion model still uses large areas of the same color to fill the generated image when obtaining the inference result. To alleviate this problem, an additional color consistency loss is designed into the training loss function of the color LoRA module. This loss is defined by calculating the difference in histogram distribution between the generated image and the color reference image in the color space, and its expression can be simplified to:
[0063]
[0064] Where B represents the number of histogram intervals, and c represents the color channel. and These represent the histograms of the generated image and the reference image in the c-th channel, respectively. Through color loss, the color distribution of the image will more closely resemble the style image, rather than simply selecting a single color from the style image and filling it in excessively.
[0065] Step 4: Decoupled reasoning generation.
[0066] During the inference phase, the trained texture LoRA and color LoRA modules are simultaneously loaded into the pre-trained diffusion model. Users can independently provide texture and color reference images (which can be the same or different images), activating the corresponding LoRA module through relevant prompts. During the denoising and generation process, the diffusion model integrates the adapted features from the two modules in parallel, ultimately synthesizing a stylized image that possesses both the specified texture and color features.
[0067] During inference, the weights of each LoRA are calculated to determine its potential contribution. These contributions are then used to weight the outputs of the two branches of the LoRA within each attention layer. First, in each of the two LoRA modules, the key matrix projection layer is multiplied by the learnable coefficient matrix in the gating control, and then the absolute value is taken. The contribution coefficients of the two LoRA modules were obtained separately, and the contribution degree was obtained by subtracting the two contribution coefficients. The contribution calculation formula is as follows:
[0068]
[0069] First, take the top k of the contributions, and get :
[0070]
[0071] Where k is the proportion of elements selected, used to activate the weight matrix:
[0072]
[0073]
[0074] in This is a matrix of all ones. The final LoRA weights used in the inference are obtained as follows:
[0075]
[0076]
[0077] This yields a value matrix and a key matrix that combine different image textures and colors, which can be used for subsequent attention map calculations and inference generation.
[0078] Furthermore, the method may also include an extended application step: combining the trained dual LoRA module with the Conditional Control Network (ControlNet). During inference, in addition to texture and color cues, the additional input content image's structure map, typically an edge map based on the Canny algorithm, is used to achieve decoupled stylization of texture and color on the real image while preserving its original structure.
[0079] In another aspect, the present invention provides an image stylization system based on dual-parallel low-rank adaptation and feature decoupling, for implementing the aforementioned image stylization method, comprising a feature decoupling module, a parallel training module, a color sensitivity enhancement module, and a stylization result map output module.
[0080] The feature decoupling module is used to construct two independent low-rank adaptation modules, namely the texture LoRA module and the color LoRA module, to achieve feature decoupling.
[0081] The parallel training module is used to provide style reference images and perform importance selection and orthogonalization parallel training on two independent low-rank adaptation modules.
[0082] The color sensitivity enhancement module is used to design a color consistency loss in the training loss function of the color LoRA module to enhance color sensitivity.
[0083] The stylized result map output module is used to load the trained texture LoRA module and color LoRA module into the pre-trained diffusion model, perform decoupled inference generation, and obtain the image stylized result map.
[0084] Table 1 uses MS-SWD, C-Hist, FID, and CLIP-I as evaluation metrics. These metrics consider both color and texture preservation, enabling the evaluation of overall style transfer quality. MS-SWD and C-Hist are used to measure the color similarity between the stylized image and the style image. FID and CLIP-I are used to evaluate the texture similarity between the stylized image and the style image. Baseline methods for comparison include SADis and InstantStyle. Experimental results show that this invention is slightly superior to SADis in color alignment and significantly surpasses all other methods. In texture representation, this method is only slightly inferior to SADis, further illustrating the advantages of this invention.
[0085] Table 1
[0086]
[0087] Figure 2 The following is an experimental result diagram of the present invention. The first column is the input color image, the second column is the input texture image, and the following columns are style transfer result diagrams generated based on the input image and the corresponding stylized image. It can be seen from the result diagram that the present invention can perform complete, accurate and efficient style transfer based on the required style image, and has broad application prospects in multiple fields of computer vision and image processing.
[0088] This invention provides an efficient, flexible, and high-quality image style decoupling and control system, which has broad application prospects in digital art creation, advertising design, film and television special effects, and other fields. For those skilled in the art, any improvements and modifications made to the training strategy, loss function, model structure, etc., without departing from the principles of this invention, should be included within the scope of protection of this invention.< / c> < / t> < / c> < / t> < / c> < / t> < / c> < / t> < / c> < / t> < / c> < / t>
Claims
1. An image stylization method based on dual parallel low-rank adaptation and feature decoupling, characterized in that, Includes the following steps: Step 1: Construct two independent low-rank adaptation modules, namely the texture LoRA module and the color LoRA module, to achieve feature decoupling; Step 2: Provide a style reference image, and perform importance selection and orthogonalization parallel training on two independent low-rank adaptation modules. The specific implementation process is as follows: Using style reference images, texture LoRA modules and color LoRA modules are trained in parallel. During training, an importance selection strategy is introduced: during training, by accumulating the weights updated in the first k rounds within each LoRA module, starting from the k+1 round of training, only the top h parameter blocks with the highest weights in each LoRA module are updated, and it is ensured that the set of parameter blocks to be updated selected by the two LoRA modules are different. Meanwhile, a learnable coefficient matrix is introduced to perform gating control on the output of each LoRA module, forcing the parameter spaces of the two modules to tend to be orthogonal and minimizing mutual interference; the gating control calculates and updates the attention map by multiplying the learnable coefficient matrix with the different cue embeddings z of the two LoRA modules. Step 3: In the training loss function of the color LoRA module, design a color consistency loss to enhance color sensitivity. The specific implementation process is as follows: Color consistency loss is defined by calculating the difference in histogram distribution between the generated image and the color reference image in the color space: and Let represent the histograms of the generated image and the reference image in the c-th channel, respectively. Indicates the first... Each interval, first through Obtain the histogram of the generated image and the reference image in the c-th channel. The minimum value of each interval is calculated, and then the minimum values of all intervals are summed to obtain the histogram distribution difference P. The mean of 1-P is calculated in the three color channels to obtain the color consistency loss. Step 4: Load the trained texture LoRA module and color LoRA module into the pre-trained diffusion model, perform decoupled inference generation, and obtain the image stylization result map.
2. The image stylization method based on dual-parallel low-rank adaptation and feature decoupling according to claim 1, characterized in that, In step 1: For the diffusion model, each LoRA module is assigned a different cue during inference and training. <t>It only corresponds to the projection layer of the LoRA texture, while <c>The key matrix is obtained by adding the products of the two different key matrix projection layers to the projection layers of the two different key matrix projection layers, corresponding only to the different cue embeddings z of the two key matrix modules. The key matrix is obtained by adding the products of the different hint embeddings z from the two LoRA modules to the projection layers of the two different value matrices. .< / c> < / t> 3. The image stylization method based on dual-parallel low-rank adaptation and feature decoupling according to claim 2, characterized in that, In step 4: During the inference phase, the trained texture LoRA module and color LoRA module are simultaneously loaded into the pre-trained diffusion model; the user can independently provide texture reference images and color reference images, and activate the corresponding LoRA module through the corresponding prompts; during the denoising generation process, the diffusion model integrates the adaptation features from the two modules in parallel, and finally synthesizes a stylized image that has both specified texture features and specified color features.
4. The image stylization method based on dual-parallel low-rank adaptation and feature decoupling according to claim 3, characterized in that, The specific process of integrating the adaptation features from the two modules is as follows: First, by calculating the weights of each LoRA after weighting, the contribution of each LoRA in the inference process is determined. Then, the outputs of the two branch LoRAs in each attention layer are weighted according to their contributions. The contribution is calculated as follows: First, in each of the two LoRA modules, the key matrix projection layer is multiplied by the learnable coefficient matrix in the gating control, and the absolute value is taken to obtain the contribution coefficients of the two LoRA modules. Finally, the contribution is obtained by subtracting the two contribution coefficients. ; Contribution Take the top k, then find the minimum value to obtain the weight threshold. Contribution All elements and weight thresholds Do the work, and then go through The function obtains ,pass To achieve the result by inverting the values, This completes the soft weighting of the weight matrix; and The key matrix is obtained by dot-producting the different hint embeddings z from the two LoRA modules and the key matrix projection layer, and then summing the results of the dot-products to obtain the final key matrix used for inference. ; and The value matrix is obtained by multiplying the different hint embeddings z from the two LoRA modules with the projection layer of the value matrix, and then summing the results of the multiplication to obtain the final value matrix used for inference. .
5. The image stylization method based on dual-parallel low-rank adaptation and feature decoupling according to claim 4, characterized in that, The image stylization method also includes an extended application step: combining the two trained LoRA modules with the Conditional Control Network ControlNet; during inference, in addition to texture and color cues, an additional structural map of the content image is input, thereby achieving decoupled stylization of texture and color on the real image while maintaining its original structure.
6. An image stylization system based on dual-parallel low-rank adaptation and feature decoupling, used to implement the image stylization method according to any one of claims 1 to 5, characterized in that, Includes the following modules: The feature decoupling module is used to construct two independent low-rank adaptation modules, namely the texture LoRA module and the color LoRA module, to achieve feature decoupling. A parallel training module is used to provide style reference images and perform importance selection and orthogonalization parallel training on two independent low-rank adaptation modules. The color sensitivity enhancement module is used to design a color consistency loss in the training loss function of the color LoRA module to enhance color sensitivity. The stylization result map output module is used to load the trained texture LoRA module and color LoRA module into the pre-trained diffusion model, perform decoupled inference generation, and obtain the image stylization result map.
