Federal style migration and diffusion generation method based on LoRA aggregation
By introducing semantic initialization of style keywords and iterative optimization of global style LoRA in the federated diffusion model, the problems of structural heterogeneity and privacy leakage of LoRA module are solved, realizing efficient integration of personalized content and unified style, and improving generation quality and privacy protection.
Patent Information
- Application Number
- CN202511515108.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2026-03-20
AI Technical Summary
Existing LoRA fusion methods struggle to effectively integrate personalized content with a unified style in federated diffusion models when faced with non-independent identical distributions and structural heterogeneity, and also pose privacy risks and inconsistent generation quality.
By introducing a semantic initialization mechanism for style keywords and an iterative optimization strategy for global style LoRA, a content-style fusion mechanism is constructed. The content LoRA and style LoRA are explicitly divided into orthogonal subspaces, and a two-stage style aggregation is performed on the server side. Combined with CLIP semantic embedding and k-means clustering, the effective fusion of personalized content and unified style is achieved.
It significantly improves the quality of generated content and privacy robustness, ensuring consistency and diversity in personalized content and unified style, and is suitable for privacy-sensitive federated diffusion generation tasks.
Smart Images

Figure CN121708133A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of federated LoRA adaptive technology, and in particular to a federated style migration diffusion generation method based on LoRA aggregation. Background Technology
[0002] Low-Rank Adaptation (LoRA) fine-tuning has emerged as a lightweight and flexible technique in recent years for customizing large-scale diffusion models to specific content or style domains. Compared to traditional full model retraining, LoRA enables rapid transfer and combination of independent content or style modules, significantly reducing computational and storage costs and enhancing its applicability in multi-concept fusion scenarios. As research has progressed, researchers have proposed several fusion strategies for LoRA, such as modular fusion methods that achieve flexible combination of style and content through layer-by-layer or attention mechanisms, generative quality enhancement methods that improve visual fidelity through noise reduction reconstruction or spatial masking, and adaptive selection methods that use semantic similarity or CLIP guidance to filter and integrate representative modules. Further developments have combined LoRA with broader style transfer frameworks, achieving decoupling of text semantics and style features, continuous customization, and fine-grained style and content separation. These advancements have driven the control of style and content within a unified framework and demonstrated the broad potential of LoRA in multimodal generation tasks.
[0003] However, existing methods are typically based on centralized training, which by default allows access to all LoRA weights and their corresponding data distributions. This not only ignores the variability caused by cross-client data differences but also introduces the risk of privacy breaches in privacy-sensitive scenarios. The generated output may unintentionally reveal user-specific style patterns or semantic priors, thus limiting its widespread adoption in distributed applications.
[0004] The rise of Parameter-Efficient Fine-Tuning (PEFT) has driven the application of LoRA in Federated Learning (FL), providing a lightweight alternative for adapting large-scale models to distributed clients. LoRA effectively reduces communication overhead and enhances personalization capabilities by updating and swapping only low-rank modules rather than the full weights, and exhibits superior scalability in heterogeneous environments. Early studies often employed simple strategies, such as directly applying Federated Averaging (FedAvg) to LoRA matrices A and B. However, in non-IID distributions, this naive decomposition often leads to negative transfer. To alleviate this problem, LoRM and FedSA-LoRA proposed matrix scheduling mechanisms, such as freezing one matrix while aggregating another, or alternating updates at different stages, thereby achieving a balance between local adaptability and global generalization. Subsequently, RoLoRA and FlexLoRA utilized Singular Value Decomposition (SVD) to mitigate rank mismatch and structural inconsistency issues, and improved the robustness of aggregation through implicit regularization.
[0005] Despite this, most existing methods still assume cross-client structural homogeneity, meaning all LoRA modules are injected with the same architectural components. However, in diffusion model generation tasks, users often fine-tune different sub-modules (such as UNet cross-attention blocks or text encoders) according to personalized goals, resulting in significant structural heterogeneity. This makes direct aggregation of LoRA modules difficult, as merging LoRA modules trained at different injection points can easily lead to semantic mismatches or degraded generation quality. Furthermore, the iterative sampling and intermediate outputs of diffusion models further amplify privacy risks. Recent centralized solutions (such as Recycling LoRA and HydraLoRA) can flexibly reuse LoRA modules or adopt expert hybrid strategies, but when applied in federated scenarios, they may unintentionally reveal client-specific style patterns. Even though FedEx-LoRA and FedDP-LoRA introduce residual correction and privacy protection, they are still insufficient in addressing the structural heterogeneity and privacy challenges unique to diffusion models. In this context, LoRA modules are often injected with different sub-components and optimized for different visual cues, making direct aggregation infeasible.
[0006] Existing LoRA fusion methods exhibit several limitations when dealing with non-independent and identically distributed (non-IID) data and structural heterogeneity, particularly when applied to Federated Diffusion Models (FDM). On one hand, many methods rely on centralized LoRA storage and synthesis, such as LoRA Recycle. While these methods are effective in improving adaptability and generation quality, the user-specific features encoded in the parameters may pose serious privacy risks. For example, in image generation tasks, the iterative decoding process may expose a user's facial style or content features, making them vulnerable to member inference or model inversion attacks.
[0007] On the other hand, some methods attempt to mitigate client-side variability by adjusting the module structure or introducing a Mixture of Experts (MoE) model. However, this strategy often results in a significant increase in model complexity, placing high demands on computational and communication resources, and is unsuitable for resource-constrained terminal environments. Furthermore, most current methods lack an effective mechanism for unifying the LoRA module injection location. Since LoRA in diffusion models may be injected into different attention layers in the text encoder or UNet, semantic misalignment and style drift problems occur during style aggregation, reducing the consistency and diversity of the generated results.
[0008] More importantly, existing federated aggregation strategies (such as FedAvg) often cause conflicts when faced with highly personalized parameters like style LoRA due to a lack of discriminativeness, failing to effectively integrate style expression and semantic fidelity. Although some optimization strategies have introduced mechanisms such as weighted aggregation or orthogonal decoupling, they still cannot prevent style information leakage and lack sufficient robustness against attacks, failing to resist strong attackers' sensitivity probing of LoRA parameters. Summary of the Invention
[0009] The purpose of this invention is to address the technical deficiencies in the existing technology by providing a federated style migration diffusion generation method based on LoRA aggregation.
[0010] The technical solution adopted to achieve the purpose of this invention is:
[0011] A federated style migration diffusion generation method based on LoRA aggregation includes the following steps:
[0012] Step 1, for each client N i Load the pre-trained weights W0 and freeze the base diffusion model SD, N for each client. i Using private dataset D i ={(x i ,p iPerform low-rank adaptation on the content LoRA, parameterizing the content LoRA into a content LoRA set.
[0013] Step 2, for each client N i Submit style keywords to the server, and the server retrieves each client's N from the publicly available LoRA style library. i Style LoRA set corresponding to style keywords Based on CLIP scores from the LoRA style set Selecting k styles of LoRA and minimizing the noise prediction loss function Training style weight matrix Then, the k style LoRAs are aggregated layer by layer to obtain the aggregated style LoRA. If the number of style LoRA layers is inconsistent during the aggregation process, the missing style LoRA layers are filled with an all-zero matrix, and the style weight matrix obtained after training is used. N is sent back to each client i ;
[0014] Step 3, each client N i The LoRA set obtained in step 1 and the style weight matrix trained in step 2 The fusion yields the initial content-style diffusion model and the updated style weight matrix. And calculate N for each client i semantic vector C i Each client N i semantic vector C i and Uploaded to the server, the server uses semantic vector C i The style weight matrix after one update Perform k-means clustering, and then normalize and average the style weight matrices within the same group based on the clustering results to obtain the global style matrix α. g The server will use the global style matrix α g The data is sent back to the corresponding client, and each client N... i For the global style matrix α g LoRA Collection The fusion yields the content-style diffusion model D. L Introducing a regularization loss function For the content-style diffusion model D LOptimization is performed to complete one iteration. Each client generates a test image using the latest content-style diffusion model. The server calculates the CLIP score of the test image. If the CLIP score is higher than the individual style LoRA CLIP score of each client, the iteration stops; otherwise, it continues until the CLIP score of the test image is higher than the individual style LoRA CLIP score of each client.
[0015] In the above technical solution, in step 2, the aggregated style LoRA The calculation formula is:
[0016]
[0017] In the formula, ⊙ represents element-wise product. For style weight matrix, The weights for style LoRA.
[0018] In the above technical solution, in step 3, the semantic vector C i The calculation formula is as follows:
[0019]
[0020] In the formula, M i For client N i The number of image data in the private dataset, x j For the j-th image in the private dataset, CLIP image The semantic vector score.
[0021] In the above technical solution, in step 3, the global style matrix α g The calculation formula is:
[0022]
[0023] In the formula, α g This represents the global style matrix, where N is the number of clients using the same style. This is the normalized style weight matrix;
[0024] The normalized style weight matrix The calculation formula is:
[0025]
[0026] In the formula, μ(α) i ) and σ(α) i These are the normalized style weight matrices. The mean and standard deviation.
[0027] In the above technical solution, in step 2, the noise prediction loss function The calculation formula is:
[0028]
[0029] In the formula, ∈ represents injected Gaussian noise. The noise predicted by the diffusion model, For the image after adding noise, p j Text prompts for the image. The aggregated style is LoRA.
[0030] In the above technical solution, in step 3, the content-style diffusion model D L The calculation formula is:
[0031]
[0032] In the formula, D L For the merged content-style diffusion model, SD is the basic diffusion model, α C Let α be the learnable weight matrix of content LoRA. i Let α be the weight of the LoRA of the i-th layer content. S For the learnable weights of the LoRA style group, α ij Let m be the weight of the i-th layer of the j-th style LoRA, and m be the number of LoRA layers.
[0033] In the above technical solution, in step 3, the regularization loss function The calculation formula is:
[0034]
[0035] In the formula, The regularization loss function is... For style loss function, Let D be the content loss function. L For the merged content-style diffusion model, SD is the basic diffusion model, α C For the learnable weight matrix of content LoRA, For content LoRA, α S Learnable weights for the LoRA style group For the LoRA style.
[0036] A second aspect of the present invention is an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.
[0037] A third aspect of the present invention is a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.
[0038] A fourth aspect of the present invention is a computer program product comprising computer-executable instructions that, when executed, implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.
[0039] Compared with the prior art, the beneficial effects of the present invention are:
[0040] 1. This invention proposes a federated low-rank adaptation and aggregation framework for personalized image generation tasks using diffusion models. This framework aims to address key challenges in traditional LoRA modules, such as privacy leakage risks in centralized adaptation and structural heterogeneity and aggregation failure in federated scenarios. By constructing a semantic initialization mechanism for style keywords and an iterative optimization strategy for global style LoRA, the consistency of the global model in terms of content and style is enhanced. To address the semantic misalignment problem of LoRA modules in federated scenarios during structural heterogeneity and style aggregation, a style keyword-guided semantic initialization mechanism is injected into the diffusion model with shareable style priors before federated training begins, thereby significantly reducing early style drift and shortening the convergence rounds.
[0041] 2. This invention addresses the problem that traditional methods struggle to simultaneously maintain personalized content features and a unified style expression in federated scenarios. It proposes a content-style fusion mechanism that, at the client level, explicitly divides the personalized low-rank adaptation module into two orthogonal subspaces: content LoRA and style LoRA. The former is used to maintain the semantic and structural consistency of the image, while the latter is used to capture personalized style features, thereby achieving effective isolation between sensitive attributes and key task representations. On the server side, a two-stage style aggregation mechanism is proposed: The first stage is a semantic initialization mechanism guided by style keywords. This mechanism downloads corresponding style LoRAs from the LoRA library through CLIP semantic embedding and a public prompt lexicon, constructs an initial hierarchical fusion matrix α, adaptively aligns latent structures, and achieves consistency and personalized stylization under federated constraints based on effective cross-client integration of heterogeneous LoRA modules. This extends the generation capabilities of LoRA to privacy-sensitive scenarios. The second stage is an iterative optimization strategy for global style LoRA. While maintaining the privacy of local data, it clusters clients based on semantic prototypes, performs federated averaging, and iteratively optimizes the fusion matrix. This improves the global generalization ability and expressive accuracy of the aggregated style representation. The final generated global style LoRA maintains the privacy of local data while taking into account the diversity and consistency of style features across multiple clients, achieving a unified cross-client personalized style transfer and global generalization capability. This provides a reliable solution for personalized image generation in privacy-sensitive environments. By introducing a content-style fusion mechanism during the diffusion generation process, the personalized content of the client and the aggregated style are effectively combined, achieving effective fusion of client-specific personalized content and aggregated style, thereby improving the generation consistency and expressive stability of the global model. This invention significantly improves the generation quality and privacy robustness of cross-client style transfer. Attached Figure Description
[0042] Figure 1 The diagram shows a federated style migration diffusion generation method based on Lora aggregation according to the present invention. Detailed Implementation
[0043] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0044] Reference Figure 1 A federated style migration diffusion generation method based on LoRA aggregation includes the following steps:
[0045] Step 1, for each client N i Maintaining a private dataset D i And a frozen base diffusion model SD initialized with pre-trained weights W0. To integrate SD with its private dataset D. iAlignment, N per client i Using private dataset D i ={(x i ,p i Training client-specific content LoRA, where x i For the image, p i The text prompt for the image. The resulting LoRA content is parameterized as follows: Low-rank adaptation of pre-trained weights W0 for encoding client-specific visual concepts is captured. The corresponding style LoRA parameterization for each client's style keywords is then performed.
[0046] Step 2, for each client N i The user uploads their style keywords to the server, which then retrieves a set of style LoRAs matching the style keywords from publicly available style LoRA databases (such as CivitAI and HuggingFace) on the internet. Based on the CLIP scores, k style LoRA models are selected from the style LoRA set. The maximum number of layers for fine-tuning in these style LoRA models is m. The server expands the style keywords uploaded by the client into complete style cue words using ChatGPT. Several images are generated based on the cue words, and the CLIP score of each image is calculated. The top-H style cue words and images with the highest CLIP scores are selected to train the style weight matrix. in, This represents the weighting coefficient of LoRA for the j-th style at the i-th level;
[0047] During training, the noise prediction loss function is minimized. Noise addition and denoising operations are performed on each image, and the noise prediction loss function is... The noise prediction loss function is defined as the similarity between the predicted noise and the actual noise of an image. The calculation formula is:
[0048]
[0049] In the formula, ∈ represents injected Gaussian noise. The noise predicted by the diffusion model. For the image, p j Text prompts for the image. The aggregated style is LoRA.
[0050] The aggregated style LoRA is obtained by performing layer-by-layer aggregation on k style LoRAs. If the number of style LoRA layers is inconsistent during the aggregation process, the missing style LoRA layers are filled with an all-zero matrix, and the style weight matrix obtained after training is used. N is sent back to each client i ;
[0051] The aggregated style LoRA The calculation using element-wise multiplication is represented as follows:
[0052]
[0053] In the formula, ⊙ represents element-wise product. For the weights of style LoRA, this invention uses an all-zero matrix to fill in the missing style LoRA layers to address the issue of missing style LoRA layers. This operation not only solves the problem of heterogeneous style LoRA obtained from the Internet, but also preserves the independence of the parameters of each layer.
[0054] Step 3, each client N i LoRA, the content obtained in step 1 and the style weight matrix trained in step 2 The fusion yields the initial content-style diffusion model and the updated style weight matrix. By integrating an enhanced pattern fusion mechanism into the client, it is possible to ensure that subsequent federated aggregation is based on semantically aligned and style-diverse local representation expansion, which helps to improve the adaptability and generalization ability of the final global model on heterogeneous client data.
[0055] The content LoRA obtained in step 1 and the style weight matrix trained in step 2 The fusion process is as follows:
[0056] Each client will use the content LoRA, style LoRA, and style weight matrix obtained in step 1. Local fusion yields the content-style diffusion model D. L The D L The formula is expressed as:
[0057]
[0058] In the formula, D L For the merged content-style diffusion model, SD is the basic diffusion model, α C Let α be the learnable weight matrix of content LoRA. i Let LoRA be the weight of the content in layer i. For content LoRA, α S For the learnable weights of the LoRA style group, αij Let m be the weight of the i-th layer of the j-th style LoRA, and m be the number of LoRA layers. For the LoRA style.
[0059] To constrain the consistency between content and style signals and prevent overfitting to one aspect of a feature, this invention introduces a regularization loss function at the fusion output. In this way, the consistency of the generated images in terms of semantic structure and subject features can be guaranteed, and the styles can be flexibly switched. This significantly alleviates the style drift and generation instability caused by differences in client data and different LoRA injection locations.
[0060] The regularization loss function The calculation formula is:
[0061]
[0062] In the formula, The regularization loss function is... For style loss function, Let D be the content loss function. L For the merged content-style diffusion model, SD is the basic diffusion model, α C For the learnable weight matrix of content LoRA, For content LoRA, α S Learnable weights for the LoRA style group For the LoRA style.
[0063] By minimizing the regularization loss function For α C and α S Optimization is performed to obtain the optimal initial content-style diffusion model and the style weight matrix after one update.
[0064] N per client i Calculate client N i semantic vector C i The semantic vector C i and Uploaded to the server, the server uses semantic vector C i The style weight matrix after one update Perform k-means clustering, and then normalize and average the style weight matrices within the same group based on the clustering results to obtain the global style matrix α. g The global style matrix α is obtained by weighted aggregation of the same style weight matrix across different clients. gIt retains the diversity of style features across multiple clients and achieves efficient integration of style LoRA and content LoRA across multiple clients, thereby enhancing the model's adaptability and generalization in multi-style transfer tasks.
[0065] The semantic vector C i The calculation formula is as follows:
[0066]
[0067] In the formula, M i For client N i The number of image data in the private dataset, x j For the j-th image in the private dataset, CLIP image The semantic vector score.
[0068] The weighted aggregation process for the same style weight matrix is as follows:
[0069] To prevent structural mismatches in the style weight matrix, it is normalized to eliminate scale differences between matrices uploaded from different clients. This normalization process is applied to the style weight matrix obtained with each update. Dimension checks are performed to remove matrices from uploaded data whose dimensions do not meet expectations, preventing structural mismatches later. After confirming dimensional consistency, the style weight matrix is normalized to eliminate scale differences between style weight matrices uploaded from different clients, thus normalizing the style weight matrix. The calculation formula is as follows:
[0070]
[0071] In the formula, μ(α) i ) and σ(α) i These are the normalized style weight matrices. The mean and standard deviation are calculated. This effectively reduces aggregation errors caused by inconsistent data distribution.
[0072] For the style weight matrix The weighted aggregation process employs an average weighting strategy for the normalized style weight matrix. The global style matrix α is obtained by performing average aggregation. g The formula is
[0073]
[0074] In the formula, α g This represents the global style matrix, where N is the number of clients using the same style. This strategy, which normalizes the style weight matrix, improves robustness to outliers and imbalanced data while ensuring the consistency of the matrix structure, thereby obtaining a more stable global style representation.
[0075] The server will use the global style matrix α g The data is sent back to the corresponding client, and each client N... i For the global style matrix α g And content LoRA The fusion yields the content-style diffusion model D. L Introducing a regularization loss function For the content-style diffusion model D L To perform optimization is to complete one iteration.
[0076] Each client utilizes the latest content-style diffusion model D L A new test image is generated. The server calculates the CLIP score of the new test image and the style cue words. This score is compared with the CLIP score of each client's individual style LoRA before aggregation. If the CLIP score is higher than the individual style LoRA CLIP score of each client, the iteration stops; otherwise, it continues until the CLIP score of the test image is higher than the individual style LoRA CLIP score of each client. The server then combines the style LoRA set and the updated style weight matrix. Distribute to the corresponding clients for subsequent local optimization and iteration.
[0077] Compared to traditional globally uniform weighting methods such as FedAvg or direct summation, the hierarchical matrix weighting strategy proposed in this invention can better adjust the importance of different layers with fine granularity, avoiding the loss of specialization ability caused by direct summation when the number of layers and structure are inconsistent. The aggregated weights not only retain the advantages of individual LoRA in their respective strengths of prompt words, but also significantly improve the generation quality in the originally weak prompt words, demonstrating that hierarchical weighted aggregation significantly improves the model's style generalization ability while enhancing synergistic effects.
Claims
1. A federated style migration and diffusion generation method based on LoRA aggregation, characterized in that, Includes the following steps: Step 1, for each client N i Load the pre-trained weights W0 and freeze the base diffusion model SD, N for each client. i Using private dataset D i ={(x i ,p i Perform low-rank adaptation on the content LoRA, parameterizing the content LoRA into a content LoRA set. Step 2, for each client N i Submit style keywords to the server, and the server retrieves each client's N from the publicly available LoRA style library. i Style LoRA set corresponding to style keywords Based on CLIP scores from the LoRA style set Selecting k styles of LoRA and minimizing the noise prediction loss function Training style weight matrix Then, the k style LoRAs are aggregated layer by layer to obtain the aggregated style LoRA. If the number of style LoRA layers is inconsistent during the aggregation process, the missing style LoRA layers are filled with an all-zero matrix, and the style weight matrix obtained after training is used. N is sent back to each client i ; Step 3, each client N i The LoRA set obtained in step 1 and the style weight matrix trained in step 2 The fusion yields the initial content-style diffusion model and the updated style weight matrix. And calculate N for each client i semantic vector C i Each client N i semantic vector C i and Uploaded to the server, the server uses semantic vector C i The style weight matrix after one update Perform k-means clustering, and then normalize and average the style weight matrices within the same group based on the clustering results to obtain the global style matrix α. g The server will use the global style matrix α g The data is sent back to the corresponding client, and each client N... i For the global style matrix α g LoRA Collection The fusion yields the content-style diffusion model D. L Introducing a regularization loss function For the content-style diffusion model D L Optimization is performed to complete one iteration. Each client generates a test image using the latest content-style diffusion model. The server calculates the CLIP score of the test image. If the CLIP score is higher than the individual style LoRA CLIP score of each client, the iteration stops; otherwise, it continues until the CLIP score of the test image is higher than the individual style LoRA CLIP score of each client.
2. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 2, the aggregated style LoRA The calculation formula is: In the formula, ⊙ represents element-wise product. For style weight matrix, The weights for style LoRA.
3. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 3, the semantic vector C i The calculation formula is as follows: In the formula, M i For client N i The number of image data in the private dataset, x j For the j-th image in the private dataset, CLIP image The semantic vector score.
4. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 3, the global style matrix α g The calculation formula is: In the formula, α g This represents the global style matrix, where N is the number of clients using the same style. This is the normalized style weight matrix; The normalized style weight matrix The calculation formula is: In the formula, μ(α) i ) and σ(α) i These are the normalized style weight matrices. The mean and standard deviation.
5. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 2, the noise prediction loss function The calculation formula is: In the formula, ∈ represents injected Gaussian noise. The noise predicted by the diffusion model, For the image after adding noise, p j Text prompts for the image. The aggregated style is LoRA.
6. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 3, the content-style diffusion model D L The calculation formula is: In the formula, D L For the merged content-style diffusion model, SD is the basic diffusion model, α C Let α be the learnable weight matrix of content LoRA. i Let α be the weight of the LoRA of the i-th layer content. S For the learnable weights of the LoRA style group, α ij Let m be the weight of the i-th layer of the j-th style LoRA, and m be the number of LoRA layers.
7. The federated style migration and diffusion generation method based on LoRA aggregation as described in claim 1, characterized in that, In step 3, the regularization loss function The calculation formula is: In the formula, The regularization loss function is... For style loss function, Let D be the content loss function. L For the merged content-style diffusion model, SD is the basic diffusion model, α C For the learnable weight matrix of content LoRA, For content LoRA, α S Learnable weights for the LoRA style group For the LoRA style.
8. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.
9. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed, are used to implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.
10. A computer program product, characterized in that, The aforementioned computer program product includes computer-executable instructions that, when executed, implement the federated style migration diffusion generation method based on LoRA aggregation as described in claim 1.