Method for generating multitask stable diffusion model based on task arithmetic

Abstract

According to the method for generating the multitask stable diffusion model based on the task arithmetic, additional training is not needed, the model is directly edited, and the multitask processing capacity is achieved. According to the method, through addition and subtraction operation of task arithmetic, model features of multiple tasks are fused into one model, and the multi-task processing capacity of the model is enhanced. Specifically, the stable diffusion model can generate models of opposite functions through'counter-moving 'and integrate into models of other concept enhancement / elimination tasks. The method comprises: a) enhancing the generated concept by task vector subtraction; and b) realizing multi-task processing through task vector addition. Experimental verification is carried out in a text-to-image generation task of three condition generation categories: 1) an object; 2) style; and 3) condition control. Results show that the model can integrate different tasks without additional training, storage and computing resources are saved, and performance is improved at the same time. The method has the advantages of simplicity, practicability and wide application prospect.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and image generation, and in particular to a method for generating a multi-task stable diffusion (SD) model based on task arithmetic. The method realizes the generation of a model with multi-task feature fusion through addition and subtraction operations of task vectors without additional training or fine-tuning. Background Art

[0002] With the rapid development of artificial intelligence (AIGC), image generation techniques based on diffusion models, such as Stable Diffusion (SD), have been widely used in the field of computer vision. These models use large-scale pre-trained models as a basis and adapt to specific task requirements through fine-tuning. However, traditional image generation model fine-tuning methods usually require a lot of computing resources and time, and are difficult to apply efficiently in multi-task scenarios.

[0003] In the existing technology, the improvement of the generative model usually relies on further training or fine-tuning of the pre-trained model to adapt to new task requirements. Although this method can improve the performance of the model on specific tasks, it requires high computing costs, and when dealing with multi-task scenarios, multiple fine-tuned models need to be loaded at the same time, resulting in a waste of computing resources and an increase in storage space. Therefore, how to integrate multiple task capabilities into one model without adding additional training overhead has become a hot topic in current research. Summary of the invention

[0004] The purpose of the present invention is to propose a method for generating a multi-task stable diffusion model based on task arithmetic in view of the limitations of existing image generation technology in multi-task processing. The method directly edits and enhances at the model level by means of task arithmetic to achieve seamless integration of multi-task processing capabilities. The method uses task arithmetic to directly edit and enhance at the model level without additional training or fine-tuning, thereby generating a new multi-task model. Specifically, the method can obtain a model with the opposite characteristics of the original task through task subtraction operations; through task addition operations, multiple models based on Stable Diffusion (SD) can be fused, so that the new model can solve the multi-task scenarios corresponding to the fused model. The method can adapt to new tasks without retraining or minor adjustments, and realizes plug-and-play model application, which can not only effectively improve the multi-task processing capabilities of the model, but also significantly reduce the consumption of computing resources.

[0005] The following is a specific technical solution to achieve the purpose of the present invention:

[0006] A method for generating a multi-task stable diffusion model based on task arithmetic is characterized in that a model with opposite characteristics of the original task is obtained through task subtraction; or a plurality of different stable diffusion-based models are fused through task addition so that the new model can solve the multi-task scenario of the task originally corresponding to the fused model. The method specifically comprises the following steps:

[0007] Step 1: Define and select the task vector corresponding to the image generation task

[0008] 1-1: In the Stable Diffusion (SD) community, collect the models of concept elimination, and the parameters corresponding to the collected models are recorded as θ1, θ2, ..., θ n ;

[0009] 1-2: Find the pre-trained stable diffusion model before fine-tuning the collected model, and its parameters are recorded as θ0;

[0010] 1-3: Calculate the task vector Δθ1, Δθ2, ..., Δθ n :

[0011] Δθ1=θ1-θ0

[0012] Δθ2=θ2-θ0

[0013] …

[0014] Δθ n =θ n -θ0;

[0015] Step 2: Subtract the task vector from the pre-trained stable diffusion model through task arithmetic operations to obtain a concept-enhanced model, or add multiple task vectors to the pre-trained stable diffusion model to obtain a new model that solves multi-task scenarios.

[0016] 2-1: When the new task requirement is opposite to the original task, that is, when a concept is to be enhanced, the new model parameters are calculated through task subtraction operation:

[0017] θ′ i =θ0-Δθ i , i∈[0,n]

[0018] The model corresponding to the new model parameters is the model enhanced by the concept;

[0019] 2-2: When a new task requires solving multiple tasks at the same time or switching between multiple tasks, calculate the new model parameters through task addition operation:

[0020] θ∑ =θ0+Δθ1+Δθ2+...+Δθ n ;

[0021] The model corresponding to the new model parameters is the new model for solving the multi-task scenario;

[0022] 2-3: In solving multi-task scenarios, the concept to be enhanced is first operated according to step 2-1 to obtain the concept enhanced model θ' i1 ,θ' i2 , ..., θ′ in Then, the task vectors Δθ' of these concept enhancements are calculated according to steps 1-3 i1 , Δθ' i2 , ..., Δθ' in Finally, through step 2-2, the task vectors that enhance these concepts and the task vectors that eliminate other concepts are calculated using task addition operations to obtain new model parameters. The obtained new model can enhance some concepts while weakening other concepts.

[0023] Step 3: Task arithmetic with fusion conditional control

[0024] 3-1: Apply the algorithm in step 2 to the stable diffusion model of the fixed part on the left side of ControlNet, that is, obtain the concept enhancement model of conditional control or the model of conditional control for solving multi-task scenarios;

[0025] Step 4: Experimental verification and performance evaluation

[0026] 4-1: Experimentally verify the model after task arithmetic operation and evaluate its performance in multi-task generation and concept enhancement;

[0027] 4-2: Through quantitative and qualitative analysis of CLIP Score and CLIP Accuracy, we compare the performance of different models in generating specified objects and styles, and verify the multi-task processing capability of the model.

[0028] Furthermore, the task arithmetic includes operations of addition and subtraction; the stable diffusion model before fine-tuning is a pre-trained stable diffusion model in the stable diffusion community.

[0029] Beneficial effects of the present invention:

[0030] The present invention proposes a method for generating a multi-task stable diffusion model based on task arithmetic. Through the addition and subtraction operations of task vectors, it can be directly edited and enhanced at the model level to achieve seamless integration of multi-task processing capabilities. This method not only retains the powerful performance of the original model, but also can significantly improve the multi-task generation capability of the model without increasing the computational cost. Compared with the traditional fine-tuning method, the method of the present invention has the advantages of no need for additional training, low computational overhead, and easy expansion, and is suitable for practical applications in a variety of image generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a process for processing a task using the method of the present invention is shown;

[0032] Figure 2 A schematic diagram showing how to use task subtraction is shown;

[0033] Figure 3 A schematic diagram showing how to use task addition is shown;

[0034] Figure 4 A schematic diagram showing a method of combining task arithmetic operations with ControlNet to achieve conditional control. DETAILED DESCRIPTION

[0035] The present invention is further described in detail through the following specific embodiments and drawings.

[0036] Example 1

[0037] See also Figure 1 According to a preferred embodiment of the present invention, a method for generating a multi-task diffusion model based on task arithmetic image generation specifically comprises the following steps:

[0038] Step 1: Define and select the task vector relevant to the image generation task

[0039] 1-1: See Figure 2 In the Stable Diffusion (SD) related community, we collect models related to the target task, denoted as θ1, θ2, ..., θ n The model has been fine-tuned by community contributors for different task scenarios, such as Figure 1 Examples of different tasks shown in;

[0040] 1-2: Find the basic Stable Diffusion version of these models before fine-tuning, denoted as θ0. This version is used as the baseline model for all tasks and has not been fine-tuned for specific tasks.

[0041] 1-3: Calculate the task vector Δθ1, Δθ2, ..., Δθ for each taskn ,in:

[0042] Δθ1=θ1-θ0

[0043] Δθ2=θ2-θ0

[0044] …

[0045] Δθ n =θ n -θ0

[0046] These task vectors represent the changes from the base model to each fine-tuned model. Figure 2 How to compute these vectors is shown in detail in .

[0047] Step 2: Add or subtract the task vector from the Stable Diffusion model through arithmetic operations

[0048] 2-1: If the new task requirements are opposite to the original task characteristics, refer to Figure 2 , the new model parameters are calculated by task subtraction operation:

[0049] θ′ i =θ0-θ i , i∈[0,n]

[0050] like Figure 2 As shown in Figure 2, this operation achieves the reverse enhancement of the original task characteristics, such as the transition from weakening a certain style to enhancing it.

[0051] 2-2: If the new task requires the model to be able to switch between multiple tasks or process in parallel, refer to Figure 3 , calculate the new model parameters through task addition operation:

[0052] θ ∑ =θ0+Δθ1+Δθ2+...+Δθ n

[0053] Through this addition operation, the new model can simultaneously integrate multiple task features and achieve multi-task generation. Figure 3 The operation flow of multi-task addition is demonstrated in

[0054] Step 3: Task arithmetic with fusion conditional control

[0055] 3-1: In the ControlNet framework, the calculated task vector is applied to the StableDiffusion model in the fixed part on the left to enhance the multi-task operation capability of the model, such as Figure 4 As shown;

[0056] 3-2: For the input of each task (such as additional conditional control inputs such as edge maps and depth maps), additional conditional controls are added at different layers of the model to guide the model to generate images that meet the requirements. Figure 4 The middle right section shows how to add conditional controls to the model to regulate the generation process;

[0057] 3-3: By fusing and combining different task vectors, a model with multi-task capabilities is generated and applied to a variety of image generation task scenarios. This method can not only retain the original generation characteristics of the model, but also flexibly adjust the generation effect according to the input conditions, such as Figure 4 Multitasking example in .

[0058] Step 4: Experimental verification and performance evaluation

[0059] 4-1: Experimental verification of the model after task arithmetic operation to evaluate its performance in multi-task generation and concept enhancement. The experimental setting is as follows Figure 4 As shown, a variety of input conditions and generated results are compared and analyzed respectively;

[0060] 4-2: Through quantitative and qualitative results analysis, we compare the performance of different models in generating specified objects and styles, and verify the multi-task processing capabilities of the models. The experimental results show that the method based on task arithmetic achieves more efficient model utilization and performance improvement without additional training.

[0061] The above is only a specific description of the present invention and should not be regarded as limiting the protection scope of the present invention. Any modification or replacement within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a multi-task stable diffusion model based on task arithmetic, characterized in that: The method specifically comprises the following steps: Step 1: Define and select the task vector corresponding to the image generation task 1-1: In the community of the stable diffusion model, the model of concept elimination is collected, and the parameters corresponding to the collected models are recorded as θ1, θ2, ..., θ n ; 1-2: Find the pre-trained stable diffusion model before fine-tuning the collected model, and its parameters are recorded as θ0; 1-3: Calculate the task vector Δθ1, Δθ2, ..., Δθ n : Δθ1=θ1-θ0 Δθ2=θ2-θ0 … Dth n =θ n -θ0; Step 2: Subtract the task vector from the pre-trained stable diffusion model through task arithmetic operations to obtain a concept-enhanced model, or add multiple task vectors to the pre-trained stable diffusion model to obtain a new model that solves multi-task scenarios. 2-1: When the new task requirement is opposite to the original task, that is, when a concept is to be enhanced, the new model parameters are calculated through task subtraction operation: θ′ i =θ0-Δθ i ,i∈[0,n] The model corresponding to the new model parameters is the model enhanced by the concept; 2-2: When a new task requires solving multiple tasks at the same time or switching between multiple tasks, calculate the new model parameters through task addition operation: i ∑ =θ0+Δθ1+Δθ2+...+Δθ n ; The model corresponding to the new model parameters is the new model for solving the multi-task scenario; 2-3: In solving multi-task scenarios, the concept to be enhanced is first operated according to step 2-1 to obtain the concept enhanced model θ′ i1 ,θ′ i2 , ..., θ′ in , where 1≤i1≤in≤n; then calculate the task vectors Δθ' of these concept enhancements according to steps 1-3 i1 , Δθ' i2 , ..., Δθ' in Finally, through step 2-2, the task vectors that enhance these concepts and the task vectors that eliminate other concepts are calculated using task addition operations to obtain new model parameters. The obtained new model can enhance some concepts while weakening other concepts. Step 3: Task arithmetic with fusion conditional control 3-1: Apply the algorithm in step 2 to C o ntr o The stable diffusion model of the fixed part on the left side of lNet is the concept enhancement model of conditional control or the model of conditional control for solving multi-task scenarios.

2. The method according to claim 1, characterized in that The task arithmetic includes operations of addition and subtraction; the stable diffusion model before fine-tuning is a pre-trained stable diffusion model in the stable diffusion community.

Images