Stable-Diffusion model fusion method

Abstract

The invention relates to model fusion, in particular to a Stable-Diffusion model fusion method, and the method comprises the steps: loading the model weights of two to-be-fused Stable-Diffusion models to a U-Net model; traversing the model weight of each layer in the U-Net model, and carrying out normalization processing; independently setting interpolation parameters for the model weight of each layer in the U-Net model; independently calculating an interpolation vector for the model weight of each layer in the U-Net model by adopting spherical linear interpolation SLERP, and storing a calculation result; interpolation parameters are stored according to a model structure, model meta-information is constructed, and new model fusion is completed; according to the technical scheme provided by the invention, the defect of relatively poor fusion effect of a Stable-Diffusion model in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to model fusion, and in particular to a Stable-Diffusion model fusion method. Background Art

[0002] The Stable-Diffusion model is an image generation model based on deep learning that can generate high-quality, high-resolution images. The model generates images by adding noise to the image and passing it into the model, so it is also called a denoising diffusion model. The Stable-Diffusion model has a wide range of applications in computer vision and image processing, such as image generation, image restoration, image editing, etc.

[0003] At present, model fusion is an important way to fine-tune the model, which can quickly obtain a new model. The existing Stable-Diffusion model fusion method mainly adopts linear interpolation, that is, a simple weighted average of the weights of the two models. The advantages of this method are simple implementation and fast calculation speed. However, this linear fusion method has the following main problems:

[0004] 1) Instability: When two models are fused, since the weights belong to the latent space, linear fusion is just a weighted superposition of features, which may produce unstable jump outputs. This is because linear interpolation does not consider the geometric structure of weights in high-dimensional space, which may cause the interpolation path to pass through unreasonable areas;

[0005] 2) Feature loss: The fused new model may lose some unique characteristics of the original model, because simple weighted averaging may cause some important features to be averaged or diluted;

[0006] 3) Lack of smooth transition: At different levels of linear fusion, the behavior of the model may change suddenly instead of smoothly, because linear interpolation cannot guarantee smooth feature changes in high-dimensional nonlinear space;

[0007] 4) Geometric structure is not preserved: In some parameter spaces, such as the sphere, simple linear interpolation cannot preserve its intrinsic geometric structure, which may cause the fused new model to perform worse than expected in some aspects.

[0008] The root cause of the above problems is that the weight space of the Stable-Diffusion model is a high-dimensional nonlinear space, and simple linear interpolation cannot capture the geometric characteristics of this space well. Therefore, it is difficult to achieve the ideal fusion effect when performing model fusion, especially when it is necessary to finely control the degree of fusion or maintain the specific characteristics of the original model. Summary of the invention

[0009] 1. Technical issues to be solved

[0010] In view of the above-mentioned shortcomings of the prior art, the present invention provides a Stable-Diffusion model fusion method, which can effectively overcome the defect of poor Stable-Diffusion model fusion effect in the prior art.

[0011] (II) Technical solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0013] A Stable-Diffusion model fusion method includes the following steps:

[0014] S1. Load the model weights of the two Stable-Diffusion models to be fused into the U-Net model;

[0015] S2, traverse the model weights of each layer in the U-Net model and normalize them;

[0016] S3, set interpolation parameters for the model weights of each layer in the U-Net model separately;

[0017] S4, using spherical linear interpolation SLERP to calculate the interpolation vector for each layer of the U-Net model separately, and save the calculation results;

[0018] S5. Save the interpolation parameters according to the model structure and construct the model meta-information. The new model fusion is completed.

[0019] Preferably, in S1, the model weights of the two Stable-Diffusion models to be fused are loaded into the U-Net model, including:

[0020] Load the model weight CheckPoint files of the two Stable-Diffusion models to be fused and locate them to the U-Net model through pyTorch.

[0021] Preferably, in S3, interpolation parameters are set separately for the model weights of each layer in the U-Net model, including:

[0022] The MergeBlockWeight method is used to set the interpolation parameters of the model weights of each layer in the U-Net model separately.

[0023] Preferably, S4 uses spherical linear interpolation SLERP to separately calculate the interpolation vector for the model weights of each layer in the U-Net model, including:

[0024] For each layer in the U-Net model:

[0025] S41, calculating the angle between the model weight vectors;

[0026] S42, calculating a scale factor according to the angle between the weight vectors and the interpolation parameter t;

[0027] S43. Perform weighted summation on the weight vector and the scale factor to obtain an interpolation vector.

[0028] Preferably, S4 uses spherical linear interpolation SLERP to separately calculate the interpolation vector for the model weights of each layer in the U-Net model, including:

[0029] The following formula is used to calculate the interpolation vector for the model weight of a certain layer in the U-Net model separately:

[0030]

[0031] in, is the interpolation vector of layer A of the U-Net model, is the model weight vector of the first Stable-Diffusion model to be fused in the A layer of the U-Net model. is the model weight vector of the second Stable-Diffusion model to be fused in the A layer of the U-Net model, t A is the interpolation parameter of layer A of the U-Net model, t A ∈[0,1],Ω is and The angle between

[0032] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the Stable-Diffusion model fusion method described above is implemented.

[0033] A computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, each step of the above-mentioned Stable-Diffusion model fusion method is implemented.

[0034] (III) Beneficial effects

[0035] Compared with the prior art, the Stable-Diffusion model fusion method provided by the present invention has the following beneficial effects:

[0036] 1) Improve model stability

[0037] Existing technology: Linear fusion is just a weighted superposition of features, which may produce unstable jump outputs;

[0038] The present invention: the spherical linear interpolation SLERP maintains the geometric structure of the sphere, and is more likely to maintain the stability of the model during the interpolation process. This stability is brought about by the spherical linear interpolation SLERP interpolating along the shortest path (i.e., the arc on the sphere) in the high-dimensional nonlinear space, which effectively avoids the numerical instability that may be caused by simple linear interpolation;

[0039] 2) Achieve smooth transition

[0040] Existing technology: Linear fusion may lead to sudden changes in model behavior;

[0041] The present invention: Spherical linear interpolation SLERP can provide a smooth transition between two vectors, which is particularly important for model parameter interpolation in high-dimensional nonlinear space. This smooth transition ensures that the model will not have unnecessary jumps or mutations during the fusion process, thereby producing a more coherent and natural fusion effect;

[0042] 3) Better preserve the original model features

[0043] Prior art: Simple weighted averaging may result in some important features being averaged or diluted;

[0044] The present invention: Spherical linear interpolation SLERP can better preserve the unique features and curvature of each original model. In a high-dimensional nonlinear space, spherical linear interpolation SLERP effectively avoids the problem of feature dilution by maintaining the features and curvature of each original model, so that the fused new model can better inherit the excellent characteristics of the original model;

[0045] 4) Adapting to continuous parameter space

[0046] Prior art: Linear interpolation performs poorly in nonlinear parameter spaces;

[0047] The present invention: In the Stable-Diffusion model, the weights are usually in a continuous parameter space, and the spherical linear interpolation SLERP is just suitable for this type of parameter space. This adaptability enables the method to more accurately capture the geometric characteristics of the model weight space, thereby achieving more accurate model fusion;

[0048] 5) Fine control of the fusion process

[0049] Existing technologies: It is difficult to differentiate and fuse different parts of the model;

[0050] The present invention: by using spherical linear interpolation SLERP to calculate interpolation vectors for different layers (input layer, middle layer, output layer) of the U-Net model, it allows for more refined differentiated fusion of different parts of the model. This hierarchical fusion strategy enables researchers to adjust different aspects of the model as needed, thereby achieving more flexible and targeted model fusion;

[0051] 6) Improve the success rate of fusion

[0052] Existing technology: Linear fusion may lead to fusion failure or poor effect;

[0053] The present invention: Since the spherical linear interpolation SLERP takes into account the geometric structure of weights in high-dimensional nonlinear space and realizes smooth transition and feature retention, the success rate of model fusion is greatly improved, which means that the use of this method can more reliably create a high-quality fusion model and reduce invalid attempts and resource waste;

[0054] The above advantages are all directly brought about by the Stable-Diffusion model fusion method proposed in the present invention. By adopting this method, researchers and developers can more effectively create new hybrid style models, improve model performance, and achieve innovation while maintaining the excellent characteristics of the original model. This not only improves the effect of model fusion, but also provides a powerful tool for the further development of the AI ​​image generation industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a schematic diagram of the process of the present invention;

[0057] Figure 2 This is a schematic diagram of the principle of using spherical linear interpolation SLERP in the present invention to separately calculate the interpolation vector for the model weights of each layer in the U-Net model. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] A Stable-Diffusion model fusion method, such as Figure 1 As shown, S1, load the model weights of the two Stable-Diffusion models to be fused into the U-Net model, specifically including:

[0060] Load the model weight CheckPoint files of the two Stable-Diffusion models to be fused and locate them to the U-Net model through pyTorch.

[0061] S2: Traverse the model weights of each layer in the U-Net model and normalize them.

[0062] S3. Set interpolation parameters for the model weights of each layer in the U-Net model separately, including:

[0063] The MergeBlockWeight method is used to set the interpolation parameters of the model weights of each layer in the U-Net model separately.

[0064] S4. Use spherical linear interpolation (SLERP) to calculate the interpolation vector for the model weights of each layer in the U-Net model separately, and save the calculation results.

[0065] Specifically, spherical linear interpolation SLERP is used to calculate the interpolation vector of each layer in the U-Net model separately, including:

[0066] For each layer in the U-Net model:

[0067] S41, calculating the angle between the model weight vectors;

[0068] S42, calculating a scale factor according to the angle between the weight vectors and the interpolation parameter t;

[0069] S43. Perform weighted summation on the weight vector and the scale factor to obtain an interpolation vector.

[0070] Specifically, spherical linear interpolation SLERP is used to calculate the interpolation vector for each layer of the U-Net model separately, such as Figure 2 As shown, including:

[0071] The following formula is used to calculate the interpolation vector for the model weight of a certain layer in the U-Net model separately:

[0072]

[0073] in, is the interpolation vector of layer A of the U-Net model, is the model weight vector of the first Stable-Diffusion model to be fused in the A layer of the U-Net model. is the model weight vector of the second Stable-Diffusion model to be fused in the A layer of the U-Net model, t A is the interpolation parameter of layer A of the U-Net model, t A ∈[0,1],Ω is and The angle between

[0074] S5. Save the interpolation parameters according to the model structure and construct the model meta-information. The new model fusion is completed.

[0075] In the Stable-Diffusion model fusion, spherical linear interpolation (SLERP) is used to achieve a smooth transition between different original models and to interpolate between two original models with different styles or characteristics to create a new mixed style model.

[0076] In the Stable-Diffusion model, the model weights are regarded as points on a high-dimensional sphere. When spherical linear interpolation (SLERP) is used for interpolation, the shortest path between the two weight vectors (i.e., the arc on the sphere) is first calculated, and then interpolation is performed along this path, such as Figure 2 This method has significant advantages over direct linear interpolation in Euclidean space because it can maintain the geometric structure of the sphere and avoid the interpolation path passing through the interior of the sphere.

[0077] The technical solution of the present application also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned Stable-Diffusion model fusion method is implemented.

[0078] The technical solution of the present application also discloses a computer-readable storage medium, characterized in that: a computer program is stored thereon, and when the computer program is executed by a processor, each step of the above-mentioned Stable-Diffusion model fusion method is implemented.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Stable-Diffusion model fusion method, characterized by: The following steps are involved: S1. Load the model weights of the two Stable-Diffusion models to be fused into the U-Net model; S2, traverse the model weights of each layer in the U-Net model and normalize them; S3, set interpolation parameters for the model weights of each layer in the U-Net model separately; S4, using spherical linear interpolation SLERP to calculate the interpolation vector for each layer of the U-Net model separately, and save the calculation results; S5. Save the interpolation parameters according to the model structure and construct the model meta-information. The new model fusion is completed.

2. The Stable-Diffusion model fusion method according to claim 1, characterized in that: In S1, the model weights of the two Stable-Diffusion models to be fused are loaded into the U-Net model, including: Load the model weight CheckPoint files of the two Stable-Diffusion models to be fused and locate them to the U-Net model through pyTorch.

3. The Stable-Diffusion model fusion method according to claim 1, characterized in that: In S3, interpolation parameters are set separately for the model weights of each layer in the U-Net model, including: The MergeBlockWeight method is used to set the interpolation parameters of the model weights of each layer in the U-Net model separately.

4. The Stable-Diffusion model fusion method according to claim 1, characterized in that: S4 uses spherical linear interpolation SLERP to calculate the interpolation vector for each layer of the U-Net model, including: For each layer in the U-Net model: S41, calculating the angle between the model weight vectors; S42, calculating a scale factor according to the angle between the weight vectors and the interpolation parameter t; S43. Perform weighted summation on the weight vector and the scale factor to obtain an interpolation vector.

5. The Stable-Diffusion model fusion method according to claim 4, characterized in that: S4 uses spherical linear interpolation SLERP to calculate the interpolation vector for each layer of the U-Net model, including: The following formula is used to calculate the interpolation vector for the model weight of a certain layer in the U-Net model separately: in, is the interpolation vector of layer A of the U-Net model, is the model weight vector of the first Stable-Diffusion model to be fused in the A layer of the U-Net model. is the model weight vector of the second Stable-Diffusion model to be fused in the A layer of the U-Net model, t A is the interpolation parameter of layer A of the U-Net model, t A ∈[0,1],Ω is and The angle between 6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the Stable-Diffusion model fusion method as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, each step of the Stable-Diffusion model fusion method as described in any one of claims 1 to 4 is implemented.

Images