Diffusion model migration method based on prediction residual guidance and related device

Abstract

The invention discloses a diffusion model migration method based on prediction residual guidance and a related device, and relates to the technical field of diffusion model knowledge migration, and the method comprises the steps: taking noise data as input at each denoising time step, and carrying out denoising on the noise data; determining a first prediction noise, a second prediction noise and a third prediction noise by using the basic model, the adaptive model and the target model respectively, calculating a deviation between the first prediction noise and the second prediction noise to obtain a prediction residual error, performing weighted summation on the third prediction noise and the prediction residual error to obtain a guide prediction noise, and outputting the guide prediction noise. And on the basis of the noise data and the guide prediction noise, noise data of the next denoising time step is calculated until the last denoising time step is reached, and an output result of the target model is obtained, and the output result is a picture, a video, a voice or a text. According to the method, the knowledge migration of the diffusion model can be completed on the premise of not accessing original training data and not training.

Description

Technical Field

[0001] This application relates to the field of knowledge transfer technology for diffusion models, and in particular to a diffusion model transfer method and related apparatus based on prediction residual guidance. Background Technology

[0002] In recent years, diffusion models, as a representative technology of generative models, have demonstrated powerful performance capabilities in various fields such as image generation, video generation, speech generation, and text generation. Open-source systems, represented by diffusion models such as Stable Diffusion, PixArt, and FLUX, have driven the development of personalized generative applications. Currently, various efficient adaptation modules such as LoRA, LyCORIS, ControlNet, and IP-Adapter can be used, allowing ordinary users to fine-tune parameters or perform structured control on the basic model without large-scale pre-training to obtain adapted models that meet their personalized needs. These personalized needs can include specific styles (such as ink painting, landscape, or Van Gogh style), specific characters (such as an anime character, a cat, or a dog), and specific structures (such as the outline of a bird).

[0003] In practice, these adaptation modules are often highly bound to a certain version of the base model. For example, many existing adaptation modules for LoRA, LyCORIS, and ControlNet are trained on the Stable Diffusion 1.5 base model. In this case, Stable Diffusion 1.5 with the adaptation module has specific personalized capabilities. However, as the base model version evolves (such as upgrading to Stable Diffusion 2.1), its model architecture, parameter shape, and other distributions change, making the original adaptation modules unusable and greatly limiting their sustainable development. In this case, the Stable Diffusion 2.1 target model no longer has specific personalized capabilities.

[0004] Currently, there are two main solutions.

[0005] (1) Readjust or train the adaptation module.

[0006] This method requires raw training data, a large amount of computing resources, and a high time cost, posing a very high barrier to entry for users, especially individual developers or small businesses.

[0007] (2) Use transfer learning methods such as knowledge distillation.

[0008] This method can be used for some compatible modules (such as LoRA) and can achieve knowledge transfer to a certain extent. However, it still requires training the target model, is complex to operate, has high deployment costs, and the transfer effect is often unstable. It is not suitable for large-scale migration of compatible modules.

[0009] Both of the above methods require adjustment of the target model parameters, which violates the engineering requirements of modern generative models in the deployment phase, namely "frozen model weights, lightweight inference, and high compatibility".

[0010] In summary, the current diffusion model for knowledge transfer faces the following technical bottlenecks.

[0011] (1) Strong architecture coupling: Adaptation modules often cannot be migrated across basic model versions.

[0012] (2) High dependence on training resources: Almost all existing transfer methods require retraining.

[0013] (3) High deployment threshold: The migration strategy requires access to the original training data, which is not conducive to industrial-grade deployment. Summary of the Invention

[0014] The purpose of this application is to provide a diffusion model transfer method and related apparatus based on prediction residual guidance, which can complete the knowledge transfer of diffusion models without accessing the original training data or without training.

[0015] To achieve the above objectives, this application provides the following solution.

[0016] Firstly, this application provides a diffusion model transfer method based on prediction residuals, the method comprising: Obtain the noise data for the current denoising time step; in the first denoising time step, the noise data is randomly distributed noise; Using the noise data at the current denoising time step as input, the first predicted noise, second predicted noise, and third predicted noise at the current denoising time step are determined using the base model, the adapted model, and the target model, respectively. The base model is a model obtained by training the first diffusion model using the first dataset, the adapted model is a model obtained by training the base model using the second dataset, and the target model is a model obtained by training the second diffusion model using the third dataset, wherein the third dataset does not include the second dataset. Calculate the deviation between the first and second predicted noise at the current denoising time step to obtain the prediction residual at the current denoising time step; The third prediction noise and prediction residual of the current denoising time step are weighted and summed to obtain the guiding prediction noise of the current denoising time step; Based on the noise data of the current denoising time step and the guided prediction noise, the noise data of the next denoising time step is calculated. If the next denoising time step is the last denoising time step, the noise data of the next denoising time step is used as the output of the target model. The output can be an image, video, speech, or text.

[0017] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the above-described diffusion model migration method based on prediction residual guidance.

[0018] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described diffusion model migration method based on prediction residual guidance.

[0019] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described diffusion model migration method based on prediction residual guidance.

[0020] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a diffusion model transfer method and related apparatus based on prediction residual guidance. In each denoising time step, noise data is used as input, and the first, second, and third prediction noises are determined using the base model, the adapted model, and the target model, respectively. The deviation between the first and second prediction noises is calculated to obtain the prediction residual. The third prediction noise and the prediction residual are weighted and summed to obtain the guided prediction noise. Based on the noise data and the guided prediction noise, the noise data for the next denoising time step is calculated until the last denoising time step is reached, at which point the output result of the target model is obtained. The output result can be an image, video, speech, or text. This application does not require access to the original training data or training. It can directly use the prediction residuals of the base model and the adapted model when predicting noise to guide the backward generation process of the target model (also known as the backward denoising process). The adapted model is a model obtained by training the base model using a second dataset. Compared with the base model, it has knowledge of the second dataset. The second dataset is a dataset set up to achieve personalization. Therefore, the prediction residuals between the two contain personalized generation features such as visual style, semantic features or structural control. At this time, the target model can learn the personalized generation features and thus have specific personalization capabilities. It can complete the diffusion model knowledge transfer without accessing the original training data or training. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an application environment diagram for a diffusion model migration method based on prediction residuals provided in Embodiment 1 of this application.

[0023] Figure 2 This is a flowchart illustrating a diffusion model migration method based on prediction residuals, provided in Embodiment 1 of this application.

[0024] Figure 3 This is a schematic diagram illustrating the principle of a diffusion model migration method based on prediction residual guidance provided in Embodiment 1 of this application.

[0025] Figure 4 This is a detailed flowchart illustrating a diffusion model migration method based on prediction residuals, provided in Embodiment 1 of this application.

[0026] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Example 1.

[0029] The diffusion model transfer method based on prediction residual guidance provided in this application can be applied to, for example... Figure 1The application environment shown illustrates this. The terminal communicates with the server via a network. A data storage system stores the data the server needs to process. This system can be set up independently, integrated into the server, or located in the cloud or on another server. The terminal can send a migration request to be processed to the server. Upon receiving the request, the server retrieves the noise data for the current denoising time step. Using this noise data as input, it determines the first, second, and third predicted noise for the current denoising time step using the base model, the adaptation model, and the target model, respectively. It calculates the deviation between the first and second predicted noises to obtain the prediction residual for the current denoising time step. It then performs a weighted summation of the third predicted noise and the prediction residual to obtain the guided prediction noise for the current denoising time step. Based on the noise data and guided prediction noise of the current denoising time step, it calculates the noise data for the next denoising time step. If the next denoising time step is the last denoising time step, the noise data for that step is used as the output of the target model, which can be an image, video, audio, or text. The server can feed back the migration result, which is the output of the target model for the migration request, to the terminal.

[0030] Furthermore, in some embodiments, the diffusion model migration method based on prediction residuals can also be implemented by a server or a terminal independently. For example, the terminal can directly process the migration request to be processed, or the server can obtain the migration request to be processed from the data storage system and process it.

[0031] In one exemplary embodiment, such as Figure 2 As shown, a diffusion model migration method based on prediction residual guidance is provided. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is applied to... Figure 1 The following steps are used as an example of a server in the example.

[0032] Step S1: Obtain the noise data for the current denoising time step; in the first denoising time step, the noise data is randomly distributed noise.

[0033] Step S2: Using the noise data of the current denoising time step as input, determine the first predicted noise, the second predicted noise, and the third predicted noise of the current denoising time step using the base model, the adapted model, and the target model respectively; the base model is the model obtained by training the first diffusion model using the first dataset, the adapted model is the model obtained by training the base model using the second dataset, and the target model is the model obtained by training the second diffusion model using the third dataset, wherein the third dataset does not include the second dataset.

[0034] Step S3: Calculate the deviation between the first and second predicted noise at the current denoising time step to obtain the prediction residual at the current denoising time step.

[0035] Step S4: Perform a weighted summation of the third prediction noise and the prediction residual at the current denoising time step to obtain the guiding prediction noise at the current denoising time step.

[0036] Step S5: Based on the noise data of the current denoising time step and the guided prediction noise, calculate the noise data of the next denoising time step. If the next denoising time step is the last denoising time step, then use the noise data of the next denoising time step as the output result of the target model. The output result is an image, video, speech, or text.

[0037] By implementing steps S1 to S5 above, this embodiment can complete the knowledge transfer of the diffusion model without accessing the original training data or undergoing training.

[0038] The core idea of ​​the diffusion model (also known as the diffusion generation model) is to diffuse the original data into noisy data by gradually adding noise, and then gradually removing the noise to recover the original data from the noisy data. The process of gradually adding noise is called the forward diffusion process, and the process of gradually removing noise is called the backward generation process. During the training process, both the forward diffusion process and the backward generation process are required to determine the network parameters of the diffusion model. During the inference process, only the backward generation process is required. By inputting randomly distributed noisy data, the original data can be generated through noise prediction and gradual noise removal.

[0039] This embodiment provides a data-agnostic knowledge transfer method applicable to diffusion models (transferring knowledge from model A to model B without original training data), specifically a cross-architecture knowledge transfer method for diffusion models based on prediction residual guidance (or injection). This method does not require access to the original training data, nor does it require parameter fine-tuning or structured control of the target model. It can transfer learned generative capabilities (such as visual style, semantic features, or structural control) between different diffusion models, and is applicable to adjusting the inference process of diffusion models, including tasks such as image generation and text generation. During the diffusion model inference process, starting with noisy data, the prediction residual is obtained by calculating the deviation of the predicted noise between the base model and its corresponding adapted model at each denoising time step. This prediction residual is then injected into the inference process of the target model, thereby achieving knowledge transfer and reuse between different model architectures.

[0040] Visual style refers to the appearance features of the generated image, including color, texture, and lighting. By controlling the visual style, images with a specific artistic style or visual effect can be generated. Semantic features refer to the semantic information of the generated content, including theme, object, and scene. By controlling semantic features, images with a specific theme or content can be generated. Structural control refers to the layout and organization of the generated content, including the position, size, and orientation of objects. By controlling the structure, images with a specific layout and organization can be generated.

[0041] The base model is the model obtained after learning general knowledge. The adaptation model is a personalized adaptation of the base model, allowing it to further learn personalized domain knowledge. This personalized adaptation process can be accomplished through adaptation modules. For example, LoRA and LyCORIS can fine-tune some parameters of the base model, and LoRA / LyCORIS and the base model together form the adaptation model. Similarly, ControlNet can introduce external control conditions to input external control signals into the base model, forming the adaptation model. Another example is the IP-Adapter, which can input real-world image features into the base model, forming the adaptation model. Yet another example is the introduction of prompt words for fine-tuning, inputting more detailed and personalized prompt words into the base model to guide its generation; these prompt words and the base model together form the adaptation model. This personalized adaptation process can also be accomplished without an adaptation module, directly utilizing... The method uses a dataset composed of personalized domain knowledge to update all parameters of the base model, thus obtaining an adapted model. Therefore, the method in this embodiment is applicable to adapted models obtained by mainstream model adaptation methods such as LoRA, LyCORIS, ControlNet, IP-Adapter, prompt word fine-tuning, and full fine-tuning (i.e., updating all parameters of the base model). It is independent of the base model architecture and sampling algorithm (the sampling algorithm restores the original data by gradually removing noise, which is equivalent to the post-generation process, such as DDPM, DDIM, Euler, Heun, etc., which can also be called a sampler). It has the characteristics of strong versatility (regardless of the base model, the model adaptation method used to process the base model to obtain the adapted model, and the sampling algorithm used for the subsequent generation process, it can be transferred to the target model), low transfer cost, fast speed, and stable transfer effect. The low transfer cost, fast speed, and stable transfer effect are due to the fact that no original training data or training is required.

[0042] like Figure 2 As shown, the diffusion model migration method based on prediction residual guidance in this embodiment includes the following steps.

[0043] (1) Obtain the noise data of the current denoising time step. In the first denoising time step, the noise data is randomly distributed noise.

[0044] The randomly distributed noise can be Gaussian noise or Bernoulli noise.

[0045] (2) Using the noise data of the current denoising time step as input, the first predicted noise, the second predicted noise and the third predicted noise of the current denoising time step are determined by using the base model, the adapted model and the target model respectively. The base model is the model obtained by training the first diffusion model using the first dataset. The adapted model is the model obtained by training the base model using the second dataset. The target model is the model obtained by training the second diffusion model using the third dataset. The third dataset does not include the second dataset.

[0046] The base model, the adapted model, and the target model are all diffusion models. After training, the base model, the adapted model, and the target model have the ability to predict noise. At this time, the noise data of the current denoising time step is used as input, and the base model, the adapted model, and the target model are used to determine the first predicted noise, the second predicted noise, and the third predicted noise of the current denoising time step, respectively.

[0047] Both the first and third datasets can be general knowledge datasets, such as any open-source dataset. For image generation tasks, these can be any type of image. The first and third datasets can be the same or different. Generally, the third dataset is larger and of higher quality than the first dataset. Both the base model and the target model are models obtained after learning general knowledge. The second dataset can be personalized domain knowledge, such as medical knowledge. For image generation tasks, these can be images from the medical field. The adaptation model is the model obtained by further learning personalized domain knowledge from the base model. It can be understood as a model obtained by adapting the base model based on personalized needs, which is equivalent to an adapted version of the base model.

[0048] For image generation tasks, if the adaptation method is LoRA or LyCORIS to fine-tune some parameters of the base model, then the second dataset includes images from the personalized domain. If the adaptation method is ControlNet to introduce external control conditions to input external control signals into the base model, then the second dataset includes external control conditions from the personalized domain, such as pose maps and edge maps. If the adaptation method is IP-Adapter to introduce real images to input real image features into the base model, then the second dataset includes real images from the personalized domain, such as cats wearing hats and running dogs. If the adaptation method is cue word fine-tuning, then the second dataset includes cue words from the personalized domain. If the adaptation method is to use a dataset composed of personalized domain knowledge to update all parameters of the base model, then the second dataset includes images from the personalized domain.

[0049] The first and second diffusion models are of the same type, including image generation diffusion models, video generation diffusion models, speech generation diffusion models, and text generation diffusion models.

[0050] When the first and second diffusion models are image generation diffusion models, the output is an image; when the first and second diffusion models are video generation diffusion models, the output is a video; when the first and second diffusion models are speech generation diffusion models, the output is speech; and when the first and second diffusion models are text generation diffusion models, the output is text.

[0051] Since some diffusion models do not predict noise, but rather predict the original data (i.e., images, videos, speech, or text) or velocity, and since noise, original data, and velocity can be converted into each other, the method of this embodiment is also applicable to such diffusion models that predict original data or velocity. Based on the predicted original data or velocity, the noise is further calculated, and the first predicted noise, second predicted noise, and third predicted noise at the current denoising time step can be obtained.

[0052] (3) Calculate the deviation between the first and second predicted noise at the current denoising time step to obtain the prediction residual at the current denoising time step.

[0053] If one adaptation method is used to adapt the base model, then there is one adapted model; if multiple adaptation methods are used to adapt the base model, then there are multiple adapted models.

[0054] If there is only one model to adapt to, then the deviation between the first and second prediction noises at the current denoising time step is calculated to obtain the prediction residual at the current denoising time step. Specifically, this includes subtracting the first prediction noise at the current denoising time step from the second prediction noise at the current denoising time step to obtain the prediction residual at the current denoising time step.

[0055] If there are multiple adaptation models, each adaptation model corresponds to the second predicted noise of the current denoising time step. In this case, the deviation between the first and second predicted noise of the current denoising time step is calculated to obtain the prediction residual of the current denoising time step. Specifically, for each adaptation model, the second predicted noise of the adaptation model at the current denoising time step is subtracted from the first predicted noise of the current denoising time step to obtain the initial prediction residual of the adaptation model at the current denoising time step. The initial prediction residuals of all adaptation models at the current denoising time step are weighted and summed to obtain the prediction residual of the current denoising time step.

[0056] Specifically, if there is only one model to adapt to, the formula for calculating the prediction residual at the current denoising time step is: ; in, For the current denoising time step t The predicted residuals For the current denoising time step t Noise data; For the current denoising time step t The second prediction noise; For the current denoising time step t The first predicted noise.

[0057] If there are multiple adaptation models, the formula for calculating the prediction residual at the current denoising time step is: ; in, For the current denoising time step t The predicted residuals For the current denoising time step t Noise data; n To accommodate the number of models; For the first i The weights of each adapted model; For the first i The adapted model at the current denoising time step t The second prediction noise; For the current denoising time step t The first predicted noise.

[0058] Traditional methods also suffer from difficulties in combining multiple modules, namely, the difficulty in simultaneously migrating multiple adaptable modules. To address this issue, this embodiment uses the prediction residual... It can originate not only from a single adapted model, but also from multiple adapted models of different types or styles (e.g., multiple base models adapted to different LoRA or ControlNet modules). Regardless of the number of adapted models, prediction residuals can be generated. This design allows users to uniformly migrate multiple style, semantic, and structural control modules to the target model, greatly enhancing the generative expressiveness after migration.

[0059] (4) The third prediction noise and prediction residual of the current denoising time step are weighted and summed to obtain the guiding prediction noise of the current denoising time step.

[0060] The formula for calculating the guided prediction noise at the current denoising time step is: ; in, For the current denoising time step t Guided prediction noise, For the current denoising time step t Noise data; For the current denoising time step t The third prediction noise; The hyperparameter used to control the injection intensity of the predicted residual can be a fixed value or based on the current denoising time step. t Sure; For the current denoising time step t The predicted residuals.

[0061] In this embodiment, It can be a fixed constant, or it can change with the current denoising time step. t Gradual decay can be achieved, for example, by using linear decay, exponential decay, cosine decay, or sigmoid decay.

[0062] The calculation formula for linear decay is as follows: ; in, For the current denoising time step t The hyperparameters used to control the intensity of predicted residual injection; for The maximum value; for The minimum value; This represents the total number of time steps in the denoising process.

[0063] The formula for calculating exponential decay is as follows: ; in, The decay rate is between 0 and 1.

[0064] The formula for calculating the cosine attenuation method is as follows: ; in, Pi is the mathematical constant of a circle.

[0065] The formula for calculating Sigmoid attenuation is as follows: ; in, Hyperparameters are used to control the steepness of intermediate changes.

[0066] (5) Based on the noise data of the current denoising time step and the guided prediction noise, calculate the noise data of the next denoising time step. If the next denoising time step is the last denoising time step, then use the noise data of the next denoising time step as the output result of the target model. The output result is an image, video, speech or text.

[0067] The formula for calculating the noise data in the next denoising time step is determined based on the sampler type. When the sampler type is DDPM, the formula for calculating the noise data in the next denoising time step is: ; in, For the next denoising time step t -1 noise data; For the current denoising time step t The preset noise gain for the diffusion step size; For the current denoising time step t Noise data; For the current denoising time step t The cumulative retention factor; For the current denoising time step t Guided prediction noise; For the current denoising time step t The standard deviation of the matching forward diffusion process; It is random Gaussian noise.

[0068] It should be noted that the above formula for calculating the noise data in the next denoising time step is the DDPM formula, and it is not unique. Different sampling algorithms (such as DDPM, DDIM, Euler, etc.) have different denoising methods, which leads to different calculation formulas for the noise data in the next denoising time step for different sampling algorithms. The calculation formula for the noise data in the next denoising time step is determined based on the sampling algorithm finally adopted.

[0069] After calculating the noise data for the next denoising time step, the diffusion model transfer method based on prediction residuals in this embodiment further includes: if the next denoising time step is not the last denoising time step, then the noise data of the next denoising time step is used as the noise data of the current denoising time step, the current denoising time step is decremented by 1, and the process returns to the step of "obtaining the noise data of the current denoising time step". Here, the initial denoising time step is T, and the last denoising time step is 1.

[0070] The method of this embodiment is proposed in the following context: (1) The current diffusion model contains a large number of adaptation modules that are only applicable to the old basic model, making migration difficult; (2) In engineering practice, it is necessary to deploy new target models (such as higher precision and stronger capabilities) and do not want to abandon existing adaptation results (i.e., the adaptation model obtained by adapting the basic model). (3) The need for knowledge transfer under the conditions of "no training required", "original training data is not available" and "frozen model parameters" is becoming increasingly urgent.

[0071] The method proposed in this embodiment is intended to solve the aforementioned practical problems, providing a diffusion model migration method that is available at inference time, independent of the basic model structure, and friendly to the reuse of adaptation modules, thus providing technical support for building a continuously evolving, open and shared diffusion model ecosystem.

[0072] To address the aforementioned issues, this embodiment proposes a novel "Delta Sampling" method based on an in-depth analysis of the sampling behavior of diffusion models. This method requires no original training data and does not modify any model parameters. It dynamically extracts the behavioral differences between the base model and its adapted model during the inference phase and uses these differences as prediction residual signals to guide the sampling process of the target model. This mechanism is essentially a "black-box model transfer," effectively enabling the reuse of general knowledge across architectures, versions, and modules.

[0073] The following uses the first and second diffusion models as examples of image generation diffusion models, combined with... Figure 3 and Figure 4 This embodiment provides a detailed description of a diffusion model transfer method based on prediction residuals. Figure 3 In x 0 to x T This represents a noisy image obtained by continuously adding noise to the original image; it is a forward diffusion process. x T arrive x 0 represents the denoised image obtained by continuously removing noise from the noise image after adding noise, until the original image is finally obtained. This is the backward generation process. In the backward generation process, the noise image at each denoising time step is input into the three models (base model, adaptation model and target model) for processing to obtain their respective predicted noise. Based on their respective predicted noise, model transfer is performed to guide the backward generation process of the target model to obtain the original image.

[0074] (1) Obtain the prediction noise of the base model and the adaptation model.

[0075] Load the pre-trained base model using a computer device (such as a graphics processing unit, GPU). and its adapted model The adaptation model can include a single or multiple LoRA, LyCORIS, ControlNet, IP-Adapter, prompt words and base model combinations, or it can be a model after full fine-tuning of the base model.

[0076] For example, the base model could be Stable Diffusion 1.5, the adaptation model could be a fully fine-tuned Stable Diffusion 1.5 or LoRA / ControlNet / prompt words + Stable Diffusion 1.5, etc., and the target model could be Stable Diffusion 2.1. This target model is incompatible with the base model in terms of structure or parameters, and the adaptation module corresponding to the base model cannot be directly loaded. The goal is to realize the migration process from the base model and the adaptation model to the target model, such as the migration process from Stable Diffusion 1.5 and its adaptation model to Stable Diffusion 2.1.

[0077] At each diffusion sampling step (i.e., the denoising time step) For the noise data input at the current denoising time step The predicted noise outputs of the two are obtained respectively: Prediction noise of the base model: ; Predictive noise of the adaptation model: ; in, This indicates general conditions, such as text prompts. It indicates specific conditions, such as text prompts or text prompts plus additional structural information / prompt words (equivalent to simultaneously receiving text prompts or structured conditions for generating the image). The structural information can be a pose map, edge map, etc.

[0078] Text prompts refer to text descriptions or instructions provided by the user in a generation task to guide the model in generating specific content. Structural information refers to the structure and organization of the input data that the model needs to consider in a generation task. Structural information can be the layout of an image, the format of text, the hierarchical structure of data, etc., to help the model better understand and generate content. A pose map is an image that represents the pose of a human or object, usually used to guide the model to generate images with specific poses. A pose map usually contains key points (such as joint positions) and skeleton information, which can help the model understand the pose and structure of the target. An edge map is an image that represents the edge information of an image, usually used to guide the model to generate images with specific edge features. Edge maps help the model understand the contour and shape of the target by highlighting the edges of the image.

[0079] (2) Calculate the prediction residuals.

[0080] Based on the prediction noise of the base model and the prediction noise of the adaptation model obtained in step (1) above, the prediction residual (i.e. behavioral difference) is calculated.

[0081] Compute the base model and its adapted model at the current denoising time step The difference between the predicted noise and the predicted residual is used as the prediction residual. : ; This difference reflects the behavioral shift of the adapted model relative to the base model under the current input conditions, and usually contains personalized generation features such as style, semantics or structure.

[0082] (3) Obtain the prediction noise of the target model.

[0083] Using the target model Input the noise data at the current denoising time step Perform a prediction to obtain its predicted noise output: Predictive noise of the target model: .

[0084] (4) Inject the prediction residual during the sampling process.

[0085] The predicted residuals calculated in step (2) The noise is injected into the predicted noise of the target model obtained in step (3) in a linear weighted form to inject the prediction residual and guide the sampling, forming the guided noise prediction (i.e., guided prediction noise). : ; in, The hyperparameter is used to control the intensity of the predicted residual injection, and is used to control the strength of the migration effect.

[0086] (5) Repeat the prediction residual injection operation to complete the entire diffusion sampling process and generate the target output image.

[0087] The guided prediction noise calculated in step (4) Substituting into the standard sampling formula of the diffusion model, perform a one-step update and then backsampling: ; in, For the next denoising time step t -1 noise data; For the current denoising time step t The preset noise gain for the diffusion step size; For the current denoising time step t The cumulative retention factor, , For noise reduction time step s The preset noise gain for the diffusion step size; For the current denoising time step tThe standard deviation of the matching forward diffusion process, For the current denoising time step t The variance of the matching forward diffusion process, ; It is random Gaussian noise. .

[0088] From the initial denoising time step Begin by repeating steps (1) through (5) until the denoising time step. The final output image This process can transfer specific personalization capabilities without modifying the parameters of the target model, in order to generate personalized images.

[0089] The above examples of cross-architecture knowledge transfer for diffusion models can also be accomplished using functional modules, specifically including the following functional modules.

[0090] (1) Input interface module: used to receive the basic model, the adaptation model, the target model and related generation condition information.

[0091] (2) Prediction residual calculation module: used to calculate the difference in prediction noise between the base model and the adapted model in each denoising time step.

[0092] (3) Guiding injection module: used to inject the calculated prediction residual into the prediction noise of the target model, so as to inject into the sampling prediction process.

[0093] (4) Sampling scheduling module: used to coordinate the calculation execution of multiple models at each denoising time step and complete the sampling steps according to the standard sampling formula.

[0094] (5) Output generation module: used to output the final generated sample (such as image or text) after all sampling steps are completed.

[0095] Specifically, the input interface module receives three types of diffusion models: Basic models (such as Stable Diffusion 1.5); Adapt the model (such as the fully fine-tuned Stable Diffusion 1.5+LoRA / ControlNet, etc.) and simultaneously receive text prompts or structured conditions for generation; Target model (such as Stable Diffusion 2.1).

[0096] The prediction residual calculation module is used to call the base model and the adapted model, predict the noise data input at each denoising time step, output the predicted noise, and calculate the residual between the two (i.e., the prediction residual), resulting in: .

[0097] The guided injection module is used to fuse the above-mentioned prediction residuals with the prediction noise of the target model to form guided prediction noise (i.e., guided prediction noise): .in, Hyperparameters set by the user to control the intensity of prediction residual injection are used to regulate migration intensity.

[0098] The sampling scheduling module controls the scheduling logic of the sampling process. This module is responsible for substituting the guiding prediction noise into the standard sampling formula of the target model to update the noise data for the next denoising time step. It then iterates through the next sampling step until sampling is complete.

[0099] The output generation module is used to output the final generated image or text after sampling is completed, which can be directly viewed or saved by the user.

[0100] The connection relationships between the modules are as follows: the input interface module is connected to the prediction residual calculation module and the sampling scheduling module; the prediction residual calculation module outputs the prediction residual to the guidance injection module; the guidance injection module generates guidance prediction noise and feeds it back to the sampling scheduling module; the sampling scheduling module controls the entire sampling process and retrieves the guidance prediction noise for each denoising time step; the final sampling result is post-processed and output by the output generation module.

[0101] The device composed of the above functional modules can be implemented in a high-performance computer by a software platform, or it can be encapsulated in an executable inference deployment system, and is suitable for tasks such as image generation and text generation.

[0102] The demand for personalized generation of diffusion models is rapidly increasing; however, traditional methods suffer from a serious "version dependency" problem: adapted models trained on a base model (such as LoRA, LyCORIS, ControlNet, or fully fine-tuned models) are difficult to migrate to newer versions of target models with different structures or parameter spaces (e.g., migrating from StableDiffusion 1.5 to Stable Diffusion 2.1). To address this challenge, traditional methods typically rely on retraining or knowledge distillation, which is not only resource-intensive but also unfeasible when the data is not reproducible.

[0103] The diffusion model transfer method based on prediction residuals proposed in this embodiment achieves cross-architecture, data-free, and training-free knowledge transfer by injecting the prediction difference (i.e., prediction residual) between the original base model and its adapted model during the sampling process. Compared with traditional methods, this embodiment has the following significant advantages.

[0104] (1) Completely data-independent migration capability.

[0105] This embodiment extracts the difference between the base model and its adapted model in each prediction step to construct a behavioral residual signal (i.e., prediction residual) that does not require training data. This bypasses the strong dependence of traditional methods on the original training data and is suitable for practical application scenarios where there is no access to training data.

[0106] (2) No parameter fine-tuning is required; it is plug-and-play during the inference stage.

[0107] This embodiment uses residual injection during inference, which only modifies the combination of noise predictions during inference, without requiring any adjustment to the weight level of the target model, thus greatly reducing deployment complexity and resource costs.

[0108] (3) A diffusion model that is highly compatible with different architectures.

[0109] Since the residual injection operation in this embodiment is based on the prediction space rather than the parameter space, it does not depend on the consistency of the model structure. This feature allows for migration even when the model architecture changes (such as different resolutions or number of channels), overcoming the limitation of existing adaptation modules that can only "bind to the original base model".

[0110] (4) Supports migration of multiple adaptation modules, enhancing generation control capabilities.

[0111] The residual signal proposed in this embodiment can naturally superimpose the influence of multiple adaptation modules, such as the total behavioral difference after combining modules like LoRA style transfer and ControlNet structure control. Therefore, it can realize multi-dimensional generation control and transfer, greatly improving the controllability and richness of the generation effect.

[0112] (5) Fully compatible with mainstream diffusion samplers and applicable to existing inference pipelines.

[0113] This embodiment does not rely on a specific sampling update formula. Its injection operation directly acts on the noise prediction value, thus it is compatible with various samplers such as Euler, DDIM, Heun, and DPM-Solver, and can be seamlessly integrated into existing industrial systems.

[0114] This application also provides an application scenario in which the above-described diffusion model transfer method guided by prediction residuals is applied. Specifically, the diffusion model transfer method guided by prediction residuals provided in this embodiment can be applied in an image generation scenario. The image generation scenario includes a construction phase and a display phase. The construction phase is used to control the image generation process of the target model based on the base model and the adaptation model to generate an image. The display phase is used to display the image to the user. The diffusion model transfer method guided by prediction residuals provided in this embodiment belongs to the construction phase.

[0115] Example 2.

[0116] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a diffusion model migration method guided by predictive residuals.

[0117] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the diffusion model migration method based on prediction residual guidance in Embodiment 1.

[0119] Example 3.

[0120] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the diffusion model migration method based on prediction residual guidance in Embodiment 1.

[0121] Example 4.

[0122] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the diffusion model migration method based on prediction residual guidance in Embodiment 1.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A diffusion model migration method based on prediction residual guidance, characterized in that, The diffusion model migration method based on the predicted residual guided diffusion model comprises the following steps: Obtaining noise data of a current denoising time step; at the first denoising time step, the noise data is random distribution noise; Taking the noise data of the current denoising time step as input, respectively determining first predicted noise, second predicted noise and third predicted noise of the current denoising time step by using a basic model, an adaptive model and a target model; the basic model is a model obtained by training a first diffusion model by using a first data set, the adaptive model is a model obtained by training the basic model by using a second data set, and the target model is a model obtained by training a second diffusion model by using a third data set, wherein the third data set does not include the second data set; Calculating the deviation between the first predicted noise and the second predicted noise of the current denoising time step to obtain the predicted residual of the current denoising time step; Performing weighted summation on the third predicted noise and the predicted residual of the current denoising time step to obtain the guided predicted noise of the current denoising time step; Based on the noise data and the guided predicted noise of the current denoising time step, the noise data of the next denoising time step is calculated, and if the next denoising time step is the last denoising time step, the noise data of the next denoising time step is taken as the output result of the target model; the output result is a picture, a video, a voice or a text.

2. The method of claim 1, wherein, The first diffusion model and the second diffusion model are of the same type, and the type includes a picture generation diffusion model, a video generation diffusion model, a voice generation diffusion model and a text generation diffusion model; When the first diffusion model and the second diffusion model are picture generation diffusion models, the output result is a picture; when the first diffusion model and the second diffusion model are video generation diffusion models, the output result is a video; when the first diffusion model and the second diffusion model are voice generation diffusion models, the output result is a voice; and when the first diffusion model and the second diffusion model are text generation diffusion models, the output result is a text.

3. The method of claim 1, wherein, If the adaptive model is one, the deviation between the first predicted noise and the second predicted noise of the current denoising time step is calculated to obtain the predicted residual of the current denoising time step, specifically including: subtracting the first predicted noise of the current denoising time step from the second predicted noise of the current denoising time step to obtain the predicted residual of the current denoising time step; If the adaptive model is multiple, one adaptive model corresponds to the second predicted noise of the current denoising time step, and the deviation between the first predicted noise and the second predicted noise of the current denoising time step is calculated to obtain the predicted residual of the current denoising time step, specifically including: for each adaptive model, subtracting the first predicted noise of the current denoising time step from the second predicted noise of the current denoising time step to obtain the initial predicted residual of the adaptive model at the current denoising time step, and performing weighted summation on the initial predicted residuals of all adaptive models at the current denoising time step to obtain the predicted residual of the current denoising time step.

4. The method of claim 3, wherein, If the adaptive model is multiple, the calculation formula of the predicted residual of the current denoising time step is: ； wherein, is a prediction residual for the current denoising time step t is noise data for the current denoising time step is a first predicted noise for the current denoising time step t is a second predicted noise for the current denoising time step n is a number of adaptation models; is a weight of the adaptation model i i t is a first predicted noise for the current denoising time step t is a second predicted noise for the current denoising time step​​​​ 5. The method of claim 1, wherein, The calculation formula of the guided prediction noise of the current denoising time step is: ； wherein, is a guided prediction noise for the current denoising time step t is a noise data for the current denoising time step is a third prediction noise for the current denoising time step t is a hyperparameter for controlling the prediction residual injection strength; is a prediction residual for the current denoising time step t is a third prediction noise for the current denoising time step is a hyperparameter for controlling the prediction residual injection strength; is a prediction residual for the current denoising time step t is a prediction residual for the current denoising time step 6. The method of claim 1, wherein, The calculation formula of the noise data of the next denoising time step is determined based on the type of the sampler, and when the type of the sampler is DDPM, the calculation formula of the noise data of the next denoising time step is: ； in, For the next denoising time step t -1 noise data; For the current denoising time step t The preset noise gain for the diffusion step size; For the current denoising time step t Noise data; For the current denoising time step t The cumulative retention factor; For the current denoising time step t Guided prediction noise; For the current denoising time step t The standard deviation of the matching forward diffusion process; It is random Gaussian noise.

7. The method of claim 1, wherein, After the noise data of the next denoising time step is calculated, the diffusion model migration method based on predicted residual guidance further comprises: if the next denoising time step is not the last denoising time step, taking the noise data of the next denoising time step as the noise data of the current denoising time step, reducing the current denoising time step by 1, and returning to the step of "obtaining the noise data of the current denoising time step".

8. A computer device comprising: A memory, a processor, and a computer program stored on the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the diffusion model migration method based on predicted residual guidance according to any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the diffusion model migration method based on predicted residual guidance according to any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the diffusion model migration method based on predicted residual guidance according to any one of claims 1-7. The computer program is executed by the processor to implement the diffusion model migration method based on predicted residual guidance according to any one of claims 1-7.

Images