A diffusion model image generation method and device of structural disturbance guidance and consistency distillation fusion

Abstract

This invention discloses a method and apparatus for generating images using a diffusion model that integrates structural perturbation guidance and consistency distillation. The method includes: initializing a first student network and a first teacher network using a diffusion model; generating an initial latent space state based on the latent space dataset and the noise scheduling function of the diffusion model; introducing a classifier-free guide with structural perturbation into the first teacher network to obtain a second teacher network; generating an augmented input for the first teacher network by performing state prediction using a solver based on the first teacher network, the second teacher network, and the initial latent space state; obtaining a consistency loss based on the first student network, the first teacher network, and the augmented input; updating the parameters of the first student network based on the consistency loss to obtain a target model; and using the target model to generate a target image. This invention can improve the quality of generated images and accelerate the generation speed of diffusion models, and can be widely applied in the field of image generation technology.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a diffusion model image generation method and apparatus based on structure perturbation guidance and consistency distillation fusion. Background Technology

[0002] Significant progress has been made in image generation in recent years. From the initial Generative Adversarial Networks (GANs) to current diffusion models, generative models have gradually demonstrated stronger image generation capabilities. Various guidance methods have played a crucial role in the development of diffusion models. Classifier guidance (CG) is a method that uses a pre-trained classifier to provide semantic gradients, thereby guiding the generation process. However, it requires an additional classifier model, resulting in relatively high computational costs. Classifier-free guidance (CFG) still has certain shortcomings in image structure, such as a lack of consistency or stability in complex scenes or detail representation. Furthermore, existing latent space diffusion models require dozens of iterations of denoising network operations to progressively estimate the noise intensity at each step. This high computational cost contradicts the real-world demand for rapid image generation. Summary of the Invention

[0003] In view of this, the main objective of the embodiments of the present invention is to provide a method and apparatus for generating diffusion model images by combining structural perturbation guidance and uniform distillation fusion, in order to solve at least one of the problems of the prior art. The present invention can improve the quality of the generated images and accelerate the generation speed of diffusion models.

[0004] To achieve the above objectives, one aspect of the present invention provides a diffusion model image generation method that integrates structural perturbation guidance and uniform distillation fusion, the method comprising the following steps:

[0005] Obtain the original training dataset and convert it into a latent space dataset;

[0006] The first student network is initialized using a diffusion model, and the first teacher network is also initialized.

[0007] The initial state of the latent space is generated based on the latent space dataset and the noise scheduling function of the diffusion model;

[0008] The classifier-free instruction after structural perturbation is introduced into the first teacher network to obtain the second teacher network;

[0009] Based on the first teacher network, the second teacher network, and the initial state of the latent space, a solver is used to predict the state and generate the enhanced input for the first teacher network.

[0010] Based on the first student network, the first teacher network, and the augmented input, obtain the consistency loss;

[0011] Based on the consistency loss, the parameters of the first student network are updated to obtain the target model;

[0012] The target model is used to generate the target image.

[0013] In some embodiments, generating the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model includes the following steps:

[0014] Random sampling is performed on the latent space dataset to obtain sampled samples;

[0015] Preset time step range;

[0016] Randomly sample the time step range to obtain the sampling time step;

[0017] The initial state of the latent space is generated based on the noise scheduling function of the diffusion model, the sampled samples, and the sampling time step.

[0018] In some embodiments, introducing the structurally perturbated, classifier-free guidance into the first teacher network to obtain the second teacher network includes the following steps:

[0019] The first self-attention matrix of the first teacher network is perturbed to generate a perturbed self-attention matrix;

[0020] The perturbation self-attention matrix is ​​multiplied by the appearance information to obtain the perturbation self-attention module;

[0021] The first self-attention module of the first teacher network is replaced with the perturbation self-attention module to obtain the second teacher network.

[0022] In some embodiments, the formula used to multiply the perturbation self-attention matrix with the appearance information to obtain the perturbation self-attention module includes:

[0023]

[0024] In the formula, PSA represents the perturbation self-attention module; Q t ,K t V t These represent query, key, and value respectively; Shuffle(·) represents the perturbation operation; A t Represents the first self-attention matrix; This represents the perturbation self-attention matrix.

[0025] In some embodiments, the step of generating enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space includes the following steps:

[0026] The solver is executed based on the first teacher network to obtain the first intermediate data;

[0027] The solver is executed based on the second teacher network to obtain the second intermediate data;

[0028] The enhanced input of the first teacher network is obtained based on the first intermediate data, the second intermediate data, and the initial state of the latent space.

[0029] In some embodiments, the formula used to generate the enhanced input for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space includes:

[0030]

[0031] In the formula, Enhanced input representing the first teacher network; z t+1 Ψ represents the initial state of the latent space; ω represents the guiding weights; t represents the sampling time step, t∈[0,T]; c represents the conditional information; φ represents the empty text; Ψ θ- This represents a solver based on a first-teacher network. Represents a solver based on a second-teacher network; θ - Parameters representing the first teacher network; The parameters represent the second teacher network.

[0032] In some embodiments, obtaining the consistency loss based on the first student network, the first teacher network, and the augmented input includes the following steps:

[0033] Obtain the first prediction of the first student network;

[0034] Based on the enhanced input from the first teacher network, obtain the distillation target output by the first teacher network;

[0035] The consistency loss is obtained based on the first prediction, the distillation objective, and the metric function.

[0036] In some embodiments, the formula used to obtain the consistency loss based on the first student network, the first teacher network, and the augmented input includes:

[0037]

[0038] In the formula, θ represents the consistency loss; θ represents the parameters of the first student network; θ - Parameters representing the first teacher network; Ψ θ- f represents a solver based on a first-teacher network; θ (z t+1 (c,t+1) represents the first prediction of the first student network; z t+1 represents the initial state of the latent space; c represents the conditional information; t represents the sampling time step. The distillation target represents the output of the first teacher network; d represents the augmented input of the first teacher network; d(·,·) represents the metric function; This represents the average expected value of random variables z, c, ω, and t.

[0039] To achieve the above objectives, another aspect of the present invention proposes a diffusion model image generation apparatus that integrates structural perturbation guidance and consistency distillation, the apparatus comprising:

[0040] The first module is used to obtain the original training dataset and convert the original training dataset into a latent space dataset.

[0041] The second module is used to initialize the first student network and the first teacher network using a diffusion model.

[0042] The third module is used to generate the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model;

[0043] The fourth module is used to introduce the classifier-free guidance after structural perturbation into the first teacher network to obtain the second teacher network;

[0044] The fifth module is used to generate enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space.

[0045] The sixth module is used to obtain the consistency loss based on the first student network, the first teacher network, and the enhanced input;

[0046] The seventh module is used to update the parameters of the first student network based on the consistency loss to obtain the target model;

[0047] The target model is used to generate the target image.

[0048] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method for generating diffusion model images by structural perturbation guidance and uniform distillation fusion.

[0049] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for generating diffusion model images by structural perturbation guidance and uniform distillation fusion.

[0050] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned diffusion model image generation method based on structure perturbation guidance and uniform distillation fusion.

[0051] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a diffusion model image generation method and apparatus that integrates structural perturbation guidance and consistency distillation. This scheme obtains an original training dataset and converts it into a latent space dataset; initializes a first student network and a first teacher network using a diffusion model; generates an initial latent space state based on the latent space dataset and the noise scheduling function of the diffusion model; introduces structurally perturbated, classifier-free guidance into the first teacher network to obtain a second teacher network; generates an augmented input for the first teacher network by performing state prediction using a solver based on the first teacher network, the second teacher network, and the initial latent space state; obtains a consistency loss based on the first student network, the first teacher network, and the augmented input; and updates the parameters of the first student network based on the consistency loss to obtain a target model; wherein the target model is used to generate the target image. The present invention transforms the complex generation process into a simplified model, allowing the teacher model to generate high-quality distillation targets, while the student model learns these targets to approximate complex generation behaviors. Furthermore, the introduction of structurally perturbated, classifier-free guidance into the teacher network further improves the quality of the generated images and makes the generation process more efficient. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the diffusion model image generation method based on structure perturbation guidance and consistency distillation fusion provided in this embodiment of the invention;

[0054] Figure 2 This is a schematic diagram illustrating the specific steps of generating a diffusion model image based on the structural perturbation guidance and uniform distillation fusion provided in this embodiment of the invention;

[0055] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0057] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0058] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0060] Significant progress has been made in image generation technology in recent years. From the initial Generative Adversarial Networks (GANs) to the current diffusion models, generative models have gradually demonstrated stronger image generation capabilities. GANs generate realistic images through adversarial training, but they have certain limitations in terms of generation diversity and training stability. In contrast, diffusion models employ a progressive denoising generation strategy, starting from random noise and gradually reconstructing a clear image through multiple iterations. This generation mechanism can accurately model the data distribution, resulting in generated images that are not only of higher quality but also possess greater diversity, thus gradually becoming the mainstream method in the field of image generation.

[0061] In the development of diffusion models, various guidance methods have played a crucial role. Classifier guidance (CG) is a method that uses a pre-trained classifier to provide semantic gradients, thereby guiding the generation process. However, it requires an additional classifier model, resulting in relatively high computational overhead. In contrast, classifier-free guidance (CFG) eliminates the dependence on classifiers by fusing conditional and unconditional distributions, simplifying the generation process and further improving the quality of the generated images. Therefore, CFG has become the mainstream guidance method. However, CFG-generated content still has certain shortcomings in terms of image structure, such as a lack of consistency or stability in the representation of complex scenes or details.

[0062] Besides generation quality, generation speed is another major challenge faced by diffusion models. Although latent space-based diffusion models improve computational efficiency and flexibility while maintaining high-quality image synthesis by reducing the computational complexity of high-dimensional pixel space, existing latent space diffusion models still require dozens of iterations of denoising network operations to gradually estimate the noise intensity at each step. This high computational cost contradicts the real-world user demand for rapid image generation.

[0063] In view of this, such as Figure 1 As shown, this embodiment of the invention provides a diffusion model image generation method that integrates structure perturbation guidance and consistency distillation fusion. This method may include, but is not limited to, steps S100 to S700:

[0064] Step S100: Obtain the original training dataset and convert the original training dataset into a latent space dataset;

[0065] Step S200: Initialize the first student network and the first teacher network using the diffusion model;

[0066] Step S300: Generate the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model;

[0067] Step S400: Introduce the classifier-free guidance after structural perturbation into the first teacher network to obtain the second teacher network;

[0068] Step S500: Based on the first teacher network, the second teacher network, and the initial state of the latent space, the solver performs state prediction to generate the enhanced input for the first teacher network;

[0069] Step S600: Obtain the consistency loss based on the first student network, the first teacher network, and the enhanced input;

[0070] Step S700: Update the parameters of the first student network according to the consistency loss to obtain the target model;

[0071] The target model is used to generate the target image.

[0072] In step S100 of some embodiments, the original training dataset D is obtained, and the original training dataset is converted into a latent space dataset D by encoder E(·). z Then we have D z = {(z,c)|z=E(x),(x,c)∈D}, where x is the original training data, z is the latent space encoding of the original training data x, and c is the corresponding conditional information. Using latent space representation can significantly reduce computational complexity, providing a more efficient foundation for subsequent model training and inference.

[0073] In step S200 of some embodiments, a first student network is initialized using the parameters θ of a pre-trained diffusion model. Simultaneously, a first teacher network is defined, with parameters θ... - The exponential moving average (EMA) can be used for updating, and the expression is as follows:

[0074] θ - ←μθ - +(1-μ)θ

[0075] In the formula, θ - θ represents the parameters of the first teacher network; θ represents the parameters of the first student network; μ represents the weight hyperparameters of the exponential moving average. The exponential moving average (EMA) mechanism can effectively maintain the stability of the training process.

[0076] In some embodiments, the diffusion model is a generative model that generates samples step by step by constructing a forward process by injecting Gaussian noise into the data and then gradually recovering the data samples from the noise through reverse denoising.

[0077] (1) Forward diffusion process:

[0078]

[0079] In the formula, t∈[0,T] is the time step; x t x represents the data state at time step t; x0 represents the original data state; α(t) and σ(t) represent the noise scheduling functions at time step t; I represents the identity matrix. Represents a Gaussian distribution; q(x) t |x0) represents the conditional probability distribution of the data state at time step t, given the original data state. As t approaches 0, x t Approximately the original data distribution p data (x); as t approaches T, x t It is distributed near the edge.

[0080] (2) Reverse generation process:

[0081] By training the denoising network ∈ θ (x t ,t), for x t Modeling is performed on t to evaluate noise intensity, gradually restoring the marginal distribution to the data distribution. Network parameters θ are optimized to minimize denoising error; the inverse generation formula is:

[0082]

[0083] In the formula, μ θ Represents the mean; Σ θ Represents covariance.

[0084] In some embodiments, step S300 may include, but is not limited to, steps S310 to S340:

[0085] Step S310: Randomly sample the latent space dataset to obtain sampled samples;

[0086] Step S320: Preset the time step range;

[0087] Step S330: Randomly sample the time step range to obtain the sampled time step;

[0088] Step S340: Generate the initial state of the latent space based on the noise scheduling function of the diffusion model, the sampled samples, and the sampling time step.

[0089] In steps S310 to S340 of some embodiments, the latent space dataset D is... z A sample pair (z, c) is randomly sampled from the training data x, where z is the latent space encoding of the original training data x and c is the conditional information. Random sampling is performed within the time step range [1, T], so the sampling time step t∈[1, T]. Then, using the noise scheduling functions α(t) and σ(t) of the diffusion model, the initial state z of the latent space can be generated. t+1 Its distribution follows a Gaussian distribution:

[0090]

[0091] This process simulates the forward diffusion process of a diffusion model, gradually introducing noise into the data.

[0092] In some embodiments, the self-attention module of the diffusion model models the structure and appearance of the image, where query-key similarity is primarily responsible for structural modeling, while value information is processed. In the self-attention module at time step t, its output can be represented as:

[0093]

[0094] in, These are the query, key, and value, respectively; d represents the number of image channels; A t It is the first self-attention matrix, used to model the relationship between different positions of the input image, especially the structural information of the image.

[0095] In some embodiments, consistent distillation is a method to approximate the generative behavior of complex models by training a simplified model. In diffusion models, the goal is to simplify complex multi-step backward generation processes into fewer or even single-step generation while maintaining generation quality. The core idea of ​​consistent distillation is based on knowledge distillation, condensing the complex generative behavior of the teacher model into an efficient generation process of the student model. Latent space consistent distillation further extends this distillation process to the latent space, thereby significantly improving computational efficiency and inference speed. Given a solution trajectory {x} of the diffusion model PF-ODE... t} t∈[0,T] Consistency function Defined as starting from any time step (x) t The function that maps (x, t) to the initial state (x0, 0). The following are the core concepts of the latent space consistent distillation scheme:

[0096] 1) A consistency function possesses the following consistency properties.

[0097] For any state (x) on the same PF-ODE trajectory t,t) and (x t′ The consistency function defined by ,t′) must have the same output, that is

[0098]

[0099] 2) The boundary condition for the uniformity function is the identity mapping.

[0100] f(x0,0)=x0

[0101] This boundary condition is crucial during the training process of uniform distillation. For any well-trained diffusion model F... θ (x,t) can be defined with a consistency function to satisfy the boundary conditions in the following form:

[0102] f θ (x,t)=c skip (t)x+c out (t)F θ (x,t)

[0103] Among them, c skip (t) and c out (t) is a differentiable function with respect to time t, satisfying c skip (0) = 1 and c out (0) = 0.

[0104] In traditional latent space consistent distillation, the teacher model is solved by the ODE solver (i.e. Starting from the initial state z in the latent space t+1 Predict the latent space state at time step t. At this point, the ODE solver is executed. If the computation is based on a teacher network, then the input that generates the teacher network is... The following expression exists:

[0105]

[0106] Based on the input of the teacher network generated above Define consistency loss as:

[0107]

[0108] Among them, f θ (z t+1 ,c,t+1) represents the prediction of the student network; d represents the distillation target of the teacher network output; d(·,·) represents the metric function (such as Euclidean distance); Let θ represent the average expected value of random variables z, c, and t. Based on consistency loss, the student model parameters θ are optimized using gradient descent. Simultaneously, the teacher network parameters θ are updated using the EMA mechanism. - .

[0109] In some embodiments, classifier-free guidance (CFG) is a method to improve the generation quality of diffusion models, particularly suitable for textual graph tasks. It does not rely on a classifier but directly utilizes the capabilities of the conditional model itself for guidance, improving generation quality and conditional consistency. By interpolating between unconditional and conditional generation, classifier-free guidance effectively enhances the guiding effect of conditions on the generation task. Applying CFG to textual graph models can be represented as:

[0110]

[0111] In the formula, x t Represents the data state at time step t; c represents conditional information; φ represents empty text (no conditions); ω represents the guidance weight, used to adjust the strength of conditional guidance; ∈ θ A noise prediction network representing a conditional model; This represents a noise prediction network enhanced without classifier guidance. This enhanced noise prediction network consists of the outputs of two branch networks with identical parameters, differing only in their input conditions: one is c, and the other is φ.

[0112] Classifier-free guidance improves the matching accuracy of specific conditions in the generation task by directly leveraging the capabilities of the conditional model itself. Ordinary classifier-free guidance enhances generation quality by introducing empty text (unconditional branch) into the diffusion model, but it primarily relies on the conditional model itself for guidance, without altering the model architecture. While this approach can improve image generation quality to some extent, it still has the following limitations:

[0113] 1) Problem of structural integrity in unconditional generation: In the absence of classifier guidance, the structure of unconditionally generated samples is relatively complete, but this cannot ensure that the model can effectively avoid potential structural collapse samples when generating conditionally.

[0114] 2) Limitations of negative sample effects: Due to the lack of a clear negative sample interference mechanism, the final generated image may still show structural degradation under conditional guidance.

[0115] In view of this, in steps S400 to S600 of some embodiments, a combination of structural perturbation mechanism and classifier-free guidance is proposed. Specifically, the structurally perturbation-guided classifier-free guidance is applied to the teacher model's generation of distilled targets through the structural perturbation guidance mechanism. By introducing structurally collapsed negative samples, the unconditionally generated samples not only differ from the conditionally generated samples in category but also exhibit significant structural collapse, thereby strengthening the guiding role of the negative sample targets. This combined approach can more explicitly guide the conditional model away from collapsed samples during the generation process, further improving the structural consistency and quality of the final generated image.

[0116] Optionally, the core idea of ​​classifier-free guided fusion structure perturbation is to modify the conditional model of unconditional branching by introducing a structure perturbation mechanism to disrupt the self-attention matrix and thus disturb the structure of unconditionally generated samples. Its formula is expressed as:

[0117]

[0118] Where c represents conditional information; φ represents empty text (no conditions); ω is the guiding weight, used to control the strength of conditional guidance; ∈ θ A noise prediction network representing a conditional model; This represents the unconditional branching model modified by structural perturbation. The difference from the ordinary classifier-free guided formula is that the parameters of the diffusion model in the unconditional branching are modified from θ to... This introduces structural disturbances.

[0119] Through the above formula, this embodiment of the invention explicitly introduces a structural perturbation mechanism in the unconditional generation branch, making the generated negative sample targets more explicit. During actual training, these structurally collapsed negative samples can significantly strengthen the learning objective of the conditional model, guiding it to avoid degenerate sample distributions, thereby better reconstructing high-quality image structures during the generation process.

[0120] In this embodiment of the invention, classifier-free guidance after structural perturbation is incorporated into latent space consistent distillation. Combining classifier-free guidance after structural perturbation with latent space consistent distillation can significantly improve the quality of generated images and inference efficiency. Distillation transforms the complex generation process into a simplified model learning process, allowing the teacher model to generate high-quality distillation targets, while the student model learns these targets to approximate complex generation behaviors. In latent space consistent distillation, the distillation targets generated by the teacher model are typically obtained through a multi-step inference process, while classifier-free guidance after structural perturbation is introduced into the teacher network, thereby further improving the quality of distillation targets generated by the teacher network, improving the quality of generated images, and making the generation process more efficient.

[0121] In some embodiments, step S400 may include, but is not limited to, steps S410 to S430:

[0122] Step S410: Perturb the first self-attention matrix of the first teacher network to generate a perturbed self-attention matrix;

[0123] Step S420: Multiply the perturbation self-attention matrix with the appearance information to obtain the perturbation self-attention module;

[0124] Step S430: Replace the first self-attention module of the first teacher network with the perturbation self-attention module to obtain the second teacher network.

[0125] In steps S410 to S430 of some embodiments, the first self-attention matrix A of the first teacher network is... t Perturbation, that is, perturbation of the first self-attention matrix A t The order of each row is shuffled to generate a perturbation self-attention matrix. This disrupts the structural information of the image. Subsequently, the self-attention matrix is ​​perturbed. With appearance information V t Multiplying these yields the module's output. The first self-attention module, after structural perturbation, is rewritten as a perturbed self-attention module (PSA), resulting in the second teacher network. The formula for the perturbed self-attention module is as follows:

[0126]

[0127] In the formula, PSA represents the perturbation self-attention module; Q t ,K t V t These represent query, key, and value respectively; Shuffle(·) represents the perturbation operation; A t Represents the first self-attention matrix; This represents the perturbation self-attention matrix.

[0128] The structural perturbation guidance mechanism, which generates negative samples of structural collapse by perturbing the query-key similarity in the first self-attention matrix, guides the model to avoid generating degenerate samples, thereby improving the robustness of the denoising process.

[0129] In some embodiments, step S500 may include, but is not limited to, steps S510 to S530:

[0130] Step S510: Execute the solver based on the first teacher network to obtain the first intermediate data;

[0131] Step S520: Execute the solver based on the second teacher network to obtain the second intermediate data;

[0132] Step S530: Based on the first intermediate data, the second intermediate data, and the initial state of the latent space, the enhanced input of the first teacher network is obtained.

[0133] In some embodiments, steps S510 to S530 involve executing the solver Ψ based on the first teacher network. θ- Obtain the first intermediate data Ψ θ- (z t+1 (t+1,t,c), the solver is executed based on the second teacher network. Obtain the second intermediate data Based on the first intermediate data, the second intermediate data, and the initial state of the latent space, the formula for the augmented input of the first teacher network can be obtained as follows:

[0134]

[0135] In the formula, Enhanced input representing the first teacher network; z t+1 Ψ represents the initial state of the latent space; ω represents the guiding weight, controlling the strength of the unconditional guidance after structural perturbation; t represents the sampling time step, t∈[0,T]; c represents the conditional information; φ represents the empty text (unconditional); Ψ * This indicates that the ODE solver is based on the diffusion model* and is responsible for simulating the denoising process; Ψ θ- This represents a solver based on a first-teacher network. Represents a solver based on a second-teacher network; θ - Parameters representing the first teacher network; The parameters represent the second teacher network.

[0136] When the unclassifier-guided approach after structural perturbation is applied to the teacher network, it means that the unconditional branch is explicitly perturbed to generate structurally degraded negative samples. These negative samples effectively help the teacher network avoid generating structurally degraded samples, thereby improving the quality of the generated images. In this way, the quality of the latent space states generated by the teacher network is significantly improved, while reinforcing the learning objectives of the student model during training. This combination ensures that the objectives generated by the teacher network are far removed from potential structurally degraded negative samples, enabling the student model to better learn how to generate high-quality, structurally consistent images. Through this improvement, the student model can rely on fewer steps during inference, or even complete high-quality image generation through single-step inference, without relying on the traditional multi-step inference process. This not only greatly improves inference efficiency but also ensures the quality and consistency of the final generated images.

[0137] In some embodiments, step S600 may include, but is not limited to, steps S610 to S630:

[0138] Step S610: Obtain the first prediction of the first student network;

[0139] Step S620: Obtain the distillation target output by the first teacher network based on the enhanced input of the first teacher network;

[0140] Step S630: Based on the first prediction, the distillation target, and the metric function, the consistency loss is obtained.

[0141] In some embodiments, steps S610 to S630 involve obtaining the first prediction f of the first student network. θ (z t+1 (c,t+1), based on the augmented input of the first teacher network Obtain the distillation target from the first teacher network output. Combining this with the metric function, we can obtain the following expression for the consistency loss:

[0142]

[0143] In the formula, θ represents the consistency loss; θ represents the parameters of the first student network; θ - Parameters representing the first teacher network; Ψ θ- f represents a solver based on a first-teacher network; θ (z t+1 (c,t+1) represents the first prediction of the first student network; z t+1 represents the initial state of the latent space; c represents the conditional information; t represents the sampling time step. The distillation target represents the output of the first teacher network; d represents the augmented input of the first teacher network; d(·,·) represents the metric function; This represents the average expected value of random variables z, c, ω, and t.

[0144] In step S700 of some embodiments, based on the consistency loss, the parameters θ of the first student model are optimized by gradient descent, while the parameters of the first teacher network are updated using the EMA mechanism, thus obtaining the target model for generating the target image. For example, by calculating the consistency loss, the parameters of the first student model and the first teacher network are updated, and it is determined whether the consistency loss meets the preset convergence condition. If not, the process returns to the step of generating the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model. If the condition is met, the target model for generating the target image is obtained. The model trained through consistency distillation can complete high-quality generation tasks in fewer steps or even a single step, significantly improving inference efficiency.

[0145] In summary, the processing flow of the diffusion model image generation method based on structure perturbation guidance and consistency distillation fusion according to an embodiment of the present invention is as follows: Figure 2 As shown:

[0146] Step 1, Latent Space Data Construction: The original training dataset is converted into a latent space dataset using an encoder;

[0147] Step 2, Model Initialization and EMA Update Mechanism: Initialize the student model using the parameters of the pre-trained diffusion model. Simultaneously, define a teacher network, where the teacher network parameters are updated using exponential moving average (EMA).

[0148] Step 3: Random sampling to generate the initial state of the latent space: In each training iteration, a sample pair is randomly sampled from the latent space dataset, and time steps are randomly sampled from the time step range. Then, the initial state of the latent space is generated using the noise scheduling function of the diffusion model. This process simulates the forward diffusion process of the diffusion model, gradually introducing noise into the data.

[0149] Step 4: Add a classifier-free guidance branch with structural perturbation. Introduce the classifier-free guidance after structural perturbation into the teacher network. Based on the teacher network, use the ODE solver to predict the state corresponding to the time step from the initial state of the latent space and generate the input of the teacher network.

[0150] Step 5: Using the predictions of the student network and the distillation target of the teacher network output, define the consistency loss. Based on the consistency loss, optimize the student model parameters through gradient descent, and at the same time update the teacher network parameters using the EMA mechanism.

[0151] Step 6, Convergence and Model Output: Repeat steps 3 to 5 until the loss function meets the preset convergence condition, the model converges, and the target model for generating the target image is obtained. The model trained through consistency distillation can complete high-quality generation tasks in fewer steps or even a single step, significantly improving inference efficiency.

[0152] This invention also provides an apparatus for generating diffusion model images using structure perturbation guidance and uniform distillation fusion, which can realize the above-described method for generating diffusion model images using structure perturbation guidance and uniform distillation fusion. The apparatus includes:

[0153] The first module is used to obtain the original training dataset and convert the original training dataset into a latent space dataset.

[0154] The second module is used to initialize the first student network and the first teacher network using a diffusion model.

[0155] The third module is used to generate the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model;

[0156] The fourth module is used to introduce the classifier-free guidance after structural perturbation into the first teacher network to obtain the second teacher network;

[0157] The fifth module is used to generate enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space.

[0158] The sixth module is used to obtain the consistency loss based on the first student network, the first teacher network, and the enhanced input;

[0159] The seventh module is used to update the parameters of the first student network based on the consistency loss to obtain the target model;

[0160] The target model is used to generate the target image.

[0161] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0162] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned method for generating diffusion model images based on structural perturbation guidance and uniform distillation fusion. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0163] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] refer to Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0165] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0166] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to execute a diffusion model image generation method based on structural perturbation guidance and uniform distillation fusion according to an embodiment of the present invention.

[0167] The 803 input / output interface is used to implement information input and output.

[0168] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0169] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0170] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0171] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating diffusion model images by structural perturbation guidance and uniform distillation fusion.

[0172] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0173] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for generating a diffusion model image based on structure perturbation guidance and uniform distillation fusion.

[0174] In summary, the diffusion model image generation method and apparatus based on structure perturbation guidance and consistency distillation fusion according to embodiments of the present invention have the following advantages:

[0175] 1. This invention applies additional perturbation to the image structure within a classifier-free guidance framework, generating samples with degenerate structures. These samples guide the denoising process away from the degenerate samples, thereby improving the structural quality and visual effect of the generated image (e.g., text-based images), enhancing the structural consistency and detail representation of the generated image, and significantly optimizing the generation quality of complex scenes.

[0176] 2. The embodiments of the present invention introduce consistent distillation, which learns and optimizes the inference path in the latent space, greatly reducing the number of inference steps. The number of steps required for inference is reduced from dozens of steps in the traditional method to single digits, and even supports single-step generation. While ensuring the generation quality, it significantly accelerates the generation process of the diffusion model, greatly improves the generation speed, and meets the user's actual needs for rapid image generation.

[0177] 3. This embodiment of the invention further reduces the computational overhead of the unclassified guidance branch by distilling the unclassified guidance after structural perturbation into the model. This optimization scheme significantly improves the model's generation efficiency while ensuring generation quality. Through dual optimization of generation quality and efficiency, the innovative method of this embodiment not only meets the requirements of high-quality image generation but also greatly improves the generation speed, providing a more efficient and reliable solution for practical application scenarios.

[0178] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0179] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0180] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0182] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0183] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0184] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0185] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0186] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for generating diffusion model images by fusing structural perturbation guidance and uniform distillation, characterized in that, Includes the following steps: Obtain the original training dataset and convert it into a latent space dataset; The first student network is initialized using a diffusion model, and the first teacher network is also initialized. The initial state of the latent space is generated based on the latent space dataset and the noise scheduling function of the diffusion model; The classifier-free instruction after structural perturbation is introduced into the first teacher network to obtain the second teacher network; Based on the first teacher network, the second teacher network, and the initial state of the latent space, a solver is used to predict the state and generate the enhanced input for the first teacher network. Based on the first student network, the first teacher network, and the augmented input, obtain the consistency loss; Based on the consistency loss, the parameters of the first student network are updated to obtain the target model; The target model is used to generate the target image; The step of introducing the classifier-free guidance after structural perturbation into the first teacher network to obtain the second teacher network includes the following steps: The first self-attention matrix of the first teacher network is perturbed to generate a perturbed self-attention matrix; The perturbation self-attention matrix is ​​multiplied by the appearance information to obtain the perturbation self-attention module; The first self-attention module of the first teacher network is replaced with the perturbation self-attention module to obtain the second teacher network.

2. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The step of generating the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model includes the following steps: Random sampling is performed on the latent space dataset to obtain sampled samples; Preset time step range; Randomly sample the time step range to obtain the sampling time step; The initial state of the latent space is generated based on the noise scheduling function of the diffusion model, the sampled samples, and the sampling time step.

3. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The perturbation self-attention module is obtained by multiplying the perturbation self-attention matrix with the appearance information, and the formula used includes: ； In the formula, Represents a perturbation self-attention module; These represent the query, key, and value, respectively. Represents a disturbance operation; Represents the first self-attention matrix; This represents the perturbation self-attention matrix.

4. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The step of generating enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space includes the following steps: The solver is executed based on the first teacher network to obtain the first intermediate data; The solver is executed based on the second teacher network to obtain the second intermediate data; The enhanced input of the first teacher network is obtained based on the first intermediate data, the second intermediate data, and the initial state of the latent space.

5. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The process of generating enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space includes the following formulas: ； In the formula, Enhanced input representing the first teacher network; Represents the initial state of the latent space; Represents guiding weight; Represents the sampling time step. ; Represents conditional information; Represents empty text; This represents a solver based on a first-teacher network. This represents a solver based on a second-teacher network. Parameters representing the first teacher network; The parameters represent the second teacher network.

6. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The step of obtaining the consistency loss based on the first student network, the first teacher network, and the enhanced input includes the following steps: Obtain the first prediction of the first student network; Based on the enhanced input from the first teacher network, obtain the distillation target output by the first teacher network; The consistency loss is obtained based on the first prediction, the distillation objective, and the metric function.

7. The diffusion model image generation method based on structural perturbation guidance and consistency distillation fusion according to claim 1, characterized in that, The formula used to obtain the consistency loss based on the first student network, the first teacher network, and the augmented input includes: ； In the formula, This represents a loss of consistency; Parameters representing the first student network; Parameters representing the first teacher network; This represents a solver based on a first-teacher network. The first prediction representing the first student network; Represents the initial state of the latent space; Represents conditional information; Represents the sampling time step. ; The distillation target represents the output of the first teacher network; Enhanced input representing the first teacher network; Represents a metric function; Represents the relationship between random variables and The average expected value.

8. A diffusion model image generation device that integrates structural perturbation guidance and uniform distillation fusion, characterized in that, include: The first module is used to obtain the original training dataset and convert the original training dataset into a latent space dataset. The second module is used to initialize the first student network and the first teacher network using a diffusion model. The third module is used to generate the initial state of the latent space based on the latent space dataset and the noise scheduling function of the diffusion model; The fourth module is used to introduce the classifier-free guidance after structural perturbation into the first teacher network to obtain the second teacher network; The fifth module is used to generate enhanced inputs for the first teacher network by performing state prediction through a solver based on the first teacher network, the second teacher network, and the initial state of the latent space. The sixth module is used to obtain the consistency loss based on the first student network, the first teacher network, and the enhanced input; The seventh module is used to update the parameters of the first student network based on the consistency loss to obtain the target model; The target model is used to generate the target image; The fourth module is specifically used for: The first self-attention matrix of the first teacher network is perturbed to generate a perturbed self-attention matrix; The perturbation self-attention matrix is ​​multiplied by the appearance information to obtain the perturbation self-attention module; The first self-attention module of the first teacher network is replaced with the perturbation self-attention module to obtain the second teacher network.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

Images