Image or video generation method and system based on diffusion model self-guidance

Abstract

The invention discloses a diffusion model self-guiding-based image or video generation method and system. The method comprises the steps that the input content is a noise potential variable and a future offset step length of a current time step; the diffusion model predicts the score network twice based on the input content to obtain a conditional score network of the current time step and the future offset time step; carrying out linear combination on the two conditional score networks to generate a combined conditional score network; denoising the noise potential variable of the current time step by using a combined condition score network; calculating and updating a potential variable of the previous time step according to a denoising result; in each time step, the self-guiding strategy is adopted to carry out denoising processing on the noise, and noise prediction is optimized; and after de-noising processing of all time steps, a de-noising potential variable is finally output as a final model generation result, and an image or a video is obtained. According to the invention, the quality of the generated image or video can be obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and in particular relates to a method and system for generating images or videos based on self-guiding of a diffusion model. Background Art

[0002] In the past decade, deep generative models have achieved remarkable progress in various application areas. Among them, diffusion models and flow-based models have performed well in generating high-resolution text-driven data, such as images, videos, etc.

[0003] For current Diffuion-based text-to-image models and text-to-video models, it is particularly important to optimize the generation quality by adopting appropriate guidance strategies only in the inference phase without additional training. However, for existing text-to-image models, simple sampling strategies often fail to achieve ideal sample quality, such as generating poor hand and face images, and images with poor foreground and background processing.

[0004] In order to solve these problems, various guidance strategies have emerged and become an effective means to improve the generation quality, including classifier-free guidance (CFG), automatic guidance (AG), perturbation attention guidance (PAG), etc.

[0005] For example, a Chinese patent document with publication number CN118886452A discloses a lightweight diffusion model based on U-net, which forms a stable diffusion model by training a latent diffusion model. The latent space is encoded from a pre-trained variational autoencoder. During the inference process, the image is constructed from the latent space decoder; the latent diffusion model explores text-to-image generation, in which a text prompt embedding c is fed into the diffusion model as a condition, and when synthesizing images, no classifier guidance is used.

[0006] The Chinese patent document with publication number CN118864295A discloses a fast two-stream denoising diffusion model and image fusion method based on soft attention control without classifier guidance. A denoising probability diffusion network is built using a convolutional neural network, and an image fusion network model and training process for integrating the desired visual concepts into existing images are provided; the soft attention control mechanism and the classification-free guidance method are introduced into the diffusion model, and weights and controllers are used to control the fusion features of the image.

[0007] Classifier-free guidance (CFG) guides the generated samples to be more consistent with the target category by introducing the difference between the category-conditional and unconditional speeds in the speed of the generative model. However, although existing guidance methods have shown significant performance improvements in practical applications, they still have certain limitations. For example, classifier-free guidance relies on calculating the unconditional speed, which requires an additional training process and may affect the performance of the model. Automatic guidance (AG) relies on a "bad version" model, which is not only difficult to train, but also requires additional memory overhead. Another widely used perturbation attention guidance (PAG), although it performs well in UNet-based diffusion models, its performance may drop significantly when dealing with Transformer architectures due to the complexity of hyperparameter adjustment. Summary of the invention

[0008] The present invention provides a method and system for generating images or videos based on self-guidance of a diffusion model, which can significantly improve the quality of generated images or videos.

[0009] A method for generating an image or video based on self-guidance of a diffusion model comprises the following steps:

[0010] (1) Prepare the input, which includes the noise latent variable x at the current time step t. t and the set future offset step size δ(t);

[0011] (2) The diffusion model performs two score network predictions based on the input content to obtain the conditional score network of the current time step t and the conditional score network of the future offset time step t+δ(t); and linearly combines the two conditional score networks to generate the final combined conditional score network s combined ;

[0012] (3) Using a combined conditional score network combined , for the noise latent variable x at the current time step t Perform denoising; based on the denoising result, calculate and update the latent variable x of the previous time step t-1 ;

[0013] (4) At each time step, the self-guided SG strategy in steps (1) to (3) above is used to denoise the noise and optimize the noise prediction;

[0014] (5) After denoising at all time steps, the denoised latent variable x0 is finally output as the final generation result of the diffusion model to generate an image or video.

[0015] The present invention adopts a novel self-guided SG strategy to improve the quality of generated samples by adding a generation speed difference to two different noise levels while keeping the default generation speed. Under this guidance, the samples can be biased towards images with higher probabilities as the denoising process of the diffusion or flow-based model progresses. SG has multiple advantages. First, SG is architecture-independent and can be integrated into almost all existing diffusion and flow-based models regardless of the UNet or Transformer architecture. Second, SG does not require training for specific guidance, which is the difference between SG and CFG and AG. Third, SG is compatible with other guidance such as CFG, AG and PAG.

[0016] Optionally, in step (1), when the diffusion model is a conditional generation model, the input content also includes a category label c, otherwise, the category label c is not included.

[0017] In step (2), the two conditional score networks are linearly combined using the hyperparameter ω, as follows:

[0018] s combined =(1+ω)*s t -ω*s t+δ(t)

[0019] In the formula, s t represents the conditional score network at the current time step t, s t+δ(t) A conditional score network representing the future offset time step t+δ(t).

[0020] If the input includes a category label c, then s t It is expressed as s(x t |c,t),s t+δ(t) It is expressed as s(x t |c,t+δ(t)).

[0021] In step (3), the latent variable x in the previous time step is calculated and updated t-1 , the formula is:

[0022] x t-1 =f(x t ,s combined ,t)

[0023] Where f represents the diffusion model denoising function, which is used to transform the current latent variable x t Denoising to compute the latent variable at the previous time step.

[0024] Preferably, in step (4), on the basis of the self-guided SG strategy, one of the guiding strategies of classifier-free guided CFG and perturbation attention guided PAG is added, and the noise prediction is comprehensively optimized through weighted factors.

[0025] Preferably, on the basis of the self-guided SG strategy, two strategies, CFG without classifier guidance and PAG with perturbation attention guidance, are added, and the weighting factor ω SG ,ω CFG and ω PAG Comprehensive optimization of noise prediction.

[0026] Based on the same inventive principle, the present invention also provides an image or video generation system based on self-guiding of a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned image or video generation method.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. No specific training required: SG does not require additional specific training. This feature clearly distinguishes SG from CFG and AG, which often require additional training of unconditional models or so-called "bad version models", thereby increasing complexity and computational cost.

[0029] 2. Not restricted by neural network architecture: The implementation of SG is independent of the neural network architecture, which means that it can be flexibly applied to diffusion models and flow models based on different architectures, such as UNet-based models or Transformer-based models. In contrast, methods such as PAG rely heavily on the UNet architecture and may have obvious performance limitations when applied to pure Transformer-based diffusion models.

[0030] 3. Strong compatibility and synergy with other guidance methods: SG has good compatibility and can be used in conjunction with other guidance methods on UNet or DiT-based diffusion models, and is suitable for generation tasks such as text-to-image and text-to-video. This compatibility shows that SG is orthogonal to other guidance methods and can further improve the generation quality without weakening the performance of other guidance methods.

[0031] 4. High-quality human generation: SG has shown surprising advantages in generating high-quality human images or videos. Generating natural and accurate human bodies (especially hands, fingers, and faces) has always been a challenging but extremely important task. Experiments have shown that with the help of SG, the naturalness of the generated hand shape and the accuracy of the number of fingers can be significantly improved. This shows that the text-to-image diffusion model may not require additional specific training or post-processing for the human body, and high-quality human generation can be achieved through SG alone. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the self-guidance strategy in the present invention;

[0033] Figure 2 A comparative schematic diagram of generating high-quality images in an embodiment of the present invention;

[0034] Figure 3 A comparative schematic diagram of generating high-quality videos in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0036] First, the working principles of the existing classifier-free guidance strategy and the self-guidance strategy designed by the present invention are introduced.

[0037] (a) Without classifier guidance

[0038] Although the diffusion model has a solid theoretical interpretation, directly using its score function to simulate generative SDEs often leads to suboptimal performance, especially for class-conditional generation. Assume c is the class label, is a conditional scoring network. The pioneer work introduced the classifier-free guidance (CFG), which transforms the original probability q t (x t |c) Replace with new probability q t (x t ) represents the unconditional distribution, which can be obtained by inputting an empty label φ as q t (x t |φ) to achieve this. In this way, the new score function becomes:

[0039]

[0040] This guidance strategy has become the default for diffusion models, such as the stable diffusion series. However, the CFG strategy has its limitations. First, CFG requires both conditional and unconditional score networks, which either requires two separate models or is difficult to train with a single model. Second, CFG is not suitable for tasks such as unconditional generation.

[0041] (b) Self-direction

[0042] Inspired by classifier-free guidance (CFG), self-guided SG uses the ratio of class conditional density to unconditional density as the guidance factor. The unconditional density is used as a “reference distribution” and has the following two key properties:

[0043] Density coverage: The unconditional distribution covers the density support area of ​​the category conditional distribution;

[0044] Probability promotion: Under this reference distribution, the probability of class conditional samples is greater than that of the base distribution, thus encouraging samples to explore areas of higher conditional probability.

[0045] Since the diffusion model is constructed through a series of step-by-step defined distributions, and the distribution at the next time step is usually smoother than the distribution at the current time step, the self-guidance strategy designed by the present invention can flexibly select a distribution with more noise as the basic distribution for building guidance.

[0046] The present invention allows δ(t) to be a positive but small perturbation function of the diffusion time t. The target distribution is p t (x t |c). Distribution p t+δ(t) (x t |c) is a suitable choice of reference distribution for two reasons. First, the distribution p t+δ(t) (x t |c) than p t (x t |c) smoother, flatter density. Secondly, we already have p t+δ(t) (x t |c). Therefore, the corresponding augmented distribution can be written as

[0047]

[0048]

[0049] Here, c represents the category label (e.g., text prompt), s(x t |c,t) represents the conditional score network. For a noise input x at time t t , the model predicts the conditional score network s(x t |c,t) and the conditional score network s(x) at time t+δ(t) t |c,t+δ(t)), and use the hyperparameter ω for linear combination to encourage samples to explore areas with higher conditional probability (p t (x t |c)) and reduce sampling to areas with low conditional probability (p t+δ(t) (x y |c)).

[0050] The guidance strategy designed by the present invention is named self-guidance because it only uses the trained model network itself, without the need for an additional unconditional score network in CFG or a bad version model in automatic guidance. Another advantage to be emphasized is that although the present embodiment uses class-conditional generation as a demonstration, the class label c is not required. Therefore, unlike CFG, self-guidance SG can also be used for unconditional generation.

[0051] Specifically, Figure 1 As shown, the image or video generation method of the present invention is further described in detail, comprising the following steps:

[0052] Step 1: Input preparation.

[0053] The input includes: the noise latent variable x at the current time step t t ; The conditional input c of the model (if it is a conditional generation model) or the default input for unconditional generation; The future offset step δ(t). These inputs provide the information needed by the model to predict the noise at the current and future time steps.

[0054] Step 2: Predict the conditional score network.

[0055] The model makes two predictions of the conditional score network based on the input content:

[0056] 1. Predict the conditional score network s(x t |c,t);

[0057] 2. Predict the conditional score network s(x) at the future time step t+δ(t) t |c,t+δ(t));

[0058] Combine the two conditional score networks according to the following linear combination rules to generate the final combined conditional score network:

[0059] s combined =(1+ω)*s(x t |c,t)-ω*s(x t |c,t+δ(t))

[0060] Step 3: Update latent variables.

[0061] s predicted using the combined conditional score network combined , for the noise latent variable x at the current time step t According to the denoising result, the latent variable x of the previous time step is calculated and updated. t-1 , the specific formula is:

[0062] x t-1 =f(x t ,scombined ,t)

[0063] Step 4: Combine multiple guidance methods.

[0064] The above three steps, namely the self-guidance (SG) strategy of the present invention, in order to further improve the quality of image or video generation, at each time step t, the model adjusts the noise in combination with the following guidance strategy:

[0065] 1. Classifier-free guidance (CFG): Combined with unconditional noise prediction, through the weighting factor ω CFG Control generation.

[0066] 2. Perturbation Attention Guidance (PAG): Introducing the perturbation attention mechanism to adjust the generated attention distribution.

[0067] 3. Self-Guidance (SG): Integrate the noise prediction information of the future time step t+δ(t) to guide the model to sample high-probability areas and reduce the possibility of generating low-probability samples.

[0068] By weighting factor ω CFG ,ω PAG and ω SG Comprehensively optimize noise prediction to improve generation consistency and quality.

[0069] Step 5: Output the results.

[0070] After denoising at all time steps, the final output is the denoised latent variable x0, which is the final generated result of the diffusion model (e.g., a high-quality image, such as Figure 2 as shown, or high-quality video, such as Figure 3 At the same time, the present invention also measures quantitative indicators on three stable diffusion models and Flux. The results show that our method effectively improves the performance during inference.

[0071] like Figure 2 As shown, a qualitative comparison between Flux.1 (baseline) and self-guided (SG) is shown. The red box in the figure indicates that Flux generation is not good, and the lower right corner shows the enlarged image. Compared with Flux, the self-guided SG strategy of the present invention significantly enhances the model's ability to generate fine-grained human hands, fixes generation errors, emphasizes themes in samples, and improves text-image consistency.

[0072] like Figure 3Figure 1 shows a qualitative comparison of CFG and CFG+SG in the text generation video model CogVideoX-5B. The red box in the figure indicates where the baseline (CFG) is poorly generated, and the lower left corner shows the enlarged image. Adding SG to CFG can effectively eliminate artifacts in video generation (broken arms and legs and misplaced limbs). SG also enhances the consistency between the prompt and the video (the number of dogs in the video).

[0073] As shown in Table 1 below, quantitative results of different stable diffusion models and Flux combined with various guidance methods are shown. FID, Clip score, and aesthetic score are evaluated on the MS-COCO 2017 validation subset. HPS score is evaluated on the human preference rating dataset. Bold values ​​highlight the best performance of each metric in each method.

[0074] Table 1

[0075]

[0076] It can be seen that the SG of the present invention can significantly improve the generation performance by itself. By combining SG, the present invention dropped 8 points in the FID indicator of SD 1.4 (50.7602 -42.7362). In addition, the method of the present invention also moderately improved the HPSv2.1 of SD 3.5 (22.74-24.24), the CLIP score of SD 3 (0.2762-0.2964), and the aesthetic score of Flux.1 (5.6224-5.6749). In addition, it is also shown that SG is compatible with CFG and PAG. By combining them all, the model not only significantly improves the details, fast alignment and error correction capabilities, but also reaches the new SoTA in all indicators (FID in SD1.4 reaches 22.1782, and HPS v2.1 in SD 3 reaches 30.63).

[0077] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating images or videos based on self-guidance of a diffusion model, characterized in that: The following steps are involved: (1) Prepare the input, which includes the noise latent variable x at the current time step t. t and the set future offset step size δ(t); (2) The diffusion model performs two score network predictions based on the input content to obtain the conditional score network of the current time step t and the conditional score network of the future offset time step t+δ(t); and linearly combines the two conditional score networks to generate the final combined conditional score network s combined ; (3) Using a combined conditional score network combined , for the noise latent variable x at the current time step t Perform denoising; based on the denoising result, calculate and update the latent variable x of the previous time step t-1 ; (4) At each time step, the self-guided SG strategy in steps (1) to (3) above is used to denoise the noise and optimize the noise prediction; (5) After denoising at all time steps, the denoised latent variable x0 is finally output as the final generation result of the diffusion model to generate an image or video.

2. The method for generating images or videos based on self-guidance of a diffusion model according to claim 1, characterized in that: In step (1), when the diffusion model is a conditional generation model, the input content also includes the category label c, otherwise, the category label c is not included.

3. The method for generating images or videos based on self-guidance of a diffusion model according to claim 1, characterized in that: In step (2), the two conditional score networks are linearly combined using the hyperparameter ω, as follows: s combined =(1+ω)*s t -ω*s t+δ(t) In the formula, s t represents the conditional score network at the current time step t, s t+δ(t) A conditional score network representing the future offset time step t+δ(t).

4. The method for generating images or videos based on self-guidance of a diffusion model according to claim 3, characterized in that: If the input includes a category label c, then s t It is expressed as s(x t |c,t),s t+δ(t) It is expressed as s(x t |c,t+δ(t)).

5. The method for generating images or videos based on self-guidance of a diffusion model according to claim 1, characterized in that: In step (3), the latent variable x in the previous time step is calculated and updated t-1 , the formula is: x t-1 =f(x t ,s combined ,t) Where f represents the diffusion model denoising function, which is used to transform the current latent variable x t Denoising to compute the latent variable at the previous time step.

6. The method for generating images or videos based on self-guidance of a diffusion model according to claim 1, characterized in that: In step (4), based on the self-guided SG strategy, one of the guidance strategies, namely, the classifier-free guided CFG and the perturbation attention guided PAG, is added, and the noise prediction is comprehensively optimized through the weighted factor.

7. The method for generating images or videos based on self-guidance of a diffusion model according to claim 6, characterized in that: In step (4), based on the self-guided SG strategy, two strategies are added: the classifier-free guided CFG and the perturbation attention guided PAG, and the weighted factor ω is used to SG ,ω CFG and ω PAG Comprehensive optimization of noise prediction.

8. A self-guided image or video generation system based on a diffusion model, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, the method for generating an image or video according to any one of claims 1 to 7 is implemented.

Images