Diffusion generation method of hierarchical condition control and dynamic jump sampling

Abstract

The invention belongs to the technical field of computer vision and model generation, and particularly relates to a diffusion generation method for hierarchical condition control and dynamic jump sampling, and the method is realized through a hierarchical control architecture: macroscopic time scheduling adopts a bidirectional LSTM dynamic combination de-noising step, and the equivalence is ensured through differential transition constraint; in microscopic space optimization, a key area is positioned through an uncertainty thermodynamic diagram, only local attention is executed on the key area, and an implicit network is used for background generation; according to the multi-modal fusion, dynamic routing gating is adopted to calculate the weight and adversarial training is combined, according to the diffusion generation method of hierarchical condition control and dynamic jump sampling, the sampling step number is compressed from 1000 + to 50-80, 95% of non-critical area calculation is reduced, the multi-modal guide quality loss is reduced by 37%, the peak video memory occupation is reduced by 60%, the efficiency is improved while the quality is guaranteed, and the method is suitable for large-scale popularization and application. And mobile terminal deployment is supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and generative model, and particularly relates to a diffusion generation method with hierarchical condition control and dynamic skip sampling. BACKGROUND

[0002] At present, the technologies related to the present application on the market mainly include diffusion model acceleration schemes (such as DDIM, knowledge distillation, and progressive distillation) and local enhancement methods (such as attention optimization in Latent Diffusion). Among them, the global step compression technology realizes acceleration by reducing the total sampling steps (such as reducing from 1000 steps to 100 steps), but sacrifices the generation quality; the model distillation technology trains a light model to imitate the behavior of a complete model, but it is difficult to maintain the details of complex scenes; and the fixed local enhancement technology manually presets key areas (such as faces), which cannot dynamically adapt to changes in content. These existing technologies have obvious defects. Step compression can cause details to be lost (such as blurred textures and text), because the global uniform step compression does not consider the information density difference of different time steps; static local optimization can ignore the dynamic changes of the content, causing background generation redundancy, because there is a lack of spatial adaptive mechanism and the uncertainty of key areas cannot be quantified; and the condition fusion is insufficient when guiding multiple modalities, because the weight of the multiple modalities is fixed and cannot be dynamically optimized to generate a path.

[0003] Therefore, in view of the above status, it is urgent to develop a diffusion generation method with hierarchical condition control and dynamic skip sampling to overcome the deficiencies in current practical applications. SUMMARY

[0004] The present application aims to provide a diffusion generation method with hierarchical condition control and dynamic skip sampling to solve the problems raised in the background.

[0005] To achieve the above object, the present application provides the following technical scheme:

[0006] A diffusion generation method with hierarchical condition control and dynamic skip sampling, comprising the following steps:

[0007] (1) Dynamic skip sampling: receiving the hidden state and multi-modal condition encoding of the previous moment through a long short-term memory network to predict the dynamic weight within the time window; based on the dynamic weight, merging the continuous denoising steps within the window to generate the hidden variable output of the equivalent time step, wherein the sum of the dynamic weights is 1;

[0008] (2) Spatial local refinement: calculating the pixel-level uncertainty heat map of the current hidden variable to determine the key area according to a preset threshold; performing a local attention enhancement operation in the key area, and directly generating in the non-key area through an implicit network;

[0009] (3) Multi-modal condition fusion: dynamically route and gate to calculate the fusion weight of multi-modal conditions, and dynamically allocate network resources combined with the current hidden state;

[0010] (4) Repeat steps (1)-(3) until the final output is generated.

[0011] As a further scheme of the present application: step (1) further comprises:

[0012] The consistency of the output of the merging step and the complete chain step is ensured by a differentiable transition constraint loss function, which calculates the mean square error between the output of the merging step and the output of the end point of the complete chain step.

[0013] As a further scheme of the present application: in step (2), the calculation of the uncertainty heat map comprises:

[0014] The gradient expectation of the KL divergence between the real reverse distribution and the model predicted reverse distribution under the current hidden variable is calculated with respect to the input, and is mapped to a heat map in the interval [0, 1] by a sigmoid function.

[0015] As a further scheme of the present application: in step (2), the local attention enhancement operation comprises:

[0016] The local attention matrix of the query, key, and value is calculated in the key region, and an edge enhancement mask is added to the attention score, which is generated by edge detection on the key region boundary.

[0017] As a further scheme of the present application: in step (3), the calculation of the fusion weight comprises:

[0018] The current hidden state and the condition encoding of each modality are spliced, projected to a common space by a learnable projection matrix, and then the inner product is calculated with a learnable direction vector to obtain the similarity, and finally normalized by a softmax function to obtain the fusion weight.

[0019] As a further scheme of the present application: further comprising: optimizing dynamic path selection by adversarial training, wherein the adversarial training uses a discriminator to distinguish between the static path of the real sample and the dynamic path of the generated sample.

[0020] As a further scheme of the present application: in step (1), the time window merging operation compresses the total sampling step from 1000 steps or more to 50-80 steps.

[0021] As a further scheme of the present application: in step (2), the spatial local refinement reduces the calculation amount of non-key regions by 95%.

[0022] As a further scheme of the present application: the method adopts a three-stage strategy in the training phase:

[0023] a pre-trained base diffusion model;

[0024] a fine-tuned dynamic jump and spatial refinement module;

[0025] strengthening the multimodal fusion path through adversarial training.

[0026] A diffusion generation system implementing the diffusion generation method of hierarchical conditional control and dynamic jump sampling described above, comprising:

[0027] a time window scheduler configured to perform dynamic jump sampling;

[0028] a spatial refinement module configured to generate an uncertainty heat map and perform local attention enhancement;

[0029] a multimodal routing module configured to calculate dynamic fusion weights;

[0030] a dual-flow processing architecture connecting the main generation network and the transition prediction network in parallel.

[0031] Compared with the prior art, the beneficial effects of the present application are:

[0032] The present application realizes multi-aspect technical improvement through hierarchical control architecture, bringing significant beneficial effects. Among them, the bidirectional dynamic time window scheduler can reduce redundant calculation and maintain generation quality, compressing the sampling step number from 1000 steps or more to 50-80 steps; thermodynamic uncertainty quantization can accurately locate the areas that need to be optimized, avoiding global calculation and reducing the calculation of non-critical areas by 95%; dynamic routing gating combined with adversarial training realizes adaptive fusion of multi-modal conditions, prevents generation path deviation, and reduces quality loss under multi-modal guidance by 37%; implicit neural background generation reduces GPU consumption, supports mobile deployment, and reduces peak GPU occupancy by 60%. These technical solutions cooperate with each other, effectively overcoming the defects in the prior art such as detail loss caused by step compression, static local optimization ignoring content dynamic change, and insufficient multi-modal condition fusion, while ensuring generation quality, significantly improving the efficiency and adaptability of diffusion generation. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 Flow chart of the diffusion generation method of hierarchical conditional control and dynamic jump sampling in the embodiments of the present application. DETAILED DESCRIPTION

[0034] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0035] The specific implementation of the present application will be described in detail below in combination with specific embodiments.

[0036] Please refer to Figure 1 The diffusion generation method provided by the embodiments of the present application solves the defects of the existing diffusion model acceleration scheme and local enhancement method, such as detail loss caused by step compression, static local optimization ignoring content dynamic change, and insufficient condition fusion in multi-modal guidance, and realizes the cooperative optimization of macro time scheduling, micro space optimization and multi-modal routing through a hierarchical control architecture, thereby significantly improving the diffusion generation efficiency while ensuring the generation quality. The specific content is as follows:

[0037] I. Overall technical architecture

[0038] The efficient diffusion generation system based on hierarchical condition control and dynamic jump sampling of the present application includes a preprocessing module, a dynamic path scheduler, a heat map generator, a multi-modal gateway, a diffusion generator, an output evaluation module and a gradient feedback module.

[0039] Among them:

[0040] The preprocessing module is used for standardizing the multi-modal input (such as text, image, audio) and extracting basic features; the dynamic path scheduler integrates a bidirectional LSTM network and a differentiable transition decision unit, and is responsible for dynamic step merging in the time dimension; the heat map generator realizes the spatial positioning of key areas by quantifying regional uncertainty; the multi-modal gateway includes a dynamic routing gate, which is used for dynamically allocating network resources of different modalities; the diffusion generator completes the final generation based on the merged time step features, the locally refined spatial features and the fused multi-modal conditions; the output evaluation module evaluates the quality of the generated results, and the feedback signal is transmitted to the dynamic path scheduler and the multi-modal gateway through the gradient feedback module, to realize path optimization.

[0041] II. Specific implementation of key technical modules

[0042] (1) Bidirectional dynamic time window scheduling module

[0043] This module dynamically merges consecutive denoising steps to reduce the amount of calculation while ensuring the chain consistency of the generation quality, and specifically includes a dynamic step merging mechanism and a differentiable transition constraint.

[0044] 1. Dynamic step merging mechanism

[0045] Let the sequence of time steps of the standard diffusion process be {t1,...,tN}, the bi-directional LSTM network receives the hidden state h k―1 and the pre-processed conditional encoding c cond of the previous time window, predicts the current time window W K = [t a ,t b ] that needs to be merged, and generates the weights w i for each time step in the window.

[0046] The hidden variable update formula of the equivalent time step t~k after merging is:

[0047]

[0048] where the weight w i satisfies ∑w i = 1, which is generated by the parameters Θ LSTM of the LSTM network, the previous hidden state h k―1 and the conditional encoding c cond to control the contribution proportion of each original time step t i (i∈[a,b]) to the merging result x t～k .

[0049] For example, when t i = 15 corresponds to w i = 0.8, it means that the feature of the 15th time step contributes 80% to the equivalent step feature after merging.

[0050] 2. Differentiable transition constraint

[0051] To ensure the consistency of the output of the merged equivalent time step and the original chain step, define the jump decision loss function:

[0052]

[0053] where fθ(x,t) is the denoising function of the diffusion model, the parameters are θ, and the input is the current state x and the time step t; tb: the end time step of the time window, for example, the tb of the window [10,20] is 20; Ext: the expectation of the data distribution xt, which ensures that the constraint is established on all possible data states.

[0054] Through the constraint of this loss function, it can be ensured that the dynamic merging step will not introduce additional errors, and the stability of the generated path is ensured.

[0055] (II) Spatial local refinement module

[0056] The module quantifies the region uncertainty, only finely processes high-priority regions, and reduces redundant calculations of background regions, including a heat map positioning function and a local attention enhancement.

[0057] 1. The heat map positioning function;

[0058] Based on pixel-level uncertainty quantification, a heat map identifying key regions is generated, with the formula being:

[0059]

[0060] wherein σ is a sigmoid function that maps the calculation result to the interval [0, 1], so that M(x t )∈[0,1] H×W (H and W are the height and width of the feature map, respectively); M(xt): heat map matrix, identifying pixel positions that need to be finely processed (the closer the value is to 1, the more critical it is); Gradient calculation for input xt reflects the sensitivity of KL divergence to input changes; q(xt―1∣xt): true reverse distribution defined by the diffusion process; pθ(xt―1∣xt): reverse distribution predicted by the model;

[0061] The closer the value of the pixel in the heat map is to 1, the higher the uncertainty of the region, and the finer processing is needed.

[0062] 2. Local attention enhancement;

[0063] For the region R in the heat map that satisfies M(x t )>τ(τ is a preset threshold), an improved local attention mechanism is implemented:

[0064]

[0065] wherein Q, K, and V are global query, key, and value matrices; QR, KR, and VR are local attention matrices limited within the key region R (for example, only the top 10% of high-heat pixels are processed); d is the dimension of the attention head (for example, 64 or 128), used to scale the dot product to prevent numerical overflow; λ is an edge enhancement coefficient that controls the weight of edge features (for example, λ = 0.5 represents moderate enhancement); EdgeMask(R) is a mask matrix generated by applying edge detection (such as the Sobel operator) to the region R, used to enhance the clarity of the region boundaries.

[0066] For non-critical regions (i.e., background regions with M(x t ) p ≤τ), direct implicit network generation is adopted without additional attention calculation, which can reduce the calculation amount of 95% of the background regions.

[0067] (III) a multi-modal condition fusion module;

[0068] The module realizes adaptive fusion of multi-modal conditions by dynamically allocating weights of different modalities, including dynamic routing gating and adversarial path optimization.

[0069] 1. Dynamic routing gating;

[0070] For multi-modal conditions c={c text ,c image ,c audio}(respectively, text, image and audio modalities of encoding), the fusion weights of each modality are calculated:

[0071]

[0072] Wherein, gm: the fusion weight of the mth modality (for example, the text modality weight gtext=0.7 indicates that the text condition is dominant); Wm: a learnable matrix, which projects the condition encoding cm of the mth modality and the hidden state ht into a common space; ht: the model hidden state at the current time (for example, the output vector of LSTM); cm: the condition encoding of the mth modality (such as text encoding ctext, image encoding cimage); v: a learnable direction vector used to calculate the modality correlation score; <·,·>: vector inner product operation, calculate similarity.

[0073] 2. Adversarial path optimization;

[0074] The discriminator D is introduced to constrain the selection of the dynamic path, to ensure the consistency of the generated results with the real data, and the loss function is:

[0075]

[0076] Wherein, D: discriminator network, input is data x and path mode identifier xreal: real data sample (such as real image); xgen: generated data sample (such as image generated by the model); Pstatic: static path identifier (fixed as 0); Pdynamic: dynamic path identifier (fixed as 1);

[0077] Through adversarial training, the samples generated by the dynamic path cannot be distinguished by the discriminator, ensuring the accuracy of the generated path under multi-modal guidance.

[0078] In summary, the hierarchical condition control and dynamic skip sampling diffusion generation method of the present application, each technical scheme and the corresponding technical effect and quantitative index are shown in the following table:

[0079]

[0080] It should be noted that in the present application, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, and the person skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments that can be understood by the person skilled in the art.

Claims

1. A diffusion generation method based on hierarchical conditional control and dynamic skip sampling, characterized in that, Includes the following steps: (1) Dynamic jump sampling: The hidden state and multimodal conditional coding of the previous moment are received through the long short-term memory network to predict the dynamic weights within the time window; based on the continuous denoising steps within the dynamic weight merging window, the hidden variable output of the equivalent time step is generated, wherein the sum of the dynamic weights is 1. (2) Spatial local refinement: Calculate the pixel-level uncertainty heatmap of the current latent variable, and determine the key region according to the preset threshold; perform local attention enhancement operation in the key region, and generate non-key regions directly through the implicit network; (3) Multimodal condition fusion: The fusion weight of multimodal conditions is calculated by dynamic routing gating, and network resources are dynamically allocated in combination with the current hidden state; (4) Repeat steps (1)-(3) until the final output is generated.

2. The diffusion generation method for hierarchical condition control and dynamic skip sampling according to claim 1, characterized in that, Step (1) also includes: The consistency of the outputs of the merging step and the complete chain step is ensured by a differentiable transition constraint loss function. The loss function calculates the mean square error between the output of the merging step and the final output of the complete chain step.

3. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, In step (2), the calculation of the uncertainty heatmap includes: Calculate the expected gradient of the KL divergence between the true inverse distribution and the model-predicted inverse distribution under the current latent variables with respect to the input, and map it to a heatmap in the [0,1] interval using the sigmoid function.

4. The diffusion generation method for hierarchical condition control and dynamic skip sampling according to claim 1, characterized in that, In step (2), the local attention enhancement operation includes: Calculate the local attention matrix of query, key, and value within the key region, and add an edge enhancement mask to the attention score. The edge enhancement mask is generated by edge detection of the boundary of the key region.

5. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, In step (3), the calculation of the fusion weights includes: The current hidden state is concatenated with the conditional codes of each modality, projected onto the common space through the learnable projection matrix, and then the inner product with the learnable direction vector is calculated to obtain the similarity. Finally, the similarity is normalized to the fusion weights by the softmax function.

6. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, Also includes: The dynamic path selection is optimized through adversarial training, which uses a discriminator to distinguish between the static path of real samples and the dynamic path of generated samples.

7. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, In step (1), the time window merging operation reduces the total number of sampling steps from over 1000 steps to 50-80 steps.

8. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, In step (2), the spatial local refinement reduces the computational load of non-critical regions by 95%.

9. The diffusion generation method for hierarchical conditional control and dynamic skip sampling according to claim 1, characterized in that, The method employs a three-stage strategy during the training phase: Pre-trained base diffusion model; Fine-tune the dynamic jump and spatial refinement modules; The multimodal fusion path is enhanced through adversarial training.

10. A diffusion generation system implementing the diffusion generation method of hierarchical condition control and dynamic skip sampling as described in any one of claims 1-9, characterized in that, include: Time window scheduler: Configured to perform dynamic skip sampling; Spatial Refinement Module: Configured to generate uncertainty heatmaps and perform local attention enhancement; Multimodal routing module: configured to calculate dynamic fusion weights; Dual-stream processing architecture: The main generator network and the transition prediction network are connected in parallel.

Images