Multi-scale diffusion optimization method for dynamic content awareness

Abstract

The invention relates to the technical field of image generation, in particular to a multi-scale diffusion optimization method for dynamic content awareness, which comprises the following steps of: (1) generating a semantic heat map of an input image through a pre-training segmentation network, and identifying semantic complexity of different regions; (2) dynamically adjusting the noise attenuation rate of each spatial position in the diffusion process based on the semantic heat map and the local pixel variance; (3) constructing a fractal attention gating network, and dynamically skipping calculation of a redundant region according to an entropy value of the feature map; and (4) forward features are multiplexed through a reversible residual diffusion chain, the back propagation calculation amount is reduced, and according to the multi-scale diffusion optimization method for dynamic content perception, the high-frequency detail generation quality is remarkably improved, calculation resource consumption is greatly reduced, the image generation speed is increased, and the generation effect stability is maintained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, and particularly relates to a dynamic content-aware multi-scale diffusion optimization method. BACKGROUND

[0002] The prior art adopts diffusion models (such as DDPM, NCSN++) and optimization schemes (such as DDIM, DPM-Solver), which rely on fixed noise scheduling strategies and static computing architectures: noise scheduling uses a globally fixed decay strategy (such as linear / cosine), ignoring the semantic differences (such as high-frequency details and low-frequency backgrounds) of different regions of the image; the computing architecture performs the same intensity of calculation on all regions through a U-Net network, resulting in redundant calculation of smooth regions (such as the sky); meanwhile, the irreversible forward process makes it impossible to reuse intermediate features, and the backward process needs to be calculated repeatedly, causing resource waste.

[0003] The above-mentioned technology has three defects of detail loss, calculation redundancy and low efficiency: the fixed noise scheduling strategy (βt=LinearSchedule(t)) leads to insufficient denoising of high-frequency regions (such as facial features), resulting in a low PSNR index; the static computing architecture lacks dynamic gating in the standard attention mechanism (Attention(Q, K, V)), causing invalid calculation of smooth regions and increasing memory usage and calculation time; the irreversible design of the traditional forward process (q(xt∣xt-1)) forces the backward propagation to be calculated completely, significantly prolonging the generation time. Therefore, in view of the above status, it is urgent to develop a dynamic content-aware multi-scale diffusion optimization method to overcome the deficiencies in current practical applications. SUMMARY

[0004] The present application aims to provide a dynamic content-aware multi-scale diffusion optimization method to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] A dynamic content-aware multi-scale diffusion optimization method, comprising the following steps:

[0007] (1) generating a semantic heat map of the input image through a pre-trained segmentation network to identify the semantic complexity of different regions;

[0008] (2) dynamically adjusting the noise decay rate of each spatial position in the diffusion process based on the semantic heat map and the local pixel variance;

[0009] (3) constructing a fractal attention gating network to dynamically skip the calculation of redundant regions according to the entropy value of the feature map;

[0010] (4) By reusing forward features through reversible residual diffusion chains, the computational load of backpropagation is reduced.

[0011] As a further aspect of the present invention: in step (2), the specific implementation of dynamically adjusting the noise attenuation rate is as follows:

[0012] Using formula

[0013] in, β is the dynamic noise coefficient of pixel position (i,j) at time step t; t The global noise figure of a traditional DDPM; Semantic heatmaps generated for a pre-trained lightweight segmentation network; For x t The variance of the 3×3 neighborhood pixels is calculated at position (i,j); γ is a learnable temperature parameter.

[0014] As a further aspect of the present invention: in step (3), the construction of the fractal attention gating network includes:

[0015] The feature map is decomposed into blocks of a quadtree structure;

[0016] Calculate the normalized information entropy of the feature map of each block;

[0017] Through gate function Generate activation coefficients;

[0018] in, is the i-th channel in the k-th quadtree block of the l-th layer; Entropy(·) is used to calculate the normalized information entropy of the feature map; τ is the trainable threshold; σ is the Sigmoid activation function; N is the number of feature channels.

[0019] As a further aspect of the present invention: step (3) further includes:

[0020] when Attention calculation is performed when the condition is met; otherwise, the calculation is skipped and the input features are preserved, where θ is the gating activation threshold.

[0021] As a further aspect of the present invention: in step (4), the implementation of the reversible residual diffusion chain includes:

[0022] Forward process:

[0023] Reverse process direct reuse:

[0024] Where fθ is the reversible coupling layer; and This is a block representation of state variables.

[0025] As a further scheme of the present application: the reversible coupling layer f θ satisfies the Lipschitz constraint condition ||f θ (x)-f θ (y)||≤L||x-y||, wherein L is a Lipschitz constant, L=1.2.

[0026] As a further scheme of the present application: in step (1), the pre-trained segmentation network is a lightweight convolutional neural network.

[0027] A dynamic content-aware multi-scale diffusion optimization system for implementing the dynamic content-aware multi-scale diffusion optimization method described above, comprising:

[0028] A semantic heat map generation module configured to generate a semantic heat map of an input image through a pre-trained segmentation network;

[0029] A dynamic noise scheduling module configured to dynamically adjust the noise decay rate of each spatial position in the diffusion process based on the semantic heat map and the local pixel variance;

[0030] A fractal attention gating module configured to construct a fractal structure and dynamically skip the calculation of redundant areas according to the feature map entropy value;

[0031] A reversible residual diffusion module configured to reuse the forward features through a reversible residual diffusion chain.

[0032] As a further scheme of the present application: the fractal attention gating module comprises:

[0033] A quadtree blocking unit configured to decompose the feature map into a multi-scale blocking structure;

[0034] An entropy value calculation unit configured to calculate the normalized information entropy of each block;

[0035] A gating decision unit configured to activate attention calculation when the entropy value weighted sum is greater than a threshold θ.

[0036] As a further scheme of the present application: the reversible residual diffusion module contains a reversible coupling layer, which satisfies the constraint condition of Lipschitz constant L=1.2.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] 1. Significantly improve the quality of high-frequency detail generation: in the high-frequency area of facial features, the PSNR index is improved by 2.1dB (verified by CelebA-HQ experiment);

[0039] 2. Dramatically reduce the consumption of computing resources: skip redundant calculations for smooth areas (such as the sky), GPU memory occupancy is reduced by 37%;

[0040] 3. Accelerate the speed of image generation: through the feature reuse mechanism, the back propagation calculation amount is reduced by 47% (LSUN-Bedroom task);

[0041] 4. Maintain the stability of the generation effect: while improving efficiency, the FID index fluctuation is not more than ±0.3. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flow chart of the dynamic content-aware multi-scale diffusion optimization method in the embodiment of the application.

[0043] Figure 2 The spatial resolution and semantic complexity double-dimensional dynamic optimization framework diagram in the embodiment of the application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0045] The specific implementation of the application will be described in detail below in combination with specific embodiments.

[0046] Please refer to Figure 1 and Figure 2 The dynamic content-aware multi-scale diffusion optimization method provided by the embodiment of the application aims to solve the problems of detail loss, calculation redundancy and low efficiency existing in the existing diffusion model. Through three core technical modules of semantic-guided noise adaptive scheduling, fractal attention gating and reversible residual diffusion chain, dynamic optimization of the diffusion process is realized, and the calculation cost is reduced while the generation quality is improved. The specific content is as follows:

[0047] I. Overall architecture

[0048] The dynamic content-aware multi-scale diffusion optimization method of the application comprises a semantic heat map generation module, a dynamic noise scheduling module, a fractal attention gating module, a reversible residual diffusion chain module connected in sequence, and a calculation efficiency evaluation module for performance monitoring.

[0049] The semantic heatmap generation module extracts semantic features from the input image and generates a semantic heatmap; the dynamic noise scheduling module dynamically adjusts the noise attenuation strategy based on the semantic heatmap and local pixel variance; the fractal attention gating module dynamically determines the attention computation intensity of each region through quadtree block division and entropy calculation; the reversible residual diffusion chain module constructs a reversible forward process to realize the reuse of intermediate features during reverse denoising; and the computational efficiency evaluation module monitors memory usage, computation time, and generated image quality indicators (such as PSNR and FID).

[0050] II. Specific implementation of each module;

[0051] 2.1 Semantic-Aware Noise Curriculum;

[0052] Traditional solutions use fixed linear / cosine noise attenuation and ignore differences in image regions. This invention innovatively designs a semantically guided noise adaptive scheduler, specifically implemented as follows:

[0053] 2.1.1 Semantic Heatmap Generation: At each diffusion step, a semantic heatmap is generated in real time using a lightweight segmentation network (MobileNetV3+DeepLabv3Lite). in This is used to identify the semantic importance of each pixel position (i,j) in the image, such as 0.9 for the eye region and 0.2 for the background region, providing spatial semantic weights for subsequent noise scheduling.

[0054] 2.1.2 Calculation of Dynamic Noise Figure: Dynamic noise figure of pixel position (i,j) at time step t Calculated using the following formula:

[0055]

[0056] Among them: Among them, β is the dynamic noise coefficient of pixel position (i,j) at time step t; t The global noise figure of a traditional DDPM; Semantic heatmaps generated for a pre-trained lightweight segmentation network; For x t The variance of the 3×3 neighborhood pixels is calculated at position (i,j); γ is a learnable temperature parameter (initial value 0.7).

[0057] Through the dynamic noise scheduling, the noise attenuation speed of high-frequency detail areas (such as facial features) is automatically reduced, and the de-noising speed of low-frequency areas (such as solid color backgrounds) is accelerated, and on the CelebA-HQ dataset, compared with DDPM, the eye area PSNR can be increased by 2.1dB, while reducing the total calculation amount by 15%.

[0058] 2.2, Fractal Attention Gating (Fractal Attention Gating);

[0059] The traditional U-Net uses the same calculation intensity at all resolution layers, and there is calculation redundancy, and the fractal attention gating mechanism is innovatively designed, and the specific implementation is as follows:

[0060] 2.2.1 Quadtree partitioning and entropy calculation: a quadtree fractal structure is constructed, the feature map Fl (the lth layer feature) is quadtree partitioned to obtain a plurality of subblocks F l (k,i) (the feature map of the i-th channel in the k-th quadtree block of the l-th layer), and the normalized information entropy Entropy(F l (k,i) ) of each subblock is calculated, which quantifies the sparsity of feature activation, and high entropy corresponds to complex regions and low entropy corresponds to smooth regions. The calculation formula of the normalized information entropy Entropy(F l (k,i) ) is as follows:

[0061]

[0062] Wherein, F l (k,i) : a two-dimensional feature map with a size of HxW; P h,w : the normalized probability value of the feature map at position (h, w); ∈: a minimum value (such as 10 -8 ), to avoid log(0) calculation error. Wherein:

[0063]

[0064] T: temperature coefficient (default T = 1.0), adjust the sharpness of the probability distribution; T→0: the distribution focuses on the maximum value→the entropy value decreases; T→∞: the distribution tends to be uniform→the entropy value increases.

[0065] 2.2.2 Dynamic gating decision: gating coefficient is calculated by the following formula:

[0066]

[0067] Wherein, is the i-th channel in the l-th layer k-th quad-tree block; Entropy(·) is the normalized information entropy calculation (value range [0, 1]) of the feature map; τ is a trainable threshold (initial value 0.4); σ is a Sigmoid activation function; and N is the number of feature channels.

[0068] When (θ is a fixed gating activation threshold of 0.3), the complete attention calculation is performed:

[0069]

[0070] wherein Q: the query representation matrix of the target sequence, encodes the information features that need to be retrieved; K: the key representation matrix of the source sequence, encodes the content index features; V: the value representation matrix of the source sequence, contains the actual content features; QK T : the un-scaled attention score of the query-key pair is calculated; The score scale is controlled to stabilize the gradient flow and prevent the softmax from entering the saturation zone.

[0071] Otherwise, the Identity mapping is performed, and the attention calculation is skipped.

[0072] The mechanism makes more than 80% of the smooth sky area automatically skip the high-level attention calculation in 512x512 image generation, and reduces the GPU memory occupancy by 37%.

[0073] 2.3, Invertible Residual Diffusion Chain;

[0074] The existing diffusion model loses irreversible information in the forward process, resulting in low efficiency. The present application innovatively designs an invertible residual diffusion chain, which is specifically implemented as follows:

[0075] 2.3.1 Invertible forward process: the hidden state x t of the diffusion process is decomposed into two blocks and (i.e. The forward process calculation formula is:

[0076]

[0077] wherein fθ is an invertible coupled layer, satisfying the Lipschitz constant constraint ||fθ(x)-fθ(y)||≤L||x-y|| (in the present scheme, L=1.2); Main feature channel: carrying the main information of the image (such as outline, texture), injecting conditioned noise in the diffusion process; Residual channel: preserving low-dimensional features (such as color distribution, low-frequency signals), used to drive coupled layer calculation and not directly added with noise; Residual channel features; t: time step embedding vector.

[0078] 2.3.2 Reverse denoising process multiplexing intermediate features: In the reverse process, directly multiplex the intermediate features calculated in the forward process, and the calculation formula is:

[0079]

[0080] Through the reversible design, in the LSUN-Bedroom generation task, the reverse propagation calculation amount is reduced by 47% compared with NCSN++, and the FID index remains the same (±0.3).

[0081] III. Overall flow;

[0082] The overall flow of the dynamic content-aware multi-scale diffusion optimization method of the application is as follows:

[0083] Input the original image x0, generate a semantic heat map through a semantic segmentation network

[0084] Diffusion forward process: The dynamic noise scheduler calculates the semantic heat map and local variance Reversible residual chain forward calculation And store the features in the feature multiplexing cache area;

[0085] Fractal attention gate performs quadtree partitioning and entropy calculation on the feature map, and decides whether to perform complete Attention or Identity mapping according to the gating decision (G> θ = 0.3?);

[0086] Reverse denoising process: reversible residual chain reverse calculation Combined with multi-scale gradient fusion, after T-step optimization, x is obtained T ;

[0087] Performance evaluation is performed, and the calculation efficiency and memory occupation are monitored, and finally the generated image x0 is output, which meets the performance indicators such as eye SSIM≥0.92, FLOPs≤3.2Tera and VRAM≤8GB@512px.

[0088] It should be noted that in the present application, it should be understood that although the present application is described in the form of embodiments, each embodiment does not contain only one independent technical solution, and the description manner of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be properly combined to form other embodiments that those skilled in the art can understand.

Claims

1. A dynamic content-aware multi-scale diffusion optimization method, characterized in that, The method comprises the following steps: (1) generating a semantic heat map of an input image by a pre-trained segmentation network to identify the semantic complexity of different regions; (2) dynamically adjusting the noise attenuation rate of each spatial position in the diffusion process based on the semantic heat map and local pixel variance; (3) constructing a fractal attention gate network to dynamically skip the calculation of redundant regions according to the entropy value of the feature map; (4) multiplexing the forward feature by a reversible residual diffusion chain to reduce the calculation amount of back propagation.

2. The dynamic content-aware multi-scale diffusion optimization method of claim 1, wherein, In step (2), the specific implementation of dynamically adjusting the noise attenuation rate is as follows: Using the formula where, is the dynamic noise coefficient for pixel position (i, j) at time step t; β t is the global noise coefficient for the traditional DDPM; is the semantic heat map generated by the pre-trained lightweight segmentation network; is the semantic heat map generated by the pre-trained lightweight segmentation network; t is the 3x3 neighborhood pixel variance computed at the (i, j) position of x; γ is a learnable temperature parameter.

3. The dynamic content-aware multi-scale diffusion optimization method of claim 1, wherein, In step (3), the construction of the fractal attention gate network comprises: decomposing the feature map into a four-tuple tree structure block; calculating the normalized information entropy of each block feature map; By a gating function Generating activation coefficients; wherein, is the i-th channel in the k-th quad-tree block in the l-th layer; Entropy(·) is the normalized information entropy calculation of the feature map; τ is the trainable threshold; σ is the Sigmoid activation function; and N is the number of feature channels.

4. The dynamic content-aware multi-scale diffusion optimization method of claim 3, wherein, Step (3) further comprises: When attention computation is performed, otherwise skip the computation and keep the input features, where θ is a gating activation threshold.

5. The dynamic content-aware multi-scale diffusion optimization method of claim 1, wherein, In step (4), the implementation of the reversible residual diffusion chain comprises: Forward process: Reverse process direct reuse: where fQ is the reversible coupling layer; and is a block representation of the state variable.

6. The dynamic content-aware multi-scale diffusion optimization method of claim 4, wherein, The reversible coupling layer f θ satisfies the Lipschitz constraint ||f θ (x)-f θ (y)||≤L||x-y||, where L is a Lipschitz constant, L=1.

2.

7. The dynamic content-aware multi-scale diffusion optimization method of claim 1, wherein, In step (1), the pre-trained segmentation network is a lightweight convolutional neural network.

8. A dynamic content-aware multi-scale diffusion optimization system, characterized in that, A dynamic content-aware multi-scale diffusion optimization method for realizing any one of claims 1-7, comprising: a semantic heat map generation module configured to generate a semantic heat map of an input image by a pre-trained segmentation network; a dynamic noise scheduling module configured to dynamically adjust the noise attenuation rate of each spatial position in the diffusion process based on the semantic heat map and local pixel variance; a fractal attention gate module configured to construct a fractal structure and dynamically skip the calculation of redundant regions according to the entropy value of the feature map; a reversible residual diffusion module configured to multiplex the forward feature by a reversible residual diffusion chain.

9. The dynamic content-aware multi-scale diffusion optimization system of claim 8, wherein, The fractal attention gate module comprises: a four-tuple tree block unit configured to decompose the feature map into a multi-scale block structure; an entropy value calculation unit configured to calculate the normalized information entropy of each block; a gate decision unit configured to activate attention calculation when the entropy value weighted sum is greater than a threshold θ.

10. The dynamic content-aware multi-scale diffusion optimization system of claim 8, wherein, The reversible residual diffusion module includes a reversible coupling layer that satisfies the constraint condition of Lipschitz constant L = 1.2.

Images