Image generation method based on diffusion model

Abstract

The invention provides an image generation method based on a diffusion model, and relates to the technical field of image processing, and the method comprises the steps: constructing and initializing an initial diffusion model applied to the field of image generation, and carrying out the binarization processing of the initial diffusion model; replacing all binary structures of the diffusion model subjected to binarization processing with a time-step-friendly binary diffusion model structure to obtain a binary diffusion model; training the binary diffusion model by utilizing spatial block distillation to obtain a trained binary diffusion model; and generating an image task by using the trained binary diffusion model. According to the method, the collapse problem during image generation of a traditional binary diffusion model is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image generation method based on a diffusion model. Background Art

[0002] Diffusion models (DMs) have attracted great attention and applications in the field of image generation due to their high-quality and diverse generation capabilities. Diffusion models can generate images from random noise with up to 1000 denoising steps. Although some accelerated sampling methods can effectively reduce the number of steps required for the generation task, the expensive floating-point calculations at each time step still limit the widespread application of such image generation models in resource-constrained scenarios. Therefore, compression of diffusion models becomes a key step for their wider application to reduce storage and computational costs while maintaining the accuracy of image generation.

[0003] In the field of image processing, quantization is considered to be a very effective model compression technology. By quantizing weights and / or activations to low-bit integers or binarization, compact storage and efficient computing in inference can be achieved, and efficient and lightweight image generation can be achieved. Binarization is considered to be the form that can achieve the highest acceleration effect. Some existing works have attempted to quantize the diffusion model in the field of image generation to 1 bit, but their exploration mainly focuses on weights and faces significant collapse when fully binarized. Binarizing the diffusion model while ensuring accuracy is expected to significantly improve the efficiency of image generation and achieve efficient and lightweight image generation.

[0004] Diffusion models (DMs) have demonstrated excellent generative capabilities in the field of image generation. However, their large-scale model architecture and the high computational cost required for multi-step reasoning limit their practical applications. To address this issue, many methods have been proposed to accelerate the reasoning process, including sampling acceleration methods and distillation methods that do not require retraining. A recent method, called DeepCache, avoids a lot of redundant computation by caching high-dimensional features and is compatible with typical sampling acceleration methods. Methods based on sampling acceleration cannot overcome the memory bottleneck and efficiency limitations in single-step reasoning, and the models still cannot be deployed in scenarios with limited storage resources, resulting in the application scope of models in the field of image generation being limited.

[0005] Quantization is a widely validated compression technique that achieves compression and acceleration by compressing weights and activations from the conventional 32 bits to 1-8 bits. Therefore, quantization has also been studied for diffusion models. Among them, binarization, as the most extreme form of quantization, usually represents weights and activations as ±1, allowing the model to achieve maximum compression and acceleration. In the field of computer vision, binarization work mainly focuses on discriminative models, such as CNN or ViTs. Due to the unique difficulty of image generation models, most post-training quantization (PTQ) methods can only quantize models to 4 bits or higher, while more precise quantization-aware training (QAT) methods encounter performance bottlenecks below 3 bits. Binarization has been less studied on generative models. Although ResNet VAE and Flow++ have achieved full binarization of VAE, their generation performance is not comparable to current state-of-the-art models. BinaryLatent Diffusion binarizes the latent space of LDM, but fails to improve the model's space occupancy or inference efficiency. The latest work BinaryDM quantizes DMs to close to W1A4, but does not solve the activation quantization problem. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides an image generation method based on a diffusion model, which introduces a variational autoencoder and a channel attention mechanism in a conditional generative adversarial network to improve the quality of the generated signal, solve the unique requirements of the diffusion model for flexible activation features during image generation, generate a model structure with limited information, and the spatial locality of the image generation task, and overcome the collapse challenge of image generation caused by the complete binarization of the diffusion model.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: an image generation method based on a diffusion model, comprising the following steps:

[0008] S1. Construct and initialize an initial diffusion model applied to the field of image generation, and perform binarization on the initial diffusion model;

[0009] S2, replacing all binary structures of the binary diffusion model with a time-step-friendly binary diffusion model structure to obtain a binary diffusion model;

[0010] S3, training the binary diffusion model using spatial block distillation to obtain a trained binary diffusion model;

[0011] S4. Generate image tasks using the trained binary diffusion model.

[0012] The beneficial effects of the present invention are as follows: From a temporal perspective, the present invention observes that the activation features of the diffusion model are highly correlated with the time step. The present invention introduces a "time-step-friendly binary structure" (TBS), matches the dynamic activation range of the diffusion model through a learnable activation binary quantizer, designs feature connections across time steps, and utilizes feature similarities between adjacent time steps to enhance the representation ability of the binary model. From a spatial perspective, the present invention notes the spatial locality in the image generation task and the convolution-based U-Net structure used by the diffusion model, proposes "spatial block distillation" (SPD), and introduces a full-precision diffusion model θ fp As supervision, by imitating the self-attention of the blocks, we focus on the local features of the generated image and better guide the optimization update direction of the binary diffusion model during image generation training.

[0013] Furthermore, the loss function of the initial diffusion model in step S1 is expressed as follows:

[0014]

[0015] Among them, L DM represents the loss function of the initial diffusion model, represents the mathematical expectation, ε t represents the noise at the tth time step, ε θ represents the predicted noise restored by the trained binary diffusion model, x t represents the original input image, t represents the time step, T represents the total time step, q(x 0 ) represents the sample space of the input image, ε t represents noise, and N(0,I) represents the distribution space of Gaussian noise.

[0016] The beneficial effect of the above further solution is that, through the diffusion principle of image generation, the initial diffusion model can be trained on the image and the image generation task can be achieved.

[0017] Furthermore, the binarization process in step S1 is as follows:

[0018] The weights of the initial diffusion model are binarized using the following formula:

[0019]

[0020] Among them, w bi represents a binary weight, σ represents a floating-point scalar, w represents a full-precision weight, and sign() represents a sign function taking ±1;

[0021] The input image is quantized using the following formula:

[0022]

[0023] Among them, a bi represents the image output by the binary diffusion model, and a represents the image output by the full-precision diffusion model.

[0024] The beneficial effect of the above further solution is that binarization can enable the diffusion model to perform image generation tasks lightly and efficiently, expand the scope of application, reduce the cost of use and improve the efficiency of image generation.

[0025] Furthermore, step S2 includes the following steps:

[0026] S201. Based on the binary diffusion model, a learnable tiny convolution scaling factor is applied to the two-dimensional information structure of all image inputs, wherein the gradient of the tiny convolution is calculated as follows:

[0027]

[0028] Where L represents the loss function of the initial diffusion model, k represents a tiny convolution, and Represent convolution operations with and without multiplication, respectively. I represents image input, W represents full-precision weight, A represents image output, and sign() represents a sign function taking ±1.

[0029] S202: Based on the processing result of step S201, the similarity of the image outputs between adjacent time steps is analyzed by applying information enhancement connection across time steps to obtain a binary diffusion model.

[0030] The beneficial effect of the above further scheme is that the learnable tiny convolution scale factor and the information enhancement connection across time steps can enhance the image generation accuracy of the binary diffusion model and obtain images with better visual effects.

[0031] Furthermore, the expression of the process of information enhancement connection across time steps is as follows:

[0032]

[0033] Among them, Concat() represents the image cutting and splicing operation. represents the image output of the mth downsampling block at the t-1th time step, represents a learnable scaling factor, represents the image output of the m+1th upsampling block in the t-1th time step, represents the image output of the m+1th downsampling block at the tth time step.

[0034] The beneficial effect of the above further scheme is that low-dimensional and high-dimensional information can be fused through information connection, and representations of different dimensions can be comprehensively utilized to generate images.

[0035] Furthermore, step S3 includes the following steps:

[0036] S301, extracting the intermediate features of the image generated by the full-precision diffusion model and the binary diffusion model, and performing block processing;

[0037] S302, based on the block processing result, calculate the self-attention for each small block;

[0038] S303, regularize the calculated self-attention, calculate and accumulate the loss of the corresponding position of each small block;

[0039] S304. According to the loss at the corresponding position of each small block, the total training loss function of the binary diffusion model is obtained, and the training of the binary diffusion model is completed.

[0040] The beneficial effect of the above further scheme is that the spatial block distillation can better guide the optimization update direction of the binary diffusion model during image generation training, so that the final binary diffusion model has better image generation quality.

[0041] Furthermore, the expression of the total training loss function of the binary diffusion model is as follows:

[0042]

[0043] Where L represents the total training loss function of the binary diffusion model, L DM represents the loss function of the initial diffusion model, d represents the number of blocks in the upsampling or downsampling process, λ represents the hyperparameter coefficient used to balance the loss term, represents the loss function of the binary diffusion model, m represents the total number of layers of the binary diffusion model, p represents the number of blocks, represents the self-attention of the image output of row i and column j of the full-precision diffusion model, represents the self-attention of the image output of the i-th row and j-th column of the binary diffusion model, represents the output block of the i-th row and j-th column image of the full-precision diffusion model, T represents transpose, represents the output block of the image in row i and column j of the binary diffusion model, represents the image output block of the i-th row and j-th column of the full-precision diffusion model, w represents the total width of the image output, h represents the total height of the image output, represents the output block of the image in row i and column j of the binary diffusion model, |||| 2 represents the L2 function.

[0044] The beneficial effect of the above further scheme is that the self-attention-based scheme can focus on the local texture information of the image in each block, so that the binary diffusion model performing image generation has a better optimization training direction and final generation effect.

[0045] Furthermore, the step S4 is specifically as follows:

[0046] Use the trained binary diffusion model to perform diffusion inference on the image noise;

[0047] The inference process is consistent with the inference process of the initial diffusion model, and the binary diffusion model structure is applied when performing image generation to output the generated image.

[0048] The beneficial effect of the above further solution is that it can simply and conveniently make the binary diffusion model run efficiently while keeping the reasoning process unchanged.

[0049] The beneficial effect of the above further solution is that it can simply and conveniently enable the binary diffusion model to efficiently generate images while keeping the inference process unchanged.

[0050] Furthermore, the expression of the trained binary diffusion model is as follows:

[0051]

[0052] Among them, p θ (x t-1 |x t ) represents the previous noise image inferred by the trained binary diffusion model when the next noise image is known, N represents the normal distribution, x t-1 represents the noise image at the t-1th time step, represents the image mean derived from the trained binary diffusion model, x t represents the noise image at the t-th time step, t represents the t-th time step, represents the cumulative statistical image standard deviation at the t-th time step, I represents the input image, It means that the binary diffusion model after training at the tth time step is composed of x t and the image mean derived from t, a t Auxiliary standard deviation and Both represent the cumulative auxiliary standard deviation ε θ represents the noise image derived from the trained binary diffusion model, β t represents the standard deviation of the image at the t-th time step.

[0053] The beneficial effect of the above further scheme is that by predicting the mean and variance, the binary diffusion model has the ability to generate images efficiently and stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention is a flow chart of the method.

[0055] Figure 2 Schematic diagram of activation features of the binary diffusion model.

[0056] Figure 3 Schematic diagram of the binary diffusion model structure TBS.

[0057] Figure 4 Visualization of the last output time step layer (TimeStepBlock) of the latent space diffusion model (LDM) on the bedroom dataset (LSUN-bedroom). DETAILED DESCRIPTION

[0058] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0059] Example

[0060] The present invention pushes the diffusion model applied in the field of image generation to the limit of compression and acceleration by fully binarizing weights and activations (step S1). The present invention includes two innovative technologies: from a temporal perspective, it is observed that the activation features of the diffusion model applied in the field of image generation are highly correlated with the time step. For this reason, the present invention introduces a "time-step-friendly binary structure" (TBS), matches the dynamic activation range of the diffusion model through a learnable activation binary quantizer (step S2), and designs feature connections across time steps, utilizing the feature similarity between adjacent time steps to enhance the representation ability of the binary model. From a spatial perspective, the spatial locality in the image generation task and the convolution-based U-Net structure used by the diffusion model are noticed. For this reason, the present invention proposes "spatial block distillation" (SPD), introduces a full-precision diffusion model as supervision, and focuses on local features through self-attention imitation of the blocks, so as to better guide the optimization direction of the binary diffusion model applied in the field of image generation. Figure 1 As shown, the present invention provides an image generation method based on a diffusion model, and its implementation method is as follows:

[0061] S1. Construct and initialize an initial diffusion model applied to the field of image generation, and perform binarization on the initial diffusion model;

[0062] In this embodiment, an initial diffusion model applied to the field of image generation is constructed and binarized.

[0063] In this embodiment, the initial diffusion model applied to the field of image generation is constructed as follows:

[0064] Given an image data distribution x 0 ~q(x 0 ), the forward process generates a random variable sequence x t ∈{x 1 ,...,x T}, the sequence is transferred through the kernel q(x t |x t-1 ) is generated, and its expression is:

[0065]

[0066] Among them, q(x 1 ,..,x T |x 0 ) represents the noise distribution after iterative noise addition to the original image, x 0 represents the original image, x T represents the final noisy image, t represents the time step, T represents the total time step, q(x t |x t-1 ) represents the image at time step t obtained by further adding noise to the noisy image at step t-1, x t-1 represents the noisy image at the t-1th time step, N represents the normal distribution, I represents the unit matrix, β t ∈(0,1) represents a noisy schedule. The Gaussian transition kernel allows marginalizing the joint distribution, so using a t :=1-β t and The present invention can be achieved by sampling a Gaussian vector and applying the transformation Get x t .

[0067] The reverse process aims to generate samples by removing noise, using a learnable transfer kernel p θ (x t-1 |x t ) approximates the unavailable conditional distribution q(x) t-1 |x t ), whose expression is:

[0068]

[0069] Mean and variance We can use the reparameterization technique to get:

[0070]

[0071] Among them, p θ (x t-1 |x t ) represents the previous noise image inferred by the trained binary diffusion model when the next noise image is known, N represents the normal distribution, x t-1 represents the noise image at the t-1th time step, represents the image mean derived from the trained binary diffusion model, x t represents the noise image at the t-th time step, t represents the t-th time step, represents the cumulative statistical image standard deviation at the t-th time step, I represents the input image, It means that the binary diffusion model after training at the tth time step is composed of x t and the image mean derived from t, a t Auxiliary standard deviation and Both represent the cumulative auxiliary standard deviation ε θ represents the noise image derived from the trained binary diffusion model, β t represents the standard deviation of the image at the t-th time step.

[0072] For the training of the initial diffusion model applied to the field of image generation, a simplified variant of the variational lower bound is usually used as the loss function to improve the sample quality, which is expressed as:

[0073]

[0074] Among them, L DM represents the loss function of the initial diffusion model, represents the mathematical expectation, ε t represents the noise at the tth time step, ε θ represents the predicted noise restored by the trained binary diffusion model, x t represents the original input image, t represents the time step, T represents the total time step, q(x 0 ) represents the sample space of the input image, ε t represents noise, and N(0,I) represents the distribution space of Gaussian noise.

[0075] In this embodiment, the full-precision diffusion model θ fpThe U-Net network is a network that can integrate low-level and high-dimensional features of images. It has become the mainstream backbone framework of diffusion models applied in the field of image processing. The input and output blocks of the U-Net network can be expressed as and Among them, the block corresponding to the smaller m is at a lower level, and the skip connection transfers the low-level information from D m (·) propagates to U m (·), therefore, U m The received input is represented as:

[0076] Concat(D m (·), U m+1 (·)) (6)

[0077] in, represents the mth downsampling block, represents the m+1th upsampling block, m represents the mth block, and D m (·) represents the image output of the mth downsampling block, U m (·) represents the image output of the mth upsampling block.

[0078] In this embodiment, the process of binarizing the diffusion model applied to the field of image generation is as follows:

[0079] Quantization compresses and accelerates the image generation model by discretizing the diffusion model weights and image inputs into low bit widths. In the baseline of the binarized diffusion model, the weight w is binarized to 1 bit:

[0080]

[0081] Among them, w bi represents a binary weight, σ represents a floating point scalar, w represents a weight, sign() represents a sign function of ±1, the sign function limits the weight w to +1 or -1, and sets 0 as the threshold, initialized to n represents the number of weights, and the image input is usually quantized through a simple BNN quantizer:

[0082]

[0083] Among them, a bi represents the image output by the binary diffusion model, and a represents the image output by the full-precision diffusion model.

[0084] When the diffusion model weights and image input are both quantized to 1 bit, the computation of the image generation model can be replaced by XNOR and bitcount operations, thus achieving significant compression and acceleration.

[0085] S2. Replace all binary structures of the binary diffusion model with a time-step-friendly binary diffusion model structure to obtain a binary diffusion model. The implementation method is as follows:

[0086] S201, based on the binary diffusion model, applying a learnable tiny convolution scaling factor to the two-dimensional information structure of all image inputs;

[0087] S202: Based on the processing result of step S201, the similarity of the image outputs between adjacent time steps is analyzed by applying information enhancement connection across time steps to obtain a binary diffusion model.

[0088] In this embodiment, based on the initial diffusion model applied to the field of image generation constructed in step S1, a learnable tiny convolution scale factor is applied to the two-dimensional information structure of the image input; based on the above process, information enhancement connections across time steps are applied to the similarity of image outputs between adjacent time steps.

[0089] In this example, before introducing the proposed method in detail, we first summarize the observations on the properties of the diffusion model applied to the field of image generation:

[0090] Observation 1: The activation range of generated images varies significantly in long-term time steps, but the activation features of generated images are similar in short-term adjacent time steps.

[0091] Previous studies, such as Temporal Dynamic Quantization (TDQ) and Diffusion Model Alignment Quantization (Q-DM), have shown that the image distribution of diffusion models applied in the field of image generation is highly dependent on the time step during denoising, which is manifested in the similarity between adjacent time steps and the large difference between distant time steps, such as Figure 2 (a). Therefore, applying a fixed scaling factor across all time steps can lead to severe distortion of the activation range. In addition to the distribution range, Deepcache emphasizes the significant temporal consistency of high-dimensional features between consecutive time steps, such as Figure 2 (b). These phenomena prompt a re-examination of existing binarization structures. Binarization, especially full binarization of weights and activations, leads to a greater loss in image generation range and accuracy compared to low-bit quantization such as 4-bit. This makes it more difficult to generate rich activation features. The lack of activation range and output features seriously impairs generative models with rich representations such as diffusion models DMs. Therefore, adopting a binary quantizer with a more flexible image input-output range and enhancing the overall expressiveness of the model by leveraging its feature output is a key strategy to improve the image generation ability of diffusion models after full binarization.

[0092] In this embodiment, we first focus on the difference between long-term time steps. Most existing activation quantizers, such as binary neural networks (BNN) and binary-real hybrid networks (Bi-Real), directly quantize activations to {+1,-1}, as shown in equation (8). This method seriously perturbs the activation features and negatively affects the expressiveness of the generated model. Some improved activation binary quantizers, such as XNOR++, use a trainable scaling factor k:

[0093]

[0094] Among them, K can be in the form of a vector or the product of multiple vectors, but it remains a constant value during inference. Although this method partially recovers the characteristic expression of activation, it does not match the highly correlated time steps and may still lead to significant performance loss.

[0095] In this embodiment, we turn our attention to the original XNOR-Net, which uses dynamically calculated mean values ​​to construct an activation binary quantizer. Its operation on two-dimensional convolution can be expressed as:

[0096]

[0097] Where I represents the input image, W represents the full-precision weight, sign(I) represents the input image after taking the sign function, sign(W) represents the weight after taking the sign function, Ka represents the product of the output scale factor map K and the weight scale factor a, ⊙ represents the matrix dot product, A represents the scale factor map of the image input, ka represents the product of the tiny convolution k and the weight scale factor a, c represents the number of channels of the input image, w in Indicates the width of the input image, h in represents the height of the input image, w represents the width of the weight, h represents the height of the weight, Ii represents the i-th channel of the input image, l1 represents the L1 norm, n represents the number of weight elements, a represents the weight scaling factor, W represents the full-precision weight, represents a two-dimensional filter, i represents the row number, j represents the column number, k represents the ij represents the tiny convolution weight element in row i and column j, * and Represent the convolution operation with and without multiplication, respectively. This method naturally preserves the range of image features and dynamically adjusts with the input range at different time steps. However, due to the rich expression of DM features, local activations show inconsistency in the range before and after the module passes, indicating that the predetermined k value cannot effectively restore the activation representation. Therefore, the present invention makes k adjustable and allows it to be learned during training to adaptively match the range changes of image distribution before and after. The gradient calculation process of the learnable tiny convolution k proposed in the present invention is as follows:

[0098]

[0099] Where L represents the loss function of the initial diffusion model, k represents a tiny convolution, and They represent convolution operations with and without multiplication respectively, I represents image input, W represents full-precision weight, A represents image output, and sign() represents a sign function taking ±1.

[0100] It is worth noting that making k learnable does not impose any additional inference burden. The computational cost remains the same, allowing for efficient binary operations.

[0101] On the other hand, the present invention focuses on the similarity between adjacent time steps. Deepcache directly extracts high-dimensional features as cache to skip a large number of deep calculations in U-Net, achieving significant inference acceleration. The process is expressed as:

[0102]

[0103] in, represents the t'th cached image, represents the image output of the m+1th downsampling block at the tth time step, Concat() represents the image cutting operation, represents the image output of the mth downsampling block at the t-1th time step, Represents the t'th cached image.

[0104] However, this approach is not suitable for binary diffusion models applied in the field of image generation, because the information content of each output of the binary network is very limited. For binary diffusion models applied in the field of image generation, which have essentially achieved significant compression and acceleration but limited expression capabilities, the present invention expects that the feature similarity between adjacent time steps will enhance the binary representation, thereby compensating for the expression challenge. The present invention constructs an information enhancement connection across time steps to enrich the expression of the current time step with the features of the previous time step. The process can be expressed as:

[0105]

[0106] Among them, Concat() represents the image cutting and splicing operation. represents the image output of the mth downsampling block at the t-1th time step, represents a learnable scaling factor, represents the image output of the m+1th upsampling block in the t-1th time step, represents the image output of the m+1th downsampling block at the tth time step.

[0107] As Figure 2 As shown in (b), the feature similarities of high-dimensional images in different blocks and time steps vary. Therefore, we set multiple independent a values, where a represents a learnable weight scaling factor initialized to 0.3, to allow the binary diffusion model to learn more effectively and adaptively during training.

[0108] In this embodiment, in summary, a time-step-friendly binary diffusion model structure (TBS) applied to the field of image generation includes learnable micro-convolutions applied to scaling factors after input averaging and concatenation across time steps. Their combined effect can adapt to the activation range changes of the diffusion model in long-term time steps and utilize the similarity of high-dimensional image features between adjacent time steps to enhance information representation.

[0109] From the perspective of error reduction, Figure 3 The visualization of the binary diffusion model structure TBS is shown in Fig. 1. First, the output of the binary diffusion model under the baseline method is abstracted into a vector B t-1 , the mismatch in scaling factors will cause a difference between it and the output vector F of the full-precision diffusion model. t-1 By using the scaling factor and learnable micro-convolution proposed by the present invention, vector B t-1 is expanded to vector L t-1 . Vector L t-1 Closer to the output vector F t-1 , but there is still a direction difference with the full-precision diffusion model. The connection across time steps further combines the output F of the previous time step t , B t and L t The high-dimensional feature similarity between adjacent time steps means that the output vector F t-1 and F t The gap between them is relatively small, which promotes the vector L t-1 and L t Finally, the output of the binary diffusion model is obtained by applying the binary diffusion model structure TBS, which is expressed as T t-1 =(1-a)·L t-1 +aL t , this output is closest to the output vector F of the full-precision diffusion modelt-1 , where B t-1 represents the image output of the naive binary diffusion model at the t-1th time step, F t-1 represents the image output of the full-precision diffusion model at the t-1th time step, L t-1 represents the image output of the binary diffusion model at the t-1th time step with tiny convolution applied, F t represents the image output of the full-precision diffusion model at the tth time step, B t represents the image output of the naive binary diffusion model at the tth time step, L t represents the image output of the binary diffusion model at the tth time step with tiny convolution applied, T t-1 The binary diffusion model of the present invention outputs the image at the t-1th time step, and a represents the mixing coefficient.

[0110] S3. Use spatial block distillation to train the binary diffusion model to obtain a trained binary diffusion model. The implementation method is as follows:

[0111] S301, extracting the intermediate features of the image generated by the full-precision diffusion model and the binary diffusion model, and performing block processing;

[0112] S302, based on the block processing result, calculate the self-attention for each small block;

[0113] S303, regularize the calculated self-attention, calculate and accumulate the loss of the corresponding position of each small block;

[0114] S304. According to the loss at the corresponding position of each small block, the total training loss function of the binary diffusion model is obtained, and the training of the binary diffusion model is completed.

[0115] In this embodiment, based on the binary diffusion model structure determined by S2, the intermediate features of the images generated by the full-precision diffusion model and the binary diffusion model are extracted and divided into blocks, and the loss function is added block by block for supervision during training.

[0116] Due to the characteristics of the generative model used in the field of image generation, the optimization process of the diffusion model has different characteristics from the previous discriminative model:

[0117] Observation 2: Traditional distillation has difficulty guiding the fully binarized binary diffusion model applied to image generation to align with the full-precision diffusion model, while the image features of the diffusion model in the generation task are spatially local.

[0118] In previous practice, it is common to add distillation loss during the training process of quantized models. Due to the limited numerical space of binary models, directly using simple losses for optimization will lead to difficulties in adjusting the direction of gradient updates, making learning challenging. Therefore, adding distillation loss to intermediate image features can better guide the local and global optimization process of the model. However, as a generative model applied in the field of image generation, the highly rich image feature representation of the diffusion model makes it extremely difficult for binary models applied in the field of image generation to imitate the full-precision diffusion model in detail. Although the L2 loss used in the training of the original diffusion model is aligned with the Gaussian noise in the diffusion process, it is not suitable for distillation matching of intermediate features. In the conventional distillation process, the commonly used L2 loss often prioritizes the optimization of pixels with large differences, resulting in more uniform and smooth optimization results. This globally constrained learning process is difficult for binary models targeting image generation, because their limited representation capabilities make it difficult for fine distillation imitation to directly adjust the model to fully match the direction of the full-precision diffusion model.

[0119] At the same time, the binary diffusion model used in the field of image generation using the U-Net network as the backbone naturally exhibits spatial locality due to its convolution-based structure and the requirements of the generation task. This is different from previous discriminative models, where tasks (such as classification) only require overall feature extraction without low-level requirements, which makes traditional distillation methods unsuitable for generation tasks with spatial locality. In addition, most existing diffusion model distillation methods focus on reducing the number of time steps and do not address the need for feature spatial locality in image generation tasks. Therefore, in view of the difficulty of existing loss functions in optimizing binary diffusion models and the spatial locality of diffusion models applied to the field of image generation, the present invention proposes spatial block distillation (SPD). Specifically, the present invention designs a new loss function that divides the features into small blocks before distillation and then calculates spatial self-attention block by block. Although the traditional L2 loss makes it difficult for binary diffusion models to achieve direct matching, resulting in optimization challenges, the attention mechanism allows distillation optimization to focus more on key parts in the image. However, this is still challenging for fully binary diffusion models because the highly discrete binary output has limited information, making it difficult for the model to capture global information. Therefore, the present invention divides the intermediate features into multiple small blocks and independently calculates the spatial self-attention for each small block, so that the binary model can better utilize local information during the optimization process.

[0120] Spatial block distillation SPD first converts the intermediate features F output by a block of the binary diffusion model and the full-precision diffusion model into bi and F fp ∈F b×c×w×h Divided into p 2small blocks, b represents the number of image batches, c represents the number of image channels, w represents the image width, and h represents the image height.

[0121]

[0122] Then, calculate the self-attention for each small block separately:

[0123]

[0124] After regularization, the loss at the corresponding position is calculated and accumulated:

[0125]

[0126] Finally, the total training loss is calculated as:

[0127]

[0128] Where L represents the total training loss function of the binary diffusion model, L DM represents the loss function of the initial diffusion model, represents the loss function of the binary diffusion model, m represents the total number of layers of the binary diffusion model, p represents the number of blocks, represents the self-attention of the image output of row i and column j of the full-precision diffusion model, represents the self-attention of the image output of the i-th row and j-th column of the binary diffusion model, represents the output block of the i-th row and j-th column image of the full-precision diffusion model, T represents transpose, represents the output block of the image in row i and column j of the binary diffusion model, represents the image output block of the i-th row and j-th column of the full-precision diffusion model, w represents the total width of the image output, h represents the total height of the image output, represents the output block of the image in row i and column j of the binary diffusion model, |||| 2 represents the L2 function, d represents the number of blocks in the upsampling or downsampling process, and finally including the intermediate blocks, there are 2d+1 intermediate features, λ represents the hyperparameter coefficient used to balance the loss term, and the default value is 4.

[0129] In this example, the intermediate features and self-attention mentioned above are visualized. Figure 4 As shown, Figure 4 In , FP32 represents the output of the full-precision diffusion model. Diff represents the difference between the output of the full-precision diffusion model and the output of the binarization model. Ours represents the self-attention mechanism. The spatial block distillation (SPD) proposed in the present invention enables the model to pay more attention to local information in each small block of the generated image.

[0130] S4. Generate an image task using the trained binary diffusion model. The implementation method is as follows:

[0131] Use the trained binary diffusion model to perform diffusion inference on the image noise;

[0132] The inference process is consistent with the inference process of the initial diffusion model, and the binary diffusion model structure is applied when performing image generation to output the generated image.

[0133] In this embodiment, based on the structure determined by step S1 and step S2 and the optimization result obtained by step S3, the binary diffusion model of the present invention performs noise diffusion reasoning according to the original denoising diffusion probability model (DDPM) when performing image generation. The overall reasoning process is consistent with the basic diffusion model constructed in step S1 (Formula (3), Formula (4)), and the time step-friendly binary structure constructed in S2 is applied when executing to the corresponding binary module (Formula (10), Formula (13)).

[0134] Compared with the existing state-of-the-art fully binarized methods, the present invention significantly improves the accuracy while maintaining the same inference efficiency, surpassing all existing baseline methods in various evaluation indicators. Specifically, in the pixel space diffusion model, the present invention reduces the Fréchet Inception Distance (FID) of the bedroom dataset (LSUN-Bedrooms) from 59.44 of the original best method to an impressive 22.74, while saving 28.0 times the storage space and 52.7 times the OPs. As the first fully binarized diffusion model method, the large number of generated samples also proves that the present invention is currently the only method that can generate acceptable fully binarized diffusion model images, thereby enabling the diffusion model to be effectively applied in low-resource scenarios.

Claims

1. An image generation method based on a diffusion model, characterized in that: The following steps are involved: S1. Construct and initialize an initial diffusion model applied to the field of image generation, and perform binarization on the initial diffusion model; S2, replacing all binary structures of the binary diffusion model with a time-step-friendly binary diffusion model structure to obtain a binary diffusion model; S3, training the binary diffusion model using spatial block distillation to obtain a trained binary diffusion model; S4. Generate image tasks using the trained binary diffusion model.

2. The image generation method based on the diffusion model according to claim 1, characterized in that: The expression of the loss function of the initial diffusion model in step S1 is as follows: Among them, L DM represents the loss function of the initial diffusion model, represents the mathematical expectation, ε t represents the noise at the tth time step, ε θ represents the predicted noise restored by the trained binary diffusion model, x t represents the original input image, t represents the time step, T represents the total time step, q(x0) represents the sample space of the input image, ε t represents noise, and N(0,I) represents the distribution space of Gaussian noise.

3. The image generation method based on the diffusion model according to claim 1, characterized in that: The binarization process in step S1 is as follows: The weights of the initial diffusion model are binarized using the following formula: Among them, w bi represents a binary weight, σ represents a floating-point scalar, w represents a full-precision weight, and sign() represents a sign function taking ±1; The input image is quantized using the following formula: Among them, a bi represents the image output by the binary diffusion model, and a represents the image output by the full-precision diffusion model.

4. The image generation method based on the diffusion model according to claim 1, characterized in that: The step S2 comprises the following steps: S201. Based on the binary diffusion model, a learnable tiny convolution scaling factor is applied to the two-dimensional information structure of all image inputs, wherein the gradient of the tiny convolution is calculated as follows: Where L represents the loss function of the initial diffusion model, k represents a tiny convolution, and Represent convolution operations with and without multiplication, respectively. I represents image input, W represents full-precision weight, A represents image output, and sign() represents a sign function taking ±1. S202: Based on the processing result of step S201, the similarity of the image outputs between adjacent time steps is analyzed by applying information enhancement connection across time steps to obtain a binary diffusion model.

5. The image generation method based on the diffusion model according to claim 4, characterized in that: The expression of the process of information enhancement connection across time steps is as follows: Among them, Concat() represents the image cutting and splicing operation. represents the image output of the mth downsampling block at the t-1th time step, represents a learnable scaling factor, represents the image output of the m+1th upsampling block in the t-1th time step, represents the image output of the m+1th downsampling block at the tth time step.

6. The image generation method based on the diffusion model according to claim 1, characterized in that: The step S3 comprises the following steps: S301, extracting the intermediate features of the image generated by the full-precision diffusion model and the binary diffusion model, and performing block processing; S302, based on the block processing result, calculate the self-attention for each small block; S303, regularize the calculated self-attention, calculate and accumulate the loss of the corresponding position of each small block; S304. According to the loss at the corresponding position of each small block, the total training loss function of the binary diffusion model is obtained, and the training of the binary diffusion model is completed.

7. The image generation method based on the diffusion model according to claim 1, characterized in that: The expression of the total training loss function of the binary diffusion model is as follows: Among them, L represents the total training loss function of the binary diffusion model, L DM represents the loss function of the initial diffusion model, d represents the number of blocks in the upsampling or downsampling process, λ represents the hyperparameter coefficient used to balance the loss term, represents the loss function of the binary diffusion model, m represents the total number of layers of the binary diffusion model, p represents the number of blocks, represents the self-attention of the image output of row i and column j of the full-precision diffusion model, represents the self-attention of the image output of the i-th row and j-th column of the binary diffusion model, represents the output block of the i-th row and j-th column image of the full-precision diffusion model, T represents transpose, represents the output block of the image in row i and column j of the binary diffusion model, represents the image output block of the i-th row and j-th column of the full-precision diffusion model, w represents the total width of the image output, h represents the total height of the image output, represents the output block of the i-th row and j-th column image of the binary diffusion model, and || ||2 represents the L2 function.

8. The image generation method based on the diffusion model according to claim 1, characterized in that: The step S4 is specifically as follows: Use the trained binary diffusion model to perform diffusion inference on the image noise; The inference process is consistent with the inference process of the initial diffusion model, and the binary diffusion model structure is applied when performing image generation to output the generated image.

9. The image generation method based on the diffusion model according to claim 8, characterized in that: The trained binary diffusion model is used to perform diffusion inference processing on the noise of the image, which is specifically as follows: According to the trained binary diffusion model, the following formula is used to perform diffusion inference on the noise of the image: Among them, p θ (x t-1 |x t ) represents the previous noise image inferred by the trained binary diffusion model when the next noise image is known, N represents the normal distribution, x t-1 represents the noise image at the t-1th time step, represents the image mean derived from the trained binary diffusion model, x t represents the noise image at the t-th time step, t represents the t-th time step, represents the cumulative statistical image standard deviation at the t-th time step, I represents the input image, It means that the binary diffusion model after training at the tth time step is composed of x t and the image mean derived from t, a t Auxiliary standard deviation and Both represent the cumulative auxiliary standard deviation ε θ represents the noise image derived from the trained binary diffusion model, β t represents the standard deviation of the image at the t-th time step.

Images