An image generation method, apparatus, device and computer readable storage medium

Abstract

The application relates to the technical field of image processing, and discloses an image generation method, device and equipment and a computer readable storage medium. Random noise output by a noise generator is used to perform noise adding processing on a training image to obtain a noisy image; a diffusion model contains a codec and a latent variable space, and the noisy image and a time vector sampled by a Gaussian distribution sampler are analyzed to obtain a noise prediction result. Based on the noise prediction result and the random noise, the contribution degrees of model parameters in the diffusion model are determined; the diffusion model is pruned by a pruner according to the contribution degrees of the model parameters to obtain a compressed diffusion model. The noise training process of the diffusion model is compressed to the latent variable space for learning, and the running efficiency of the diffusion model on a server is improved. The compressed diffusion model is used to analyze a newly acquired noisy image to generate a denoised image, and the calculation overhead of the image generation process is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image generation method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Server vendors and artificial intelligence (AI) product departments, when providing customized generative model applications such as AI-assisted writing, AI-powered image editing, and AI-driven creative design, need to retrain the given pre-trained models in a customized manner. This process requires a huge amount of computation, resulting in long preparation times for model deployment on distributed servers and high demands on storage devices and computing resources. Significant progress in generative models is largely due to the introduction of diffusion models (DM). However, diffusion models require considerable computational overhead during training and inference, thus necessitating a trade-off between performance and efficiency when server computing resources are limited.

[0003] Currently, to reduce the computational cost of models, a common approach is to perform structured pruning. This involves eliminating redundant parameters and substructures to effectively reduce model size, thus achieving a better balance between performance and efficiency. However, diffusion models are iterative in image generation. The perturbation effect on the loss function varies at different iteration stages, and the entire diffusion process generates complex interactions between iterations. Therefore, traditional structured pruning methods cannot be directly applied to diffusion models due to their unique characteristics. While diffusion models have good processing capabilities and can generate high-quality images from noisy images, the computational cost during image generation is high.

[0004] It is evident that reducing the computational overhead of the image generation process is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an image generation method, apparatus, device, and computer-readable storage medium that can solve the problem of high computational overhead in the image generation process.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide an image generation method, comprising:

[0007] The training image is noise-added based on the random noise output by the noise generator to obtain a noisy image;

[0008] By utilizing the encoder-decoder and latent variable space contained in the diffusion model, the noisy image and the time vector sampled by the Gaussian distribution sampler are analyzed to obtain the noise prediction results;

[0009] Based on the noise prediction results and the random noise, the contribution of each model parameter in the diffusion model is determined.

[0010] The diffusion model is pruned using a pruning tool based on the contribution of each model parameter to obtain a compressed diffusion model.

[0011] The newly acquired noisy image is analyzed using the compressed diffusion model to generate a denoised image.

[0012] On the one hand, the analysis of the noisy image and the time vector sampled by the Gaussian distribution sampler using the encoder-decoder and latent variable space contained in the diffusion model to obtain noise prediction results includes:

[0013] The noisy image and the time vector are encoded based on the encoder contained in the diffusion model to obtain input features;

[0014] The input features are analyzed using the latent variable space contained in the diffusion model to obtain the output features;

[0015] The output features are decoded based on the decoder included in the diffusion model to obtain noise prediction results.

[0016] On the one hand, regarding the generation process of the time vector, the method includes:

[0017] A time vector is obtained by sampling from a conditional Gaussian distribution using the Gaussian distribution sampler; wherein the mean of the conditional Gaussian distribution is less than half the training period time.

[0018] On the one hand, it also includes:

[0019] The mean of the conditional Gaussian distribution is adjusted based on the contribution of each model parameter in the diffusion model at different sampling times; wherein the time vector includes multiple different sampling times.

[0020] On the one hand, adjusting the mean of the conditional Gaussian distribution based on the contribution of each model parameter in the diffusion model at different sampling times includes:

[0021] The contribution of each model parameter in the diffusion model at different sampling times is statistically analyzed, and the target sampling time with the highest contribution is selected.

[0022] Determine whether the mean of the conditional Gaussian distribution is greater than the target sampling time;

[0023] If the mean of the conditional Gaussian distribution is greater than the target sampling time, the mean of the conditional Gaussian distribution is reduced.

[0024] On the one hand, before analyzing the noisy image and the time vector sampled by the Gaussian distribution sampler using the encoder-decoder and latent variable space contained in the diffusion model to obtain the noise prediction result, the method further includes:

[0025] Determine whether the random noise meets the condition for regular noise;

[0026] When the random noise does not meet the condition of regular noise, training constraint text is set according to the training image;

[0027] The analysis of the noisy image and the time vector sampled by the Gaussian distribution sampler, utilizing the encoder-decoder and latent variable space contained in the diffusion model, to obtain noise prediction results includes:

[0028] The noisy image, the time vector, and the training constraint text are analyzed using the encoder-decoder and latent variable space contained in the diffusion model to obtain noise prediction results.

[0029] On the one hand, determining the contribution of each model parameter in the diffusion model based on the noise prediction results and the random noise includes:

[0030] Based on the noise prediction results and the random noise at different sampling times, the loss values ​​at different sampling times are determined.

[0031] Based on the loss values ​​at different sampling times, the model parameters in the diffusion model, and the parameter gradients, the contribution of each model parameter in the diffusion model at different sampling times is determined.

[0032] On the one hand, determining the loss value at different sampling times based on the noise prediction results and the random noise at different sampling times includes:

[0033] Cross-entropy calculation is performed on the noise prediction results and the random noise at different sampling times to obtain the loss value at different sampling times.

[0034] On the one hand, before determining the contribution of each model parameter in the diffusion model at different sampling times based on the loss value at different sampling times, each model parameter in the diffusion model, and the parameter gradient, the method further includes:

[0035] Based on the loss values ​​at different sampling times, the segmentation time when the loss value is less than the set loss threshold is determined;

[0036] For all sampling times prior to the segmentation time, the step of determining the contribution of each model parameter in the diffusion model at different sampling times is performed based on the loss value at different sampling times, each model parameter in the diffusion model, and the parameter gradient.

[0037] Set the contribution of the segmentation time and all sampling times after the segmentation time to zero.

[0038] On the one hand, determining the contribution of each model parameter in the diffusion model at different sampling times based on the loss value at different sampling times, the model parameters in the diffusion model, and the parameter gradients includes:

[0039] Select the target sampling time that is located within the confidence interval of the conditional Gaussian distribution from all the sampling times;

[0040] Based on the loss value at each target sampling time, the model parameters in the diffusion model, and the parameter gradient, the contribution of each model parameter in the diffusion model at each target sampling time is determined.

[0041] On the one hand, determining the contribution of each model parameter in the diffusion model at each target sampling time based on the loss value at each target sampling time, each model parameter in the diffusion model, and the parameter gradient includes:

[0042] The contribution calculation formula is used to analyze the loss value, model parameters, and parameter gradients at each target sampling time to determine the contribution of each model parameter at each target sampling time. The contribution calculation formula is as follows:

[0043]

[0044] Among them, I(θ ik ,x) represents the contribution of the model parameter in the i-th row and k-th column, θ ik Let represent the model parameters in the i-th row and k-th column of the model parameter matrix of the diffusion model, x represent the noisy image, μ represent the mean of the conditional Gaussian distribution, σ represent the standard deviation of the conditional Gaussian distribution, and Z represent the standard score in the standard normal distribution table. Let L represent the parameter gradient of the model parameters in the i-th row and k-th column, and t represent the target sampling time. t (θ,x) represents the loss value at sampling time t.

[0045] On the one hand, the process of pruning the diffusion model based on the contribution of each model parameter to obtain a compressed diffusion model includes:

[0046] The contribution of each model parameter at each sampling time is sorted in descending order;

[0047] The model parameters, arranged in descending order, are input into the pruning tool to perform structured pruning on the model parameters that meet the pruning requirements, thereby obtaining the compressed diffusion model.

[0048] On one hand, the step of inputting the model parameters arranged in descending order into the pruning device to perform structured pruning on the model parameters that meet the pruning requirements, in order to obtain the compressed diffusion model, includes:

[0049] The model parameters whose contribution is less than the threshold at each sampling time are structurally pruned to obtain the compressed diffusion model.

[0050] On the one hand, the step of structurally pruning the model parameters whose contribution is less than a threshold at each sampling time to obtain the compressed diffusion model includes:

[0051] At each sampling time, set the model parameters whose contribution is less than the threshold to zero.

[0052] On the one hand, before setting the model parameters whose contribution is less than the threshold at each sampling time to zero, the method further includes:

[0053] Based on the contribution of each model parameter at each sampling time, the average contribution value corresponding to each sampling time is determined;

[0054] Query the list of correspondences between contribution ranges and thresholds to determine the threshold that matches the average contribution at different sampling times.

[0055] On one hand, the step of inputting the model parameters arranged in descending order into the pruning device to perform structured pruning on the model parameters that meet the pruning requirements, in order to obtain the compressed diffusion model, includes:

[0056] Set the last N model parameters at each sampling time to zero.

[0057] On one hand, the step of adding noise to the training image based on the random noise output by the noise generator to obtain a noisy image includes:

[0058] Obtain the random noise output by the noise generator;

[0059] The random noise is gradually diffused into the training image using the diffusion model to obtain the noisy image.

[0060] On the one hand, the step of analyzing the newly acquired noisy image using the compressed diffusion model to generate a denoised image includes:

[0061] Determine whether the newly acquired noisy image satisfies the condition of regular noise;

[0062] If the newly acquired noisy image does not meet the regular noise condition, constraint text is set according to the image generation requirements;

[0063] The newly acquired noisy image, the time vector, and the constraint text are input into the compressed diffusion model to obtain the denoised image.

[0064] On the one hand, after analyzing the newly acquired noisy image using the compressed diffusion model to generate a denoised image, the method further includes:

[0065] Extract the semantic information contained in the denoised image;

[0066] Determine the image semantic information contained in the denoised image and the image semantic information contained in the constrained text.

[0067] Does the information match?

[0068] If the image semantic information contained in the denoised image matches the image semantic information contained in the constraint text, the denoised image is saved.

[0069] If the semantic information contained in the denoised image does not match the semantic information contained in the constraint text, a prompt message indicating that the denoised image does not match the constraint text is generated.

[0070] This invention also provides an image generation apparatus, including a noise-adding unit, an analysis unit, a determination unit, a pruning unit, and a noise reduction unit;

[0071] The noise-adding unit is used to add noise to the training image based on the random noise output by the noise generator to obtain a noisy image.

[0072] The analysis unit is used to analyze the noisy image and the time vector sampled by the Gaussian distribution sampler using the encoder and decoder and latent variable space contained in the diffusion model, so as to obtain the noise prediction result.

[0073] The determining unit is used to determine the contribution of each model parameter in the diffusion model based on the noise prediction result and the random noise.

[0074] The pruning unit is used to prune the diffusion model according to the contribution of each model parameter using a pruner to obtain a compressed diffusion model.

[0075] The denoising unit is used to analyze the newly acquired noisy image using the compressed diffusion model to generate a denoised image.

[0076] On the one hand, the analysis unit includes an encoding subunit, a latent variable analysis subunit, and a decoding subunit;

[0077] The encoding subunit is used to encode the noisy image and the time vector based on the encoder included in the diffusion model to obtain input features;

[0078] The latent variable analysis subunit is used to analyze the input features using the latent variable space contained in the diffusion model to obtain the output features;

[0079] The decoding subunit is used to decode the output features based on the decoder included in the diffusion model to obtain noise prediction results.

[0080] On the one hand, regarding the generation process of the time vector, the device includes a sampling unit;

[0081] The sampling unit is used to sample from a conditional Gaussian distribution through the Gaussian distribution sampler to obtain a time vector; wherein the mean of the conditional Gaussian distribution is less than half the training period time.

[0082] On the one hand, it also includes adjustment units;

[0083] The adjustment unit is used to adjust the mean of the conditional Gaussian distribution according to the contribution of each model parameter in the diffusion model at different sampling times; wherein, the time vector includes multiple different sampling times.

[0084] On the one hand, the adjustment unit includes a statistics subunit, a judgment subunit, and a reduction subunit;

[0085] The statistical subunit is used to calculate the contribution of each model parameter in the diffusion model at different sampling times, and select the target sampling time with the highest contribution.

[0086] The judgment subunit is used to determine whether the mean of the conditional Gaussian distribution is greater than the target sampling time;

[0087] The adjustment subunit is used to reduce the mean of the conditional Gaussian distribution when the mean of the conditional Gaussian distribution is greater than the target sampling time.

[0088] On the one hand, it also includes a noise judgment unit and a setting unit;

[0089] The noise judgment unit is used to determine whether the random noise meets the condition of regular noise;

[0090] The setting unit is used to set training constraint text based on the training image when the random noise does not meet the regular noise condition.

[0091] The analysis unit is used to analyze the noisy image, the time vector, and the training constraint text using the encoder-decoder and latent variable space contained in the diffusion model to obtain noise prediction results.

[0092] On the one hand, the determining unit includes a loss value determining subunit and a contribution degree determining subunit;

[0093] The loss value determination subunit is used to determine the loss value at different sampling times based on the noise prediction results and the random noise at different sampling times.

[0094] The contribution determination subunit is used to determine the contribution of each model parameter in the diffusion model at different sampling times based on the loss value at different sampling times, each model parameter in the diffusion model, and the parameter gradient.

[0095] On the one hand, the loss value determination subunit is used to perform cross-entropy calculation on the noise prediction results and the random noise at different sampling times to obtain the loss value at different sampling times.

[0096] On the one hand, it also includes the segmentation time determination unit and the zeroing unit;

[0097] The segmentation time determination unit is used to determine the segmentation time when the loss value is less than a set loss threshold based on the loss value at different sampling times.

[0098] The contribution determination subunit is used to perform the step of determining the contribution of each model parameter in the diffusion model at different sampling times based on the loss value, each model parameter in the diffusion model, and the parameter gradient at all sampling times before the segmentation time.

[0099] The zeroing unit is used to set the contribution of the segmentation time and all sampling times after the segmentation time to zero.

[0100] On the one hand, the contribution determination subunit is used to select target sampling times that are located within the confidence interval of the conditional Gaussian distribution from all the sampling times; based on the loss value at each target sampling time, each model parameter in the diffusion model and the parameter gradient, the contribution of each model parameter in the diffusion model at each target sampling time is determined.

[0101] On one hand, the contribution determination subunit is used to call the contribution calculation formula to analyze the loss value, model parameters, and parameter gradients at each target sampling time, so as to determine the contribution of each model parameter in the diffusion model at each target sampling time; wherein, the contribution calculation formula is as follows:

[0102]

[0103] Among them, I(θ ik ,x) represents the contribution of the model parameter in the i-th row and k-th column, θ ik Let represent the model parameters in the i-th row and k-th column of the model parameter matrix of the diffusion model, x represent the noisy image, μ represent the mean of the conditional Gaussian distribution, σ represent the standard deviation of the conditional Gaussian distribution, and Z represent the standard score in the standard normal distribution table. Let L represent the parameter gradient of the model parameters in the i-th row and k-th column, and t represent the target sampling time. t (θ,x) represents the loss value at sampling time t.

[0104] On the one hand, the pruning unit includes an arrangement subunit and a compression subunit;

[0105] The sorting subunit is used to sort the contribution of each model parameter in descending order at each sampling time.

[0106] The compression subunit is used to input the model parameters arranged in descending order into the pruner to perform structured pruning on the model parameters that meet the pruning requirements, so as to obtain the compressed diffusion model.

[0107] On the one hand, the compression subunit is used to perform structured pruning on the model parameters whose contribution is less than a threshold at each sampling time to obtain the compressed diffusion model.

[0108] On the one hand, the compression subunit is used to set the model parameters whose contribution is less than a threshold at each sampling time to zero.

[0109] On the one hand, it also includes a mean determination unit and a threshold determination unit;

[0110] The mean determination unit is used to determine the mean contribution value corresponding to each sampling time based on the contribution value of each model parameter at each sampling time.

[0111] The threshold determination unit is used to query the correspondence list between contribution range and threshold to determine the threshold matched by the average contribution at different sampling times.

[0112] On the one hand, the compression subunit is used to set the last N model parameters of each model parameter at each sampling time to zero.

[0113] On one hand, the noise-adding unit is used to acquire random noise output by the noise generator; and the random noise is gradually diffused into the training image using the diffusion model to obtain the noisy image.

[0114] On one hand, the denoising unit is used to determine whether the newly acquired noise image meets the regular noise condition; if the newly acquired noise image does not meet the regular noise condition, constraint text is set according to the image generation requirements; the newly acquired noise image, the time vector and the constraint text are input into the compressed diffusion model to obtain the denoised image.

[0115] On the one hand, it also includes an extraction unit, a matching unit, a saving unit, and a prompting unit;

[0116] The extraction unit is used to extract the image semantic information contained in the denoised image;

[0117] The matching unit is used to determine whether the image semantic information matches the image semantic information contained in the constraint text;

[0118] The storage unit is used to save the denoised image when the image semantic information matches the image semantic information contained in the constraint text;

[0119] The prompting unit is used to generate a prompt message indicating that the denoised image does not match the constraint text when the semantic information of the image does not match the semantic information of the image contained in the constraint text.

[0120] This invention also provides an image generation device, comprising:

[0121] Memory, used to store computer programs;

[0122] A processor for executing the computer program to implement the steps of the image generation method described above.

[0123] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image generation method described above.

[0124] As can be seen from the above technical solution, the training image is noise-added using random noise output by the noise generator to obtain a noisy image. The noisy image and the time vector sampled by the Gaussian distribution sampler are analyzed using the encoder / decoder and latent variable space contained in the diffusion model to obtain noise prediction results. Based on the noise prediction results and random noise, the contribution of each model parameter in the diffusion model is determined. The diffusion model is then pruned using a pruning tool based on the contribution of each model parameter to obtain a compressed diffusion model. The compressed diffusion model is then used to analyze newly acquired noisy images to generate denoised images. The beneficial effect of this invention is that by compressing the noise training process of the diffusion model into the latent variable space for learning, the number of model parameters in the diffusion model is effectively reduced, improving computational efficiency. Subsequent pruning work will also be performed in the latent variable space, thereby improving the running efficiency of the diffusion model on the server. The compressed diffusion model effectively reduces model parameters and lowers the computational overhead of the image generation process while ensuring image generation quality. Attached Figure Description

[0125] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0126] Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention;

[0127] Figure 2 A schematic diagram illustrating a process for generating a denoised image using a compressed diffusion model, provided as an embodiment of the present invention;

[0128] Figure 3 A schematic diagram of a conditional Gaussian distribution provided in an embodiment of the present invention;

[0129] Figure 4 This is a schematic diagram of a process for processing noisy images using an improved noise estimation module, provided by an embodiment of the present invention.

[0130] Figure 5 This is a schematic diagram of the structure of an image generation device provided in an embodiment of the present invention;

[0131] Figure 6 This is a structural diagram of an image generation device provided in an embodiment of the present invention. Detailed Implementation

[0132] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0133] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0134] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0135] Structured pruning is a classic model compression technique that effectively reduces model size by eliminating redundant parameters and substructures, achieving a good balance between performance and efficiency. However, diffusion models are iterative in image generation, and the perturbation of the loss function varies at different iteration stages. Furthermore, the entire diffusion process generates complex interactions between iterations. Therefore, traditional structured pruning requires specific improvements to address the unique characteristics of diffusion models.

[0136] The diffusion process contributes differently to the generated samples at different stages. The diffusion process can include stages from 0 to T. In the later iterations (T to t, i.e., T->t), the diffusion model prioritizes generating high-level semantic content; while in the earlier iterations (t to 0, i.e., t->0), it focuses more on the details of image generation, such as edges and colors. Therefore, a trade-off needs to be made between content, detail, noise, and computational resources. The images corresponding to the stages from 0 to T can be sequentially processed using x0 to x... T This indicates that from x0 to x T The process belongs to the training phase, starting from x T Up to x0 is the application phase. In the application phase, the diffusion model can generate useful denoised images from noisy images.

[0137] The model parameters of a diffusion model can be represented by θ. A diffusion model has multiple model parameters, which can be presented in the form of a parameter matrix.

[0138] Given pre-trained model parameters θ, the goal of structured pruning is to find a sparse parameter matrix θ′ that maximizes the performance of the diffusion model. Therefore, a common choice is to minimize the loss perturbation caused by pruning.

[0139] min|L(θ′)-L(θ)|;

[0140] st||θ′||0 <s;

[0141] Where L(θ) represents the loss value corresponding to the model parameter θ, L(θ′) represents the loss value corresponding to the sparse parameter matrix θ′, ||θ′||0 represents the L0 norm of the parameter matrix, that is, the number of non-zero row vectors of the matrix; s represents the sparsity of the parameter matrix after pruning.

[0142] Because the diffusion model has iterative properties, the loss function L can be represented as the composition of T interconnected tasks: {L1, L2, ..., L...} T Each task influences and depends on other tasks, so each loss component L must be analyzed. t Contribution to the pruning process.

[0143] First, it is necessary to model L. t Contribution to structural pruning. This work utilizes L... t We use the Taylor expansion to calculate the linear approximation loss perturbation, where x is any point in the neighborhood other than x0. Then there exists at least one ξ in the neighborhood such that...

[0144]

[0145] Where f(x) represents the Taylor expansion of x, f′(x0) represents the first-order Taylor expansion of x0, and f””x0) represents the second-order Taylor expansion of x0.

[0146] Substituting the Taylor first-order expansion into the model parameters:

[0147] L t (θ′)=L t (θ)+L′ t (θ)(θ′-θ)+O(||θ′-θ|| 2 );

[0148] It can be inferred that: L t (θ′)-L t (θ)=L′ t (θ)(θ′-θ)+O(||θ′-θ|| 2 );

[0149] Among them, L t (θ′) represents the loss value corresponding to the sparse parameter matrix θ′ at sampling time t, L t (θ) represents the loss value corresponding to the model parameter θ at sampling time t. L′ t(θ) represents the first-order loss value corresponding to the model parameter θ at sampling time t.

[0150] Taylor's formula gives the perturbation of the loss estimated from the first-order gradient of the loss function. To evaluate the importance of individual weight parameters, assume θ ik Since ′ = 0, the importance criterion is deduced as:

[0151]

[0152] Among them, I(θ ik (x) represents the model parameter θ ik Contribution It is the model parameter θ ik The gradient of , where i represents the row of the model parameter matrix and k represents the column of the model parameter matrix.

[0153] A key task of pruning is to eliminate the entire θ i This row vector only considers the loss difference between the initial and final states. Due to the iterative nature of the diffusion model, sequential removal of θ should be considered. ik The process is such that importance can be estimated using the following formula:

[0154]

[0155] Among them, L t (θ,x) represents the loss value of the diffusion model at sampling time t.

[0156] At a given time period t, the loss term L t =|∈-∈ θ |This will ultimately affect the generated image x0, where ∈ is random noise drawn from a Gaussian distribution, ∈ θ This is the learned noise prediction result. Assume that an error δ is generated during the pruning process at time t. t The reverse process allows us to obtain the final effect δ0 approximately equal to x0:

[0157]

[0158] because As the sampling process progresses, the error δ t The resulting generation distortion gradually increases, where α and β both represent trade-off parameters. t and β t This represents the trade-off parameter corresponding to sampling time t.

[0159] Therefore, α can be introduced into this method. t ∈(0,1) balances image content and detail. Since applying this method across all time stages of the Taylor expansion is too inefficient, α is... tModify to only take {0, 1}, and only use some steps for pruning.

[0160] The stage responsible for generating content is not only in the later stages of (T->t). On the contrary, there are many noisy and redundant stages in the later stages that contribute very little to the overall generation. When the loss converges, the loss is mainly affected by higher-order terms rather than the first-order gradient. In the later stages of (T->t), the loss converges to 0 quickly.

[0161] This invention, considering the aforementioned characteristics of diffusion models, designs an image generation method for applications involving the rapid deployment of generative models on enterprise data center servers. By compressing the noisy training process of the diffusion model into the latent variable space for learning, computational efficiency is improved. A loss perturbation calculation method based on Taylor first-order expansion is designed to determine the contribution of each model parameter in the diffusion model. Based on the contribution of each model parameter, the diffusion model is pruned to obtain a compressed diffusion model. The compressed diffusion model effectively reduces the number of model parameters and lowers the computational cost of the image generation process while maintaining image generation quality.

[0162] Next, we will describe in detail an image generation method provided by an embodiment of the present invention. Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention, the method comprising:

[0163] S101: Add noise to the training image based on the random noise output by the noise generator to obtain a noisy image.

[0164] In practical applications, a diffusion model can be used to gradually diffuse random noise into the training image to obtain a noisy image. For ease of description, we can use ∈ to represent random noise, x0 to represent the training image, and x T This indicates a noisy image.

[0165] A noise generator can be a server or a component independent of the server, used to generate random noise.

[0166] S102: Using the encoder-decoder and latent variable space contained in the diffusion model, analyze the time vectors sampled by the noisy image and the Gaussian distribution sampler to obtain noise prediction results.

[0167] For ease of description, we can use ∈ θ This indicates the noise prediction result.

[0168] The key to the diffusion model is training the noise estimation module, which estimates the actual noise. This module can employ a semantic segmentation (UNet) model, and the loss value can be calculated from the noise prediction results output by the module and the actual random noise. To train diversity at different time steps, time vectors sampled throughout the entire training period can be used. The training period can be represented by T, and the moments contained in the time vector can be represented by t.

[0169] In this embodiment of the invention, in order to improve the running efficiency of the diffusion model on the server, the architecture of the noise estimation module of the diffusion model can be modified. The modified noise estimation module may include an encoder, an input layer, a latent variable space, an output layer, and a decoder.

[0170] The latent variable space is the space remaining after breaking down a complex problem and removing useless extraneous information. The description in this space is concise and to the point.

[0171] Compared to traditional diffusion models, this embodiment of the invention compresses the training process of the diffusion model into the latent variable space, which can significantly reduce the number of model parameters. Furthermore, subsequent pruning can also be performed in the latent variable space, thereby improving the running efficiency of the diffusion model on the server.

[0172] In practical applications, after acquiring a noisy image, the input features can be obtained by encoding the noisy image and the time vector sampled by the Gaussian distribution sampler based on the encoder contained in the diffusion model. The input features are then analyzed using the latent variable space contained in the diffusion model to obtain the output features; finally, the noise prediction results can be obtained by decoding the output features based on the decoder contained in the diffusion model.

[0173] A Gaussian distribution sampler is a component used to sample time vectors. A Gaussian distribution sampler can be a component deployed on a server or a component independent of the server.

[0174] S103: Based on the noise prediction results and random noise, determine the contribution of each model parameter in the diffusion model.

[0175] The time vector includes different sampling times. In practical applications, the loss value at different sampling times can be determined based on the noise prediction results and random noise at different sampling times. Based on the loss values ​​at different sampling times, the model parameters in the diffusion model, and the parameter gradients, the contribution of each model parameter in the diffusion model at different sampling times can be determined.

[0176] The loss value can be determined by cross-entropy calculation. Cross-entropy calculation can be performed on the noise prediction results and random noise at different sampling times to obtain the loss value at different sampling times.

[0177] S104: Based on the contribution of each model parameter, the diffusion model is pruned using a pruner to obtain a compressed diffusion model.

[0178] The diffusion model has multiple model parameters, and each parameter has its own contribution at different sampling times. In this embodiment of the invention, pruning operations can be performed on the diffusion model at each sampling time.

[0179] In practical applications, the contribution of each model parameter at each sampling time can be sorted in descending order; the sorted model parameters are then input into a pruning tool to perform structured pruning on the model parameters that meet the pruning requirements, so as to obtain a compressed diffusion model.

[0180] A pruner is a component used to perform structured pruning on a diffusion model. A pruner can be a component deployed on a server or a component independent of the server.

[0181] There are several ways to prune. The first pruning method can prune model parameters whose contribution is less than a threshold in the diffusion model. The second pruning method can prune model parameters whose contribution ranks lower in the diffusion model.

[0182] Taking the first pruning method as an example, the model parameters whose contribution is less than the threshold at each sampling time can be structurally pruned to obtain the compressed diffusion model.

[0183] In practical applications, structured pruning can be achieved by setting model parameters to zero. That is, model parameters whose contribution is less than a threshold at each sampling time can be set to zero.

[0184] In this embodiment of the invention, the contribution of model parameters can be evaluated using the same threshold at different sampling times. Furthermore, considering that the later stages of diffusion contribute less to the training of the diffusion model, different thresholds can be set for different sampling times to further improve the pruning effect.

[0185] In this embodiment of the invention, before setting the model parameters whose contribution is less than the threshold to zero at each sampling time, the average contribution value corresponding to each sampling time can be determined based on the contribution value of each model parameter at each sampling time; the correspondence list between contribution range and threshold is queried to determine the threshold matched by the average contribution value at different sampling times.

[0186] Taking the second pruning method as an example, the last N model parameters at each sampling time can be set to zero. The value of N can be set based on the actual pruning requirements of the diffusion model, and is not limited here.

[0187] It should be noted that, in this embodiment of the invention, the noise generator, codec, Gaussian distribution sampler, and pruner can all be deployed on the same server, or they can be set up independently.

[0188] S105: Analyze the newly acquired noisy image using the compressed diffusion model to generate a denoised image.

[0189] When a new noisy image is acquired, in order to obtain a denoised image, the newly acquired noisy image can be input into a compressed diffusion model. The compressed diffusion model can then output a denoised image. To distinguish between different images, the image output by the compressed diffusion model can be called the denoised image.

[0190] Considering that the noise contained in the newly acquired noisy image may not be regular, the denoised image obtained by analyzing the compressed diffusion model may not have practical value. Therefore, in this embodiment of the invention, in order to output a denoised image with practical value, after acquiring the new noisy image, it can be determined whether the newly acquired noisy image meets the condition of regular noise.

[0191] Regular noise conditions can include the distribution characteristics of regular noise. Regular noise conditions can be set manually and are not limited here.

[0192] If the newly acquired noisy image does not meet the conditions for regular noise, it indicates that the denoised image output by the diffusion model may not be usable. To ensure that the diffusion model can output a usable denoised image, constraint text can be set according to the image generation requirements. The newly acquired noisy image, time vector, and constraint text are then input into the compressed diffusion model to obtain the denoised image.

[0193] After generating the denoised image, to evaluate whether it meets the actual requirements, the semantic information contained in the denoised image can be extracted. This involves determining whether the semantic information contained in the denoised image matches the semantic information contained in the constraint text.

[0194] If the semantic information contained in the denoised image matches the semantic information contained in the constraint text, the generated denoised image meets the actual requirements, and can be saved. If the semantic information contained in the denoised image does not match the semantic information contained in the constraint text, the generated denoised image does not meet the actual requirements, and a prompt message indicating that the denoised image and constraint text do not match can be generated.

[0195] Figure 2 This is a flowchart illustrating a process for generating a denoised image using a compressed diffusion model, as provided in an embodiment of the present invention. After obtaining a training image and random noise, adding the random noise to the training image yields a first noisy image, which can be represented by x1. Through the diffusion process of the diffusion model, the random noise is gradually diffused into the training image, ultimately resulting in a noisy image, which can be represented by x1. t This represents a noisy image. The key to the diffusion model is training the noise estimation module, which, after modification, includes an encoder, input layer, latent variable space, output layer, and decoder. The latent variable space can include multiple latent variables. Figure 2 The example used here is the first latent variable and the second latent variable. In this embodiment of the invention, a Transformer model can be used as the encoder-decoder framework.

[0196] In practical applications, the noisy image and time vector can be concatenated and then fed into an encoder for encoding to obtain encoded high-dimensional features. These encoded high-dimensional features are then processed through an input layer, a first latent variable layer, a second latent variable layer, and an output layer to output another high-dimensional feature. The decoder then decodes the output high-dimensional feature to obtain the noise prediction result. In this embodiment of the invention, the contribution of each model parameter can be determined based on the noise prediction result and random noise. A pruning mechanism can perform structured pruning operations on the diffusion model based on the contribution of each model parameter, thereby obtaining a compressed diffusion model. Inputting the newly acquired noisy image into the compressed diffusion model yields the output denoised image.

[0197] Based on the loss values, model parameters, and parameter gradients at different sampling times, the contribution of each model parameter in the diffusion model at different sampling times can be calculated. Based on the contribution of each model parameter, parameters with smaller contributions can be structurally pruned to reduce the number of parameters in the diffusion model. The pruned diffusion model retains the performance of the original diffusion model while reducing the number of parameters, essentially compressing it; therefore, the pruned diffusion model can be called a compressed diffusion model. Using the compressed diffusion model to analyze newly acquired noisy images can quickly generate denoised images.

[0198] As can be seen from the above technical solution, the training image is noise-added using random noise output by the noise generator to obtain a noisy image. The noisy image and the time vector sampled by the Gaussian distribution sampler are analyzed using the encoder / decoder and latent variable space contained in the diffusion model to obtain noise prediction results. Based on the noise prediction results and random noise, the contribution of each model parameter in the diffusion model is determined. The diffusion model is then pruned using a pruning tool based on the contribution of each model parameter to obtain a compressed diffusion model. The compressed diffusion model is then used to analyze newly acquired noisy images to generate denoised images. The beneficial effect of this invention is that by compressing the noise training process of the diffusion model into the latent variable space for learning, the number of model parameters in the diffusion model is effectively reduced, improving computational efficiency. Subsequent pruning work will also be performed in the latent variable space, thereby improving the running efficiency of the diffusion model on the server. The compressed diffusion model effectively reduces model parameters and lowers the computational overhead of the image generation process while ensuring image generation quality.

[0199] The time vector includes multiple sampling times. To satisfy the diversity of sampling at different times, random sampling can be performed from 1 to T to obtain multiple sampling times that follow a uniform distribution.

[0200] Considering that the later stages of the diffusion process contribute less to the training of the diffusion model, in addition to random sampling, a Gaussian distribution sampler can also be used for time sampling.

[0201] In this embodiment of the invention, a time vector can be obtained by sampling from a conditional Gaussian distribution using a Gaussian distribution sampler.

[0202] The mean of a traditional Gaussian distribution is equal to half the training cycle time. In this invention, the mean of a conditional Gaussian distribution is less than half the training cycle time.

[0203] In practical applications, a Gaussian distribution sampler installed on a server can be designed for sampling. The sampling time during the training process of the noise estimation module is modified to use a Gaussian distribution with a smaller mean, i.e., t ~ Gaussian(μ, σ). 2 ).

[0204] To focus the model diffusion model on the early stages of the diffusion process and achieve rapid convergence within a smaller time step, we can set μ < 2 / T. μ represents the mean of the conditional Gaussian distribution, and σ represents the standard deviation of the conditional Gaussian distribution, which is less than 1 by default.

[0205] The specific value of the mean can be adjusted based on the sampling time with the largest contribution in the diffusion model. In this embodiment of the invention, the mean of the conditional Gaussian distribution can be adjusted according to the contribution of each model parameter in the diffusion model at different sampling times.

[0206] In practical implementation, the contribution of each model parameter in the diffusion model at different sampling times can be statistically analyzed, and the target sampling time with the highest contribution can be selected. It is then determined whether the mean of the conditional Gaussian distribution is greater than the target sampling time. If the mean of the conditional Gaussian distribution is greater than the target sampling time, the mean of the conditional Gaussian distribution can be further reduced.

[0207] Figure 3 This is a schematic diagram of a conditional Gaussian distribution provided in an embodiment of the present invention. Since the mean of the conditional Gaussian distribution is less than half the training period time, the sampling time distribution is relatively dense in the early sampling stage and relatively sparse in the later sampling stage.

[0208] For example, assuming a training period of 100 seconds, random sampling can start from 0 and take a sampling moment every 2 seconds. Using a conditional Gaussian distribution sampler, sampling might take a moment every 2 seconds initially, every 10 seconds later, and finally every 15 seconds.

[0209] In this embodiment of the invention, considering that the later stages of the diffusion process contribute little to the training of the diffusion model, a Gaussian distribution sampler with a small mean can be used for time sampling during training. This reduces the amount of data sampled in the later stages of the diffusion model and improves the training efficiency of the diffusion model.

[0210] Considering that the random noise generated by the server may not be regular, the noisy image obtained by the diffusion process of the diffusion model may not accurately cover the image characteristics of the training image. Therefore, before using the encoder and decoder and the latent variable space contained in the diffusion model to analyze the time vector sampled by the noisy image and the Gaussian distribution sampler to obtain the noise prediction result, it can be determined whether the random noise meets the condition of regular noise.

[0211] In cases where random noise does not meet the conditions for regular noise, in order to improve the training effect of the diffusion model, training constraint text can be set based on the training images. The noisy images, time vectors, and training constraint text contained in the diffusion model can be analyzed to obtain noise prediction results.

[0212] Figure 4This is a schematic diagram of a process for processing noisy images using an improved noise estimation module, which is provided in an embodiment of the present invention. The improved noise estimation module includes an encoder, an input layer, a latent variable space, an output layer, and a decoder. Figure 4 Taking the first and second latent variables as examples, the noise generator produces random noise, which is added to the training images in a progressively expanding manner to obtain noisy images. The Gaussian sampler samples the time vector. After obtaining the noisy image, time vector, and training constraint text, these can be input into the encoder. The encoder encodes these features to obtain high-dimensional input features. These high-dimensional input features are then transmitted to the first and second latent variables for analysis, resulting in another high-dimensional output feature. This output feature is then transmitted to the decoder via the output layer. The decoder decrypts this output feature to obtain the noise prediction result.

[0213] By adding training constraint text during the training phase of the diffusion model when random noise is irregular, the noise prediction results obtained from the training can better cover the image characteristics of the training image, thereby improving the training effect of the diffusion model. As a result, the noise prediction results can be accurately output by using the trained diffusion model to perform anti-diffusion on the noise prediction results.

[0214] The lower the loss value, the less it contributes to the diffusion model. In this embodiment of the invention, to reduce computational load, sampling times with smaller loss values ​​may be excluded from the contribution calculation.

[0215] In practical applications, before determining the contribution of each model parameter in the diffusion model at different sampling times based on the loss value, model parameters, and parameter gradients at different sampling times, the segmentation time when the loss value is less than a set loss threshold can be determined based on the loss value at different sampling times. For all sampling times before the segmentation time, the step of determining the contribution of each model parameter in the diffusion model at different sampling times based on the loss value, model parameters, and parameter gradients at different sampling times is performed. The contribution value corresponding to the segmentation time and all sampling times after the segmentation time is set to zero.

[0216] The value of the loss threshold can be set based on the actual application requirements of the diffusion model, and is not limited here.

[0217] By determining the segmentation time based on the relationship between the loss value and the loss threshold at different sampling times, the contribution calculation can be divided. Only the contribution value of the sampling time before the segmentation time needs to be calculated, which reduces the overall computational cost of contribution and improves the training efficiency of the diffusion model.

[0218] In this embodiment of the invention, a confidence interval method can be used at the contribution calculation level to determine the contribution to be calculated.

[0219] In practical applications, target sampling times that fall within the confidence interval of a conditional Gaussian distribution can be selected from all sampling times. Based on the loss value, model parameters, and parameter gradients at each target sampling time, the contribution of each model parameter in the diffusion model at each target sampling time can be determined.

[0220] In practical implementation, the contribution calculation formula can be called to analyze the loss value, model parameters, and parameter gradients at each target sampling time to determine the contribution of each model parameter in the diffusion model at each target sampling time. The contribution calculation formula is as follows:

[0221]

[0222] Among them, I(θ ik ,x) represents the contribution of the model parameter in the i-th row and k-th column, θ ik Let represent the model parameters in the i-th row and k-th column of the model parameter matrix of the diffusion model, where x represents the noisy image, μ represents the mean of the conditional Gaussian distribution, σ represents the standard deviation of the conditional Gaussian distribution, and Z represents the standard score in the standard normal distribution table. Let L represent the parameter gradient of the model parameters in the i-th row and k-th column, where t represents the target sampling time. t (θ,x) represents the loss value at sampling time t.

[0223] In this embodiment of the invention, by using a confidence interval approach to calculate the contribution, the contribution of the diffusion model is only calculated at sampling times within the confidence interval, which reduces the overall computational load of the contribution and accelerates computational efficiency.

[0224] Figure 5 This is a schematic diagram of the structure of an image generation device provided in an embodiment of the present invention, including a noise addition unit 51, an analysis unit 52, a determination unit 53, a pruning unit 54, and a noise reduction unit 55;

[0225] The noise-adding unit 51 is used to add noise to the training image based on the random noise output by the noise generator to obtain a noisy image.

[0226] Analysis unit 52 is used to analyze the noisy image and the time vector sampled by the Gaussian distribution sampler using the encoder and decoder and latent variable space contained in the diffusion model to obtain noise prediction results;

[0227] The determination unit 53 is used to determine the contribution of each model parameter in the diffusion model based on the noise prediction results and random noise.

[0228] The pruning unit 54 is used to prune the diffusion model according to the contribution of each model parameter through a pruner to obtain a compressed diffusion model.

[0229] The denoising unit 55 is used to analyze the newly acquired noisy image using the compressed diffusion model to generate a denoised image.

[0230] In some embodiments, the analysis unit includes an encoding subunit, a latent variable analysis subunit, and a decoding subunit;

[0231] The encoding subunit is used to encode the noisy image and time vector based on the encoder contained in the diffusion model to obtain the input features;

[0232] The latent variable analysis subunit is used to analyze the input features using the latent variable space contained in the diffusion model in order to obtain the output features;

[0233] The decoding subunit is used to decode the output features based on the decoder contained in the diffusion model to obtain the noise prediction result.

[0234] In some embodiments, the apparatus includes a sampling unit for the generation of the time vector;

[0235] The sampling unit is used to sample from a conditional Gaussian distribution using a Gaussian distribution sampler to obtain a time vector; wherein the mean of the conditional Gaussian distribution is less than half the training period time.

[0236] In some embodiments, an adjustment unit is also included;

[0237] The adjustment unit is used to adjust the mean of the conditional Gaussian distribution based on the contribution of each model parameter in the diffusion model at different sampling times; wherein, the time vector includes multiple different sampling times.

[0238] In some embodiments, the adjustment unit includes a statistics subunit, a judgment subunit, and a reduction subunit;

[0239] The statistical subunit is used to calculate the contribution of each model parameter in the diffusion model at different sampling times and select the target sampling time with the highest contribution.

[0240] The judgment sub-unit is used to determine whether the mean of the conditional Gaussian distribution is greater than the target sampling time.

[0241] The smaller sub-unit is used to reduce the mean of the conditional Gaussian distribution when the mean of the conditional Gaussian distribution is greater than the target sampling time.

[0242] In some embodiments, the system further includes a noise determination unit and a setting unit;

[0243] The noise judgment unit is used to determine whether random noise meets the conditions for regular noise.

[0244] The setting unit is used to set training constraint text based on the training image when random noise does not meet the condition of regular noise.

[0245] The analysis unit is used to analyze noisy images, temporal vectors, and training constraint text using the encoder-decoder and latent variable space contained in the diffusion model to obtain noise prediction results.

[0246] In some embodiments, the determining unit includes a loss value determining subunit and a contribution determining subunit;

[0247] The loss value determination subunit is used to determine the loss value at different sampling times based on the noise prediction results and random noise at different sampling times.

[0248] The contribution determination subunit is used to determine the contribution of each model parameter in the diffusion model at different sampling times based on the loss value, each model parameter in the diffusion model, and the parameter gradient at different sampling times.

[0249] In some embodiments, the loss value determination subunit is used to perform cross-entropy calculation on the noise prediction results and random noise at different sampling times to obtain the loss value at different sampling times.

[0250] In some embodiments, the system further includes a segmentation time determination unit and a zeroing unit;

[0251] The segmentation time determination unit is used to determine the segmentation time when the loss value is less than a set loss threshold based on the loss value at different sampling times.

[0252] The contribution determination subunit is used to perform the following steps for all sampling times before the segmentation time: based on the loss value, model parameters and parameter gradients at different sampling times, determine the contribution of each model parameter in the diffusion model at different sampling times.

[0253] The zeroing unit is used to set the contribution of the segmentation time and all subsequent sampling times to zero.

[0254] In some embodiments, the contribution determination subunit is used to select target sampling times that are located within the confidence interval of a conditional Gaussian distribution from all sampling times; and to determine the contribution of each model parameter in the diffusion model at each target sampling time based on the loss value, each model parameter in the diffusion model, and the parameter gradient at each target sampling time.

[0255] In some embodiments, the contribution determination subunit is used to call the contribution calculation formula to analyze the loss value, model parameters in the diffusion model, and parameter gradients at each target sampling time, so as to determine the contribution of each model parameter in the diffusion model at each target sampling time; wherein, the contribution calculation formula is as follows:

[0256]

[0257] Among them, I(θ ik ,x) represents the contribution of the model parameter in the i-th row and k-th column, θ ik Let represent the model parameters in the i-th row and k-th column of the model parameter matrix of the diffusion model, where x represents the noisy image, μ represents the mean of the conditional Gaussian distribution, σ represents the standard deviation of the conditional Gaussian distribution, and Z represents the standard score in the standard normal distribution table. Let L represent the parameter gradient of the model parameters in the i-th row and k-th column, where t represents the target sampling time. t (θ,x) represents the loss value at sampling time t.

[0258] In some embodiments, the pruning unit includes an arrangement subunit and a compression subunit;

[0259] The sorting sub-unit is used to sort the contribution of each model parameter in descending order at each sampling time.

[0260] The compression subunit is used to input the model parameters arranged in descending order into the pruner to perform structured pruning on the model parameters that meet the pruning requirements, so as to obtain the compressed diffusion model.

[0261] In some embodiments, the compression subunit is used to perform structured pruning on model parameters whose contribution is less than a threshold at each sampling time to obtain a compressed diffusion model.

[0262] In some embodiments, the compression subunit is used to set the model parameters whose contribution is less than a threshold at each sampling time to zero.

[0263] In some embodiments, the system further includes an average determination unit and a threshold determination unit;

[0264] The mean determination unit is used to determine the mean contribution of each model parameter at each sampling time based on the contribution of each sampling time.

[0265] The threshold determination unit is used to query the correspondence list between contribution range and threshold to determine the threshold that matches the average contribution at different sampling times.

[0266] In some embodiments, the compression subunit is used to set the last N model parameters in each sampling time to zero.

[0267] In some embodiments, the noise-adding unit is used to acquire random noise output by the noise generator; and the random noise is gradually diffused into the training image using a diffusion model to obtain a noisy image.

[0268] In some embodiments, the denoising unit is used to determine whether the newly acquired noise image meets the regular noise condition; if the newly acquired noise image does not meet the regular noise condition, constraint text is set according to the image generation requirements; the newly acquired noise image, time vector and constraint text are input into the compressed diffusion model to obtain the denoised image.

[0269] In some embodiments, the system further includes an extraction unit, a matching unit, a saving unit, and a prompting unit;

[0270] Extraction unit, used to extract the semantic information contained in the denoised image;

[0271] The matching unit is used to determine whether the semantic information of the image matches the semantic information of the image contained in the constraint text.

[0272] A storage unit is used to save the denoised image when the image semantic information matches the image semantic information contained in the constraint text;

[0273] The prompting unit is used to generate a prompt message indicating that the denoised image does not match the constraint text when the semantic information of the image does not match the semantic information of the constraint text.

[0274] Figure 5 For a description of the features in the corresponding embodiments, please refer to Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0275] As can be seen from the above technical solution, the training image is noise-added using random noise output by the noise generator to obtain a noisy image. The noisy image and the time vector sampled by the Gaussian distribution sampler are analyzed using the encoder / decoder and latent variable space contained in the diffusion model to obtain noise prediction results. Based on the noise prediction results and random noise, the contribution of each model parameter in the diffusion model is determined. The diffusion model is then pruned using a pruning tool based on the contribution of each model parameter to obtain a compressed diffusion model. The compressed diffusion model is then used to analyze newly acquired noisy images to generate denoised images. The beneficial effect of this invention is that by compressing the noise training process of the diffusion model into the latent variable space for learning, the number of model parameters in the diffusion model is effectively reduced, improving computational efficiency. Subsequent pruning work will also be performed in the latent variable space, thereby improving the running efficiency of the diffusion model on the server. The compressed diffusion model effectively reduces model parameters and lowers the computational overhead of the image generation process while ensuring image generation quality.

[0276] Figure 6 This is a structural diagram of an image generation device provided in an embodiment of the present invention, such as... Figure 6 As shown, the image generation device includes: a memory 60 for storing computer programs;

[0277] The processor 61 is used to execute computer programs to implement the steps of the image generation method as described in the above embodiments.

[0278] The image generation device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0279] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0280] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the image generation method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, random noise, training images, noisy images, denoised images, etc.

[0281] In some embodiments, the image generating device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0282] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the image generation device and may include more or fewer components than illustrated.

[0283] It is understood that if the image generation method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0284] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image generation method described above.

[0285] The foregoing has provided a detailed description of an image generation method, apparatus, device, and computer-readable storage medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0286] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0287] The foregoing has provided a detailed description of an image generation method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. An image generation method characterized by, The method comprises: adding noise to a training image according to random noise output by a noise generator to obtain a noisy image; analyzing the noisy image and a time vector sampled by a Gaussian distribution sampler according to a codec and a latent variable space included in a diffusion model to obtain a noise prediction result; determining a contribution degree of each model parameter in the diffusion model based on the noise prediction result and the random noise; pruning the diffusion model according to the contribution degrees of the model parameters to obtain a compressed diffusion model; analyzing a newly acquired noisy image using the compressed diffusion model to generate a denoised image; The method further comprises: determining a loss value at different sampling time points based on the noise prediction result and the random noise at the different sampling time points; calculating a linear approximation of a loss disturbance using a Taylor expansion when calculating the loss value; determining the contribution degree of each model parameter in the diffusion model at different sampling time points based on the loss value at the different sampling time points, the model parameters in the diffusion model, and a parameter gradient.

2. The image generation method of claim 1, wherein, The method further comprises: encoding the noisy image and the time vector using an encoder included in the diffusion model to obtain input features; analyzing the input features using a latent variable space included in the diffusion model to obtain output features; decoding the output features using a decoder included in the diffusion model to obtain the noise prediction result.

3. The image generation method of claim 2, wherein, For the generation process of the time vector, the method comprises: sampling from a Gaussian distribution with a condition using the Gaussian distribution sampler to obtain the time vector; wherein a mean value of the Gaussian distribution with the condition is less than half of a training period time.

4. The image generation method of claim 3, wherein, The method further comprises: adjusting the mean value of the Gaussian distribution with the condition according to the contribution degrees of the model parameters in the diffusion model at different sampling time points; wherein the time vector comprises a plurality of different sampling time points.

5. The image generation method of claim 4, wherein, The method further comprises: counting the contribution degrees of the model parameters in the diffusion model at different sampling time points to select a target sampling time point with the highest contribution degree; determining whether the mean value of the Gaussian distribution with the condition is greater than the target sampling time point; in a case where the mean value of the Gaussian distribution with the condition is greater than the target sampling time point, adjusting the mean value of the Gaussian distribution with the condition to be smaller.

6. The image generation method of claim 1, wherein, Before the analyzing the time vector sampled by the Gaussian distribution sampler according to the codec and the latent variable space included in the diffusion model to obtain the noise prediction result, the method further comprises: determining whether the random noise meets a regular noise condition; in a case where the random noise does not meet the regular noise condition, setting a training constraint text according to the training image. The codec and the latent variable space contained in the diffusion model are used to analyze the time vector sampled by the noisy image and the Gaussian distribution sampler to obtain a noise prediction result. The codec and the latent variable space contained in the diffusion model are used to analyze the noisy image, the time vector, and the training constraint text to obtain a noise prediction result.

7. The image generation method of claim 1, wherein, The loss value at different sampling time points is determined based on the noise prediction result and the random noise at different sampling time points. The loss value at different sampling time points is determined by cross-entropy operation on the noise prediction result and the random noise at different sampling time points.

8. The image generation method of claim 1, wherein, Before determining the contribution degree of each model parameter in the diffusion model at different sampling time points based on the loss value at different sampling time points, the model parameters in the diffusion model, and the parameter gradient, the following steps are further included: According to the loss value at different sampling time points, a segmentation time point is determined, at which the loss value is less than a set loss threshold. The step of determining the contribution degree of each model parameter in the diffusion model at different sampling time points based on the loss value at different sampling time points, the model parameters in the diffusion model, and the parameter gradient is performed for all sampling time points before the segmentation time point. The contribution degrees corresponding to the segmentation time point and all sampling time points after the segmentation time point are set to zero.

9. The image generation method of claim 8, wherein, The loss value at different sampling time points is determined based on the noise prediction result and the random noise at different sampling time points. Target sampling time points within the confidence interval of the conditional Gaussian distribution are selected from all the sampling time points. The contribution degree of each model parameter in the diffusion model at each target sampling time point is determined based on the loss value at each target sampling time point, the model parameters in the diffusion model, and the parameter gradient.

10. The image generation method of claim 9, wherein, The contribution degree of each model parameter in the diffusion model at each target sampling time point is determined based on the loss value at each target sampling time point, the model parameters in the diffusion model, and the parameter gradient. The contribution degree of each model parameter in the diffusion model at each target sampling time point is determined based on the loss value at each target sampling time point, the model parameters in the diffusion model, and the parameter gradient. ； wherein, denotes the contribution degree of the model parameter in the i-th row and the k-th column, denotes the model parameter in the i-th row and the k-th column in the model parameter matrix of the diffusion model, x denotes a noisy image, μ denotes a mean of the conditional Gaussian distribution, σ denotes a standard deviation of the conditional Gaussian distribution, and Z denotes a standard score in a standard normal distribution table, denotes the parameter gradient of the model parameter in the i-th row and the k-th column, and t denotes the target sampling time, denotes the loss value at the sampling time t, and θ denotes the model parameter of the diffusion model.

11. The image generation method of claim 1, wherein, The contribution degree of each model parameter in the diffusion model at each target sampling time point is determined based on the loss value at each target sampling time point, the model parameters in the diffusion model, and the parameter gradient. The diffusion model is pruned according to the contribution degree of each model parameter by a pruner to obtain a compressed diffusion model. The contribution degrees of each model parameter at each sampling time point are arranged in descending order.

12. The image generation method of claim 11, wherein, The compressed diffusion model is obtained by inputting the model parameters arranged in descending order into the pruner to perform structural pruning on the model parameters that meet the pruning requirements. The compressed diffusion model is obtained by inputting the model parameters arranged in descending order into the pruner to perform structural pruning on the model parameters that meet the pruning requirements. The model parameters with a contribution degree less than a threshold value among the model parameters at each sampling time are structurally pruned to obtain the compressed diffusion model.

13. The image generation method of claim 12, wherein, The structurally pruning the model parameters with a contribution degree less than a threshold value among the model parameters at each sampling time to obtain the compressed diffusion model comprises: The model parameters with a contribution degree less than a threshold value among the model parameters at each sampling time are set to zero.

14. The image generation method of claim 13, wherein, Before the model parameters with a contribution degree less than a threshold value among the model parameters at each sampling time are set to zero, the method further comprises: According to the contribution degree of each model parameter at each sampling time, a contribution degree average value corresponding to each sampling time is determined; A corresponding relationship list of the contribution degree range and the threshold value is queried to determine the threshold value matched with the contribution degree average value at different sampling times.

15. The image generation method of claim 11, wherein, The method further comprises: The last N model parameters among the model parameters at each sampling time are set to zero.

16. The image generation method of claim 1, wherein, The method further comprises: Random noise output by the noise generator is obtained; The random noise is gradually diffused to the training image by using the diffusion model to obtain the noisy image.

17. The image generation method of claim 1, wherein, The method further comprises: It is determined whether the newly obtained noise image satisfies a regular noise condition; In a case where the newly obtained noise image does not satisfy the regular noise condition, a constraint text is set according to an image generation requirement; The newly obtained noise image, the time vector and the constraint text are input into the compressed diffusion model to obtain the denoised image.

18. The image generation method of claim 17, wherein, The method further comprises: Image semantic information contained in the denoised image is extracted; It is determined whether the image semantic information contained in the denoised image matches the image semantic information contained in the constraint text; In a case where the image semantic information contained in the denoised image matches the image semantic information contained in the constraint text, the denoised image is saved; In a case where the image semantic information contained in the denoised image does not match the image semantic information contained in the constraint text, prompt information that the denoised image does not match the constraint text is generated.

19. An image generation apparatus characterized by comprising: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises The pruning unit is configured to prune the diffusion model by a pruner according to the contribution degrees of the model parameters to obtain a compressed diffusion model. The denoising unit is configured to analyze a newly acquired noisy image by using the compressed diffusion model to generate a denoised image. The determining unit includes a loss value determining subunit and a contribution degree determining subunit. The loss value determining subunit is configured to determine loss values at different sampling moments based on the noise prediction results at the different sampling moments and the random noise, and to calculate linear approximate loss perturbations by using a Taylor expansion when calculating the loss values; and the contribution degree determining subunit is configured to determine contribution degrees of the model parameters in the diffusion model at the different sampling moments based on the loss values at the different sampling moments, the model parameters in the diffusion model, and the parameter gradients.

20. An image generation apparatus characterized by comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image generation method according to any one of claims 1 to 18. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image generation method according to any one of claims 1 to 18. ​ 21. A computer-readable storage medium, characterized in that, ​

Images