A lightweight image super-resolution learning method based on diffusion model

Abstract

The present invention relates to a lightweight image super-resolution learning method based on a diffusion model, and belongs to the field of image super-resolution technology. The method comprises the following steps: S1: forward process; S2: predicting noise; S3: sparse momentum; S4: training process; S5: reverse process. Through the technical solution of the present invention, the speed of image super-resolution can be improved, so that the diffusion model does not need to perform image denoising according to the set maximum time step in the reverse reasoning stage, which helps the diffusion model to optimize the prediction noise mode, and the sparse momentum method adopted can further reduce the parameters of the model, so that the model can achieve the performance of a dense network under a sparse network, significantly improving the speed and efficiency of image super-resolution, while maintaining high-precision output on the basis of reducing computing requirements.

Description

Technical Field

[0001] The present invention relates to a lane line detection method, and in particular to a lightweight image super-resolution learning method based on a diffusion model, and belongs to the technical field of image super-resolution. Background Art

[0002] In the field of image super-resolution, traditional techniques aim to restore high-resolution details from low-resolution images and face many limitations. First, interpolation-based techniques, such as bilinear or bicubic interpolation, are easy to implement, but they tend to smooth, which leads to loss of high-frequency details, and the final output image has blurred edges and poor texture information. Secondly, reconstruction-based models strive to reproduce the features learned in the training set on new data, but often encounter generalization difficulties, which manifests as unnatural structures, blurred edges, and missing details in the generated images. In addition, although deep learning-based methods can achieve remarkable results, their high dependence on large-scale annotated datasets and high-performance computing resources has become a major bottleneck in practical applications, and model performance is highly dependent on the quality of training data.

[0003] In recent years, diffusion models (DMs), as an emerging category of generative models, have shown great potential in generating high-fidelity images by simulating the gradual noise diffusion process of data and its inverse denoising path, especially in high-resolution image synthesis. Compared with traditional techniques, DMs can generate more diverse and realistic image content, but in real-time applications and resource-constrained environments, the complex iterative operations and high computational requirements of the original diffusion model constitute a significant obstacle.

[0004] In view of the above problems, the present invention introduces an innovative image processing method, which cleverly integrates the jump coding optimization model (JSEO) to optimize the feature representation ability, and adopts a momentum update strategy inspired by the sparse optimization idea. The jump coding optimization model (JSEO) is composed of a Unet model, which mainly uses dynamic skipping and parallel technology for optimization. The JSEO model uses the Unet architecture with jump connections to gradually refine clear image features from the full noise image during the image denoising process. The model significantly accelerates denoising reasoning and improves the execution efficiency of super-resolution tasks by reusing early coding features in the decoding stage. The sparse momentum model introduces a momentum-guided weight pruning and redistribution strategy, uses exponentially smoothed gradient information to identify key weights, and realizes efficient optimization of the network structure. The specific operations include: weight pruning according to the average momentum size of the weights of each layer, and redistributing the remaining weights to the more influential layers; and increasing the weights at the zero-value weight position according to the momentum indicator, thereby accelerating the model training process without sacrificing performance.

[0005] By integrating JSEO and sparse momentum, the present invention not only solves the speed-accuracy balance problem of super-resolution tasks, but also provides a feasible solution for real-time and resource-sensitive application scenarios, marking an important step forward towards efficient and high-quality image super-resolution technology. Summary of the invention

[0006] The purpose of the present invention is to provide a lightweight image super-resolution learning method based on a diffusion model to solve the problems of complex iterative operations and high computing requirements in the prior art.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A lightweight image super-resolution learning method based on a diffusion model comprises the following steps:

[0009] S1. Forward process: Input the low-resolution image LR, the magnification factor s and the corresponding high-resolution image x0, add noise to x0 through T accumulation until it becomes a completely noisy x T , which is used for subsequent noise prediction.

[0010] S2. Predict noise: First, the corresponding time t and the image with noise are input into Unet. Unet performs in-depth analysis and fusion of the input time step information and image features, and then predicts the noise to obtain the noise map Z of a single time step.

[0011] S3. Sparse momentum: Before model training, the model is added to the mask for mask operation, and the weights are sparse through pruning, cross-layer weight configuration, and weight regeneration;

[0012] S4. Training process: First, calculate the noise of the corresponding time step, then calculate the loss, and gradually reduce the loss by continuously adjusting the model parameters. Repeat the above steps until the model reaches satisfactory performance indicators and the preset number of training times.

[0013] S5. Reverse process: Starting from a randomly generated noise tensor y with the same shape as the high-resolution image x0, noise prediction is performed through the Unet network. In the Unet network, each e time steps is set as a cycle, and a total of F cycles are set. The noise map of e time steps is generated at a time through the skipping and parallel methods, and then denoising is performed to finally output a high-resolution image.

[0014] Furthermore, the forward process specifically includes: inputting a low-resolution image LR, a magnification factor s, and a corresponding high-resolution image x0, extracting abstract features from the low-resolution image LR through a deep residual network EDSR to form a feature map f, and generating a random noise tensor y that matches the shape of LR, concatenating the feature map f with the noise tensor y to obtain a composite feature representation c, and the calculation formula is as follows:

[0015] ,

[0016] ,

[0017] ,

[0018] ,

[0019] Among them, x t represents the data state at time step t, is the noise variance at time step t, represents a high-resolution image, EDSR(.) represents a deep residual network, and conditional probability distribution Represents the state of the data at a given previous time step In the case of The distribution of , the mean is , the variance is ,in is the identity matrix, the joint probability distribution From the initial state At the beginning, the distribution of the entire diffusion process after T time steps is, Represents the concatenation operation. After obtaining the composite feature representation c, randomly select a time step and add the noise y to x0 to generate a noisy image x noise .

[0020] Furthermore, the noise prediction specifically includes: transforming the noise image x noise and the corresponding time step t n Input to Unet, the incoming time step t n After an MLP, it is mapped to the corresponding noise intensity , x noiseFirst, downsample, pass through an intermediate layer B consisting of two residual blocks, then upsample to get f(1), and finally pass through a Block convolution block. The convolution block first performs group normalization on the input, then applies the Swish activation function, and finally passes through a 3x3 convolution layer to get the predicted noise Z for one time step. The calculation formula is as follows:

[0021] ,

[0022] ,

[0023] ,

[0024] Among them, x noise represents a noise image, f(1) represents the feature map obtained after downsampling and upsampling, Z represents the feature map of noise obtained after f(1) is subjected to group normalization and Swish activation function, Up(.) represents an upsampling operation, and Down(.) represents a downsampling operation. Indicates the noise intensity, t n represents the corresponding time step, MLP(.) is a multi-layer perceptron.

[0025] Furthermore, the sparse momentum specifically includes:

[0026] Smooth gradient: Input image, initialize network weights, calculate weights of each layer, and use exponential smoothing to accelerate convergence. The calculation formula is as follows:

[0027] ,

[0028] Where α is the smoothing factor, Represents the weight in the i-th layer The momentum of represents the loss function of the pth iteration, p represents the number of iterations, Indicates that at the pth iteration, the weight The momentum value of Initialized to 0, this is the loss function About weight The partial derivative at the pth iteration, i.e. the gradient;

[0029] Momentum contribution: Take the sum of the momentum of the non-zero weights of each layer, divide it by the sum of the momentum of the weights of the total layer, and then normalize it to get the momentum contribution of each layer. The formula is as follows:

[0030] ,

[0031] in, is the total number of weights of the i-th layer, L represents the total number of layers in the network, is the momentum of the jth weight in the i-th layer, represents the contribution of momentum to the i-th layer, is the momentum of the jth weight in the kth layer, represents the total number of weights of the kth layer;

[0032] Weight initialization: Initialize the pruning rate, pruning mode, regrowth mode, pruning rate decay speed, and weight distribution strategy. Set a mask tensor for each parameter of the model. 1 indicates that the weight should be retained, and 0 indicates that the weight should be removed. Through pruning based on the weight amplitude, decide whether the weight should be retained;

[0033] Cross-layer weight configuration: Identify all non-zero weight parameters, obtain the momentum vectors corresponding to these non-zero weights through the SGD optimizer, calculate the average non-zero weight momentum of each layer, and sort the contribution rates based on the momentum contribution of each layer to get a list of absolute weight values. The layers with low contribution rates are the absolute values ​​of the last 85% of the weights in the list of absolute weight values. Remove 85% of the weights from the layers with low contribution rates and allocate these weights according to C. i Redistribute to other layers, the cross-layer weight configuration process formula is as follows:

[0034] ,

[0035] ,

[0036] represents the number of new weights assigned to the i-th layer for regrowth, A i represents the average momentum value of the i-th layer, C i represents the average momentum contribution of the i-th layer, R represents the total number of weights that need to be regenerated, represents the sum of the average momentum contributions of all layers;

[0037] Weight regeneration: New weights are regenerated at the position with the largest momentum contribution rate among the pruned weight positions. The new weights are reactivated at the position with the greatest potential to reduce errors indicated by momentum. The number of regenerated weights is determined by the total number of pruned weights in the entire network and the momentum contribution ratio of each layer. The number of weights regenerated in each layer is equal to the momentum contribution ratio of the layer multiplied by the total number of weights that need to be regenerated. Then, the weights that are regenerated are determined based on the momentum contribution of each layer and the number of removed weights. The calculation formula for the weight regeneration process is as follows:

[0038] ,

[0039] ,

[0040] ,

[0041] represents the expected number of newly assigned weights on the i-th layer, represents the number of weights actually newly assigned on the i-th layer, represents the sparsity, Indicates the sparsity and The minimum value of L represents the total number of layers in the network, R allocated is the total weight actually allocated according to the vacancy limit, R excess It is the total weight amount that exceeds the allocable spaces, that is, the excess weight. After the above steps, the weight distribution is dynamically adjusted.

[0042] Furthermore, the training process specifically includes: and the corresponding time step t n Input into Unet for denoising ,calculate The loss between y is as follows:

[0043] ,

[0044] Where n is the number of samples, Indicates the absolute value operation. is the random noise added at the i-th time step, is the estimated value of the i-th time step after processing by Unet noise.

[0045] Furthermore, the reverse process specifically includes:

[0046] Transfer time step: In the reverse reasoning process of the diffusion model, the total time step of the model is set to T time steps in Unet, where e time steps are set as a cycle, and the first time step in the cycle is selected is the critical time step, and the rest of the time steps are non-critical time steps. A time step attribute is set for Unet. In the cyclic sampling step of the reverse diffusion of the diffusion model, the time steps of each cycle are stored as a list, and then the critical time steps of each cycle are dynamically passed to Unet to change the attribute value of the time step of Unet;

[0047] Skip step: The feature graph obtained by the convolutional neural network during the Unet network training process is saved in the key time step, and the current time step is judged to be the key time step. The encoding operation of the encoder E is performed in the key time step, and the encoding operation of the encoder E is not performed in the non-key time step. The decoder D reuses the features of the encoder E in the key time step of the current cycle. In the skip encoder optimization model JSEO, the decoder D reuses the features of the encoder E in the key time step of the current cycle. , perform the complete encoding operation, that is, use the encoder , including downsampling operations , 2, E3, encodes the features and passes them to the decoder through an intermediate layer B with two residual blocks , decoder D has upsampling operation 1, 2, D3, perform upsampling operation, at non-critical time steps , reuse the previous key time step Encoder The characteristics of characteristics.

[0048] Parallel: Input Copy a cycle along the batch dimension, select key time steps from the copied tensor and put the key time steps into a list, convert each time step in the list into a tensor, copy each time step tensor obtained on the batch dimension, the number of copies is half of the cycle, then concatenate the obtained time step tensors into a tensor on the batch dimension, and finally copy it on the batch dimension to obtain a noise set of a cycle , the calculation formula is as follows:

[0049] ,

[0050] in, represents a noise image as the initial data tensor, ⊕ represents the concatenation operation of the tensor, is the set of key time steps, represents the operation of selecting the key time step k, Indicates a transformation that replicates half the length of a period in a particular dimension;

[0051] The noise set According to the preset number of cycles F, each cycle is divided into e batches, where each batch corresponds to the noise of one time step. The noise map of e time steps is obtained by noise prediction on y. y and the last noise map are taken out, and the denoising operation is performed using the following formula to obtain The characteristic graph at time x T-1 , repeat the denoising process until , and finally obtain the super-resolution image , the specific formula is as follows:

[0052] ,

[0053] ,

[0054] in, represents the total time step, represents the number of time steps to calculate a cycle, Indicates rounding up operation, x t represents the diffusion state of x0 at time step t, represents a coefficient related to time step t, which is related to the noise variance. is the noise variance coefficient at time step t, yes The cumulative multiplication of The Unet model is based on the current diffusion state and the noise of the prediction at time step t, is the coefficient to adjust the prediction noise, is the coefficient The inverse square root of Repeat this step to finally get a super-resolution image .

[0055] The advantages of the present invention are:

[0056] The JSEO model of the present invention combines with the Unet model through dynamic skipping operation, so that it can effectively save the encoding time of the encoder in the process of reverse reasoning, and can accelerate the speed of image super-resolution. The skipping connection in the traditional UNet architecture indiscriminately passes all low-level features to the high-level. This method explores ways to use these connections more intelligently, such as selectively fusing or adjusting skipping information to more efficiently utilize computing resources and promote information flow, which helps to improve the quality and diversity of generated samples. At the same time, the JSEO model introduces a parallel processing mechanism so that all key denoising steps in a recovery cycle can be executed synchronously. This design greatly compresses the time required for image denoising and further accelerates the overall processing flow. In response to the deployment requirements of resource-constrained environments, the present invention cleverly uses sparse momentum technology, which is an advanced parameter optimization strategy. By analyzing the average momentum contribution of each network layer, and selectively reducing the weight accordingly, and then implementing the weight regrowth process, the JSEO model can significantly reduce the number of parameters and computational complexity while ensuring that the model performance does not decay, and achieve an accuracy level that is equal to or even beyond the original dense model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0058] Figure 1 It is the super-resolution diffusion model SRDM of the present invention;

[0059] Figure 2 Optimizing the JSEO model for the skip encoder of the present invention;

[0060] Figure 3 It is a parallel model of the present invention;

[0061] Figure 4 A sparse momentum network of the present invention;

[0062] Figure 5 This is a visualization result diagram of the comparative experiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Example 1

[0065] Combine the following Figures 1 to 4 A lightweight image super-resolution learning method based on a diffusion model implemented by the present invention is specifically described.

[0066] In this embodiment, the present invention provides a lightweight image super-resolution learning method based on a diffusion model, and the specific steps include:

[0067] 1) Forward process: Input the low-resolution image LR, the magnification factor s, and the corresponding high-resolution image x0, and add noise to x0 through T accumulation until it becomes a completely noisy x T , which is used for subsequent noise prediction.

[0068] 2) Predicting noise: First, the corresponding time t and the image with noise are input into Unet. Unet performs in-depth analysis and fusion of the input time step information and image features, and then predicts the noise to obtain the noise map Z of a single time step.

[0069] 3) Sparse momentum: Before model training, the model is added to the mask for mask operation, and the weights are sparse through pruning, cross-layer weight configuration, and weight regeneration;

[0070] 4) Training process: First, calculate the noise of the corresponding time step, then calculate the loss, and gradually reduce the loss by continuously adjusting the model parameters. Repeat the above steps until the model reaches satisfactory performance indicators and the preset number of training times.

[0071] 5) Reverse process: Starting from a randomly generated noise tensor y with the same shape as the high-resolution image x0, noise prediction is performed through the Unet network. In the Unet network, each e time steps is set as a cycle, and a total of F cycles are set. The noise map of e time steps is generated at a time through skipping and parallel methods, and then denoising is performed to finally output a high-resolution image.

[0072] Specifically, the forward process includes: inputting a low-resolution image LR, a magnification factor s, and a corresponding high-resolution image x0, extracting abstract features from the low-resolution image LR through a deep residual network EDSR to form a feature map f, and generating a random noise tensor y that matches the shape of LR, concatenating the feature map f with the noise tensor y to obtain a composite feature representation c, and the calculation formula is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] Among them, x t represents the data state at time step t, is the noise variance at time step t, represents a high-resolution image, EDSR(.) represents a deep residual network, and conditional probability distribution Represents the state of the data at a given previous time step In the case of The distribution of , the mean is , the variance is ,in is the identity matrix, the joint probability distribution From the initial state At the beginning, the distribution of the entire diffusion process after T time steps is, Represents the concatenation operation. After obtaining the composite feature representation c, randomly select a time step and add the noise y to x0 to generate a noisy image x noise .

[0078] Specifically, the noise prediction comprises: transforming the noise image x noise and the corresponding time step t n Input to Unet, the incoming time step t n After an MLP, it is mapped to the corresponding noise intensity , x noise First, downsample, pass through an intermediate layer B consisting of two residual blocks, then upsample to get f(1), and finally pass through a Block convolution block. The convolution block first performs group normalization on the input, then applies the Swish activation function, and finally passes through a 3x3 convolution layer to get the predicted noise Z for one time step. The calculation formula is as follows:

[0079]

[0080]

[0081]

[0082] Among them, x noise represents a noise image, f(1) represents the feature map obtained after downsampling and upsampling, Z represents the feature map of noise obtained after f(1) is subjected to group normalization and Swish activation function, Up(.) represents an upsampling operation, and Down(.) represents a downsampling operation. Indicates the noise intensity, t n represents the corresponding time step, MLP(.) is a multi-layer perceptron.

[0083] Specifically, the sparse momentum specifically includes:

[0084] Smooth gradient: Input image, initialize network weights, calculate weights of each layer, and use exponential smoothing to accelerate convergence. The calculation formula is as follows:

[0085] ,

[0086] Where α is the smoothing factor, Represents the weight in the i-th layer The momentum of represents the loss function of the pth iteration, p represents the number of iterations, Indicates that at the pth iteration, the weight The momentum value of Initialized to 0, this is the loss function About weight The partial derivative at the pth iteration, i.e. the gradient;

[0087] Momentum contribution: Take the sum of the momentum of the non-zero weights of each layer, divide it by the sum of the momentum of the weights of the total layer, and then normalize it to get the momentum contribution of each layer. The formula is as follows:

[0088] ,

[0089] in, is the total number of weights of the i-th layer, L represents the total number of layers in the network, is the momentum of the jth weight in the i-th layer, represents the contribution of momentum to the i-th layer, is the momentum of the jth weight in the kth layer, represents the total number of weights of the kth layer;

[0090] Weight initialization: Initialize the pruning rate, pruning mode, regrowth mode, pruning rate decay speed, and weight distribution strategy. Set a mask tensor for each parameter of the model. 1 indicates that the weight should be retained, and 0 indicates that the weight should be removed. Through pruning based on the weight amplitude, decide whether the weight should be retained;

[0091] Cross-layer weight configuration: Identify all non-zero weight parameters, obtain the momentum vectors corresponding to these non-zero weights through the SGD optimizer, calculate the average non-zero weight momentum of each layer, and sort the contribution rates based on the momentum contribution of each layer to get a list of absolute weight values. The layers with low contribution rates are the absolute values ​​of the last 85% of the weights in the list of absolute weight values. Remove 85% of the weights from the layers with low contribution rates and allocate these weights according to C. i Redistribute to other layers, the cross-layer weight configuration process formula is as follows:

[0092] ,

[0093] ,

[0094] represents the number of new weights assigned to the i-th layer for regrowth, A i represents the average momentum value of the i-th layer, C i represents the average momentum contribution of the i-th layer, R represents the total number of weights that need to be regenerated, represents the sum of the average momentum contributions of all layers;

[0095] Weight regeneration: New weights are regenerated at the position with the largest momentum contribution rate among the pruned weight positions. The new weights are reactivated at the position with the greatest potential to reduce errors indicated by momentum. The number of regenerated weights is determined by the total number of pruned weights in the entire network and the momentum contribution ratio of each layer. The number of weights regenerated in each layer is equal to the momentum contribution ratio of the layer multiplied by the total number of weights that need to be regenerated. Then, the weights that are regenerated are determined based on the momentum contribution of each layer and the number of removed weights. The calculation formula for the weight regeneration process is as follows:

[0096] ,

[0097] ,

[0098] ,

[0099] represents the expected number of newly assigned weights on the i-th layer, represents the number of weights actually newly assigned on the i-th layer, represents the sparsity, Indicates the sparsity and The minimum value of L represents the total number of layers in the network, R allocated is the total weight actually allocated according to the vacancy limit, R excess It is the total weight amount that exceeds the allocable spaces, that is, the excess weight. After the above steps, the weight distribution is dynamically adjusted.

[0100] Specifically, the training process includes: and the corresponding time step t n Input into Unet for denoising ,calculate The loss between y is as follows:

[0101]

[0102] Where n is the number of samples, Indicates the absolute value operation. is the random noise added at the i-th time step, is the estimated value of the i-th time step after processing by Unet noise.

[0103] Specifically, the reverse process includes:

[0104] Transfer time step: In the reverse reasoning process of the diffusion model, the total time step of the model is set to T time steps in Unet, where e time steps are set as a cycle, and the first time step in the cycle is selected is the critical time step, and the rest of the time steps are non-critical time steps. A time step attribute is set for Unet. In the cyclic sampling step of the reverse diffusion of the diffusion model, the time steps of each cycle are stored as a list, and then the critical time steps of each cycle are dynamically passed to Unet to change the attribute value of the time step of Unet;

[0105] Skip step: The feature graph obtained by the convolutional neural network during the Unet network training process is saved in the key time step, and the current time step is judged to be the key time step. The encoding operation of the encoder E is performed in the key time step, and the encoding operation of the encoder E is not performed in the non-key time step. The decoder D reuses the features of the encoder E in the key time step of the current cycle. In the skip encoder optimization model JSEO, the decoder D reuses the features of the encoder E in the key time step of the current cycle. , perform the complete encoding operation, that is, use the encoder , including downsampling operations , 2, E3, encodes the features and passes them to the decoder through an intermediate layer B with two residual blocks , decoder D has upsampling operation 1, 2, D3, perform upsampling operation, at non-critical time steps , reuse the previous key time step Encoder The characteristics of characteristics.

[0106] Parallel: Input Copy a cycle along the batch dimension, select the key time step from the copied tensor and write the key time step Put it into a list, convert each time step in the list into a tensor, copy each time step tensor in the batch dimension, the number of copies is half the period, and then splice the time step tensors into a tensor in the batch dimension, and finally copy them in the batch dimension. For example, we have After calculating the characteristics of the encoder, Can be reused The characteristics of the moment, while decoding the encoder E of a period, we can get a noise set of a period at one time , the calculation formula is as follows:

[0107] ,

[0108] in, represents a noise image as the initial data tensor, ⊕ represents the concatenation operation of the tensor, is the set of key time steps, represents the operation of selecting the key time step k, Indicates a transformation that replicates half the length of a period in a particular dimension;

[0109] The noise set According to the preset number of cycles F, each cycle is divided into e batches, where each batch corresponds to the noise of one time step. The noise map of e time steps is obtained by noise prediction on y. y and the last noise map are taken out, and the denoising operation is performed using the following formula to obtain The characteristic graph at time x T-1 , repeat the denoising process until , and finally obtain the super-resolution image , the specific formula is as follows:

[0110] ,

[0111] ,

[0112] in, represents the total time step, represents the number of time steps to calculate a cycle, Indicates rounding up operation, x t represents the diffusion state of x0 at time step t, represents a coefficient related to time step t, which is related to the noise variance. is the noise variance coefficient at time step t, yes The cumulative multiplication of The Unet model is based on the current diffusion state and the noise of the prediction at time step t, is the coefficient to adjust the prediction noise, is the coefficient The inverse square root of Repeat this step to finally get a super-resolution image .

[0113] In this embodiment, the skip encoder optimization model JSEO and sparse momentum are combined: when training the model, we add sparse momentum, use the average momentum of each layer to prune the model weights, and then increase the model weights. In this way, during the training process, the model can achieve the same performance as the dense network with sparse weights. In addition, in the reverse reasoning process of the diffusion model, the skip encoder optimization model sets a cycle as e time steps. Assuming that the time step is is the critical time step, and the remaining arrive For non-critical time steps, we perform the encoding operation of the encoder on the critical time step, and reuse the encoder features of the critical time step on the non-critical time step. In addition, we can calculate the noise map of one cycle in parallel at one time, and iteratively denoise the noise map Z of one cycle generated by Unet, which can effectively save the encoding time of the encoder and speed up the process of image super-resolution.

[0114] Example 2

[0115] Table 1

[0116]

[0117] In this embodiment, in order to verify the effectiveness of the jump encoder optimization model (JSEO) and sparse momentum, this paper conducted comparative experiments with three other models for image super-resolution tasks. These models include SR3, Regression, and IDM. All models use a unified window size of 16×16 and a magnification of 8. The experiment was conducted on the Celeba-HQ dataset. PSNR (Peak Signal-to-Noise Ratio): The higher the PSNR value, the smaller the difference between the reconstructed image and the original image, that is, the better the image quality. The experimental results show that the SRDM model achieves the highest score on PSNR, which means that the model can generate super-resolution images that are closest to the real image. SSIM (Structural Similarity Index): SSIM takes into account the structural information of the image. The closer the value is to 1, the more similar the images are. SRDM also performs well on the SSIM indicator, which shows that the model can not only accurately restore image details, but also maintain the structural consistency of the image. Number of non-zero parameters: The number of non-zero parameters of the model reflects the complexity and sparsity of the model. The SRDM model achieves fewer non-zero parameters through sparse momentum, which helps reduce the demand for computing resources. The number of MACs (multiplication-accumulation operations): The number of MACs is an important indicator to measure the computational complexity of the model. The SRDM model shows a lower number of MACs, which further confirms the high efficiency of the model. We named the model that combines sparse momentum with JSEO SRDM (Super-Resolution Diffusion Model). Among all four models, SRDM achieved the best results in each indicator, which strongly verified the effectiveness and superiority of our proposed model.

[0118] In this embodiment, in order to illustrate the effectiveness of the method of the present invention, we selected three groups of comparison images: Figure 5This is a visualization result diagram of the comparative experiment of the present invention. The first column is the low-resolution image LR, the last column is the high-resolution image HR, and the middle column is the image generated by the method of the present invention and the other three methods. It can be seen from the figure that compared with the other three methods, the super-resolution image generated by the SRDM of the present invention has the highest clarity, which shows that the denoising effect of the method of the present invention is better.

[0119] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lightweight image super-resolution learning method based on a diffusion model, characterized in that: The following steps are involved: S1. Forward process: Input the low-resolution image LR, the magnification factor s and the corresponding high-resolution image x0, add noise to x0 through T accumulations until it becomes a completely noisy x T , used for subsequent noise prediction; S2. Predict noise: First, the corresponding time t and the image with noise are input into Unet. Unet performs in-depth analysis and fusion of the input time step information and image features, and then predicts the noise to obtain the noise map Z of a single time step. S3. Sparse momentum: Before model training, the model is added to the mask for mask operation, and the weights are sparse through pruning, cross-layer weight configuration, and weight regeneration; S4. Training process: First, calculate the noise of the corresponding time step, then calculate the loss, and gradually reduce the loss by continuously adjusting the model parameters. Repeat the above steps until the model reaches a satisfactory performance index and the preset number of training times; S5. Reverse process: Starting from a randomly generated noise tensor y with the same shape as the high-resolution image x0, noise prediction is performed through the Unet network. In the Unet network, each e time steps is set as a cycle, and a total of F cycles are set. The noise map of e time steps is generated at a time through the skipping and parallel methods, and then denoising is performed to finally output the super-resolution image.

2. According to the light-weight image super-resolution learning method based on diffusion model in claim 1, it is characterized in that: The forward process in step S1 specifically includes: Input the low-resolution image LR, the magnification factor s and the corresponding high-resolution image x0. The low-resolution image LR is subjected to the deep residual network EDSR to extract abstract features to form a feature map f, and a random noise tensor y that matches the shape of LR is generated. The above feature map f is concatenated with the noise tensor y to obtain a composite feature representation c. The calculation formula is as follows: ， ， ， ， Among them, x t represents the data state at time step t, is the noise variance at time step t, represents a high-resolution image, EDSR(.) represents a deep residual network, and conditional probability distribution Represents the state of the data at a given previous time step In the case of The distribution of , the mean is , the variance is ,in is the identity matrix, the joint probability distribution From the initial state At the beginning, the distribution of the entire diffusion process after T time steps is, Represents the concatenation operation. After obtaining the composite feature representation c, randomly select a time step and add the noise y to x0 to generate a noisy image x noise .

3. According to the light-weight image super-resolution learning method based on diffusion model in claim 1, it is characterized in that: The noise prediction in step S2 specifically includes: The noise image x noise and the corresponding time step t n Input to Unet, the incoming time step t n After an MLP, it is mapped to the corresponding noise intensity , x noise First, downsample, pass through an intermediate layer B consisting of two residual blocks, then upsample to get f(1), and finally pass through a Block convolution block. The convolution block first performs group normalization on the input, then applies the Swish activation function, and finally passes through a 3x3 convolution layer to get the predicted noise Z for one time step. The calculation formula is as follows: ， ， ， Among them, x noise represents a noise image, f(1) represents the feature map obtained after downsampling and upsampling, Z represents the feature map of noise obtained after f(1) is subjected to group normalization and Swish activation function, Up(.) represents an upsampling operation, and Down(.) represents a downsampling operation. Indicates the noise intensity, t n represents the corresponding time step, MLP(.) is a multi-layer perceptron.

4. According to the light-weight image super-resolution learning method based on diffusion model in claim 1, it is characterized in that: The sparse momentum in step S3 specifically includes: S31. Smooth gradient: Input image, initialize network weights, calculate weights of each layer, and use exponential smoothing to accelerate convergence. The calculation formula is as follows: ， Where α is the smoothing factor, Represents the weight in the i-th layer The momentum of represents the loss function of the pth iteration, p represents the number of iterations, Indicates that at the pth iteration, the weight The momentum value of Initialized to 0, this is the loss function About weight The partial derivative at the pth iteration, i.e. the gradient; S32. Momentum contribution: Take the sum of the momentum of the non-zero weights of each layer, divide it by the sum of the momentum of the weights of the total layer, and then normalize it to get the momentum contribution of each layer. The formula is as follows: ， in, is the total number of weights of the i-th layer, L represents the total number of layers in the network, is the momentum of the jth weight in the i-th layer, represents the contribution of momentum to the i-th layer, is the momentum of the jth weight in the kth layer, represents the total number of weights of the kth layer; S33. Weight initialization: Initialize the pruning rate, pruning mode, regrowth mode, pruning rate decay speed, and weight allocation strategy. Set a mask tensor for each parameter of the model. 1 indicates that the weight should be retained, and 0 indicates that the weight should be removed. Through pruning based on the weight amplitude, decide whether the weight should be retained; S34. Cross-layer weight configuration: Identify all non-zero weight parameters, obtain the momentum vectors corresponding to these non-zero weights through the SGD optimizer, calculate the average value of the non-zero weight momentum of each layer, and sort the contribution rates based on the momentum contribution of each layer, i.e., the contribution rate, to obtain a list of absolute weight values. The layers with low contribution rates are the absolute values ​​of the last 85% of the weights in the list of absolute weight values. Remove 85% of the weights from the layers with low contribution rates, and assign these weights according to C. i Redistribute to other layers, the cross-layer weight configuration process formula is as follows: ， ， represents the number of new weights assigned to the i-th layer for regrowth, A i represents the average momentum value of the i-th layer, C i represents the average momentum contribution of the i-th layer, R represents the total number of weights that need to be regenerated, represents the sum of the average momentum contributions of all layers; S35. Weight regeneration: regenerate new weights at the position with the largest momentum contribution rate among the pruned weight positions. The new weights are reactivated at the position with the greatest potential to reduce errors indicated by momentum. The number of regenerated weights is determined by the total number of pruned weights in the entire network and the momentum contribution ratio of each layer. The number of weights regenerated in each layer is equal to the momentum contribution ratio of the layer multiplied by the total number of weights that need to be regenerated. Then, the weights that are regenerated are determined based on the momentum contribution of each layer and the number of removed weights. The calculation formula for the weight regeneration process is as follows: ， ， ， represents the expected number of newly assigned weights on the i-th layer, represents the number of weights actually newly assigned on the i-th layer, represents the sparsity, Indicates the sparsity and The minimum value of L represents the total number of layers in the network, R allocated is the total weight actually allocated according to the vacancy limit, R excess It is the total weight amount that exceeds the allocable spaces, that is, the excess weight. After the above steps, the weight distribution is dynamically adjusted.

5. The method for lightweight image super-resolution learning based on diffusion model according to claim 1, characterized in that: The training process in step S4 specifically includes: Will and the corresponding time step t n Input into Unet for denoising ,calculate The loss between y is as follows: ， Where n is the number of samples, Indicates the absolute value operation. is the random noise added at the i-th time step, is the estimated value of the i-th time step after processing by Unet noise.

6. The method for learning lightweight image super-resolution based on a diffusion model according to claim 1, characterized in that: The step S5 specifically includes: S51. Transfer time step: In the reverse reasoning process of the diffusion model, the total time step of the model is set to T time steps in Unet, where e time steps are set as a cycle, and the first time step in the cycle is selected is the critical time step, and the rest of the time steps are non-critical time steps. A time step attribute is set for Unet. In the cyclic sampling step of the reverse diffusion of the diffusion model, the time steps of each cycle are stored as a list, and then the critical time steps of each cycle are dynamically passed to Unet to change the attribute value of the time step of Unet; S52. Skip step: Save the feature map obtained by the convolutional neural network during the Unet network training process in the key time step, determine whether the current time step is a key time step, perform the encoding operation of encoder E in the key time step, and do not perform the encoding operation of encoder E in the non-key time step. Decoder D reuses the features of encoder E in the key time step of the current cycle. In the skip encoder optimization model JSEO, at the key time step , perform the complete encoding operation, i.e. use the encoder , including downsampling operations , 2, E3, encodes the features and passes them to the decoder through an intermediate layer B with two residual blocks , decoder D has upsampling operation 1, 2, D3, perform upsampling operation, at non-critical time steps , reuse the previous key time step Encoder The characteristics of Features; S53. Parallel: Input Copy a cycle along the batch dimension, select key time steps from the copied tensor and put the key time steps into a list, convert each time step in the list into a tensor, copy each time step tensor obtained on the batch dimension, the number of copies is half of the cycle, then concatenate the obtained time step tensors into a tensor on the batch dimension, and finally copy it on the batch dimension to obtain a noise set of a cycle , the calculation process formula is as follows: ， in, represents a noise image as the initial data tensor, ⊕ represents the concatenation operation of the tensor, is the set of key time steps, represents the operation of selecting the key time step k, Indicates a transformation that replicates half the length of a period in a particular dimension; The noise set According to the preset number of cycles F, each cycle is divided into e batches, where each batch corresponds to the noise of one time step. The noise map of e time steps is obtained by noise prediction on y. y and the last noise map are taken out, and the denoising operation is performed using the following formula to obtain The characteristic graph at time x T-1 , repeat the denoising process until , and finally obtain the super-resolution image , the specific formula is as follows: ， ， in, represents the total time step, represents the number of time steps to calculate a cycle, Indicates rounding up operation, x t represents the diffusion state of x0 at time step t, represents a coefficient related to time step t, which is related to the noise variance. is the noise variance coefficient at time step t, yes The cumulative multiplication of The Unet model is based on the current diffusion state and the noise of the prediction at time step t, is the coefficient to adjust the prediction noise, is the coefficient The inverse square root of Repeat this step to finally get a super-resolution image .

Images