Image super-resolution method based on frequency domain residual diffusion model

Abstract

The invention discloses an image super-resolution method based on a frequency domain residual diffusion model. The image super-resolution method comprises the following steps: selecting a frequency domain wavelet signal of a low-resolution image as the input of a first segment of residual CNN network to generate a high-frequency component and a low-frequency component of an enhanced image; the difference value between the high-frequency component of the enhanced image and the high-frequency component of the high-resolution target image is used for training a noise estimation network in a diffusion model; generating an estimated residual high-frequency component by using a noise estimation network; adding the estimated residual high-frequency component and the high-frequency component of the enhanced image, and then superposing the low-frequency component of the enhanced image to obtain a high-frequency component and a low-frequency component of a final image; and performing inverse discrete wavelet transform on the obtained high-frequency component and low-frequency component to obtain a final super-resolution image. The image super-resolution method is used for converting a low-resolution image into a high-resolution image. A diffusion model is adopted to replace a GAN network, and the training process is more stable.

Description

Technical Field

[0001] The invention relates to an image super-resolution method, in particular to an image super-resolution method based on a frequency domain residual diffusion model. Background Art

[0002] Image super-resolution is a basic task in the field of computer vision, which aims to reconstruct high-resolution images from low-resolution images. With the popularization of informatization, images, as one of the multimedia information carriers, are becoming increasingly important in terms of their intuitive and efficient information transmission function. High-resolution images can provide more data points and precise spatial information, supporting more accurate data analysis and model building. However, due to factors such as device hardware performance, network transmission bandwidth limitations, or image compression, the acquired images often have low resolution, resulting in image blur, loss of details, and edge jagged effects, which affect the execution of subsequent tasks.

[0003] In recent years, perception-oriented methods such as GAN (Generative Adversarial Network) have been introduced into the field of image super-resolution to improve the reconstruction effect. GAN networks have attracted attention due to their remarkable effects in image generation, but they also bring problems such as training instability, mode collapse and model complexity. In contrast, the diffusion model has gradually gained attention due to its simple structure, easy training and understanding. The diffusion model gradually generates high-resolution images through probability propagation and has good robustness.

[0004] The current problem is:

[0005] 1) Training instability: When using a GAN network, the training process may be unstable and prone to mode collapse, that is, the generated images tend to be single-mode and lack diversity;

[0006] 2) Model complexity: The GAN network structure is relatively complex and has many parameters, which makes the training and tuning process more difficult;

[0007] 3) Waste of computing power: The existing diffusion model super-resolution algorithm may have the problem of wasting computing power, especially in the image reconstruction process, which is not targeted enough, resulting in the reconstruction effect to be improved;

[0008] 4) Image reconstruction quality: Existing algorithms may not be able to effectively restore high-frequency details when reconstructing images, resulting in image quality that does not meet high standards;

[0009] 5) Computational complexity: When processing images, existing technologies may not fully utilize the frequency domain characteristics of the image, resulting in excessively high computational complexity, affecting the efficiency and practicality of the algorithm. Summary of the invention

[0010] The object of the present invention is to provide an image super-resolution method based on a frequency domain residual diffusion model, wherein the image super-resolution method is used to convert a low-resolution image into a high-resolution image.

[0011] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0012] An image super-resolution method based on a frequency domain residual diffusion model, the image super-resolution method comprising:

[0013] S1, training of residual CNN network: selecting the frequency domain wavelet signal of the low-resolution image as the input of the first residual CNN network to generate high-frequency and low-frequency components of the enhanced image;

[0014] S2, training of the noise prediction network: the difference between the high-frequency component of the enhanced image and the high-resolution target image is used to train the noise estimation network in the diffusion model;

[0015] S3, iterative generation of residual high-frequency components: using the noise estimation network to generate an estimated residual high-frequency component; adding the estimated residual high-frequency component and the high-frequency component of the enhanced image, and then superimposing the low-frequency component of the enhanced image to obtain the high-frequency component and low-frequency component of the final image; performing an inverse discrete wavelet transform on the obtained high-frequency component and low-frequency component to make it the final super-resolution image.

[0016] Furthermore, step S1 specifically includes:

[0017] S11, inputting a data set for training into the residual CNN network, where the training data set includes a low-resolution image and its corresponding high-resolution image;

[0018] S12, initialize the parameters of the residual CNN network, sample a small batch of high- and low-resolution image pairs, and perform two-dimensional discrete wavelet transform on both the upsampled low-resolution image and the high-resolution image. The high-resolution image is decomposed into four frequency domain components in different directions F HR , the low-resolution image is decomposed into four frequency domain components F in different directions LR ;

[0019] S13, the four frequency domain components F LR Superimpose in the channel dimension to form a new input signal, which is used as the input of the residual CNN network;

[0020] The residual CNN network processes the input signal and generates four frequency domain components F of the enhanced image based on the input signal. CR ;

[0021] S14, calculate the four frequency domain components F of the enhanced image CRThe frequency domain component F of the real high-resolution image HR The difference between them is calculated, and the mean square error is used as the loss function. Then back propagation is performed and the model parameters are updated according to the loss value to minimize the loss function.

[0022] S15, repeat step S13 to step S14 until a preset stop condition is reached, the maximum number of iterations is reached, or the loss function converges.

[0023] Furthermore, step S2 specifically includes:

[0024] S21, input the data set into the noise prediction network and the total number of diffusion steps T;

[0025] S22, load the trained residual CNN network and initialize the noise prediction network parameters;

[0026] S23, sampling small batch images for frequency domain components F LR And the frequency domain component F HR , the frequency domain component F LR The channel dimension superposition signal is input into the residual CNN network to obtain the enhanced image frequency domain component F CR , calculate the frequency domain component F CR and the frequency domain component F HR The residual image frequency domain component F RR ;

[0027] S24, randomly sample noise ε of the same dimension as the high-frequency component from a standard normal distribution, and sample the time step t from the integer set 1, 2, ..., T;

[0028] S25, the residual image frequency domain component F RR The high frequency part {R V ,R H ,R D} as the initial x for the forward diffusion step 0 , according to the formula

[0029]

[0030] Combined with the hyperparameter β, calculate the potential signal x at the current time step t t and the potential signal x at the previous time step t-1 ;

[0031] S26, the potential signal x t and enhance the high frequency components of the image {C V ,C H ,C D} Input UNet noise prediction network predicts the current noise ∈ θ (x t,t), according to the potential signal x t and x t-1 Calculate the true noise ∈ t ;

[0032] S27, according to the predicted noise ∈ θ (x t ,t) and the real noise ∈ t The root mean square error Update noise prediction network parameters;

[0033] S28, repeat the above steps S23 to S27 until the preset stop condition is reached, the maximum number of iterations or the loss function converges.

[0034] Furthermore, the step S3 specifically includes:

[0035] S31, input low-resolution image x L and the number of complete diffusion steps T;

[0036] S32, loading noise prediction model ∈ θ and residual CNN network;

[0037] S33, discrete wavelet transform the upsampled image of the low-resolution image to obtain its frequency domain component F LR ;

[0038] S34, frequency domain component F LR Generate the frequency domain component F of the enhanced image through the residual CNN network CR , and first take the channel superposition signal of its high-frequency component {C V ,C H ,C D};

[0039] S35, Random sampling from a standard normal distribution and high frequency components {C V ,C H ,C D}x of the same dimension T ;

[0040] S36, starting from t = T, randomly sample x from the standard normal distribution T The noise signal z of the same dimension is predicted by the noise prediction model ∈ θ Predict the current noise ∈ θ (x t ,t), according to the formula

[0041]

[0042]

[0043]

[0044] Combined with the current potential image x t , get the image x at the previous time step t-1 , until t = 0 returns to x 0 End; x 0 To estimate the high-frequency signal of the residual image

[0045] S37, put x 0 With F CR The frequency domain components of the super-resolution image are obtained by superposition. Perform inverse discrete wavelet transform on the frequency domain component to obtain the super-resolution image x SR .

[0046] Furthermore, in the training dataset input into the residual CNN network, the low-resolution images use the low-quality facial dataset CelebA, and the high-resolution images use the high-quality extensive dataset DIV2K.

[0047] Furthermore, for the training data set input into the residual CNN network, the training data set is divided into a training set, a validation set, and a test set according to a division ratio of 8:1:1.

[0048] Furthermore, for the residual CNN network and the noise prediction network, the mean square error is used as the loss function.

[0049] Compared with the prior art, the image super-resolution method of the present invention has the following advantages: a diffusion model is used instead of a GAN network, and since the diffusion model has a simpler structure, the training process is more stable; high-resolution images are gradually generated through probability propagation, reducing the risk of pattern collapse. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The pseudo code of the algorithm of the residual CNN network involved in the image super-resolution method of the present invention;

[0051] Figure 2 This is the image preprocessing module diagram;

[0052] Figure 3 The pseudo code of the algorithm of the noise prediction network involved in the image super-resolution method of the present invention;

[0053] Figure 4 The pseudo code of the algorithm for super-resolution image generation;

[0054] Figure 5 It is the diffusion module diagram;

[0055] Figure 6 It is the residual CNN framework diagram;

[0056] Figure 7 This is a core noise prediction network structure diagram of the image super-resolution method of the present invention;

[0057] Figure 8 This is a comparison of super-resolution images of various models in the CelebA dataset;

[0058] Fig. 9 This is a demonstration of super-resolution results on the DIV2K dataset. DETAILED DESCRIPTION

[0059] The present invention will be further described below with specific embodiments:

[0060] This embodiment provides an image super-resolution method based on a frequency domain residual diffusion model. Based on the image super-resolution method, a low-resolution image can be converted into a high-resolution image.

[0061] See also Figure 1 The image super-resolution method of this embodiment mainly includes steps S1 to S3.

[0062] S1, training of residual CNN network: The frequency domain wavelet signal of the low-resolution image is selected as the input of the first residual CNN network to generate high-frequency and low-frequency components of the enhanced image.

[0063] Specifically, the step S1 includes the following steps S11 to S15.

[0064] S11, inputting a data set for training into a preset residual CNN network, where the training data set includes a low-resolution image and its corresponding high-resolution image.

[0065] It should be noted that, in this embodiment, the training datasets used include the low-quality facial dataset CelebA and the high-quality extensive dataset DIV2K.

[0066] It should be noted that CelebA is an existing dataset, which is a widely used facial image dataset containing 202,599 face images.

[0067] In this embodiment, 162,770 images from the CelebA dataset are selected to train the residual CNN network, accounting for 80% of the total number of images in the dataset, and the remaining images are used as validation sets and test sets, accounting for 10% each. When processing all images in CelebA, a center alignment operation is first performed, and then all images are cropped to 160×160 pixels using the standard MATLAB bicubic interpolation kernel as high-resolution target images. The corresponding low-resolution images are obtained by downsampling the high-resolution images using the bicubic interpolation kernel.

[0068] It should be noted that DIV2K is an existing dataset, which is a high-quality image dataset commonly used for image super-resolution tasks, containing 1,000 high-quality images with a resolution of 2K. These images cover a wide range of scenes and content, from urban scenery to natural landscapes, with high resolution and rich details.

[0069] In this embodiment, 800 images in DIV2K are used for training, 100 images are used for verification, and 100 images are used for testing. The ratio is the same as the division of the CelebA dataset, which is 8:1:1 (training set: verification set: test set).

[0070] As with CelebA, the images in the DIV2K dataset are cropped into multiple small images of 160×160 (pixels) as the subsequent high-resolution target images. The low-resolution images are also obtained by downsampling the cropped small images using bicubic interpolation.

[0071] S12, initialize the parameters of the residual CNN network, sample a small batch of high- and low-resolution image pairs (one high-resolution image corresponds to one low-resolution image, and the two images are combined in pairs), and perform two-dimensional discrete wavelet transform on both the upsampled low-resolution image and the high-resolution image. In the two-dimensional discrete wavelet transform process, the high-resolution image will be decomposed into four frequency domain components FH in different directions R； The low-resolution image will be decomposed into four frequency domain components F in different directions LR .

[0072] S13, the four frequency domain components F LR The channels are superimposed to form a new input signal, which is used as the input of the residual CNN network. The residual CNN network processes the input signal and generates four frequency domain components F of the enhanced image based on the input signal. CR .

[0073] S14, calculate the four frequency domain components F of the enhanced image CR The frequency domain component F of the real high-resolution image HR The difference between the two is calculated and the mean square error is used as the loss function. Back propagation is then performed and the model parameters are updated according to the loss value to minimize the loss function.

[0074] S15, repeat the above steps S13 and S14 until the preset stop condition is reached, the maximum number of iterations is reached, or the loss function converges.

[0075] Through this process, the residual CNN network gradually learns how to transform the frequency domain components F of the low-resolution imageLR Generate the frequency domain component F of the enhanced image CR The residual CNN network structure diagram is shown in Figure 2 .

[0076] S2, training of noise prediction network: using the difference between the high-frequency component of the enhanced image generated in step S1 and the high-frequency component of the high-resolution target image to train the noise estimation network in the diffusion model.

[0077] It should be noted that the diffusion model is pre-built and set up, and its basic idea is to gradually introduce noise and then learn how to reverse this process during training, that is, how to restore a clear image from noisy data. This method is different from traditional generative adversarial networks (GANs), which usually have a more stable training process and a more controllable generation process.

[0078] It should be noted that the noise estimation network plays an important role in the image super-resolution algorithm. The main purpose of the noise estimation network is to predict and estimate the noise distribution in the image so as to more accurately remove the noise and improve the image quality during the image restoration and enhancement process.

[0079] Specifically, see Figure 3 , the step S2 specifically includes the following steps S21 to 28.

[0080] S21, input the data set into the pre-set noise prediction network, and also need to input the total diffusion steps T for the image super-resolution method.

[0081] S22, load the previously trained residual CNN network and initialize the noise prediction network parameters.

[0082] S23, sampling small batch images for frequency domain components F LR And the frequency domain component F HR , the frequency domain component F LR The channel dimension superposition signal is input into the residual CNN network to obtain the enhanced image frequency domain component F CR , calculate the frequency domain component F CR and the frequency domain component F HR The residual image frequency domain component F RR .

[0083] S24, randomly sample noise ε of the same dimension as the high-frequency component from a standard normal distribution, and sample the time step t from the integer set 1, 2, ..., T.

[0084] S25, the residual image frequency domain component F RR The high frequency part {R V ,R H ,R D} as the initial x for the forward diffusion step 0 , according to the formula

[0085]

[0086] Combined with the hyperparameter β, calculate the potential signal x at the current time step t t and the potential signal x at the previous time step t-1 .

[0087] The hyperparameter β is a constant, pre-calculated by the Variance Scheduler, between 0 and 1, and needs to satisfy β 1 <β 2 <…<β T , that is, β t Gradually increases. t =1-β t ,

[0088] S26, the potential signal x t and enhance the high frequency components of the image {C V ,C H ,C D} Input UNet noise prediction network predicts the current noise ∈ θ (x t ,t), according to the potential signal x t and x t-1 Calculate the true noise ∈ t .

[0089] S27, according to the predicted noise ∈ θ (x t ,t) and the real noise ∈ t The root mean square error Update the noise prediction network parameters.

[0090] S28, repeat the above steps S23 to S27 until the preset stop condition is reached, the maximum number of iterations is reached, or the loss function converges. Through this iterative training process, the noise prediction network gradually learns to accurately predict the noise in the high-frequency components of the image, thereby providing the necessary foundation for high-quality image generation.

[0091] S3, iterative generation of residual high frequency components: In this step S3,

[0092] Generate an estimated residual high-frequency component using a noise estimation network;

[0093] The estimated residual high-frequency component and the high-frequency component of the enhanced image are added, and then the low-frequency component of the enhanced image is superimposed to obtain the high-frequency component and the low-frequency component of the final image;

[0094] The obtained high-frequency components and low-frequency components are subjected to inverse discrete wavelet transform to form the final super-resolution image.

[0095] Specifically, see Figure 4 , the step S3 specifically includes the following steps S31 to 37.

[0096] S31, input low-resolution image x L and the number of complete diffusion steps T.

[0097] S32, loading noise prediction model ∈ θ and residual CNN networks.

[0098] S33, discrete wavelet transform the upsampled image of the low-resolution image to obtain its frequency domain component F LR .

[0099] S34, F LR Generate the frequency domain component F of the enhanced image through the residual CNN network CR , and first take the channel superposition signal of its high-frequency component {C V ,C H ,C D}.

[0100] S35, Random sampling from a standard normal distribution and high frequency components {C V ,C H ,C D}x of the same dimension T .

[0101] S36, starting from t = T, randomly sample x from the standard normal distribution T The noise signal z of the same dimension is predicted by the noise prediction model ∈ θ Predict the current noise ∈ θ (x t ,t), according to the formula

[0102]

[0103]

[0104]

[0105] Combined with the current potential image x t , get the image x at the previous time step t-1 , until t = 0 returns to x 0 End. 0 To estimate the high-frequency signal of the residual image

[0106] S37, put x 0 With F CRThe frequency domain components of the super-resolution image are obtained by superposition. Perform inverse discrete wavelet transform on the frequency domain component to obtain the super-resolution image x SR .

[0107] The core noise prediction network of image super-resolution method is as follows Figure 5 shown.

[0108] In this implementation, for the CelebA face dataset, an 8x scaled super-resolution algorithm is used for testing. The pre-trained residual CNN network is set with a batch of 64 and iterated for 600 epochs. The main noise prediction network of the image super-resolution method is set with a batch of 48 and a maximum number of iterations of 160,000. The maximum number of diffusion steps T of the diffusion part is 100. According to the maximum number of steps T, β t The number of steps is 100. For the DIV2K dataset, which is more widely used, a 4-fold scaling super-resolution algorithm is used. The pre-trained residual CNN network is set to a batch of 200 and iterates for 400 epochs. The main noise prediction network of the image super-resolution method is set to a batch of 64, with a maximum number of iterations of 300,000 times. The maximum number of diffusion steps T of the diffusion part is 100. According to the maximum number of steps T, β t The two different scaling factors are divided into 0.00001 and 0.01. Through these two different scaling factors, we can more comprehensively evaluate the performance of the model when processing images of different resolutions, thereby verifying its applicability and effectiveness in different scenarios and applications.

[0109] The mean square error is used as the loss function for both the residual CNN network and the core noise prediction network of the image super-resolution method. The loss function of the residual CNN network is

[0110] Loss C =3MSE(H A ,C A )+MSE(H V ,C V )+MSE(H H ,C H )+MSE(H D ,C D ) (1-4)

[0111] The frequency domain components of the target high-resolution image are {H A ,H V ,H H ,H D}, the frequency domain components of the enhanced image generated by the residual CNN network are {C A ,C V ,CH ,C D After calculating the mean square error for each frequency domain component, the error of the low-frequency component is multiplied by three and then added to the errors of the three high-frequency components to form the final loss function. This weight distribution is intended to make the residual CNN network more focused on the reconstruction of low-frequency signals. The internal structure of the residual CNN network is as follows: Figure 6 shown.

[0112] The core noise prediction network loss function of the image super-resolution method is

[0113]

[0114] Where ∈ is a randomly sampled noise signal, x 0 is the high frequency component of the residual image {R V ,R H ,R D}, the high frequency component is calculated by the difference between the high frequency components of the enhanced image and the high resolution image. Specifically, the high frequency component calculation formula of the residual image is: {R V ,R H ,R D}={H V -C V ,H H -C H ,H D -C D}, where H V , H H and H D Represent different high-frequency components of high-resolution images, C V , C H and C D is the corresponding high frequency component of the enhanced image. The loss function is to find the mean square error between the values ​​of the two noise signals. The internal structure of the noise prediction network is as follows: Figure 7 shown.

[0115] In order to evaluate the performance of the single image super-resolution algorithm of the image super-resolution method, the following evaluation indicators were used in the experiment: Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Low Resolution Peak Signal-to-Noise Ratio (LR-PSNR). PSNR is used to evaluate the similarity between the generated high-resolution image and the original high-resolution image, and LR-PSNR is used to evaluate the similarity between the generated high-resolution image and the low-resolution image after downsampling. They are usually regarded as indicators reflecting the overall difference of the image, and the value is often greater than 20. The larger the value, the smaller the error between the generated image and the original high-resolution image, and the higher the image quality. SSIM pays more attention to the structural information of the image and can more accurately reflect the structural similarity of the image. The value range of SSIM is between 0 and 1. The closer the value is to 1, the higher the similarity between the generated image and the reference image, and the higher the image quality. LPIPS is a method that uses the deep features generated by the network as a perceptual metric. The calculation method of this method cannot be simply expressed by mathematical formulas, and it is completed using Python built-in functions. LPIPS is closer to human perception of images and has visual significance. Its value range is also between 0 and 1. The closer it is to 0, the better the perceived quality of the image and the higher the image quality.

[0116] Table 1 is provided below, which shows the comparison results of various model indicators of the CelebA dataset.

[0117] Table 1. Comparison results of various model indicators of the CelebA dataset

[0118] method PSNR SSIM LPIPS LR-PSNR Bicubic 23.38 0.642 0.483 34.66 ESRGAN 23.26 0.671 0.115 39.91 SRDiff 25.34 0.738 0.108 52.43 SR3 24.93 0.725 0.112 51.06 Method of the present invention 25.40 0.743 0.101 53.12

[0119] According to the data in Table 1, we can see the performance differences of different models on the CelebA face dataset. The bilinear interpolation method is used as the basic control group for all super-resolution algorithms. In terms of PSNR, the image super-resolution method model is 0.06 better than the best performing SRDiff model, in terms of SSIM, the image super-resolution method model is 0.005 better than the best performing SRDiff model, in terms of LPIPS, the image super-resolution method model is 0.007 better than the best performing SRDiff model, and in terms of LR-PSNR, the image super-resolution method model is 0.69 better than the best performing SRDiff model.

[0120] Table 2 is provided below, which shows the comparison results of various model indicators of the DIV2K dataset.

[0121] Table 2. Comparison results of various model indicators of DIV2K dataset

[0122] method PSNR SSIM LPIPS LR-PSNR Bicubic 26.70 0.770 0.409 38.70 ESRGAN 26.22 0.752 0.124 39.03 SRDiff 27.41 0.792 0.136 55.21 SR3 25.90 0.750 0.175 52.34 Method of the present invention 27.73 0.801 0.118 55.29

[0123] The data in Table 2 shows the performance of different models on various indicators of DIV2K wide-type datasets. The bilinear interpolation method is used as the basic control group for all super-resolution algorithms. In terms of PSNR, the image super-resolution method model is 0.32 better than the best performing SRDiff model, in terms of SSIM, the image super-resolution method model is 0.009 better than the best performing SRDiff model, in terms of LPIPS, the image super-resolution method model is 0.006 better than the best performing ESRGAN model, and in terms of LR-PSNR, the image super-resolution method model is 0.08 better than the best performing SRDiff model.

[0124] Through experimental verification, the data in Table 1 and Table 2 show that the image super-resolution method model performs well in the four indicators of PSNR, SSIM, LPIPS and LR-PSNR, and comprehensively surpasses the comparison model. This shows that the image super-resolution method model has significant advantages in the image super-resolution task, and can better process images of different resolutions and improve the visual effect and quality of the image. Especially in the high-magnification super-resolution task, the image super-resolution method model can still maintain stable performance, which further proves its ability and potential in handling complex image reconstruction tasks.

[0125] Figure 8 This is a comparison of the super-resolution images generated by each model on the test set after training on the CelebA dataset. The first column is the original low-resolution image, the second column is the image generated by RCNN using the residual CNN network proposed in this chapter, and the last column is the target high-resolution image. From the images generated by the residual CNN network, it can be seen that the network has initially achieved super-resolution restoration of the original low-resolution image, and the overall frame and shape have been well restored. However, these generated images are relatively smooth, with insufficient details, a certain haziness, and cannot fully display the high-frequency details of the image. The images generated by the image super-resolution method model have been improved in various details. For example, in terms of human faces, subtle wrinkles are truly restored instead of being smooth and traceless; in terms of hair, the hair is clearly separated into filaments instead of being stacked into a block of color; in terms of teeth, the teeth are separated by teeth instead of being connected into a piece of white. The preprocessing residual CNN network completes the initial frame, and the image super-resolution method model significantly enhances the ability to express details in the image.

[0126] Fig. 9This is a display of the super-resolution images generated by the image super-resolution method model on the test set after training on the DIV2K dataset. The first column is the target high-resolution image, the second column is the enlarged image of the low-resolution image in the selected area, the third column is the enlarged image of the image generated by the residual CNN network, the fourth column is the enlarged image of the image generated by the image super-resolution method model, and the fifth column is the enlarged image of the target high-resolution image. By observation, it can be found that the super-resolution image generated by the image super-resolution method model is very close to the original high-resolution image.

[0127] These data results show that the image super-resolution method model has excellent performance in restoring image details and improving visual quality, further verifying its advancement and effectiveness in image super-resolution tasks. Overall, the image super-resolution method model has excellent performance in image super-resolution tasks.

[0128] The image super-resolution method of this embodiment has the following advantages:

[0129] 1) The diffusion model is used instead of the GAN network because the diffusion model has a simpler structure and a more stable training process. It gradually generates high-resolution images through probability propagation, reducing the risk of mode collapse.

[0130] 2) By designing a two-stage cascade network, the processing process is divided into low-frequency and high-frequency parts, simplifying the model structure; using residual modules reduces the complexity of the model, making the network easier to train and understand;

[0131] 3) By optimizing the algorithm process and introducing discrete wavelet transform, the number of channels is increased to improve the feature expression ability, while the spatial dimension is reduced to reduce the computational complexity, effectively avoiding the waste of computing power;

[0132] 4) Using the noise estimation network to generate the estimated residual high-frequency components, focusing on the recovery of high-frequency signals, ensuring that image details are better preserved and enhanced; in addition, the inverse discrete wavelet transform helps to synthesize higher quality super-resolution images;

[0133] 5) The image is preprocessed by discrete wavelet transform, and the image is decomposed into high-frequency components and low-frequency components, which reduces the amount of calculation in the processing process; the optimized network structure and algorithm flow reduce unnecessary calculations and improve the overall computing efficiency.

[0134] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image super-resolution method based on a frequency domain residual diffusion model, characterized in that: The image super-resolution method comprises: S1, training of residual CNN network: selecting the frequency domain wavelet signal of the low-resolution image as the input of the first residual CNN network to generate high-frequency and low-frequency components of the enhanced image; S2, training of the noise prediction network: the difference between the high-frequency component of the enhanced image and the high-resolution target image is used to train the noise estimation network in the diffusion model; S3, iterative generation of residual high-frequency components: using the noise estimation network to generate an estimated residual high-frequency component; adding the estimated residual high-frequency component and the high-frequency component of the enhanced image, and then superimposing the low-frequency component of the enhanced image to obtain the high-frequency component and low-frequency component of the final image; performing an inverse discrete wavelet transform on the obtained high-frequency component and low-frequency component to make it the final super-resolution image.

2. The image super-resolution method based on the frequency domain residual diffusion model according to claim 1, characterized in that: Step S1 specifically includes: S11, inputting a data set for training into the residual CNN network, where the training data set includes a low-resolution image and its corresponding high-resolution image; S12, initialize the parameters of the residual CNN network, sample a small batch of high- and low-resolution image pairs, and perform two-dimensional discrete wavelet transform on both the upsampled low-resolution image and the high-resolution image. The high-resolution image is decomposed into four frequency domain components in different directions F HR , the low-resolution image is decomposed into four frequency domain components F in different directions LR ; S13, the four frequency domain components F LR Superimpose in the channel dimension to form a new input signal, which is used as the input of the residual CNN network; The residual CNN network processes the input signal and generates four frequency domain components F of the enhanced image based on the input signal. CR ; S14, calculate the four frequency domain components F of the enhanced image CR The frequency domain component F of the real high-resolution image HR The difference between them is calculated, and the mean square error is used as the loss function. Then back propagation is performed and the model parameters are updated according to the loss value to minimize the loss function. S15, repeat step S13 to step S14 until a preset stop condition is reached, the maximum number of iterations is reached, or the loss function converges.

3. The image super-resolution method based on the frequency domain residual diffusion model according to claim 2, characterized in that: Step S2 specifically includes: S21, input the data set into the noise prediction network and the total number of diffusion steps T; S22, load the trained residual CNN network and initialize the noise prediction network parameters; S23, sampling small batch images for frequency domain components F LR And the frequency domain component F HR , the frequency domain component F LR The channel dimension superposition signal is input into the residual CNN network to obtain the enhanced image frequency domain component F CR , calculate the frequency domain component F CR and the frequency domain component F HR The residual image frequency domain component F RR ; S24, randomly sample noise ε of the same dimension as the high-frequency component from a standard normal distribution, and sample the time step t from the integer set 1, 2, ..., T; S25, the residual image frequency domain component F RR The high frequency part {R V , R H , R D } is taken as the initial x0 of the forward diffusion step, according to the formula Combined with the hyperparameter β, the potential signal x at the current time step t is calculated t and the potential signal x at the previous time step t-1 ; S26, the potential signal x t and enhance the high frequency components of the image {C V , C H , C D } Input UNet noise prediction network predicts the current noise ∈ θ (x t , t), according to the potential signal x t and x t-1 Calculate the true noise ∈ t ; S27, according to the predicted noise ∈ θ (x t , t) and the real noise ∈ t The root mean square error Update noise prediction network parameters; S28, repeat the above steps S23 to S27 until the preset stop condition is reached, the maximum number of iterations or the loss function converges.

4. The image super-resolution method based on the frequency domain residual diffusion model according to claim 3, characterized in that: The step S3 specifically includes: S31, input low-resolution image x L and the number of complete diffusion steps T; S32, loading noise prediction model ∈ θ and residual CNN network; S33, discrete wavelet transform the upsampled image of the low-resolution image to obtain its frequency domain component F LR ; S34, frequency domain component F LR Generate the frequency domain component F of the enhanced image through the residual CNN network CR , and first take the channel superposition signal of its high-frequency component {C V , C H , C D }; S35, Random sampling from a standard normal distribution and high frequency components {C V , C H , C D }x of the same dimension T ; S36, starting from t = T, randomly sample x from the standard normal distribution T The noise signal z of the same dimension is predicted by the noise prediction model ∈ θ Predict the current noise ∈ θ (x t , t), according to the formula Combined with the current potential image x t , get the image x at the previous time step t-1 , until t = 0 returns to x0; x0 is the high-frequency signal of the estimated residual image S37, put x0 and F CR The frequency domain components of the super-resolution image are obtained by superposition. Perform inverse discrete wavelet transform on the frequency domain component to obtain the super-resolution image x SR .

5. The image super-resolution method based on the frequency domain residual diffusion model according to claim 2, characterized in that: In the training dataset input into the residual CNN network, the low-resolution images use the low-quality face dataset CelebA, and the high-resolution images use the high-quality extensive dataset DIV2K.

6. The image super-resolution method based on the frequency domain residual diffusion model according to claim 2, characterized in that: For the training data set input into the residual CNN network, the training data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

7. The image super-resolution method based on the frequency domain residual diffusion model according to claim 1, characterized in that: For the residual CNN network and the noise prediction network, the mean square error is used as the loss function.

Images