Image super-resolution reconstruction method and system based on diffusion implicit model, and readable storage medium
Through the image super-resolution reconstruction method based on diffusion implicit model, the problems of low efficiency and accuracy of traditional algorithms are solved, more efficient image recovery and computing resource optimization are achieved, and the quality and generalization ability of image super-resolution reconstruction are improved. .
Patent Information
- Application Number
- CN202411059193.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-03
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional image super-resolution algorithms have low efficiency and accuracy, relying on too strong training data sets leads to insufficient generalization capabilities, high computing resources consumption, noise sensitivity, lack of image understanding capabilities, and it is difficult to make a trade-off between reconstruction quality and computing efficiency. .
Using the image super-resolution reconstruction method based on diffusion implicit model, by learning the data distribution of normal images, inputting abnormal samples and detecting abnormal parts, using diffusion implicit model for noise image generation and denoising, optimizing network structure, and simplifying calculations process.
It improves the efficiency and quality of image super-resolution reconstruction, reduces the number of sampling steps and calculation time, improves performance on different data sets, and achieves better image recovery effect.
Smart Images

Figure CN118918008A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image reconstruction, and more particularly to an image super-resolution reconstruction method, system and readable storage medium based on a diffusion implicit model. Background Art
[0002] At present, image super-resolution algorithm, as a technology to effectively improve image resolution, has been widely studied in various fields.
[0003] However, traditional image super-resolution algorithms have problems such as low efficiency and accuracy in practical applications, which leads to certain limitations in their comprehensive performance and poor super-resolution performance. Deep learning-based methods have advantages in image quality and performance, especially those based on generative adversarial networks and attention mechanisms. However, these methods still face some challenges. The current mainstream deep learning algorithms often rely too much on training data sets in the process of image super-resolution reconstruction, resulting in insufficient generalization ability on unseen images, the so-called "overfitting" problem. Secondly, these deep learning models usually require a lot of computing resources and memory, resulting in low operating efficiency in practical applications. In addition, existing image super-resolution technologies are highly sensitive to image noise and are prone to introduce artifacts and distortion during the reconstruction process. Furthermore, current image super-resolution technologies often lack sufficient image understanding capabilities. Most of them learn the degradation distribution of images based on a large amount of data, but ignore the in-depth analysis of image content, resulting in poor results in restoring object structure and texture. Finally, image super-resolution technology often needs to balance reconstruction quality and computational efficiency. How to reduce computational costs while ensuring high-quality reconstruction is still a major challenge facing current technology.
[0004] Therefore, how to provide an image super-resolution reconstruction method that can solve the above problems is an issue that those skilled in the art urgently need to solve. Summary of the invention
[0005] In view of this, the present invention provides an image super-resolution reconstruction method, system and readable storage medium based on a diffusion implicit model, which learns the data distribution of normal images during the training process, inputs abnormal samples during the reasoning process and reconstructs normal images by learning the model of healthy distribution, and subtracts the input image from the output image to realize the detection of abnormal parts.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A method for image super-resolution reconstruction based on a diffusion implicit model comprises the following steps:
[0008] Obtain high-resolution images and corresponding low-resolution image datasets;
[0009] Constructing a diffusion implicit model, and inputting the high-resolution image into the diffusion implicit model for processing to obtain a noise image, and processing the low-resolution image data set to obtain a corresponding image;
[0010] Constructing a processing model, and using the image and the noise image as a training data set to train the processing model;
[0011] The trained processing model is used to process the input image to obtain a high-resolution image.
[0012] Preferably, the specific process of obtaining the noise image includes:
[0013] Constructing a diffusion implicit model and defining a forward process of the diffusion implicit model;
[0014] Noise is added to the high-resolution image multiple times using the forward process of the diffusion implicit model to obtain a corresponding Gaussian noise image, which is the noise image.
[0015] Preferably, the specific process of processing the low-resolution image to obtain the corresponding image includes:
[0016] A bicubic interpolation degradation operation is performed on the low-resolution image to obtain the corresponding image.
[0017] Preferably, the specific processing of connecting the image with the noise image includes:
[0018] A denoising calculation model is constructed, and the low-resolution image and the noise image are input into the denoising calculation model, wherein the low-resolution image is used as a guide condition, and the noise image is defined as:
[0019]
[0020] In the formula, y t represents a noisy image, represents the noise variance, y0 represents the high-resolution image, and ε represents the noise vector;
[0021] The denoising calculation model takes the noise variance as input, and the specific expression of the corresponding objective function is:
[0022]
[0023] In the formula, f θ represents the denoising calculation model, x represents the image, (x,y t ) is sampled from the training data set, and p indicates the use of L1 or L2 norm;
[0024] The above process realizes the noise addition and noise prediction of the image.
[0025] Preferably, the processing model is a U-Net network, comprising an encoder and a decoder connected in sequence.
[0026] The present invention also provides an image super-resolution reconstruction system based on a diffusion implicit model, comprising:
[0027] An acquisition module, used to acquire high-resolution images and corresponding low-resolution image data sets;
[0028] A processing module, used for constructing a diffusion implicit model, and inputting the high-resolution image into the diffusion implicit model for processing to obtain a noise image, and processing the low-resolution image data set to obtain a corresponding image;
[0029] A training module, used for constructing a processing model, and using the image and the noise image as a training data set to train the processing model;
[0030] The reconstruction module is used to process the input image using the trained processing model to obtain a high-resolution image.
[0031] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of the above items is implemented.
[0032] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a method, system and readable storage medium for image super-resolution reconstruction based on a diffusion implicit model. By introducing a low-resolution image as a conditional input and optimizing the network structure of the diffusion implicit model, the defects of the original diffusion model are overcome, and the forward process is no longer restricted to a Markov chain, the sampling speed is improved, and the entire sampling process is accelerated. At the same time, the process of generating samples from noise by the diffusion implicit model is determined, and there is no random noise added in the middle, which simplifies the calculation and can be better optimized. In order to verify the effectiveness of the method provided by the present invention, experiments were carried out on the FFHQ dataset and the CelebA-HQ dataset, and the image super-resolution evaluation indicators PSNR and SSIM were used to compare the results with the diffusion probability model. The experimental results show that the method of realizing image super-resolution reconstruction using the diffusion implicit model proposed in the present invention not only greatly reduces the number of sampling steps and saves sampling time, but also achieves good results on different datasets. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 The present invention provides an image super-resolution reconstruction method based on a diffusion implicit model;
[0035] Figure 2 A flow chart of the forward process of the diffusion implicit model provided by an embodiment of the present invention;
[0036] Figure 3 A schematic diagram of the U-Net network architecture provided by an embodiment of the present invention;
[0037] Figure 4 A schematic diagram of a downsampling phase provided by an embodiment of the present invention;
[0038] Figure 5 A schematic diagram of an upsampling stage provided in an embodiment of the present invention;
[0039] Figure 6 The present invention provides a structural principle block diagram of an image super-resolution reconstruction system based on a diffusion implicit model. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] See also Figure 1 As shown, an embodiment of the present invention discloses an image super-resolution reconstruction method based on a diffusion implicit model, comprising the following steps:
[0042] Obtain high-resolution images and corresponding low-resolution image datasets;
[0043] Construct a diffusion implicit model and input the high-resolution image y0 into the diffusion implicit model for processing to obtain the noise image y t , and at the same time, the low-resolution image dataset is subjected to bicubic interpolation to obtain the corresponding image x;
[0044] Build a processing model and compare image x with the noise image y tAfter the connection is made, the processing model is trained as a training data set;
[0045] The input image is processed using the trained processing model to obtain a high-resolution image.
[0046] In a specific embodiment, the specific process of obtaining the noise image includes:
[0047] Construct a diffusion implicit model and define the forward process of the diffusion implicit model;
[0048] The forward process of the diffusion implicit model is used to add noise to the high-resolution image multiple times to obtain the corresponding Gaussian noise image, which is the noise image.
[0049] Specifically, the forward and reverse processes of the diffusion implicit model can be found in Figure 2 As shown, for a given high-resolution image y0, the forward diffusion process is defined as:
[0050]
[0051] In the formula, y t and t-1 represents the noise image at time t and t-1, t∈{1,2,...,T}.
[0052] Here we need to satisfy:
[0053]
[0054] Where N represents Gaussian noise distribution, α T ∈(0,1) is the noise variance and also a hyperparameter. Related to the training duration, I represents the identity matrix with the same dimension as the initial state.
[0055] When t≥2, formula (2) is:
[0056]
[0057] In the formula, the variance is a real number with different distributions for different settings, so q(y 1:T |y0) is actually a series of inference distributions. We can see that the distribution q(y t-1 |y t ,y0) is also defined as a function that depends on y0 and y t The combined function, q(y T |y0) can be proved by mathematical induction that for all t, the following expression is satisfied:
[0058]
[0059] That is, at any time q(y t y0) still obeys the normal distribution.
[0060] Derived from the Bayesian formula:
[0061]
[0062] In a specific embodiment, the specific process of processing the low-resolution image to obtain the corresponding image includes:
[0063] The bicubic interpolation degradation operation is performed on the low-resolution image to obtain the corresponding image x.
[0064] In a specific embodiment, the specific process of connecting the image with the noise image includes:
[0065] Building a denoising computational model The low-resolution image and the noisy image are input into the denoising calculation model, where the low-resolution image is used as the guiding condition and the noisy image is defined as:
[0066]
[0067] In the formula, y t represents a noisy image, represents the noise variance, y0 represents the high-resolution image, ε~N(0,I) represents the noise vector, and the purpose of formula (6) is to t Recover the noise-free target image;
[0068] In addition to the low-resolution image x and the noisy image y t In addition, the denoising calculation model The noise variance As input, the denoising calculation model can know the level of noise. The training goal of the model is to predict the noise vector ε, so the denoising calculation model The specific expression of the corresponding objective function is:
[0069]
[0070] In the formula, represents the denoising calculation model, x represents the processed low-resolution image, (x, y t ) is sampled from the training data set, and p indicates the use of L1 or L2 norm;
[0071] Specifically, in this embodiment, it is assumed that the inference distribution of the diffusion implicit model is a Gaussian distribution, and the mean is the high-resolution image y0 and the noise image y t A linear function of , and the variance is a function of the time step, then let:
[0072]
[0073] In the formula, λ t , k t , σ t In theory, there are infinite solutions to the undetermined coefficients. In order to avoid retraining the model, the distribution q(y t-1 y t ) and q(y t y0), so we only need to find a set of solutions The reasoning distribution of the diffusion implicit model satisfies the above distribution. The specific process is as follows:
[0074] When the time step is 1, the diffusion implicit model can satisfy the following equation:
[0075]
[0076] When the time step is t, the diffusion implicit model can satisfy the following equation:
[0077]
[0078] By mathematical induction, we can see that we only need to ensure Then we only need to satisfy the marginal distribution conditions:
[0079]
[0080] untie So that:
[0081]
[0082] By sampling y t-1 and t , and using the additivity of the normal distribution, we can get:
[0083]
[0084] In the formula, ε′ t ,ε′ t-1 ~N(0,I),ε′ t , ε′ t-1 represent the noise at time t and time t-1 respectively.
[0085] In order to satisfy the marginal conditional distribution, we have:
[0086]
[0087] There are three unknowns and two equations in equation (14), and there are obviously infinite sets of solutions that can satisfy them. tis regarded as a variable, and its size can be regarded as the degree of randomness of the sampling process. Therefore, the sampling distribution of the diffusion implicit model is:
[0088]
[0089] In the diffusion implicit model, assuming that the original generated sequence is L = [T, T-1, ..., 1], and its length is dim(L) = T, then a subsequence τ = [τ s ,τ s-1 ,...,τ1], whose length is dim(τ)=S, S≤T. In the generation process, sampling is performed according to the constructed sequence τ. The sampling recursive sequence at this time is:
[0090]
[0091] In this way, the generation process can be directly performed on the subsequence, that is, the training is the complete sequence, and the generation is the subsequence. As long as the subsequence is small and the effect is not very poor, it can be accelerated. Through the above process, this embodiment explains the principle of adding noise and denoising to the diffusion implicit model and how to accelerate the generation.
[0092] In a specific embodiment, the processing model is a U-Net network, including an encoder and a decoder connected in sequence.
[0093] Specifically, the U-Net network architecture based on the diffusion implicit model can be found in Figure 3 As shown in the figure, U-Net is a U-shaped network structure designed based on the Fully Convolutional Networks (FCN).
[0094] In the U-Net network structure of the diffusion implicit model, the encoder is divided into different stages: it is composed of three ResNet Block modules and a Self-Attention module repeatedly, where the ResNet Block module is composed of two convolutions plus a mean pooling layer, and each stage contains a downsampling module to reduce the spatial size of the feature. The decoder is just the opposite of the encoder. The decoder gradually restores the features compressed by the encoder. In addition, jump connections are introduced in the decoder to connect the same-dimensional features obtained in the middle of the encoder, which is beneficial to network optimization. The downsampling stage and the upsampling stage are as follows: Figure 4-Figure 5 shown.
[0095] See also Figure 6 As shown, an embodiment of the present invention further provides a reconstruction system using an image super-resolution reconstruction method based on a diffusion implicit model according to any one of the above embodiments, comprising:
[0096] An acquisition module, used to acquire high-resolution images and corresponding low-resolution image data sets;
[0097] A processing module, used for constructing a diffusion implicit model, and inputting the high-resolution image into the diffusion implicit model for processing to obtain a noise image, and processing the low-resolution image data set to obtain a corresponding image;
[0098] A training module, used for constructing a processing model, and using the image and the noise image as a training data set to train the processing model;
[0099] The reconstruction module is used to process the input image using the trained processing model to obtain a high-resolution image.
[0100] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of the above embodiments is implemented.
[0101] In order to verify the effectiveness of the method provided by the embodiment of the present invention, two existing face datasets, FFHQ dataset and CelebA-HQ dataset, were selected for experimental verification, specifically:
[0102] The FFHQ dataset is used as a training set. The FFHQ (Flickr-Faces-High-Quality) dataset is a high-quality dataset commonly used in the field of computer vision and machine learning, and is used for tasks such as face recognition, image generation, and image synthesis. The dataset consists of high-resolution face images collected from the Flickr website, and contains a total of 70,000 high-resolution face images in PNG format.
[0103] The CelebA-HQ dataset is used as a validation set. The CelebA-HQ (CelebFaces Attributes-HighQuality) dataset is an extension of the original CelebA (CelebFaces Attribute) dataset, containing a total of 30,000 celebrity face images with a resolution of , which can provide more detailed visual information compared to the low-resolution images in the original CelebA dataset.
[0104] The evaluation indicators selected in the embodiment of the present invention mainly include PSNR and SSIM. Peak Signal-to-Noise Ratio (PSNR) is an objective standard for measuring image quality. It measures image quality by calculating the mean square error between the original image and the processed image, expressed in decibels (dB). The larger the PSNR value, the better the quality of the reconstructed image and the higher the degree of closeness to the original image. The calculation formula is as follows:
[0105]
[0106] In the formula, MaxI represents the maximum value of the pixel in the image, and MSE represents the square mean of the difference between the corresponding pixels of the two images. For multi-channel images, the MSE calculation formula is as follows:
[0107]
[0108] In the formula, C, M, and N represent the number of channels, height, and width of the image, respectively. ijk 、f' ijk represent the current image and the reference image respectively.
[0109] Structural Similarity Index (SSIM) is an index to measure the similarity between two images. It evaluates image quality by comparing the brightness, contrast and structural information of the images. The closer the SSIM value is to 1, the more similar the two images are, that is, the better the image quality of super-resolution reconstruction is. Its calculation formula is as follows:
[0110] SSIM(X,Y)=L(X,Y)×C(X,Y)×S(X,Y) (19)
[0111] In the formula, μ x , μ y Represents the mean of the pixels of image X and image Y, σ x , σ y Represent the standard values of image X and image Y pixels, σ xy Represents the covariance of image X and image Y, C1, C2, C3 are all constants, C1=(K1×L) 2 、C2=(K2×L) 2 , Generally, K1=0.01, K2=0.03, L=255.
[0112] The results of image quality evaluation using PSNR and SSIM on the validation set CelebA-HQ dataset are shown in Table 1 and Table 2 respectively.
[0113] Table 1 Comparison of PSNR and SSIM values in super-resolution tasks achieved on the CelebA-HQ dataset
[0114]
[0115] Table 2 Comparison of PSNR and SSIM values in super-resolution tasks achieved on the CelebA-HQ dataset
[0116]
[0117] Under the same sampling step number, compared with the SR3 model, the method provided by the embodiment of the present invention achieves significantly improved PSNR and SSIM for different super-resolution tasks on the CelebA-HQ dataset. After increasing the sampling step number of the SR3 model by 10 times, the two evaluation index values of the present invention at t = 200 are close. When the same model is used to achieve super-resolution tasks, when t = 2000, the performance of SR3 far exceeds the result at t = 200, while the performance of the method provided by the embodiment of the present invention has a small improvement in the evaluation index at t = 2000, but the required sampling steps are also improved.
[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0119] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image super-resolution reconstruction method based on diffusion implicit model, It is characterized in that The following steps are involved: Obtain high-resolution images and corresponding low-resolution image datasets; Constructing a diffusion implicit model, and inputting the high-resolution image into the diffusion implicit model for processing to obtain a noise image, and processing the low-resolution image data set to obtain a corresponding image; Constructing a processing model, and using the image and the noise image as a training data set to train the processing model; The trained processing model is used to process the input image to obtain a high-resolution image.
2. The image super-resolution reconstruction method based on the diffusion implicit model according to claim 1, It is characterized in that The specific process of obtaining the noisy image includes: Constructing a diffusion implicit model and defining a forward process of the diffusion implicit model; Noise is added to the high-resolution image multiple times using the forward process of the diffusion implicit model to obtain a corresponding Gaussian noise image, which is the noise image.
3. The image super-resolution reconstruction method based on the diffusion implicit model according to claim 1, It is characterized in that The specific process of processing the low-resolution image to obtain the corresponding image includes: A bicubic interpolation degradation operation is performed on the low-resolution image to obtain the corresponding image.
4. The image super-resolution reconstruction method based on the diffusion implicit model according to claim 3, It is characterized in that The specific processing process of connecting the image with the noise image includes: A denoising calculation model is constructed, and the low-resolution image and the noise image are input into the denoising calculation model, wherein the low-resolution image is used as a guide condition, and the noise image is defined as: In the formula, y t represents a noisy image, represents the noise variance, y 0 represents a high-resolution image, ε represents a noise vector; The denoising calculation model takes the noise variance as input, and the specific expression of the corresponding objective function is: In the formula, f θ represents the denoising calculation model, x represents the image, (x,y t ) is sampled from the training data set, and p indicates the use of L1 or L2 norm; The above process realizes the noise addition and noise prediction of the image.
5. The image super-resolution reconstruction method based on the diffusion implicit model according to claim 1, It is characterized in that The processing model is a U-Net network, including an encoder and a decoder connected in sequence.
6. A reconstruction system using the image super-resolution reconstruction method based on a diffusion implicit model according to any one of claims 1 to 5, It is characterized in that include: An acquisition module, used to acquire high-resolution images and corresponding low-resolution image data sets; A processing module, used for constructing a diffusion implicit model, and inputting the high-resolution image into the diffusion implicit model for processing to obtain a noise image, and processing the low-resolution image data set to obtain a corresponding image; A training module, used for constructing a processing model, and using the image and the noise image as a training data set to train the processing model; The reconstruction module is used to process the input image using the trained processing model to obtain a high-resolution image.
7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image super-resolution reconstruction method according to any one of claims 1 to 5 is implemented.
