Image super-resolution method based on multiple learnable prior depth expansion networks

Abstract

The invention discloses an image super-resolution method based on a multiple learnable prior depth expansion network, and belongs to the technical field of computer vision, and the method comprises the steps: selecting a high-resolution image from a data set, generating a low-resolution image, enabling the high-resolution image and the corresponding low-resolution image to form a sample pair, and carrying out the recognition of the sample pair, dividing the sample pairs into a training set and a test set; modeling a super-resolution problem into an optimization problem based on a degradation model and multiple priori, and obtaining two sub-problems needing to be iteratively solved for solving the optimization problem by using an optimization algorithm; multiple priori in the sub-problem is popularized into multiple learnable priori, and a deep expansion network is constructed based on the multiple learnable priori; and training the expanded network by using the training set to obtain a trained model, and testing the super-resolution performance of the model by using the test set. According to the method, a new multiple learnable prior is introduced to a deep expansion super-resolution model, simultaneous learning of a transformation operator and a prior function is realized based on a convolutional layer and a common image operator, the super-resolution reconstruction performance is further improved, and the method has a wide application scene.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and proposes an image super-resolution method based on a multiple learnable priori deep unfolding network. Background Art

[0002] Super-resolution (SR) is a technology used to obtain potential high-resolution (HR) images from known low-resolution (LR) images. Super-resolution technology plays an important role in many fields. In the medical field, SR technology can improve image clarity, allowing doctors to diagnose diseases more accurately. In the field of satellite remote sensing, SR technology can improve the resolution of low-resolution satellite images, obtain more information from them, and help complete tasks such as resource monitoring and disaster warning. In the field of video surveillance, SR technology can improve the quality of monitoring images, improve the ability to identify and track targets, and help to promptly detect potential safety hazards and criminal behavior. At the same time, it also plays a key role in fields such as virtual reality, augmented reality, and military reconnaissance.

[0003] In recent decades, researchers have proposed a large number of SR methods. Most of the early SR methods were based on interpolation. Although such methods are easy to implement and have a fast calculation speed, they are difficult to restore image details well, and the image quality needs to be further improved. In order to obtain clearer HR images, model-based SR methods consider the image degradation process. First, a super-resolution reconstruction model is established by the degradation model and the image prior, and then it is optimized and solved to achieve super-resolution reconstruction. Such methods can flexibly handle low-resolution images caused by different blur kernels and magnifications. However, traditional model-based SR methods often use artificially set priors, which are difficult to accurately characterize the characteristics of HR images and have a relatively large limitation on the improvement of SR performance. In addition, the selection of hyperparameters in such methods is also a difficult problem to solve. In recent years, deep learning has developed rapidly and has been successfully applied to image super-resolution reconstruction. Deep learning-based methods use deep neural networks to fit the mapping relationship between LR images and HR images in a "black box" manner, thereby restoring HR images from LR images. Such methods have shown obvious advantages over traditional methods in terms of performance indicators and image quality, and have become mainstream SR methods. However, existing deep learning models usually consider the SR problem of a specific degradation process. It is difficult to handle super-resolution reconstruction tasks with different scale factors, blur kernels, and noise levels using one network model, and it lacks flexibility.

[0004] In recent years, some scholars have combined model-based methods with deep learning techniques to construct model-driven deep networks, called deep unfolding networks. The main idea of ​​deep unfolding is to unfold the optimization steps of solving traditional models into neural networks, convert model optimization variables and hyperparameters into learnable network parameters, and solve the model through the trained network. The prior terms of existing deep unfolding methods often directly use prior functions based on the original image domain, or first use a transformation operator to transform the image to a specific transformation domain, and then learn the prior function based on this transformation domain from the data. However, these unfolding methods often directly use predefined transformation operators commonly used in traditional SR models (such as total variation operators and Laplace operators), ignoring the possibility of simultaneously learning transformation operators and prior functions from data, which limits the extraction of image prior information and SR reconstruction performance. Summary of the invention

[0005] The technical problem to be solved by the present invention is: to overcome the shortcomings of existing super-resolution methods, propose an image super-resolution method based on a deep unfolding network with multiple learnable priors, introduce multiple learnable transformation operators and priors, model the image super-resolution problem, and construct a corresponding unfolding network based on the model, and propose a super-resolution method with high performance, interpretability and flexibility.

[0006] The technical solution adopted by the present invention is: an image super-resolution method based on depth expansion, comprising:

[0007] Step 1: Select high-resolution images from the data set, generate low-resolution images, then combine the high-resolution images with the corresponding low-resolution images to form sample pairs, and divide these sample pairs into training sets and test sets;

[0008] Step 2: Model the super-resolution problem as an optimization problem based on the degradation model and multiple priors, and use the optimization algorithm to obtain two sub-problems that need to be solved iteratively to solve the optimization problem;

[0009] Step 3: Generalize the multiple priors in the subproblem of step 2 into multiple learnable priors, and build a deep unfolding network based on them;

[0010] Step 4: Use the training set obtained in step 1 to train the expanded network constructed in step 3 to obtain a trained network model;

[0011] Step 5: Use the test set obtained in step 1 to test the network model obtained in step 4 to obtain a super-resolution result;

[0012] As described in step 1 above, high-resolution images are selected from the data set to generate low-resolution images, and then the high-resolution images and the corresponding low-resolution images are combined into sample pairs, and these sample pairs are divided into training sets and test sets, including:

[0013] ① The process of generating LR image y from HR image x can be expressed as the following degradation model:

[0014]

[0015] Among them, k is the blur kernel. Typical blur kernels include Gaussian kernel and motion blur kernel. is a two-dimensional convolution,↓ s is an s-fold downsampling operation (s is the scale factor), and n is an additive Gaussian noise with a mean of zero and a standard deviation of σ. The high-resolution image and the corresponding low-resolution image form a sample pair.

[0016] ② Divide the image pairs into training set and test set.

[0017] In step 2 above, the super-resolution problem is modeled as an optimization problem based on the degradation model and multiple priors, and an optimization algorithm is used to obtain two sub-problems that need to be solved iteratively to solve the optimization problem, including:

[0018]

[0019] Among them, the first item is the data item, is the prior on the image in L different domains, D l represents the lth transformation operator, is the result of applying multiple transformation operators to x, f(·) represents the prior function, and λ represents the balance coefficient.

[0020] By introducing the auxiliary variable z, the data term and the prior term are decoupled, resulting in the following equivalent problem:

[0021]

[0022] It can be approximately equivalent to:

[0023]

[0024] Where μ is the penalty parameter. Based on the semi-quadratic splitting method, the original problem can be decomposed into iterative solutions to the data subproblem and the prior subproblem, namely:

[0025]

[0026] Where k represents the number of iterations, x k and z k Represent the values ​​of x and z at the kth iteration respectively. Using a fixed penalty parameter μ is often difficult to balance the convergence speed and convergence accuracy of the optimization algorithm. Therefore, different penalty parameters are used in the iteration process. The penalty parameter of the kth iteration is recorded as μ kThe former sub-problem can be solved analytically by using fast Fourier transform, and the latter sub-problem can be regarded as a noise level in the multiple transform domain. To simplify the expression, assume that α k =2μ k σ 2 and The solutions to the above two sub-problems can be expressed as:

[0027]

[0028] in,

[0029]

[0030]

[0031] In the above formula, ⊙s means dividing the previous operand into s×s blocks and multiplying them with the next operand. and represents the discrete Fourier transform and the conjugate of the discrete Fourier transform, is the inverse discrete Fourier transform, and ↑ s They are downsampling and upsampling operations with a scale factor of s, ⊙ represents a dot product operation, 1 represents a full-1 tensor with the same size as x, Representative for D l is the transpose of , Φ(·) is the denoising function that depends on the prior function f(·).

[0032] The above step 3 describes the generalization of the multiple priors in the sub-problem of step 2 into multiple learnable priors, and based on this, a deep unfolding network is constructed, including:

[0033] The multiple transformation operators are generalized into learnable transformation operators. The transformation operator Dl is modeled using a convolutional layer, and the reparameterization block (RepBlock) is used to implement the multiple transformation operators Dl in the expanded network. l Specifically, nine priors are used, namely: L = 9, where the first transformation operator D1 is a unit matrix to retain the features of the original image domain, and the remaining eight transformation operators are different learnable operators to extract image features in different domains. Based on this, multiple learnable transformation operators are applied to x k ,Right now: It can be expressed as:

[0034]

[0035] Among them, Conv i×j (·) indicates a convolutional layer with a convolution sum of i×j, while Sobelv (·), Sobel h (·) and Laplace(·) represent the Sobel operator and Laplace operator in the horizontal and vertical directions respectively. To promote, record

[0036] The solutions to the two sub-problems after generalization can be written as:

[0037]

[0038] in,

[0039]

[0040] Based on the above iterative solution algorithm, the corresponding unfolded network is constructed. The network consists of K stages, each stage corresponds to an iteration, including the data module XNet and the prior module ZNet, which correspond to the update of variables x and z respectively. In addition, a hyperparameter network HNet is used to learn the hyperparameters α of each stage by the noise level σ and the scale factor s k and β k HNet consists of three convolutional layers (the convolution kernel size is 1) and activation functions. The first two activation functions are ReLU functions, and the last activation function is softPlus function. The initial input z0 is an all-zero tensor with the same size as the HR image.

[0041] The data module XNet is used to find the closed solution of the data subproblem. In XNet, the input z k-1 Through the transformation operator Then the results obtained with LR image y, blur kernel k and hyperparameter α k Calculate the intermediate variable d, and Then use tensor 1 as the transformation operator and The input is composed of the output of the operator and the hyperparameter α k Determine the intermediate variable α k F(M). Finally, these intermediate variables are used to calculate the estimated image x k The relationship between XNet's input and output is as follows:

[0042] x k =XNet(z k-1 ,s,k,y,α k )

[0043] The prior module ZNet corresponds to the update of z in the prior subproblem. Traditional model-based methods usually need to first determine the specific expression of the prior top in the subproblem and then solve it. The deep expansion method can learn the prior function from the data. In this paper, we regard the prior subproblem as a denoising problem in a multi-transform domain. The module can be expressed as:

[0044] z k = ZNet(x k , β k )

[0045] Specifically, the operator x k Transform and compare the transformation result with the hyperparameter β k The images are spliced ​​in the channel dimension as the input of the denoiser. This module uses the residual U-shaped network (ResUNet) with the classic U-Net architecture as the denoiser. ResUNet as a whole consists of an encoder and a decoder. The encoder extracts the features of the image at different scales, while the decoder integrates the features of the encoder into the image to be restored. The denoiser adopts a four-layer structure. As the number of encoder and decoder layers increases, the number of channels of its input and output features continues to increase, while the size continues to decrease. The number of channels in each layer is 16, 32, 64 and 64 respectively. The encoder and decoder are composed of basic modules connected in series, and the basic module is the residual block (ResNet Block).

[0046] The above step 4 uses the training set obtained in step 1 to train the expanded network constructed in step 3 to obtain a trained network model, including:

[0047] The proposed network is trained using the Least Absolute Deviation (LAD) loss function, which is defined as follows:

[0048]

[0049] Where N represents the number of HR images in the training set, x i is the i-th HR image in the training set, y i For x i The corresponding LR image, Represents the estimated value of the HR image output by the expanded network when the network parameter is Θ.

[0050] The network model obtained in step 4 is tested using the test set obtained in step 1 to obtain a super-resolution result, including:

[0051] The trained model parameters are loaded into the network model, and then the low-resolution image in the test set is used as input to obtain the reconstructed high-resolution image. The algorithm performance is measured by calculating the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) between the real high-resolution image and the reconstructed image.

[0052] The beneficial effects of the present invention are:

[0053] A novel super-resolution method based on deep expansion is proposed. This method solves the problem that existing algorithms ignore the simultaneous learning of transformation operators and prior functions from data. By proposing a new multiple learnable prior, the simultaneous learning of transformation operators and prior functions is realized based on convolutional layers and common image operation operators, breaking the limitations of predefined transform domains or prior functions on the extraction of image prior information and the improvement of SR reconstruction performance. It has a wide range of application scenarios in the fields of medicine, remote sensing and video surveillance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the present invention.

[0055] Figure 2 A schematic diagram of a network model implemented in the present invention.

[0056] Figure 3 It is a schematic diagram of the structure of RepBlock in the network model implemented in the present invention.

[0057] Figure 4 This is a schematic diagram of the XNet module structure in the network model implemented in the present invention.

[0058] Figure 5 It is a schematic diagram of the structure of the ZNet module in the network model implemented in the present invention.

[0059] Figure 6 These are the experimental results of the present invention on the WHU-RS19 database. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. The following embodiments are used to illustrate the present invention but are not used to limit the protection scope of the present invention.

[0061] Embodiment 1:

[0062] This example conducts super-resolution experiments on the UC Merced Land-Use database. The UC Merced Land-Use dataset is an image remote sensing dataset of land use in urban areas across the United States. It contains 21 categories, such as farmland, airports, and bushes, and each category has 100 remote sensing images. Figure 1 As shown, the specific implementation steps are as follows:

[0063] In step 1 of the example, high-resolution images are selected from the data set to generate low-resolution images. The high-resolution images and the corresponding low-resolution images are then combined into sample pairs, and these sample pairs are divided into training sets and test sets:

[0064] Ten categories are selected from the dataset, 40 images in each category are used as training set, and 10 images are used as test set.

[0065] The specific steps to generate a low-resolution image from a high-resolution image include blurring, downsampling, and adding additive noise.

[0066] In step 2 of the example, the super-resolution problem is modeled as an optimization problem based on the degradation model and multiple priors, and the optimization algorithm is used to obtain two sub-problems that need to be solved iteratively to solve the optimization problem:

[0067] For the selection of blur, three typical blur kernels are used: isotropic Gaussian blur kernel, heterotropic Gaussian blur kernel and motion blur kernel. Four different kernels are used for each type of blur kernel to ensure that the generated LR images are diverse and representative. In order to fully test the network performance under different degradation conditions, experiments are conducted with noise intensity of 0 and scale factors of 2, 3, and 4.

[0068] In step 3 of the example, the multiple priors in the subproblem of step 2 are generalized into multiple learnable priors, and a deep unfolding network is constructed based on this:

[0069] The constructed expanded network is as follows Figure 2 As shown, the RepBlock is as follows Figure 3 As shown, the XNet module is Figure 4 As shown in Figure 5 shown.

[0070] In step 4 of the example, the training set obtained in step 1 is used to train the expanded network constructed in step 3 to obtain a trained network model:

[0071] For the image pairs in the training set, they are cropped to a size of 96×96, and the data is enhanced by operations such as rotation or mirror inversion. The optimizer selects Adam with β1=0.9 and β2=0.99, and the learning rate is initialized to 10 -4 , the training was performed 5000 times.

[0072] In example step 5, the network model obtained in step 4 is tested using the test set obtained in step 1 to obtain a super-resolution result.

[0073] To verify the effectiveness of this method, this method (MLPUN) is compared with three existing deep learning methods and two deep unfolding methods under the same experimental environment settings. The deep learning-based methods include Information Multi-Distillation Network (IMDN), Multi-Scale Attention Network (MAN) and Lightweight Network Based on Reparameterization (RepRFN). The deep unfolding-based methods include Deep Unfolding Network for Remote Sensing Images (LDUM) and Unfolding Super-Resolution Network (USRNet). In order to measure the performance of different methods, we evaluate the algorithms with the average values ​​of PSNR and SSIM of the high-resolution images reconstructed in the test set.

[0074] It can be seen from the above table that compared with other algorithms, especially algorithms based on deep learning, the method of the present invention shows obvious advantages in PSNR and SSIM.

[0075] Table 1 Super-resolution results of different methods on the UC Merced Land-Use database

[0076] method PSNR(dB) SSIM(%) Method of the present invention 28.50 77.93 IMDN 24.29 61.08 MAN 23.57 59.88 RepRFN 23.94 59.12 LDUM 22.88 55.98 USRNet 28.25 77.25

[0077] Embodiment 2:

[0078] This example conducts a super-resolution experiment on the WHU-RS19 database. WHU-RS19 was released by Wuhan University in 2011 and contains 19 remote sensing images including football fields, mountains and forests. Each image has 50 samples. These images come from satellites in different regions and have strong diversity in direction, angle, etc. The other operation steps are the same as those in Example 1. The super-resolution comparison of the present invention and other comparison algorithms on the WHU-RS19 database Figure 6 shown.

[0079] Table 2 Super-resolution results of different methods on the WHU-RS19 database

[0080] method PSNR(dB) SSIM(%) Method of the present invention 31.10 83.38 IMDN 27.07 69.20 MAN 24.92 62.50 RepRFN 26.45 66.47 LDUM 25.40 62.56 USRNet 30.99 83.41

[0081] From the perspective of the two performance indicators of super-resolution, the present invention has a great advantage over other methods. Since the present invention takes into account multiple learnable priors of the image, its performance is better than that of USRNet which only uses image domain priors.

Claims

1. An image super-resolution method based on multiple learnable prior deep unfolding networks, characterized in that: The steps include: Step 1: Select high-resolution images from the data set, generate low-resolution images, then combine the high-resolution images with the corresponding low-resolution images to form sample pairs, and divide these sample pairs into training sets and test sets; Step 2: Model the super-resolution problem as an optimization problem based on the degradation model and multiple priors, and use the optimization algorithm to obtain two sub-problems that need to be solved iteratively to solve the optimization problem; Step 3: Generalize the multiple priors in the subproblem of step 2 into multiple learnable priors, and build a deep unfolding network based on them; Step 4: Use the training set obtained in step 1 to train the expanded network constructed in step 3 to obtain a trained network model; Step 5: Use the test set obtained in step 1 to test the network model obtained in step 4 to obtain super-resolution results.

2. The image super-resolution method based on depth expansion according to claim 1, characterized in that: In step 1, a high-resolution image is selected from the data set to generate a low-resolution image, and then the high-resolution image and the corresponding low-resolution image are combined into sample pairs, and these sample pairs are divided into a training set and a test set, including: The process of generating a low-resolution image y from a high-resolution image x can be expressed as the following degradation model: Among them, k is the blur kernel. Typical blur kernels include Gaussian kernel and motion blur kernel. is a two-dimensional convolution,↓ s It is an s-fold downsampling operation (s is the scale factor), n is the additive Gaussian noise with mean zero and standard deviation σ, and the high-resolution image and the corresponding low-resolution image are combined into sample pairs, and finally the image pairs are divided into training set and test set.

3. The image super-resolution method based on depth expansion according to claim 1, characterized in that: In step 2, the super-resolution problem is modeled as an optimization problem based on the degradation model and multiple priors, and an optimization algorithm is used to obtain two sub-problems that need to be solved iteratively to solve the optimization problem, including: Among them, the first item is the data item, is the prior on the image in L different domains, D l represents the lth transformation operator, f(·) represents the prior function, and λ represents the balance coefficient. By introducing the auxiliary variable z, the data term and the prior term are decoupled, and the equivalent problem shown below is obtained: It can be approximately equivalent to: Where μ is the penalty parameter. Based on the semi-quadratic splitting method, the original problem can be decomposed into iterative solutions to the data subproblem and the prior subproblem, namely: Where k represents the number of iterations, x k and z k Represent the values ​​of x and z at the kth iteration respectively. It is often difficult to balance the convergence speed and convergence accuracy of the optimization algorithm by using a fixed penalty parameter μ. Therefore, different penalty parameters are used in the iteration process. The penalty parameter of the kth iteration is recorded as μ k The former sub-problem can be solved analytically by using fast Fourier transform, and the latter sub-problem can be regarded as a noise level in the multiple transform domain. To simplify the expression of the denoising problem, we assume that α k =2μ k σ 2 and The solutions to the above two sub-problems can be expressed as: in, In the above formula, ⊙s means dividing the previous operand into s×s blocks and multiplying them with the next operand. and represents the discrete Fourier transform and the conjugate of the transform result, is the inverse discrete Fourier transform, and ↑s are downsampling and upsampling operations with a scale factor of s, respectively. ⊙ represents a dot product operation, and 1 represents an all-1 tensor of the same size as x. Representative for D l is the transpose of , Φ(·) is the function determined by f(·).

4. The image super-resolution method based on depth expansion according to claim 1, characterized in that: The step 3 described in the above generalizes the multiple priors in the sub-problem of step 2 into multiple learnable priors, and constructs a deep unfolding network based on this, including: The multiple transformation operators are generalized into learnable transformation operators, and the transformation operator D is transformed using convolutional layers. l Modeling is performed and the reparameterization block (RepBlock) is used to implement the multiple transformation operators D in the unfolding network. l Specifically, nine priors are used, namely: L = 9, where the first transformation operator D1 is the unit matrix to retain the features of the original image domain, and the remaining eight transformation operators are different learnable operators to extract image features in different domains. Based on this, multiple learnable transformation operators are applied to x k ,Right now: It can be expressed as: Among them, Conv i×j (·) indicates a convolutional layer with a convolution sum of i×j, while Sobel v (·), Sobel h (·) and Laplace(·) represent the Sobel operator and Laplace operator in the horizontal and vertical directions respectively. To promote, record The solutions to the two sub-problems after generalization can be written as: in, Based on the above iterative solution algorithm, the corresponding unfolding network is constructed. The network consists of K stages, each stage corresponds to an iteration, including the data module XNet and the prior module ZNet, which correspond to the update of variables x and z respectively. In addition, a hyperparameter network HNet is used to learn the hyperparameters α of each stage by the noise level σ and the scale factor s k and β k ,HNet consists of three convolutional layers (the convolution kernel size is 1) and activation functions. The first two activation functions are ReLU functions, and the last activation function is sofiPlus function. The initial input z0 is an all-zero tensor with the same size as the HR image; The data module XNet is used to find the closed solution of the data subproblem. In XNet, the input z k-1 Through the transformation operator Then the obtained results with LR image y, blur kernel k and hyperparameter α k Calculate the intermediate variable d, and Then use tensor 1 as the transformation operator and The input is composed of the output of the operator and the hyperparameter α k Determine the intermediate variable α k F(M), and finally use these intermediate variables to calculate the estimated image x k , the relationship between XNet’s input and output is as follows: x k =XNet(z k-1 ,s,k,y,α k ) The prior module ZNet corresponds to the update of z in the prior subproblem. Traditional model-based methods usually need to first determine the specific expression of the prior term in the subproblem and then solve it. The deep expansion method can learn the prior function from the data and regard the prior subproblem as a denoising problem in multiple transformation domains. The module can be expressed as: from k =ZNet(x k ,β k ) Specifically, the operator x k Transform and compare the transformation result with the hyperparameter β k The images are spliced ​​in the channel dimension as the input of the denoiser. This module uses the residual U-shaped network (ResUNet) with the classic U-Net architecture as the denoiser. The ResUNet as a whole is composed of an encoder and a decoder. The encoder extracts the features of the image at different scales, while the decoder integrates the features of the encoder into the image to be restored. The denoiser adopts a four-layer structure. As the number of encoder and decoder layers increases, the number of channels of its input and output features continues to increase, while the size continues to decrease. The number of channels in each layer is 16, 32, 64 and 64 respectively. The encoder and decoder are composed of basic modules in series, and the basic module is the residual block (ResNet Block).

5. The image super-resolution method based on depth expansion according to claim 1, characterized in that: In step 4, the proposed network is trained using the minimum absolute deviation loss function.

Images