Remote sensing image super-resolution method based on depth expansion and generative denoising device
By using a combination method of deep expansion network and a generative denoiser in the remote sensing image super-resolution, the problems of insufficient interpretability and poor generalization capabilities in the prior art are solved, and a more efficient and generalized remote sensing image super-resolution effect is achieved.
Patent Information
- Application Number
- CN202411364293.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-02-25
AI Technical Summary
The existing remote sensing image super-resolution methods have problems with insufficient interpretability and poor generalization capabilities, and the existing deep expansion-based methods are limited to image degradation of fixed scale factors and specific blurred cores.
Using a remote sensing image super-resolution method based on deep expansion and generative denoiser, by building a deep expansion network, the traditional optimization algorithm is expanded into a neural network that can be trained end-to-end, and combined with a generative denoiser, we learn super-resolution models suitable for different image degradation situations.
The effect of the super-resolution of remote sensing images is improved, the interpretability and versatility of the method are enhanced, and the image degradation of different fuzzy cores, proportional factors and noise levels can be effectively handled.
Smart Images
Figure CN119515680A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a remote sensing image super-resolution method based on deep expansion and a generative denoiser. Background Art
[0002] Image super-resolution reconstruction refers to the process of restoring a natural and clear high-resolution (HR) image from a low-resolution (LR) image. It is one of the classic problems in computer vision, has a wide range of applications in the real world, and plays a cornerstone role in high-level computer vision tasks. The existing mainstream methods in the field of remote sensing image super-resolution can be roughly divided into three categories: model-based methods, deep learning-based methods, and hybrid methods that combine the first two methods.
[0003] Although the model-based method has strong theoretical support, it often needs to rely on artificial experience to design prior constraints and lacks adaptability. Due to the powerful modeling ability of deep neural network models, convolutional neural network models are also widely used in remote sensing image restoration tasks and have greatly improved the quality of image super-resolution. Unlike traditional model-based remote sensing image super-resolution methods, deep learning-based methods regard remote sensing image super-resolution tasks as function fitting problems. By designing different network structures and loss functions suitable for different image restoration tasks, the mapping relationship between low-quality LR images and high-quality HR images is fitted in an end-to-end manner on a large number of paired LR images and corresponding HR images. Although the deep learning-based method has achieved good results, it still has some defects. On the one hand, most of the network modules in the existing deep learning methods are designed based on experience, and the entire model structure can be regarded as a "black box", so it is difficult to analyze the role of different modules and understand their working mechanism, resulting in the lack of interpretability of the model and limiting the further improvement of the performance of deep learning methods. On the other hand, most methods directly learn the mapping between LR images and HR images, while ignoring the physical model of image degradation. Therefore, the neural network model is prone to overfitting during training and leads to poor generalization ability.
[0004] In order to combine the advantages of both model-based methods and deep learning-based methods, remote sensing image super-resolution methods based on deep unfolding have been proposed. Such methods usually unfold the traditional optimization algorithm into a deep neural network that can be trained end-to-end, and one stage of the network corresponds to one iteration of the algorithm. The hybrid method inherits the interpretability and adaptability of the first two methods, and can also achieve better super-resolution performance. However, the existing remote sensing image super-resolution methods based on deep unfolding are usually limited to image degradation with fixed scale factors and specific blur kernels, and the learned models cannot be directly applied to super-resolution tasks caused by image degradation caused by other scale factors and blur kernels. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects and shortcomings of existing image super-resolution algorithms, and to provide a remote sensing image super-resolution method based on depth expansion and a generative denoiser, so as to further improve the effect of remote sensing image super-resolution.
[0006] To achieve the above purpose, the technical solution adopted is as follows.
[0010] The present invention discloses a remote sensing image super-resolution method based on deep expansion and a generative denoiser, comprising the following steps: S1. Construct a data set: Generate corresponding LR images from the original HR images in the data set. The HR images and the corresponding LR images form image pairs, and the image pairs are divided into training sets and test sets. S2. Construct an optimization solution algorithm: Model the super-resolution problem as a minimization problem based on the degradation model, and construct a corresponding optimization solution algorithm. S3. Construct a deep unfolding network: Expand the optimization solution algorithm constructed in step S2, and combine it with a generative denoiser to construct a deep unfolding network that can be trained end-to-end. S4. Train the deep unfolding network constructed in step S3: Based on the training set obtained in step S1, given a loss function, use the back propagation algorithm and the stochastic gradient descent algorithm to continuously optimize the parameters in the deep unfolding network until the loss function value is stable. S5. Use the trained deep unfolding network for image super-resolution: input the LR image, and the network output is the reconstructed HR image.
[0011] Preferably, in the remote sensing image super-resolution method based on deep expansion and generative denoiser, in S1, each HR image is downsampled, blurred and noised respectively, that is:
[0012] Among them, y is the LR image, x is the HR image, is the convolution operation, k is the blur kernel, ↓ srepresents downsampling with a scale factor of s, and n represents Gaussian noise with a mean of zero and a standard deviation of σ. The original HR image and its corresponding LR image are composed into image pairs, and the image pairs are divided into training sets and test sets.
[0013] Preferably, in the remote sensing image super-resolution method based on deep expansion and generative denoiser, in S2, based on the degradation model (1) used in S1, the super-resolution problem is modeled as the following minimization problem: in is the data term, φ(x) is the prior term, and λ is the trade-off parameter. In order to optimize the solution to problem (2), a semi-quadratic splitting algorithm is used to decouple the data term and the prior term by introducing an auxiliary variable z, thereby obtaining the following approximate equivalent problem: Where μ is the penalty parameter; the original problem can be solved by iteratively solving the subproblems of z and x, namely the data subproblem and the prior subproblem: Where i represents the number of iterations, z i and x i They represent the variables updated in the i-th iteration. To speed up the convergence of the algorithm, different μ is used in each iteration. For convenience, μ in the i-th iteration is replaced by μ i For subproblem (4), the closed solution can be obtained by fast Fourier transform, namely: Where d is: in F() and F -1 () represent Fourier transform and inverse Fourier transform respectively, represents the conjugate of F(k) after Fourier transformation, ⊙ s Indicates that first Splice along the horizontal and vertical directions, and then compare the splicing results with Element-wise multiplication, = means that the downsampled object is divided into multiple non-overlapping independent blocks of size s×s, and the average value of each block is taken for downsampling; ↑s means s-fold upsampling; for subproblem (5), from a Bayesian perspective, it actually corresponds to a noise level of denoising problem. Preferably, in the remote sensing image super-resolution method based on deep unfolding and generative denoiser, in S3, the optimization solution algorithm constructed in S2 is unfolded, combined with the generative denoiser, to construct a deep unfolding network that can be trained end-to-end; this network consists of K repeated stages, each of which includes two modules, DataNet and PriNet. At the beginning of the unfolding network, the original LR image y is interpolated to obtain an image x0 with the same size as the expected HR image, and x0 is used as the input of the first stage of the unfolding network; based on the scale factor s and the noise level port, HPNet is used to learn the hyperparameters α required for each stage i and β i .
[0014] For the closed solution of the data subproblem (6), the DataNet module is constructed, and x0 is obtained by nearest neighbor interpolation of the LR image y with a scale factor s. Based on formula (6), the DataNet module can be expressed as: z i =DataNet(x i-1 ,s,k,y,α i ) (7)
[0015] The PriNet module uses a generative denoiser to optimise the noise level β i Image z i Perform denoising to obtain a higher resolution image x i , which can be expressed as: x i =PriNet(z i , β i ) (8) This module uses the generator of Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) as a denoiser. The generator of ESRGAN consists of an RRDB module, an upsampling module and a convolution module: the RRDB module uses a nested residual structure to enhance network capacity, the upsampling module uses nearest neighbor interpolation to upsample the image, and the convolution module changes the number of image channels through a convolution layer.
[0016] The HPNet module is used to learn the hyperparameters required for DataNet and PriNet. HPNet consists of three convolutional layers, each with a convolution kernel size of 1. The first two layers use the ReLU activation function, and the last layer uses the Softplus activation function. Let α = {α1, α2, ..., α i}, β={β1, β2,...,β i}, the HPNet module can be expressed as: {α, β} = HPNet(σ, s) (9)
[0017] Preferably, in the above-mentioned remote sensing image super-resolution method based on deep expansion and generative denoiser, in S4, based on a training data set and a given loss function, back propagation and stochastic gradient descent algorithms are used to continuously optimize the parameters in the deep expansion network until the loss function value is stable, thereby obtaining the optimal parameters of the deep expansion network.
[0018] Preferably, in the remote sensing image super-resolution method based on deep expansion and generative denoiser, in S4, the training set constructed in S1 is used, and the mean absolute error is used as the loss function of the network: Where M represents the total number of image pairs in the training set, y i is the result of the deep expansion network output, x i is the original image, and θ is the parameter of the deep expansion network.
[0019] Preferably, in the above-mentioned remote sensing image super-resolution method based on deep expansion and generative denoiser, in S5, the trained model parameters are loaded into the deep expansion network model, and then the LR image in the test set constructed in S1 is used as input, and the output is the image after super-resolution.
[0020] Beneficial effects of the present invention:
[0021] The remote sensing image super-resolution method based on deep expansion and generative denoiser of the present invention can reconstruct high-quality HR remote sensing images from LR remote sensing images; compared with the existing model-based remote sensing image super-resolution algorithm and deep learning-based remote sensing image super-resolution algorithm, the method of the present invention has better interpretability and higher reconstruction accuracy; compared with the existing remote sensing image super-resolution algorithm based on deep expansion network, the present invention expands its versatility while maintaining the good interpretability of the deep expansion network, and can solve the super-resolution problem of remote sensing images with different blur kernels, scale factors and noise levels. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is an implementation flow chart of the remote sensing image super-resolution method based on deep expansion and generative denoiser of the present invention
[0023] Figure 2 It is a structural diagram of the deep expansion network involved in the method of the present invention
[0024] Figure 3 The structure diagram of the PriNet module in the depth expansion network involved in the present invention
[0025] Figure 4 This is the structural diagram of the HPNet module in the depth expansion network involved in the present invention
[0026] Figure 5 The results of image super-resolution of the present invention and other comparison algorithms in the UC Merced Land-Use database DETAILED DESCRIPTION
[0027] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.
[0028] Example 1:
[0029] This example conducts a remote sensing image super-resolution experiment on the UC Merced Land-Use database. UC MercedLand-Use is a remote sensing image dataset that contains 21 types of scenes, each with 100 images, and each image size is 256×256.
[0030] like Figure 1 As shown, the specific implementation steps are as follows:
[0031] S1. Construct a dataset: Generate corresponding LR images from HR images in the dataset. HR images and their corresponding LR images form image pairs, and divide the image pairs into training sets and test sets. Select the first ten categories of scenes from UC Merced Land-Use, select 40 images from each category as training samples, and select 5 images as test samples. A total of 400 images are used for training and 50 for testing. The steps of generating corresponding LR images from HR images in the dataset include blurring, downsampling, and adding additive noise.
[0032] S2. Construct an unwrapped optimization method: Based on the degradation model, the super-resolution problem is modeled as a minimization problem, and the corresponding solution algorithm is constructed; the blur kernels used in the degradation model include 4 isotropic Gaussian kernels, 4 anisotropic Gaussian kernels, and 4 motion blur kernels. The scale factors are 2, 3, and 4. When the scale factors are 2 and 4, the noise level is set to 0, and when the scale factor is 3, the noise level is set to 0, 2.5, and 7.5.
[0033] S3. Construct a deep unfolding network: Construct a neural network module to solve the data sub-problem and the prior sub-problem. These two sub-problems are solved alternately through the neural network module to obtain an iterative network that can be trained end-to-end. The constructed unfolding network is as follows: Figure 2 As shown, the number of network stages K = 8, and the PriNet module is as follows Figure 3 As shown, the HPNet module is Figure 4 shown.
[0034] S4. Training the deep unfolding network: Based on the training data set and a given loss function, the back propagation and gradient descent algorithms are used to continuously optimize the parameters in the deep unfolding network until the loss function is stable. All image pairs consisting of high-resolution images and low-resolution images in the training set are used as training data. The proposed model is trained for 5000 rounds using the Adam optimizer, and the learning rate is set to 10 -4 .
[0035] S5, apply the trained deep expansion network for image super-resolution: input the LR image, and the network output is the image after super-resolution. Through the training process of S4, the optimal deep expansion network parameters can be determined. Based on the trained model, when performing image super-resolution, the LR image is sent to the trained deep expansion network, and the image output by the network is the image after super-resolution.
[0036] In the experiment, the present invention was tested on UC Merced Land-Use. The super-resolution of the present invention and other comparison algorithms on the UC Merced Land-Use database is as follows: Figure 5 As shown, the scale factor is 3 and the noise intensity is 0. In order to objectively evaluate the super-resolution accuracy of different methods, the peak signal-to-noise ratio (PSNR) and the average value of structural similarity (SSIM) are used as comparison indicators.
[0037] Table 1. Average peak signal-to-noise ratio (PSNR, in dB) and average structural similarity (SSIM) results of different methods under different combinations of scale factor, blur kernel and noise level. The best results are marked in bold. s σ IMDN MAN DPSR USRNet The present invention x2 0 31.08dB / 85.76% 30.81dB / 85.25% 30.36dB / 84.31% 30.45dB / 84.52% 31.66dB / 87.56% x3 0 28.36dB / 78.01% 28.18dB / 77.60% 28.03dB / 77.24% 28.19dB / 77.83% 28.81dB / 79.94% x3 2.5 27.49dB / 74.10% 27.37dB / 73.82% 27.31dB / 73.59% 27.44dB / 74.46% 27.79dB / 75.32% x3 7.5 26.18dB / 68.12% 26.12dB / 67.97% 26.01dB / 67.15% 26.20dB / 68.64% 26.39dB / 69.09% x4 0 26.06dB / 68.89% 26.01dB / 68.59% 25.86dB / 68.21% 26.12dB / 69.41% 26.49dB / 71.02%
[0038] As shown in Table 1 above, the deep unfolding network proposed in the present invention is compared with five remote sensing image super-resolution methods under different scale factors, noise levels and blur kernel combinations. The compared methods include: IMDN, MAN, HSENet, DPSR, and USRNet. It can be seen that the deep unfolding network proposed in the present invention can achieve higher super-resolution accuracy at different scale factors, noise levels and blur kernels.
[0039] Example 2:
[0040] This example conducts super-resolution experiments on the WHU-RS19 database. The WHU-RS19 database is a database specifically used for remote sensing image super-resolution tasks. The database was released by Wuhan University and contains 19 different scene categories, each with up to 50 samples, and the resolution of each sample is 600×600 pixels. The first ten categories of scenes are selected from WHU-RS19, 40 images are selected from each category as training samples, and 5 images are selected as test samples; a total of 400 images are used for training and 50 images are used for testing. The other steps are the same as Example 1.
[0041] Table 2 Average peak signal-to-noise ratio (PSNR, in dB) and average structural similarity (SSIM) results of different methods under different combinations of scale factor, blur kernel and noise level. The best results are marked in bold. s σ IMDN MAN DPSR USRNet The present invention x2 0 33.60dB / 90.42% 33.44dB / 90.14% 33.44dB / 89.98% 33.61dB / 90.65% 34.11dB / 91.18% x3 0 30.59dB / 82.81% 30.52dB / 82.87% 30.53dB / 82.82% 30.76dB / 83.71% 31.07dB / 84.28% x3 2.5 29.58dB / 78.87% 29.51dB / 78.85% 29.52dB / 78.73% 29.62dB / 79.05% 29.90dB / 79.86% x3 7.5 28.09dB / 72.45% 28.06dB / 72.47% 27.96dB / 71.74% 28.15dB / 72.88% 28.32dB / 73.20% x4 0 28.36dB / 74.48% 28.36dB / 74.71% 28.35dB / 74.71% 28.59dB / 75.88% 28.84dB / 76.34%
[0042] As shown in Table 2 above, the deep unfolding network proposed in the present invention is compared with five remote sensing image super-resolution methods under different scale factors, noise levels and blur kernel combinations. It can be seen from Table 2 that the deep unfolding network proposed in the present invention can still achieve higher super-resolution accuracy on the WHU-RS19 database.
Claims
1. A remote sensing image super-resolution method based on deep expansion and generative denoiser, characterized in that: The steps include: S1. Construct a data set: Generate corresponding LR images from the original HR images in the data set. The HR images and the corresponding LR images form image pairs, and the image pairs are divided into training sets and test sets. S2. Construct an optimization solution algorithm: Model the super-resolution problem as a minimization problem based on the degradation model, and construct a corresponding optimization solution algorithm; S3, constructing a deep unfolding network: unfolding the optimization solution algorithm constructed in step S2, combining it with a generative denoiser, to construct a deep unfolding network that can be trained end-to-end; S4, training the deep unfolded network constructed in step S3: based on the training set obtained in step S1, given a loss function, using a back propagation algorithm and a stochastic gradient descent algorithm to continuously optimize the parameters in the deep unfolded network until the loss function value is stable; S5. Use the trained deep unfolding network for image super-resolution: input the LR image, and the network output is the reconstructed HR image.
2. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S1, each HR image is downsampled, blurred and noised respectively, that is: Wherein, y is LR image, x is HR image, is the convolution operation, k is the blur kernel, ↓ s represents downsampling with a scale factor of s, and n represents Gaussian noise with a mean of zero and a standard deviation of σ. The original HR image and its corresponding LR image are composed into image pairs, and the image pairs are divided into training sets and test sets.
3. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S2, based on the degradation model (1) used in S1, the super-resolution problem is modeled as the following minimization problem: in is the data term, φ(x) is the prior term, and λ is the trade-off parameter. In order to optimize the solution to problem (2), a semi-quadratic splitting algorithm is used to decouple the data term and the prior term by introducing an auxiliary variable z, thereby obtaining the following approximate equivalent problem: Where μ is the penalty parameter; the subproblems of z and x can be solved iteratively, namely the data subproblem and the prior subproblem, Solve the original problem: Where i represents the number of iterations, z i and x i They represent the variables updated in the i-th iteration. To speed up the convergence of the algorithm, different μ is used in each iteration. For convenience, μ in the i-th iteration is replaced by μ i For subproblem (4), the closed solution can be obtained by fast Fourier transform, namely: Where d is: in F() and F -1 () represent Fourier transform and inverse Fourier transform respectively, represents the conjugate of F(k) after Fourier transformation, ⊙s represents the conjugate of s×s Splice along the horizontal and vertical directions, and then compare the splicing results with Element-wise multiplication, = means that the downsampled object is divided into multiple non-overlapping independent blocks of size s×s, and the average value of each block is taken for downsampling; ↑s means s-fold upsampling; for subproblem (5), from a Bayesian perspective, it actually corresponds to a noise level of denoising problem.
4. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S3, the optimization solution algorithm constructed in S2 is expanded and combined with a generative denoiser to construct a deep expanded network that can be trained end-to-end; the network consists of K repeated stages, each of which includes two modules, DataNet and PriNet. At the beginning of the expanded network, the original LR image y is interpolated to obtain an image x0 with the same size as the expected HR image, and x0 is used as the input of the first stage of the expanded network; Based on the scale factor s and the noise level σ, HPNet is used to learn the hyperparameters α required for each stage. i and β i .
5. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S4, based on the training data set and a given loss function, back propagation and stochastic gradient descent algorithms are used to continuously optimize the parameters in the deep unfolding network until the loss function value is stable, thereby obtaining the optimal parameters of the deep unfolding network.
6. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S4, the training set constructed in S1 is used, and the mean absolute error is adopted as the loss function of the network: Where M represents the total number of image pairs in the training set, and y i is the result of the deep expansion network output, x i is the original image, and θ is the parameter of the deep expansion network.
7. The remote sensing image super-resolution method based on deep expansion and generative denoiser according to claim 1, characterized in that: In step S5, the trained model parameters are loaded into the deep expansion network model, and the LR image in the test set constructed in S1 is used as input, and the output is the image after super resolution.