A real degradation based spatially variable kernel perceptual blind super-resolution reconstruction method

Abstract

The application discloses a remote sensing image super-resolution reconstruction method based on spatial variable degradation perception, and comprises the following steps: S1, acquiring a high-resolution remote sensing image dataset; S2, constructing an image degradation model; S3, constructing a spatial variable degradation perception blind super-resolution model; S4, training a remote sensing image blind super-resolution model based on spatial variable degradation perception; and S5, reconstructing a high-resolution remote sensing image through the trained blind super-resolution model. The blind super-resolution reconstruction task is decomposed into a degradation process and a reconstruction process, on the one hand, the complex degradation process of an image is simulated, and on the other hand, the degradation conditions of different spatial positions of the image are considered, the degradation kernel closer to the real world is evaluated while the perception degradation range is expanded, serious kernel estimation deviation is avoided, and more accurate remote sensing image super-resolution reconstruction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, more particularly to a remote sensing image super-resolution reconstruction method based on spatially variable degradation perception. BACKGROUND

[0002] In recent years, remote sensing images are widely used in various computer vision tasks, such as target detection, semantic segmentation and scene understanding. Due to the influence of factors such as radiation, platform vibration, optical system defocus, image compression, etc. in the capture and transmission process of remote sensing images, most remote sensing images have low resolution and noise, which limits the practical application range of remote sensing images. Image super-resolution (SR) refers to the process of recovering a high-resolution (HR) image from a corresponding low-resolution (LR) image, which can improve the spatial resolution of the image without being limited by hardware. Therefore, as the basis for other computer vision tasks, image super-resolution technology has received widespread attention from scholars.

[0003] Super-resolution algorithms based on deep learning have achieved excellent performance on paired LR-HR training datasets by learning the mapping relationship from LR to HR. However, paired training data is usually difficult to obtain, and LR images are generally obtained from HR images through interpolation algorithms, ignoring the complex degradation process of images in the real world, so the reconstruction performance is unstable. Subsequently, non-blind super-resolution methods guide image reconstruction by regarding the blur kernel as a fixed degradation kernel (such as a bicubic interpolation kernel), however, the method relies on the similarity between the degradation kernel and the true kernel, and the performance declines severely in real scenes. Blind super-resolution methods focus on simulating complex image degradation processes and designing special networks to extract blur kernels to eliminate noise and blur, which have better generalization ability and significantly better performance than non-blind SR methods. However, existing blind super-resolution methods use one blur kernel for the entire image when reconstructing the blur kernel, while images usually contain different degradations at different locations. Estimating the kernel on the entire image will inevitably produce serious estimation bias, and cannot perceive the overall degradation of the image. SUMMARY

[0004] In view of the defects in the prior art, the present application aims to provide a remote sensing image super-resolution reconstruction method based on spatially variable degradation perception.

[0005] The technical solution adopted by the present application to solve the above technical problems comprises the following steps:

[0006] Step S1, obtaining a high-resolution remote sensing image dataset: the training dataset is 31500 high-quality un-degraded remote sensing images in the NWPU-RESISC45 dataset;

[0007] Step S2, constructing an image degradation model: randomly generating an anisotropic Gaussian blur kernel k l and a Gaussian white noise n l , degrading a high-resolution image I HR into a low-resolution image I LR ; inputting I LR into the kernel evaluator and the noise evaluator to obtain the degradation kernel k h and the noise n h ;

[0008] Step S3, constructing a spatially variable degradation-aware blind super-resolution model: including a spatially variable kernel evaluation subnetwork, a noise level evaluation subnetwork, and a degradation-aware reconstruction subnetwork;

[0009] Step S4, training a spatially variable degradation-aware remote sensing image blind super-resolution model: setting hyperparameters, updating network parameters using the Adam optimizer to minimize the loss function, and completing model training;

[0010] Step S5, reconstructing a high-resolution remote sensing image through the trained blind super-resolution model: selecting a set of network weights with the highest evaluation index value, saving it as the optimal super-resolution reconstruction model, and using it to reconstruct high-resolution remote sensing images later.

[0011] Further, in step S2, the image degradation model is:

[0012]

[0013] where I LR and I HR represent the degraded low-resolution image and the high-resolution image, respectively, k, ↓ s , and n represent the blur kernel, the downsampling with a scale factor s, and the noise, respectively; the downsampling method is bicubic interpolation sampling.

[0014] Further, in step S3, the kernel evaluator is composed of a 5x5 convolution layer, five residual convolution groups (one 3x3 convolution layer, one LeakyReLu, and one 3x3 convolution layer in series), and a channel attention module, which is used to evaluate the image degradation kernel; the noise evaluator is composed of four transpose convolution groups (one transpose convolution and one PRelu activation function in series), four convolution groups (3x3 convolution and PRelu activation function in series), one sigmoid function, and one 3x3 convolution, which is used to evaluate the noise feature.

[0015] Further, in step S3, the spatially variable kernel evaluation subnetwork passes through a blur kernel extraction layer, a weight evaluation layer, and a mask aggregation layer to obtain the evaluated real blur kernel, which includes:

[0016] The low-resolution image is input into a blur kernel extraction layer, and a kernel evaluator is used to extract blur kernels at different positions of the image to form candidate blur kernel features;

[0017] The candidate blur kernel is input into a weight evaluation layer, and a Gumbel Softmax reparameterization operation is used to obtain different kernel weights to form a kernel weight set;

[0018] The kernel weight set and the candidate blur kernel are mask aggregated to obtain the evaluated real blur kernel.

[0019] Further, in step S3, in the noise level evaluation subnetwork, the input image is first extracted by the noise evaluator to extract the noise features of the input image, and then the noise level is obtained by calculating the variance of the variable.

[0020] Further, in step S3, the degradation perception reconstruction subnetwork includes a feature extraction module, an adaptive degradation perception module, and an upsampling module. The feature extraction module alternately uses three 3x3 convolution layers and two LeakyReLU activation layers to extract shallow features of the low-resolution image; the adaptive degradation perception module is responsible for estimating multi-kernel degradation information and restoring damaged feature information using the same; and the upsampling module is composed of PixelShuffle, convolution layers, and LeakyReLu activation functions, and obtains a high-resolution image by convolution and recombination between multiple channels.

[0021] Further, the shallow features, the real blur kernel, and the noise level are input into the adaptive degradation perception module to eliminate blur and noise, including:

[0022] The blur kernel and the noise are combined into a low-dimensional feature;

[0023] The low-dimensional feature is input into a multi-layer perception and a softmax layer for feature transformation, and is mask aggregated with the shallow features to form residual features;

[0024] The low-dimensional feature is stretched and embedded into a high-dimensional space, and a high-dimensional feature is formed through a shallow convolutional neural network;

[0025] The residual features and the high-dimensional features are residual connected and transmitted into the next adaptive degradation perception module.

[0026] Further, in step S4, the training of the spatially variable kernel perception blind super-resolution reconstruction method based on real degradation is based on L1loss to construct multiple loss functions for training, so that the image degradation model and the spatially variable kernel perception blind super-resolution model are coordinated with each other to complete parameter training, wherein the multiple loss functions are:

[0027]

[0028] wherein, denotes a multi-term loss function, is a loss function of the kernel estimator, is a loss function of the noise estimator, is a loss function of the spatially variable kernel-aware blind super-resolution network, and λ1 and λ2 are balance terms.

[0029] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0030] (1) The kernel estimator and the noise estimator designed in the present application make the extracted degradation kernel and noise level closer to the real degradation process, thereby assisting in reconstructing high-resolution remote sensing images.

[0031] (2) The spatially variable degradation-aware blind super-resolution model designed in the present application expands the perceived degradation range while selecting a degradation kernel closer to the real degradation, thereby achieving more accurate super-resolution reconstruction.

[0032] (3) The present application can effectively eliminate blur and noise while reconstructing high-resolution images with rich texture information, and when applied to remote sensing image blind super-resolution technology, better super-resolution enhancement effect can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of the remote sensing image super-resolution reconstruction method based on spatially variable degradation-aware according to an embodiment of the present application

[0034] Figure 2 is a framework diagram of the image degradation model according to an embodiment of the present application

[0035] Figure 3 is a framework diagram of the spatially variable degradation-aware blind super-resolution model according to an embodiment of the present application

[0036] Figure 4 is a framework diagram of the degradation-aware reconstruction subnetwork according to an embodiment of the present application DETAILED DESCRIPTION

[0037] The technical scheme of the present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0038] EMBODIMENT

[0039] As shown in Figure 1 , a remote sensing image super-resolution reconstruction method based on spatially variable degradation-aware includes the following steps:

[0040] Step S1, obtain a high-resolution remote sensing image dataset: the training dataset is 31500 high-quality un-degraded remote sensing images in the NWPU-RESISC45 dataset.

[0041] Step S2, constructing an image degradation model: the image degradation model structure is as shown in Figure 2 First, an anisotropic Gaussian blur kernel k l and a Gaussian white noise n l are randomly generated, and a high-resolution image I HR is degraded into a low-resolution image I LR , then I LR is input into the kernel evaluator and the noise evaluator to obtain the degradation kernel k h and the noise n h .

[0042] In this embodiment, the image degradation model is:

[0043]

[0044] wherein I LR and I HR represent the degraded image and the high-resolution image respectively. k, ↓ s , and n represent the blur kernel, the down-sampling with a scale factor s, and the noise respectively, and the down-sampling method is bicubic interpolation sampling.

[0045] The kernel evaluator is composed of a 5x5 convolution layer, five residual convolution groups (one 3x3 convolution layer, one LeakyReLu activation function, and one 3x3 convolution layer in series), and one channel attention module, to obtain the degradation kernel k h ; the noise evaluator is composed of four transpose convolution groups (transpose convolution and PReLU activation function in series), four convolution groups (3x3 convolution and PReLU activation function in series), one sigmoid function, and one 3x3 convolution, to obtain the evaluation noise n h .

[0046] Step S3, constructing a spatially variable degradation-aware blind super-resolution model: the spatially variable degradation-aware blind super-resolution model structure is as shown in Figure 3 , which comprises a spatially variable kernel evaluation subnetwork, a noise level evaluation subnetwork, and a degradation-aware reconstruction subnetwork.

[0047] (S21) Spatially variable kernel evaluation subnetwork:

[0048] Since the previous method uses one blur kernel for the whole image, and the image usually contains different degradations at different positions, estimating the kernel on the whole image will inevitably cause serious estimation bias and cannot represent the overall degradation of the image. To solve the above problems, the present application proposes a variable kernel evaluation method, that is, using the kernel evaluator to randomly extract blur kernels at different positions of the input low-resolution image, and then selecting the most real blur kernel according to the weights of different blur kernels.

[0049] Specifically, the variable kernel extraction performs blur kernel extraction on three different block locations of the low-resolution image to obtain candidate blur kernel features k. patch The specific formula can be expressed as:

[0050]

[0051] Where, k LR_patch1 k LR_patch2 k LR_patch3 These represent the three extracted blur kernels, f kernel () denotes the kernel estimator.

[0052] Specifically, the weight calculation method first uses element-wise addition and 1x1 convolution to combine the candidate features k. patch The kernel weights are then merged, and subsequently, the Gumbel Softmax reparameter method is used to adaptively learn the degenerate kernel weights to form a kernel weight set. Finally, the kernel weight set is masked and aggregated with the candidate fuzzy kernels to obtain the evaluated true fuzzy kernel k. The specific formula can be expressed as:

[0053]

[0054] Among them, Conv 1x1 This represents a 1x1 convolutional layer. The sampling method for re-parameters is indicated. This indicates element-wise addition.

[0055] (S22) Noise Level Assessment Subnet:

[0056] The noise level assessment subnet uses a noise rating system to model the input LR image as a latent variable and calculates the variance of the variable to obtain the noise level n. The specific formula can be expressed as:

[0057] n = std(f noise (I LR ))

[0058] Where std() represents variance calculation, f noise () indicates a noise evaluator.

[0059] (S33) Degradation-aware reconstruction subnetwork:

[0060] The structure of the degradation-sensing reconstruction subnet is as follows: Figure 4 As shown, it includes a feature extraction module, an adaptive degradation perception module, and an upsampling module. The feature extraction module extracts shallow features from the low-resolution image; the adaptive degradation perception module is responsible for estimating multi-kernel degradation information and using it to recover damaged feature information; the upsampling module obtains a high-resolution image through convolution and multi-channel reconstruction.

[0061] Specifically, the feature extraction module alternately uses three 3x3 convolutional layers and two LeakyReLU activation layers to obtain a shallow feature F s , and the specific formula is:

[0062] F s = Conv 3x3 (LeakyReLu(Conv 3x3 (Leaky_ReLu(Conv 3x3 (I LR )))))

[0063] wherein, Com 3x3 represents a 3x3 convolutional layer, and LeakyReLu represents an activation function.

[0064] Specifically, the shallow feature F s , blur kernel k and noise level n are input into the adaptive degradation perception module to eliminate blur and noise, including:

[0065] The blur kernel k and noise n are merged into a low-dimensional feature F l , and the specific formula is:

[0066] F l = concat(k, n)

[0067] wherein, concat() represents a channel merging operation.

[0068] The low-dimensional feature F l is input into a multi-layer perception and a Softmax layer for feature transformation, and is aggregated with the shallow feature F s to form a residual feature F r , and the specific formula can be expressed as:

[0069]

[0070] wherein, MLP represents a multi-layer perception, including an input layer, multiple hidden layers and an output layer, softmax represents a normalized exponential function, represents a cross product.

[0071] The low-dimensional feature F l is stretched and embedded into a high-dimensional space, and a 3x3 convolutional layer, a ReLu activation layer and a 3x3 convolutional layer are used to form a high-dimensional feature F h , and the expression formula is as follows:

[0072] F h = Conu 3x3 (ReLu(Conv 3x3 (stretch(F l ))))

[0073] Among them, Conv 3x3 This indicates a 3x3 convolutional layer, ReLU represents the activation function, and stretch represents dimensional stretching.

[0074] Using the residual feature F r and high-dimensional features F h The residual connections are then fed into the next adaptive degradation perception module, ultimately forming the deep feature F. d The specific formula is as follows:

[0075]

[0076] in, This represents the nth adaptive degradation sensing module (n=8).

[0077] The upsampling model sampling module consists of PixelShuffle, convolutional layers, and the LeakyReLu activation function. First, the deep features F... d With shallow features F s The images are added together, and then recombined through convolution and multi-channel reconstruction to obtain a high-resolution image I. HR The specific calculation formula is as follows:

[0078] I HR =Conv 9x9 (LeakyReLu(PixelShuffle(Conv 3x3 (F d +F s ))))

[0079] Among them, Conv 3x3 Represents a 3x3 convolutional layer, Conv 9x9 This indicates a 9x9 convolutional layer, and PixelShuffle represents the pixel recombination function.

[0080] Therefore, the final output of the blind super-resolution model for spatially variable degradation sensing can be expressed as:

[0081] I HR =f rec (k, n, I) LR )

[0082] Where k represents the true fuzzy kernel, n represents noise, and f rec () represents a blind super-resolution model for spatially variable degradation sensing.

[0083] Step S4, training the remote sensing image blind super-resolution model based on spatially variable degradation perception: setting the hyperparameters, updating the network parameters using the Adam optimizer, training the multi-loss function based on the Llloss, coordinating the image degradation model and the spatially variable kernel perception blind super-resolution model, setting the learning rate to be updated once every 100 iterations in the process of training the model for 500 iterations.

[0084] The multi-loss function is:

[0085]

[0086] wherein, indicates the multi-loss function, is the loss function of the kernel evaluator, is the loss function of the noise evaluator, is the loss function of the spatially variable kernel blind super-resolution network, and λ1 and λ2 are balancing terms.

[0087] Step S5, reconstructing the high-resolution remote sensing image by using the trained blind super-resolution model: selecting a group of network weights with the highest evaluation index value, saving the optimal network model for later use in reconstructing the high-resolution remote sensing image.

[0088] In summary, the embodiment provides a blind super-resolution method that does not need to train a paired data set, decomposes the blind super-resolution reconstruction task into a degradation process and a generation process, simulates the complex degradation process of the image on one hand, considers the similarity between the degradation kernel and the real kernel, and avoids serious kernel estimation bias. On the other hand, the real degradation situation of different spatial positions of the image is considered, and the blur kernel cannot represent the overall degradation situation of the image. The reconstructed high-resolution image is rich in texture information, effectively eliminates blur and noise, and thus the remote sensing image super-resolution reconstruction method based on spatially variable degradation perception is an end-to-end high-quality blind super-resolution reconstruction.

[0089] The above embodiments are detailed descriptions of specific implementation steps and advantages of the present application. It should be understood that the embodiments are only preferred embodiments of the present application and do not limit the present application. Those skilled in the art can make various changes or modifications within the scope of the claims. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for super-resolution reconstruction of remote sensing images based on spatially variable degradation sensing, characterized in that, Specifically, the following steps are included: Step S1, obtain the high-resolution remote sensing image dataset: the training dataset consists of 31,500 high-quality, non-degraded remote sensing images from the NWPU-RESISC45 dataset; Step S2, construct the image degradation model: randomly generate anisotropic Gaussian blur kernels. and Gaussian white noise High-resolution images Degraded to low resolution image ;Will Input to the kernel evaluator and noise evaluator to obtain degenerate kernels. and noise ; Step S3: Construct a blind super-resolution model for spatially variable degradation sensing, which includes a spatially variable kernel evaluation subnet, a noise level evaluation subnet, and a degradation sensing reconstruction subnet. The spatially variable kernel evaluation subnet, after passing through a fuzzy kernel extraction layer, a weight evaluation layer, and a mask aggregation layer, obtains the true fuzzy kernel for evaluation, specifically including: The low-resolution image is input into the blur kernel extraction layer, and the kernel evaluator is used to extract blur kernels at different locations in the image to form candidate blur kernel features; The candidate fuzzy kernels are input into the weight evaluation layer, and the Gumbel Softmax reparameter operation is used to obtain different kernel weights to form a set of kernel weights; The kernel weight set and candidate fuzzy kernels are masked together to obtain the evaluated true fuzzy kernel; In the noise level assessment subnet, the noise features of the input image are first extracted by the noise estimator, and then the variance of the variables is calculated to obtain the noise level. The degradation-aware reconstruction subnetwork includes a feature extraction module, an adaptive degradation-aware module, and an upsampling module. The feature extraction module alternately uses three convolutional layers and two LeakyReLU activation layers to extract shallow features from low-resolution images. The adaptive degradation-aware module is responsible for estimating multi-kernel degradation information and using it to recover damaged feature information. The upsampling model sampling module consists of PixelShuffle, convolutional layers, and LeakyReLU activation functions, which obtain high-resolution images through convolution and multi-channel recombination. Step S4, train the remote sensing image super-resolution reconstruction method based on spatially variable degradation perception: set hyperparameters, use the Adam optimizer to update network parameters to minimize the loss function, and complete model training; Step S5: Reconstruct high-resolution remote sensing images using the trained blind super-resolution model: Select the set of network weights with the highest evaluation index value and save it as the optimal super-resolution reconstruction model for later reconstruction of high-resolution remote sensing images.

2. The method for super-resolution reconstruction of remote sensing images based on spatially variable degradation sensing according to claim 1, characterized in that, The image degradation model mentioned in step S2 is: ； in, and These represent degraded low-resolution and high-resolution images, respectively. , , These represent the fuzzy kernel and the scaling factor, respectively. The downsampling and noise; the downsampling method is a bicubic interpolation sampling method.

3. The method for super-resolution reconstruction of remote sensing images based on spatially variable degradation sensing according to claim 1, characterized in that, The kernel evaluator in step S3 consists of one convolutional layer, five residual convolutional groups, and one channel attention module, used to evaluate image degradation kernels; wherein each residual convolutional group consists of one convolutional layer, one LeakyReLu activation function, and one convolutional layer connected in series. The noise estimator consists of four transposed convolutional groups, four convolutional groups, a sigmoid function, and a convolution, used to evaluate noise features; wherein each transposed convolutional group is composed of a transposed convolution and a PRelu activation function concatenated, and each convolutional group is composed of a convolution and a PRelu activation function concatenated.

4. The method for super-resolution reconstruction of remote sensing images based on spatially variable degradation sensing according to claim 1, characterized in that, The shallow features, the true blur kernel, and the noise level are input into the adaptive degradation perception module to eliminate blur and noise, including: The fuzzy kernel and noise are combined into a low-dimensional feature; The low-dimensional features are input into a multilayer perceptron and a softmax layer for feature transformation, and then aggregated with a shallow feature mask to form residual features. The low-dimensional features are stretched and embedded into a high-dimensional space, and high-dimensional features are formed through a shallow convolutional neural network. The residual features and high-dimensional features are then concatenated and passed to the next adaptive degradation perception module.

5. The method for super-resolution reconstruction of remote sensing images based on spatially variable degradation sensing according to claim 1, characterized in that, Step S4 describes training a remote sensing image super-resolution reconstruction method based on spatially variable degradation sensing. This method uses L1 loss as a foundation to construct multiple loss functions for training, enabling the image degradation model and the spatially variable kernel sensing blind super-resolution model to coordinate and complete parameter training. The multiple loss functions are: ； in, Represents a multinomial loss function. Let the loss function be that of the kernel evaluator. Let the loss function be that of the noise estimator. Let be the loss function of the spatially variable kernel sensing blind super-resolution network. As a balancing term, and These represent degraded low-resolution and high-resolution images, respectively. , These represent the fuzzy kernel and noise, respectively.

Images