Unsupervised learning based remote sensing image blind super-resolution method and system

Abstract

The application discloses a remote sensing image blind super-resolution method and system based on unsupervised learning and belongs to the technical field of remote sensing image reconstruction. The application is applied to a remote sensing image super-resolution scene in which a degradation model is unknown. First, spatial reconstruction information is transferred from a natural image field to a remote sensing image field by using transfer learning. Then, the spatial-spectral reconstruction information is optimized based on unsupervised learning. By using the transfer learning, the application does not need to be trained on remote sensing images with limited training data, and the advantages of data-driven are explicitly used. By using the unsupervised learning, the application can be generalized to remote sensing image super-resolution reconstruction tasks in various degradation scenes, the spectral reconstruction information is fully learned, and the blind super-resolution precision of the remote sensing image is improved.

Description

Technical Field

[0001] This invention relates to a method and system for blind super-resolution of remote sensing images based on unsupervised learning, and more particularly to a method and system for obtaining high-quality remote sensing images in scenarios where the degradation model is unknown, belonging to the field of remote sensing image processing technology. Background Technology

[0002] Remote sensing images contain rich information about ground features and are widely used in many fields such as urban planning, environmental monitoring, and disaster assessment. However, due to external environmental interference and limitations in sensor manufacturing processes, the quality of remote sensing images degrades to varying degrees. Super-resolution reconstruction, as a hardware-independent method for improving image quality, has been extensively studied.

[0003] Currently, most existing remote sensing image super-resolution methods are based on supervised learning, requiring pre-assumptions of degradation models to generate pairs of high-resolution and low-resolution training data. For example, the bidirectional quasi-recurrent neural network proposed by Fu et al., and the spatial-spectral prior learning network proposed by Jiang et al., have both achieved good results in single remote sensing image super-resolution reconstruction. However, supervised learning-based methods cause the neural network to fit the pre-assumpted degradation model, failing to generalize to various degradation scenarios. Furthermore, the degradation model in real-world scenarios may not be accurately obtained, or its mathematical form may be very complex, a problem that supervised learning-based methods cannot effectively address. This type of super-resolution problem with unknown degradation models is called blind super-resolution.

[0004] Most existing blind super-resolution methods for remote sensing images based on unsupervised learning require a high-resolution natural image or multispectral image as an auxiliary image to obtain a high-resolution remote sensing image. However, in real-world scenarios, aligned high-resolution auxiliary images are difficult to obtain; often, only a low-resolution remote sensing image is available as input, with the expectation of outputting a super-resolution reconstructed remote sensing image.

[0005] Unlike natural images, remote sensing images suffer from data scarcity, making training difficult for remote sensing image reconstruction methods based on few-shot learning. Data augmentation is widely used in the field of remote sensing image reconstruction. However, even with augmentation to expand the training dataset, the network may still overfit, a technical problem that current methods cannot adequately address. Summary of the Invention

[0006] The purpose of this invention is to address the problem that existing technologies cannot achieve blind super-resolution of single remote sensing images. Specifically, supervised learning-based remote sensing image super-resolution reconstruction methods cannot generalize to various degradation models, unsupervised blind super-resolution of remote sensing images requires additional high-resolution auxiliary images, and there are a series of technical problems such as the scarcity of remote sensing image data and the existence of few-sample learning. This invention creatively proposes a method and system for blind super-resolution of remote sensing images based on unsupervised learning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] The proposed method for blind super-resolution of remote sensing images based on unsupervised learning is applied to blind super-resolution reconstruction of single remote sensing images where the degradation model is unknown.

[0009] First, transfer learning is used to transfer spatial reconstruction information from the domain of natural images to the domain of remote sensing images. Through transfer learning, this invention eliminates the need for training on remote sensing images with limited training data, explicitly leveraging the advantages of data-driven approaches.

[0010] Secondly, the spatial spectrum reconstruction information is optimized based on unsupervised learning. Through unsupervised learning, this invention can be generalized to remote sensing image super-resolution reconstruction tasks in various degradation scenarios, fully learning the spectral reconstruction information and improving the blind super-resolution accuracy of remote sensing images.

[0011] A method for blind super-resolution of remote sensing images based on unsupervised learning includes the following steps:

[0012] Step 1: Transfer learning phase, create a natural image training dataset, and iteratively modify the parameter dictionary of the algorithm model.

[0013] Specifically, step 1 includes the following steps:

[0014] Step 1.1: Preprocess the natural image dataset (including image enhancement and cropping) to generate the training dataset.

[0015] For example, the following methods can be used:

[0016] Download publicly available natural image datasets (such as DIV2K), convert each natural image from the RGB color gamut to the YUV color gamut, and perform super-resolution reconstruction only on the Y channel, i.e., the luminance channel. Crop the images spatially to obtain 64×64 spatial resolution image patches. Use each image patch as a training sample, and treat all patches as training data.

[0017] Step 1.2: Generate the degradation model. First, a Gaussian blur kernel function (which can be 8×8) is used to smooth the high-resolution natural image, and then downsampling is performed to obtain a low-resolution natural image.

[0018] By inputting pairs of high-resolution and low-resolution natural image data, a neural network is trained to obtain a super-resolution model.

[0019] Step 1.3: Let the size of the low-resolution remote sensing image Y be N×B, where N represents the spatial resolution of the hyperspectral image and B represents the number of channels in the hyperspectral image. Based on the super-resolution model obtained in Step 1.2, super-resolution reconstruction of the remote sensing image Y is performed through transfer learning to generate the corresponding pseudo-high-resolution remote sensing image X. p Suppose a pseudo-high resolution remote sensing image X p The size is rN×B, where r is the super-resolution scale.

[0020] Step 2: Unsupervised learning phase.

[0021] Specifically, it includes the following steps:

[0022] Step 2.1: Based on the low-resolution remote sensing image Y and pseudo-high-resolution remote sensing image X obtained in the transfer learning stage of Step 1... p A pair of training data is formed and used as input to the unsupervised learning network.

[0023] Unsupervised learning networks simultaneously integrate low-resolution remote sensing image Y and pseudo-high-resolution remote sensing image X. p Perform nonnegative matrix decomposition:

[0024] Y = UV

[0025] X p =UW (1)

[0026] In the above formula, It is an endmember matrix. Let B represent the real number field, B represent the number of channels in the hyperspectral image, and L represent the number of endmembers. It is the abundance matrix of a low-resolution hyperspectral image, and each column of the abundance matrix represents the endmember component of each pixel. N represents the spatial resolution of the hyperspectral image. The abundance matrix represents a high-resolution hyperspectral image.

[0027] The nonnegative matrix decomposition is constrained using the L1 loss function:

[0028]

[0029]

[0030] Among them, ||·|| F This represents the Frobenius norm.

[0031] Step 2.2: To better generalize to various degradation models, a degradation loss function is designed, using a single convolutional layer to simulate the fuzzy kernel function:

[0032]

[0033] In Equation 2, It is a pseudo-high resolution remote sensing image X p After obtaining the corresponding endmember matrix U and abundance matrix V through nonnegative matrix decomposition, the high-resolution remote sensing image is then obtained through matrix multiplication. k is the fuzzy kernel function in the degradation model. ↓ r is the downsampling operator, and r is the super-resolution scale.

[0034] Therefore, the loss function of the entire unsupervised learning network is expressed as:

[0035]

[0036] In the formula, η1, η2, and η3 represent the weights of each item.

[0037] Furthermore, in order to effectively implement the above method, this invention further proposes a system for blind super-resolution of remote sensing images based on unsupervised learning, including a transfer learning subsystem and an unsupervised learning subsystem.

[0038] The transfer learning subsystem transfers spatial reconstruction information from the domain of natural images to the domain of remote sensing images. It explicitly leverages the advantages of data-driven learning by eliminating the need for training on remote sensing images with limited training data.

[0039] The unsupervised learning subsystem optimizes the spatial spectrum reconstruction information based on unsupervised learning. It can be generalized to remote sensing image super-resolution reconstruction tasks in various degradation scenarios, fully learns the spectral reconstruction information, and improves the blind super-resolution accuracy of remote sensing images.

[0040] The output of the transfer learning subsystem is connected to the input of the unsupervised learning subsystem.

[0041] Beneficial effects

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. This invention is a remote sensing image super-resolution reconstruction technology based on unsupervised learning, which can be generalized to various degradation models.

[0044] 2. This invention does not require training on remote sensing images. By transferring spatial reconstruction information from the natural image domain to the remote sensing image domain through transfer learning, it avoids the problem of difficulty in training with few samples.

[0045] 3. This invention utilizes unsupervised learning to perform nonnegative matrix factorization on the input remote sensing image and estimates the degradation model of the remote sensing image, which can improve the quality of the reconstructed remote sensing image.

[0046] 4. The method and system for blind super-resolution of remote sensing images based on unsupervised learning disclosed in this invention have high reconstruction quality and are applicable to multiple fields such as geological exploration, agricultural production and biomedicine. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention.

[0048] Figure 2 This is a schematic diagram of the overall framework of the method of the present invention.

[0049] Figure 3 This is a schematic diagram of the transfer learning subsystem in the core algorithm model of the method of this invention.

[0050] Figure 4 This is a schematic diagram of the unsupervised subsystem in the core algorithm model of the method of this invention.

[0051] Figure 5 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0052] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] Example

[0054] This embodiment discloses a method and system for blind super-resolution of remote sensing images based on unsupervised learning. The method includes a transfer learning stage and an unsupervised learning stage; the system includes a transfer learning subsystem and an unsupervised learning system. The flowchart of the method and the schematic diagram of the system composition of this embodiment are shown below. Figure 1 and Figure 5 As shown.

[0055] A method for blind super-resolution of remote sensing images based on unsupervised learning is proposed. It is applied to blind super-resolution reconstruction scenarios of single remote sensing images where the degradation model is unknown. First, transfer learning is used to transfer spatial reconstruction information from the natural image domain to the remote sensing image domain. Through transfer learning, this invention does not require training on remote sensing images with limited training data, explicitly utilizing the advantages of data-driven approaches. Second, unsupervised learning is used to optimize the spatial-spectral reconstruction information. Through unsupervised learning, this invention can generalize to remote sensing image super-resolution reconstruction tasks in various degradation scenarios, fully learning spectral reconstruction information and improving the accuracy of blind super-resolution of remote sensing images. The flowchart of this embodiment is as follows. Figure 2 As shown.

[0056] A method for blind super-resolution of remote sensing images based on unsupervised learning includes the following steps:

[0057] Step 1: Transfer Learning Phase. Create a natural image training dataset and iteratively modify the parameter dictionary of the algorithm model.

[0058] Specifically, step 1 includes the following steps:

[0059] Step 1.1: Preprocess the natural image dataset (including image enhancement and cropping) to generate the training dataset.

[0060] The specific method is as follows:

[0061] Download the publicly available natural image dataset DIV2K. Convert each natural image from the RGB color space to the YUV color space, and perform super-resolution reconstruction only on the Y channel, i.e., the luminance channel. Crop the images spatially to obtain 64×64 spatial resolution image patches. Use each image patch as a training sample, and treat all patches as training data.

[0062] Step 1.2: Generate the degradation model: First, smooth the high-resolution natural image using an 8×8 Gaussian blur kernel function, then downsample it to obtain a low-resolution natural image. Input the high-resolution and low-resolution natural image data pairs to train the neural network and obtain the super-resolution model f.

[0063] Step 1.3: The low-resolution remote sensing image Y has a size of N×B. Based on the super-resolution model f obtained in Step 1.2, super-resolution reconstruction is performed on the low-resolution remote sensing image Y through transfer learning, generating the corresponding pseudo-high-resolution remote sensing image X in a channel-by-channel manner. p :

[0064] X p =f(Y) (5)

[0065] Where f is the super-resolution model.

[0066] Pseudo-high resolution remote sensing image X p The size is rN×B, where r is the super-resolution scale. The flowchart of this embodiment is as follows: Figure 3 As shown.

[0067] Step 2: Unsupervised learning phase.

[0068] Step 2.1: Based on the low-resolution remote sensing image Y and pseudo-high-resolution remote sensing image X obtained in the transfer learning stage of Step 1... p These two sets of data form a pair of training data, which are then used as input to the unsupervised learning network. The unsupervised learning network simultaneously uses the low-resolution remote sensing image Y and the pseudo-high-resolution remote sensing image X. p Perform nonnegative matrix decomposition:

[0069] Y = UV

[0070] X p =UW (6)

[0071] In formula (6) It is an endmember matrix, where L is the number of endmembers. It is an abundance matrix, and each column of the abundance matrix represents the endmember component in each pixel.

[0072] Constraints on nonnegative matrix decomposition using the L1 loss function:

[0073]

[0074]

[0075] Step 2.2: To better generalize to various degradation models, a degradation loss function was designed, using a single convolutional layer to simulate the fuzzy kernel function:

[0076]

[0077] In formula (8) It is a pseudo-high resolution remote sensing image X p After obtaining the corresponding endmember matrix U and abundance matrix V through nonnegative matrix factorization, the output high-resolution remote sensing image is obtained through matrix multiplication. k is the fuzzy kernel function in the degradation model, ↓ r is the downsampling operator, and r is the super-resolution scale.

[0078] Therefore, the loss function of the entire unsupervised learning network can be expressed as:

[0079]

[0080] In the formula, η1, η2, and η3 represent the weights of each item. The flowchart of this embodiment is as follows: Figure 4 As shown.

[0081] Based on the above method, this embodiment further proposes a system for blind super-resolution of remote sensing images based on unsupervised learning, including a transfer learning subsystem and an unsupervised learning subsystem.

[0082] Among them, the transfer learning subsystem transfers spatial reconstruction information from the natural image domain to the remote sensing image domain, without the need for training on remote sensing images with limited training data, thus explicitly taking advantage of the data-driven approach.

[0083] Optimizing spectral reconstruction information based on unsupervised learning can be generalized to remote sensing image super-resolution reconstruction tasks in various degradation scenarios. It fully learns spectral reconstruction information and improves the blind super-resolution accuracy of remote sensing images.

[0084] The connection between the above-mentioned components is as follows: the output of the transfer learning subsystem is connected to the input of the unsupervised subsystem.

[0085] In remote sensing image reconstruction scenarios where the degradation model is unknown, this invention demonstrates better reconstruction quality compared to other comparative algorithms in testing.

Claims

1. A method for blind super-resolution of remote sensing images based on unsupervised learning, characterized in that, Includes the following steps: Step 1: Transfer learning phase, create a natural image training dataset, and iteratively modify the parameter dictionary of the algorithm model; Step 1.1: Preprocess the natural image dataset, including image enhancement and cropping, to generate the training dataset; Step 1.2: Generate the degradation model; First, a Gaussian blur kernel function is used to smooth the high-resolution natural image, and then downsampling is performed to obtain a low-resolution natural image. Input high-resolution and low-resolution natural image data pairs, train the neural network to obtain the super-resolution model f; Step 1.3: Let the size of the low-resolution remote sensing image Y be N×B, where N represents the spatial resolution of the hyperspectral image and B represents the number of channels of the hyperspectral image; Based on the super-resolution model f obtained in step 1.2, the remote sensing image Y is super-resolution reconstructed through transfer learning to generate the corresponding pseudo-high-resolution remote sensing image X. p Assume a pseudo-high resolution remote sensing image X p The size is rN×B, where r is the super-resolution scale; X p = f(Y), where f is the super-resolution model; Step 2: Unsupervised learning phase; Step 2.1: Based on the low-resolution remote sensing image Y and pseudo-high-resolution remote sensing image X obtained in the transfer learning stage of Step 1... p A pair of training data is formed and used as input to an unsupervised learning network; Unsupervised learning networks simultaneously integrate low-resolution remote sensing image Y and pseudo-high-resolution remote sensing image X. p Perform nonnegative matrix decomposition: Y=UV,X p YOUR In the above formula, It is an endmember matrix. Let B represent the real number field, B represent the number of channels in the hyperspectral image, and L represent the number of endmembers. It is the abundance matrix of a low-resolution hyperspectral image, and each column of the abundance matrix represents the endmember component of each pixel. N represents the spatial resolution of the hyperspectral image. The abundance matrix representing a high-resolution hyperspectral image; Constraints on nonnegative matrix decomposition: Among them, ||·|| F Denotes the Frobenius norm; Step 2.2: Design the degradation loss function, using a single convolutional layer to simulate the blur kernel function: In the above formula, It is a pseudo-high resolution remote sensing image X p After obtaining the corresponding endmember matrix U and abundance matrix V through nonnegative matrix decomposition, the high-resolution remote sensing image is then obtained through matrix multiplication; k is the fuzzy kernel function in the degradation model, ↓ r This is the downsampling operator, and r is the super-resolution scale; The loss function of the entire unsupervised learning network is expressed as: In the formula, η1, η2, and η3 represent the weights of each item.

2. The method for blind super-resolution of remote sensing images based on unsupervised learning as described in claim 1, characterized in that, In step 1.2, the natural image dataset is downloaded, and each natural image is converted from the RGB color gamut to the YUV color gamut. Super-resolution reconstruction is performed only on the Y channel, which is the luminance channel. The image is cropped in the spatial dimension to obtain 64×64 spatial resolution image patches. Each image patch is used as a training sample, and all patches are treated as training data.

3. A system for blind super-resolution of remote sensing images based on unsupervised learning, implementing the method of claim 1, characterized in that, It includes a transfer learning subsystem and an unsupervised learning subsystem; Among them, the transfer learning subsystem transfers spatial reconstruction information from the domain of natural images to the domain of remote sensing images, without the need for training on remote sensing images with limited training data. The unsupervised learning subsystem optimizes the spatial spectrum reconstruction information based on unsupervised learning. It can be generalized to remote sensing image super-resolution reconstruction tasks in various degradation scenarios, fully learns spectral reconstruction information, and improves the blind super-resolution accuracy of remote sensing images. The output of the transfer learning subsystem is connected to the input of the unsupervised learning subsystem.

Images