Remote sensing image super-resolution recovery method based on GAN neural network

Abstract

The invention provides a remote sensing image super-resolution recovery method based on a GAN neural network. The remote sensing image super-resolution recovery method comprises the following steps: constructing a sub-region uniform clipping module to generate a corresponding low-resolution training pair for a high-resolution image; building a GAN neural network for adversarial generation; an inverse transformation image decoder is constructed to serve as a generator network fitting super-resolution process in a GAN network framework, and then a discriminator network is used for discriminating the true degree of a generated picture for gradient descent training of the GAN network. Some embodiments of the invention realize remote sensing image super-resolution reconstruction based on the adversarial neural network and local semantic information fusion, and have the characteristics of general applicability to image content, generation rationality and diversity, robustness and the like.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of remote sensing image super-resolution, and specifically to a remote sensing image super-resolution restoration method based on a GAN neural network, which is a deep learning technology that generates a corresponding super-resolution image by inputting a high-resolution image. Background Art

[0002] Satellite images are an important source of geographic information that can be used in different application areas, such as environmental monitoring, crop type identification, land use / cover mapping, disaster management, decision support systems, weather forecasting, climate modeling, geographic object detection, and biophysical parameter estimation, and have been widely used by various disciplines of earth science. Due to optical and sensor limitations, the spatial and spectral resolution of earth observation satellites usually does not reach the expected high level. Therefore, it is crucial to develop software-based algorithms to improve the spatial and spectral quality of satellite images. Single remote sensing image super-resolution provides an effective solution for increasing the spatial and / or spectral resolution of low-resolution satellite image inputs, improving image processing applications and denoising performance. Therefore, it has become one of the hottest research topics in the field of remote sensing.

[0003] With the rapid development and significant progress of deep learning technology in various visual tasks such as scene classification, object detection, and image segmentation, the number of remote sensing image super-resolution methods based on deep learning has increased significantly, greatly improving the quality of satellite images. Summary of the invention

[0004] The technical problem to be solved by the present disclosure is: in view of the phenomenon that the resolution of current remote sensing images cannot reach the expected level, a remote sensing image super-resolution method based on the GAN neural network architecture is developed. This method generates corresponding low-resolution training pairs for high-resolution images by constructing a sub-region uniform cropping module; building a GAN neural network for adversarial generation; constructing an inverse transform image decoder as a generator network in the GAN network framework to fit the super-resolution process, and then using the discriminator network to discriminate the authenticity of the generated image for gradient descent training of the GAN network. The present disclosure realizes super-resolution reconstruction of remote sensing images based on adversarial neural networks and local semantic information fusion, and has the characteristics of universal applicability to image content, rationality and diversity of generation, and robustness.

[0005] The technical solution adopted by the present disclosure is: a remote sensing image super-resolution restoration method based on a GAN neural network, comprising the following four steps:

[0006] Step 1: Generate low-resolution images by simulating the scene semantic understanding process: construct a sub-region uniform cropping module to filter out coordinates containing high semantic information in the image, generate image blocks with consistent semantics but blurred content, and form high-resolution-low-resolution training data pairs;

[0007] Step 2: Generate a high-resolution image from the inverse transform image decoder: Input the low-resolution image into the inverse transform image decoder, deblur the image through a network structure consisting of nested convolutional layers, regularization layers, and activation function layers, generate a high-resolution image, and calculate the mean square error loss, energy density loss, and adversarial loss of the generated image and the paired original high-resolution image for subsequent network training;

[0008] Step 3: The discriminator network determines the authenticity of the generated image: the generated image is input into the discriminator of the adversarial network. The discriminator calculates the authenticity value of each pixel of the generated image and the predicted probability that the overall image is true, and calculates the loss of the GAN network based on the predicted probability.

[0009] Step 4: Gradient descent training of the GAN neural network: The loss of the generator network is obtained by adding the mean square error loss, energy density loss and adversarial loss calculated in steps 2 and 3. The loss of the discriminator network uses the binary cross entropy loss function. The loss of the generator network and the loss of the discriminator network both use the gradient descent method to optimize the parameters of the network and perform independent training. After multiple rounds of training, the generator can better simulate the resolution improvement process, thereby converting low-resolution remote sensing images into super-resolution images.

[0010] The principles of this disclosure are:

[0011] (1) The scene semantic understanding process is simulated to generate low-resolution images to form training pairs. By adaptively clustering the areas where objects in the remote sensing images are densely distributed, sub-regions with rich semantic information are obtained, which improves the data distribution advantage of small-scale sub-regions.

[0012] (2) A GAN neural network is used to simulate the process of resolution restoration and to generate realistic samples through adversarial training between the generator and the discriminator.

[0013] (3) Adding energy density loss on the basis of mean square error loss in image reconstruction and adversarial loss in GAN network can make the image generated by the network clearer. At the same time, it also constrains the low-level feature maps of the original image and the super-resolution image to be as close as possible, thus retaining the content information of the original image.

[0014] Compared with the prior art, the advantages of the present disclosure are as follows: a sub-region uniform cropping module is proposed. The module can filter out the coordinates containing high semantic information in the image and generate image blocks with consistent semantics but blurred content. The training pairs constructed by this module have better scene semantic information recognition and retention capabilities than the training pairs generated by the prior art. An energy density loss function for super-resolution image generation is proposed, which improves the quality of the generated image by constraining the energy error between the generated image and the original image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flowchart of the remote sensing image super-resolution method based on GAN

[0016] Figure 2 This is a network structure diagram of the inverse transformation image decoder in the GAN neural network in the present disclosure.

[0017] Figure 3 Schematic diagram of the network structure of the discriminator in the GAN neural network in the present disclosure. DETAILED DESCRIPTION

[0018] Figure 1 The overall process of the remote sensing image super-resolution method based on GAN is given, and the present disclosure is further explained below in conjunction with other drawings and specific implementation methods.

[0019] This disclosure provides a remote sensing image super-resolution method based on GAN, the main steps are introduced as follows:

[0020] 1. The constructed sub-region uniform cropping module simulates the scene semantic understanding process to generate low-resolution images. This module includes the following five calculation steps:

[0021] The first step is to obtain the corresponding image semantic annotation box for an input image.

[0022] In the second step, for each given semantic annotation box, (x 1 ,y 1 ,x 2 ,y 2 ) indicates that the coordinates of its center point are calculated Round the result.

[0023] The third step is to classify the coordinates of each center point through the K-means clustering algorithm.

[0024] Step 4: For each category i, calculate the maximum and minimum values ​​of x and y in the set of GT boxes corresponding to the center point coordinates to form the sub-region bounding box

[0025] The fifth step is to The pixels within constitute the cropped tile, a 30% pixel mask is performed, and then downsampled to generate a low-resolution tile.

[0026] 2. Construct the inverse transform image decoder and discriminator to form the GAN network architecture. The inverse transform image decoder uses a hybrid residual connection to connect four basic modules in series. The basic module is composed of a convolution layer, a regularization layer and a LeakyRelu activation function. The discriminator adopts a global-local detection fusion network structure design, accepts the image generated by the inverse transform image decoder and then outputs the authenticity value of each pixel and the probability of the image being judged as true.

[0027] 3. Input the low-resolution image-high-resolution image pair into the GAN network and calculate the loss function. The calculation steps include:

[0028] The first step is to perform pixel encoding on the input low-resolution image: (H, W, C) represents the three dimensions of the input image: length, width, and number of channels. The pixel encoding operation converts the image size into (H / r, W / r, C×r 2 ), converting the spatial dimension of the image into the length of the channel.

[0029] In the second step, the inverse transform image decoder deblurs the low-resolution image and generates image I G .

[0030] The third step is to calculate the mean square error loss, energy density loss and inverse transform image decoder adversarial loss of the generated image and the paired original high-resolution image. G To generate an image, I HR For the original high-resolution image, the specific calculation method of the mean square error loss is as follows:

[0031]

[0032] Where H, W, C represent the length, width and number of channels of the image; Φ represents the convolution function. The specific calculation method of energy density loss is as follows:

[0033]

[0034] The inverse transform image decoder adversarial loss is calculated as follows:

[0035]

[0036] The fourth step is to calculate the inverse transform image decoder loss function:

[0037]

[0038] Step 5: Calculate the discriminator loss function:

[0039]

[0040] In the sixth step, the Adam optimizer is used to optimize the neural network using the gradient descent algorithm. The neural network gradually adjusts the parameters during training.

[0041] The technical contents not elaborated in detail in this disclosure belong to the common knowledge of those skilled in the art.

[0042] Although the above describes the illustrative specific embodiments of the present disclosure to facilitate technical personnel in this technical field to understand the present disclosure, it should be clear that the present disclosure is not limited to the scope of the specific embodiments. For ordinary technical personnel in this technical field, as long as various changes are within the spirit and scope of the present disclosure defined and determined by the attached claims, these changes are obvious, and all inventions and creations conceived using the present disclosure are protected.

Claims

1. A remote sensing image super-resolution restoration method based on GAN neural network, characterized by: It includes the following four steps: Step 1: Generate low-resolution images by simulating the scene semantic understanding process: construct a sub-region uniform cropping module to filter out coordinates containing high semantic information in the image, generate image blocks with consistent semantics but blurred content, and form high-resolution-low-resolution training data pairs; Step 2: Generate a high-resolution image from the inverse transform image decoder: Input the low-resolution image into the inverse transform image decoder, deblur the image through a network structure consisting of nested convolutional layers, regularization layers, and activation function layers, generate a high-resolution image, and calculate the mean square error loss, energy density loss, and adversarial loss of the generated image and the paired original high-resolution image for subsequent network training; Step 3: The discriminator network determines the authenticity of the generated image: the generated image is input into the discriminator of the adversarial network. The discriminator calculates the authenticity value of each pixel of the generated image and the predicted probability that the overall image is true, and calculates the loss of the GAN network based on the predicted probability. Step 4: Gradient descent training of the GAN neural network: The loss of the generator network is obtained by adding the mean square error loss, energy density loss and adversarial loss calculated in steps 2 and 3. The loss of the discriminator network uses the binary cross entropy loss function. The loss of the generator network and the loss of the discriminator network both use the gradient descent method to optimize the parameters of the network and perform independent training. After multiple rounds of training, the generator can better simulate the resolution improvement process, thereby converting low-resolution remote sensing images into super-resolution images.

2. The method for generating a low-resolution image according to claim 1, wherein: The sub-region uniform clipping module constructed in step 1 includes the following five calculation steps: The first step is to obtain the corresponding image semantic annotation box for an input image; The second step is to calculate the coordinates of the center point of each given semantic annotation box, represented by (x1, y1, x2, y2). Round the result; The third step is to classify the coordinates of each center point through the K-means clustering algorithm; Step 4: For each category i, calculate the maximum and minimum values ​​of x and y in the set of GT boxes corresponding to the center point coordinates to form the sub-region bounding box The fifth step is to The pixels within constitute the cropped tile, a 30% pixel mask is performed, and then downsampled to generate a low-resolution tile.

3. The method of generating a high-resolution image according to claim 1, wherein: The specific steps of generating a high-resolution image by an inverse transform image decoder are as follows: The first step is to preprocess the low-resolution image. The inverse transform image decoder first performs pixel encoding on the input low-resolution image: (H, W, C) represents the length, width and number of channels of the input image. The pixel encoding operation converts the image size into (H / r, W / r, C×r 2 ), convert the spatial size of the image into the length of the channel; The second step is to design a network structure consisting of nested convolutional layers, regularization layers, and activation function layers to deblur the image. The most basic module is composed of a convolutional layer, a regularization layer, and a LeakyRelu activation function. The inverse transform image decoder uses a hybrid residual connection to connect the four basic modules in series. The third step is to calculate the mean square error loss, energy density loss and adversarial loss of the generated image and the paired original high-resolution image, denoted by I G To generate an image, I HR For the original high-resolution image, the specific calculation method of the mean square error loss is as follows: Among them, H, W, C represent the length, width and number of channels of the image, and Φ represents the convolution function; The specific calculation method of energy density loss is as follows: The adversarial loss is calculated as follows:

4. The discriminator network for determining the authenticity of generated images and the gradient descent training of the GAN neural network according to claim 1, characterized in that: The discriminator is designed based on the global-local detection fusion network structure. It accepts the image generated by the inverse transform image decoder and then outputs the authenticity value of each pixel and the probability of the image being judged as true. The generator calculates the cross entropy loss based on the returned true probability. The specific calculation method is as follows: Among them, I G is the image generated by the inverse transformed image decoder, is the probability that the discriminator judges the image to be real, is the probability that the discriminator judges the original image to be real, and the overall loss of the inverse transformed image decoder network is calculated as follows: The Adam optimizer is used to optimize the network parameters using the gradient descent algorithm.

Images