An unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction
Through an unsupervised generative adversarial network model, low-resolution images and degraded images are trained, which solves the problem of data acquisition difficulties in super-resolution reconstruction of remote sensing images, and realizes high-quality high-resolution image reconstruction, breaks the traditional method data dependency.
Patent Information
- Application Number
- CN202210314012.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-28
AI Technical Summary
In the super-resolution reconstruction of remote sensing images, existing generative adversarial network models usually require paired image data or auxiliary high-resolution images with super-resolution reconstruction of unsupervised remote sensing images for training. It is difficult to obtain data, making it difficult to achieve super-resolution reconstruction of unsupervised remote sensing images.
Unsupervised method based on constraint reconstruction is adopted, and unsupervised training is performed by generating an adversarial network model using low-resolution images and reconstructing image degraded low-resolution images. The generator module, discriminator module and image degradation module are combined to use Improved loss function and network structure to realize reconstruction of high-resolution images.
It breaks the constraint relationship of traditional remote sensing image super-resolution reconstruction, uses only low-resolution images to achieve super-resolution reconstruction, improves image quality, improves convolutional neural network structure, and produces better high-resolution images.
Smart Images

Figure CN114897677B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of digital image processing, and more particularly to an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction. Background Art
[0002] At present, super-resolution reconstruction is an image processing technology that reconstructs low-resolution images into high-resolution images with rich texture details. Single-frame image super-resolution reconstruction refers to the reconstruction of a single-frame high-resolution image when the input image data is a single-frame image. For super-resolution reconstruction of remote sensing images, researchers generally use high-resolution images paired with real low-resolution images for constraints to find credible solutions. Since the generation of adversarial networks, neural network parameterized generative models have also developed rapidly.
[0003] However, in the context of actual remote sensing images, most generative adversarial network models require either paired image data or original low-resolution images and auxiliary high-resolution images with certain supervisory information for training. It is very difficult to obtain data for super-resolution reconstruction of remote sensing images.
[0004] Therefore, providing an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction is an urgent problem to be solved by those skilled in the art. Summary of the invention
[0005] In view of this, the present invention provides an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction, which generates a degraded low-resolution image of the reconstructed image through unsupervised learning, and trains a generative adversarial network model with the low-resolution image and the degraded low-resolution image of the reconstructed image, so as to achieve the purpose of high-resolution image reconstruction, and the reconstructed image quality is better.
[0006] In order to achieve the above object, the present invention adopts the following technical scheme: an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction, including a training process of a generative adversarial network model: the generative adversarial network model includes a generator module, a discriminator module and an image degradation module; the training process is as follows:
[0007] Step 1: Take the low-resolution image to be trained as input; generate a high-resolution reconstructed image through the generator module; Step 2: The high-resolution reconstructed image generates a degraded low-resolution image through the image degradation module; Step 3: Take the low-resolution image and the degraded low-resolution image as input and perform discrimination through the discriminator module; the discrimination process performs unsupervised training on the discriminator module to distinguish true from false based on the mapping relationship between the low-resolution image and the degraded low-resolution image; and output the discrimination result; after the training is completed, take the unreconstructed low-resolution remote sensing image as input to the generator module to generate the corresponding high-resolution remote sensing image.
[0008] Preferably, the structure of the generator module includes a basic module, a SAF module, an upsampling module, and a convolutional layer module; the low-resolution image to be trained is passed through the convolutional layer module to generate output result one; the output result one is passed through several of the basic modules and convolutional layer modules to generate output result two; the output result one and the output result two are added and passed through the SAF module, the upsampling module and two convolutional layer modules to generate a high-resolution reconstructed image.
[0009] Preferably, the structure of the discriminator modules are all convolutional layers, which are used to extract features of the low-resolution image and the degraded low-resolution image.
[0010] Preferably, the generative adversarial network loss includes a network overall loss function and a discriminator loss function; the network overall loss function constrains the generator module, and the discriminator module loss function constrains the discriminator module.
[0011] Preferably, the overall network loss function is:
[0012] L G_total =λ1·L cb +λ per ·L per +λ adv ·L G_adv
[0013] Among them, L G_total is the overall loss function of the network, λ1 represents L cb The weight, λ per Indicates L per The weight, λ adv is the adversarial loss weight of the generator module, L cb Pixel loss, L per represents the perceptual loss, L G_adv is the adversarial loss of the generator module.
[0014] Preferably, the adversarial loss function of the generator module is:
[0015]
[0016] in, is the first constraint of the generator module, is the second constraint of the generator module, SR is the high resolution of reconstruction, and F(I LR ; θ) is the degenerate operator, I LR is a low-resolution image, θ is a parameter, D(I LR , F(I LR ;θ)) and D(F(I LR ; θ), I LR ) is the constraint of the discriminator on the low-resolution image and the degraded low-resolution image of the reconstructed image. The first constraint and the second constraint of the generator module make the reconstructed degraded image close to the real low-resolution image.
[0017] Preferably, the discriminator module loss function is:
[0018]
[0019] Among them, D(I LR , F(I LR ;θ)) and D(F(I LR ; θ), I LR ) is the constraint of the discriminator on the low-resolution image and the reconstructed low-resolution image, I LR is a low-resolution image, θ is a parameter, F(I LR ; θ) is the degeneration operator.
[0020] Preferably, the pixel loss formula is:
[0021]
[0022] Where N is the total number of pixels, i is the i-th pixel, ρ(x) is the intermediate result, and F(I LR ; θ) is the degradation operator of the input image, ε is a very small constant, x is a high-resolution image, ρ(F(I LR ;θ)-I LR ) is the pixel loss between the low-resolution image and the low-resolution image of the degraded image, θ is a parameter, I LR For low-resolution images.
[0023] Preferably, the basic module includes a dense block and a channel attention module.
[0024] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction, which has the following benefits:
[0025] (1) Breaking the constraints of the original generative adversarial network used for super-resolution reconstruction of remote sensing images, unsupervised super-resolution reconstruction of remote sensing images can be achieved, and super-resolution reconstruction can be achieved using only low-resolution images.
[0026] (2) A generative adversarial network is used to achieve image super-resolution reconstruction. The constraint function is improved, that is, the loss function is improved. In the pixel loss part, Lcb is used to replace the traditional L1 and L2 loss functions, so as to improve the performance of the unsupervised super-resolution network of remote sensing images.
[0027] (3) The structure of the traditional convolutional neural network used for super-resolution reconstruction is improved. The modified network reconstruction has better results and the reconstructed image quality is higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0029] Figure 1 This is a schematic diagram of the unsupervised GAN constraints of the present invention.
[0030] Figure 2 Schematic diagram of the supervised GAN constraints of the existing technology.
[0031] Figure 3 It is a schematic diagram of the generator module structure of the present invention.
[0032] Figure 4 This is a diagram showing the reconstructed image of the present invention. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] The embodiment of the present invention discloses an unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction. Given a low-resolution image, the goal is to find a signal that is as close as possible to the corresponding real high-resolution image. However, a low-resolution image may have many different high-resolution images corresponding to it. Therefore, the present invention adds a priori information to perform normalization constraints when solving the high-resolution image, that is, the image reconstruction formula is:
[0035]
[0036] Among them, p Y|X (y|x) is the likelihood of a high-resolution image x given a low-resolution image y, and p X (x) is the prior probability at point x; X is the low-resolution image, and Y is the true high-resolution image.
[0037] G:Y→X maps each measurement y to its associated MAP estimate x in the training set. * Make a connection and transform the objective function as follows:
[0038]
[0039] (1) Find the likelihood term
[0040] The likelihood term of the objective function is marginalized according to the following formula:
[0041]
[0042] That is, using the independence between X and parameter θ,
[0043] Substitute the image degradation module formula Y = F (X; Θ) + n into the above logp Y|X,Θ (y|G(y), θ), we get the following formula:
[0044] Substituting formula 4 into formula 3, the expected term formula is:
[0045]
[0046] Among them, c1 is a constant, c2 is a constant; σ 2 is the variance; Y is the spatial description of the degradation function, and F(X; Θ) is the degradation operator of the input image;
[0047] Under the parameter Θ and the downsampled measurement y of the high-resolution image x, we then constrain G so that Close to zero, the purpose is to keep the low-resolution image y consistent with the reconstructed image.
[0048] (2) Find the prior term p X (G(y)):p X Maximizing (G(y)) is similar to learning a mapping G such that The distribution is close to p X distribution.
[0049] (3) In the process of reconstructing high-resolution images, the prior term and the likelihood term are based on probability calculation, which will produce probability loss. The probability loss is:
[0050]
[0051]
[0052]
[0053] l prior is the probability loss of the prior term, l likeli is the probability loss of the likelihood term, p X is the prior distribution, is the output distribution of the generator, To reconstruct the degraded low-resolution image, λ is the probability loss weight of the likelihood term.
[0054] In this embodiment, the present invention is modeled and trained based on the above theoretical basis, including the following steps: using the overall network function to train the generative adversarial network model, the generative adversarial network model includes a generator module, a discriminator module and an image degradation module; step 1: taking the low-resolution image to be trained as input; generating a high-resolution reconstructed image through the generator module; step 2: the high-resolution reconstructed image is passed through the image degradation module to generate a degraded low-resolution image; step 3: taking the low-resolution image and the degraded low-resolution image as input and discriminating through the discriminator module; the discriminator module is unsupervisedly trained to discriminate true and false according to the mapping relationship between the low-resolution image and the degraded low-resolution image during the discrimination process; and the discrimination result is output. After the training is completed, the low-resolution remote sensing image is taken as input, and the corresponding high-resolution remote sensing image is generated through the generator module.
[0055] In this embodiment, the super-resolution reconstruction network used in the present invention is a generative adversarial network, and the generator of the generative adversarial network is as follows: Figure 3As shown, it includes a basic module, a SAF module, an upsampling module, and a convolutional layer module; the low-resolution image to be trained is passed through the convolutional layer module to generate output result 1; output result 1 is passed through several basic modules and convolutional layer modules to generate output result 2; output result 1 and output result 2 are added and passed through the SAF module, the upsampling module and two convolutional layer modules to generate a high-resolution reconstructed image. The basic module of the network is a combination module of a dense block and a channel attention module (CA), and the batch normalization BN (BatchNormalization) layer is not used in the overall network. Because adding the BN layer is easy to cause artificial artifacts and easy to affect the generalization ability of the model, the SAF module is added to the generator module of the present invention, so that the expression ability of the model is further enhanced.
[0056] In this embodiment, a patch discriminator is selected as the discriminator. The patch discriminator also solves the two defects of VGG-128 used in the super-resolution generative adversarial network: 1) VGG-128 limits the size of the generated image to 128, which is difficult to train at multiple scales. 2) VGG-128 has a deeper network structure, and its fixed fully connected layer makes the discriminator pay more attention to global features and ignore local features. The patch discriminator used by IRSR is a fully convolutional network, which consists of a three-layer network corresponding to a 70×70 patch. The output of the patch discriminator is a two-dimensional matrix, which is an N*N matrix; each element in the matrix represents a local area of the input image. If the local area is real, it will get 1, otherwise it is 0. Each value represents the judgment of the local area of the image, which makes each output value of the discriminator only related to the patch of the local fixed area, and then the patch loss will be fed back to the generator to optimize the gradient of local details. The final error is the average of all local errors to ensure global consistency.
[0057] In this embodiment, the generative adversarial network loss includes a network overall loss function and a discriminator loss function; the generator module is constrained by the network overall loss function, and the discriminator module is constrained by the discriminator module loss function.
[0058] In this embodiment, the overall network loss function is:
[0059] L G_total =λ1·L cb +λ per ·L per +λ adv ·L G_adv #(9)
[0060] Among them, L G_totalis the overall loss function of the network, λ1 represents L cb The weight, λ per Indicates L per The weight, λ adv is the adversarial loss weight of the generator module, L cb Pixel loss, L per represents the perceptual loss, L G_adv is the adversarial loss of the generator module.
[0061] In this embodiment, the pixel loss formula is:
[0062]
[0063] Where N is the total number of pixels, i is the i-th pixel, F(I LR ; θ) is the degradation operator of the input image, ρ(x) is the intermediate result, ε is a minimum constant, x is a high-resolution image, ρ(F(I LR ;θ)-I LR ) is the pixel loss between the low-resolution image and the low-resolution image of the degraded image, θ is a parameter, I LR For low-resolution images.
[0064] In this embodiment, the perceptual loss is constrained by converting the low-resolution image and the generated high-resolution image and then the degraded low-resolution image into the feature domain. The perceptual loss is implemented using the trained VGG-19, and converting it into the feature space can effectively solve the sparsity of the features after the activation layer.
[0065] In this embodiment, the adversarial loss function of the generator module is:
[0066]
[0067] in, is the first constraint of the generator, is the second constraint of the generator, SR is the high resolution of reconstruction, and F(I LR ; θ) is the degenerate operator, I LR is a low-resolution image, θ is a parameter, D(I LR , F(I LR ;θ)) and D(F(I LR ; θ), I LR ) is the constraint of the discriminator on the low-resolution image and the reconstructed image degraded low-resolution image.
[0068] In this embodiment, the loss function of the discriminator module is:
[0069]
[0070] Among them, D(ILR , F(I LR ;θ)) and D(F(I LR ; θ), I LR ) is the constraint of the discriminator on the low-resolution image and the reconstructed image degraded low-resolution image, I LR is a low-resolution image, θ is a parameter, F(I LR ; θ) is the degradation operator, and LR is the low resolution.
[0071] In this embodiment, the present invention reconstructs the constraints of the generative adversarial network for remote sensing image super-resolution to perform unsupervised remote sensing image super-resolution reconstruction. The image used in the present invention is the public remote sensing dataset Inria, and the evaluation indicators are PSNR, SSIM, ERGAS and NIQE. The higher the PSNR and SSIM, the higher the image quality, while the lower the ERGAS and NIQE, the higher the quality of the reconstructed image.
[0072] Table 1 Comparison of the method of the present invention with other methods
[0073] method PSNR(dB) SSIM ERGAS NIQE IBP 25.18 0.6667 3.141 20.21 BDB 24.19 0.6243 3.589 21.98 GPR 24.89 0.6311 3.232 20.97 FSR 23.79 0.5586 3.897 22.98 EUSR 25.21 0.6798 3.132 18.79 UGSR 18.13 0.3987 6.153 25.89 ZSSR 27.02 0.7001 3.025 18.16 UnSRGAN(ours) 26.13 0.7101 3.001 15.91
[0074] As can be seen from Table 1, in terms of quantitative results, compared with other unsupervised super-resolution reconstruction methods, the present invention is second only to the ZSSR method in terms of PSNR and ESRGAS indicators, and has achieved performance that exceeds other methods in terms of SSIM indicators and NIQE indicators. NIQE is relatively consistent with the perception and judgment of the human eye, which also reflects the improvement of the visual effect of the reconstructed image by the present invention. The visualization results are shown in Figure 1. Figure 2 As shown. The image reconstructed by the present invention has the finest texture details. In terms of visualization results, the reconstructed image achieves the best visual effect. In the reconstruction of the gray roof, only the present invention achieves detail performance close to that of the high-resolution image texture, which also verifies the effectiveness of UnSRGAN (unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction) and its good detail recovery ability. This proves the great application value of the present invention for unsupervised super-resolution reconstruction of remote sensing images.
[0075] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0076] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction, characterized in that: The training process includes a generative adversarial network model: the generative adversarial network model includes a generator module, a discriminator module, and an image degradation module; the training process is as follows: Step 1: Take the low-resolution image to be trained as input and generate a high-resolution reconstructed image through the generator module; Step 2: The high-resolution reconstructed image is passed through the image degradation module to generate a degraded low-resolution image; Step 3: taking the low-resolution image and the degraded low-resolution image as input and performing discrimination through the discriminator module; The discrimination process performs unsupervised training on the discriminator module to discriminate true from false based on the mapping relationship between the low-resolution image and the degraded low-resolution image; And output the judgment result; After the training is completed, the unreconstructed low-resolution remote sensing image is used as input and the corresponding high-resolution remote sensing image is generated through the generator module; The generative adversarial network loss includes a network overall loss function and a discriminator loss function; the network overall loss function constrains the generator module; the discriminator module loss function constrains the discriminator module; The overall loss function of the network is: L G_total =λ1·L cb +λ per ·L per +λ adv ·L G_adv Among them, L G_total is the overall loss function of the network, L cb Pixel loss, L per represents the perceptual loss, L G_adv represents the adversarial loss of the generator module, and λ1 represents L cb The weight, λ per Indicates L per The weight, λ adv represents the adversarial loss weight of the generator module; The adversarial loss function of the generator module is: in, is the first constraint of the generator module, is the second constraint of the generator module, SR is the high resolution of reconstruction, F(I LR ; θ) is the degenerate operator, I LR is a low-resolution image, θ is a parameter, D(I LR ,F(I LR ;θ)) and D(F(I LR ;θ),I LR ) is the constraint of the discriminator module on the low-resolution image and the reconstructed image degraded low-resolution image; The pixel loss formula is: Among them, N is the total number of pixels, i is the i-th pixel, F(I LR ; θ) is the degradation operator of the input image, ρ(x) is the intermediate result, ε is a minimum constant, x is a high-resolution image, ρ(F(I LR ;θ)-I LR ) is the pixel loss between the low-resolution image and the low-resolution image of the degraded image, θ is a parameter, I LR It is a low-resolution image; The discriminator module loss function is: Among them, D(I LR ,F(I LR ;θ)) and D(F(I LR ;θ),I LR ) is the constraint of the discriminator module on the low-resolution image and the reconstructed image degraded low-resolution image, I LR is a low-resolution image, LR is low resolution, θ is a parameter, F(I LR ; θ) is the degenerate operator.
2. The unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction according to claim 1, characterized in that: The structure of the generator module includes a basic module, a SAF module, an upsampling module, and a convolutional layer module; The low-resolution image to be trained is passed through the convolutional layer module to generate output result 1; the output result 1 is passed through several basic modules and convolutional layer modules to generate output result 2; the output result 1 and the output result 2 are added and passed through the SAF module, the upsampling module and the two convolutional layer modules to generate a high-resolution reconstructed image.
3. The unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction according to claim 1, characterized in that: The structure of the discriminator module includes a fully convolutional network layer for extracting features of the low-resolution image and the degraded low-resolution image.
4. The unsupervised remote sensing image super-resolution reconstruction method based on constrained reconstruction according to claim 2, characterized in that: The basic module includes a dense block and a channel attention module.
