Remote sensing image super-resolution reconstruction method

Abstract

The invention belongs to the field of high-magnification image super-resolution, and particularly relates to a remote sensing image super-resolution reconstruction method. The method comprises the following steps: acquiring a high-quality and high-resolution remote sensing image data set, and performing cutting and down-sampling operation on a remote sensing image to obtain a low-resolution image; randomly generating noise and generating a noise image according to the noise; processing the low-resolution image and the noise image by adopting a conditional truncation noise generator to obtain a truncation image and conditional noise; performing denoising processing on the truncated image according to the conditional noise to obtain a high-resolution image; calculating the total loss of the model and adjusting model parameters according to the total loss of the model to obtain a trained dual-output truncation diffusion model; according to the method, the image generation quality is greatly improved, the time used for sampling the image and the parameter quantity of the model are greatly reduced, the model output result is more accurate, and the method has a good application prospect.

Description

Technical Field

[0001] The invention belongs to the field of large-magnification image super-resolution, and in particular relates to a remote sensing image super-resolution reconstruction method. Background Art

[0002] The resolution of remote sensing images is the key to remote sensing image parsing. Images with higher resolution always have better performance in downstream remote sensing image processing tasks, such as image segmentation, object detection, and image classification. Although the spatial resolution of images captured by modern optical lenses is close to 1 meter / pixel, it is still not enough to distinguish remote sensing images with complex textures and backgrounds, such as dense vehicles and dense vegetation. As the physical performance of imaging equipment is getting closer and closer to its limits, researchers in the remote sensing academic community have begun to focus on using deep learning-based models to attempt to reconstruct high-resolution remote sensing (HRRS) images. In general, super-resolution (SR) tasks with more common super-resolution factors (×2, ×4) on remote sensing images are relatively easy. Many traditional methods can easily handle it. However, when the goal of super-resolution is to construct HRRS images from extremely low-resolution remote sensing (LRRS) images (SR factor ≥ ×8), it becomes extremely difficult. This is because the texture of LRRS is very blurred and the clues are extremely limited. If the method is not changed, the reconstruction results are often unsatisfactory. The large-scale super-resolution (LFSR) task is to attempt to restore a satisfactory image in LRRS.

[0003] Inspired by the success of generative models, some researchers began to formulate the LFSR task as a generative problem: y = g(x), where g(·) represents a generative method. Currently, models based on generative adversarial models (GANs) are the mainstream solution in LFSR tasks. Compared with traditional deep learning methods, images reconstructed by generative models obtain more realistic and natural high-resolution textures and backgrounds. However, the textures generated by GANs are uncontrolled, resulting in lower fidelity. In addition, GANs are unstable, difficult to train, and prone to producing erroneous images when trained on small-scale datasets. Therefore, researchers have tried to extract prior knowledge from high-resolution images to obtain better visual quality in LFSR tasks. However, unlike face images with specific objects and backgrounds, remote sensing images contain a variety of categories and complex textures. Therefore, it is extremely difficult to choose a reasonable prior in remote sensing image LFSR.

[0004] Diffusion models have received extensive attention due to their ability to generate high-quality images. Compared to GANs, DMs have advanced generation performance and are not affected by training instabilities or mode collapse. Researchers proposed a diffusion model-based method SR3, which achieved state-of-the-art objective results and impressive visual performance in general SR tasks. However, similar to GANs, due to the lack of explicit pixel-level constraints, the SR3 model tends to be blurry when reconstructing areas with rich textures. In addition, the computational burden of SR3's iterative sampling and the sampling time are also unacceptable for the processing of large-scale remote sensing images.

[0005] In summary, there is an urgent need for a remote sensing image super-resolution reconstruction method to solve the problems of low quality and lack of authenticity when performing large-scale super-resolution, as well as the problem of long sampling time when using existing diffusion models for super-resolution tasks. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a remote sensing image super-resolution reconstruction method, which comprises: obtaining a low-resolution remote sensing image to be processed, inputting the noise image and the low-resolution remote sensing image to be processed into a trained dual-output truncated diffusion model to obtain a truncated image and conditional noise; denoising the truncated image according to the conditional noise to obtain a high-resolution image;

[0007] The training process of the dual-output truncated diffusion model includes:

[0008] S1: Collect high-quality and high-resolution remote sensing image datasets, crop and downsample the remote sensing images to obtain low-resolution images;

[0009] S2: randomly generate noise and generate a noise image based on the noise; use a conditional truncated noise generator to process the low-resolution image and the noise image to obtain a truncated image and conditional noise;

[0010] S3: De-noising the truncated image according to the conditional noise to obtain a high-resolution image;

[0011] S4: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained dual-output truncated diffusion model.

[0012] Preferably, the process of processing the low-resolution image and the noise image by the conditional truncation noise generator includes:

[0013] Perform position-wise downsampling on the low-resolution image and the noisy image to obtain the sampled low-resolution image and the noisy image;

[0014] Perform convolution processing on the low-resolution image and the noise image to obtain convolution low-resolution features and convolution noise features;

[0015] The convolution low-resolution features and convolution noise features are processed by multiple FCA modules in sequence to obtain Fourier low-resolution features and Fourier noise features;

[0016] Concatenate the convolution low-resolution features and the Fourier low-resolution features to obtain low-resolution concatenated features; concatenate the convolution noise features and the Fourier noise features to obtain noise concatenated features;

[0017] An encoder is used to process low-resolution splicing features and noise splicing features to obtain deep image features and deep noise features;

[0018] The depth image features and depth noise features are processed by position-wise upsampling to obtain truncated images and conditional noise.

[0019] Furthermore, the number of FCA modules is 12.

[0020] Furthermore, discrete Fourier transform is performed on the convolution low-resolution features and convolution noise features respectively; the features after discrete Fourier transform are multiplied with two learnable filters respectively to obtain frequency domain low-resolution features and frequency domain noise features; the frequency domain low-resolution features and frequency domain noise features are respectively input into two cross-self-attention modules for processing to obtain cross-self-attention low-resolution features and cross-self-attention noise features; discrete Fourier inverse transform is performed on the cross-self-attention low-resolution features and cross-self-attention noise features, and then they are successively input into the GELU activation function, LayerNorm layer and FFN layer for processing to obtain two outputs of the FCA module.

[0021] Furthermore, the cross self-attention module consists of three linear layers and a logmax function.

[0022] Preferably, the total model loss is a weighted sum of truncation loss, classical diffusion loss, and cross entropy loss.

[0023] Furthermore, the calculation formula of truncation loss is:

[0024]

[0025] Among them, T loss represents the truncation loss, t r is expressed as the number of truncation steps, Represents a truncated image. represents a noisy image, It means that the next truncated image is calculated by the classic denoising formula in the diffusion model. Represents the true noisy image for the next step.

[0026] The beneficial effects of the present invention are:

[0027] Observing the frequency domain characteristics of remote sensing images in the diffusion model, the present invention uses the Fourier transform method to couple the characteristics of noise and truncated noise images into the Fourier domain, and designs a new conditional truncated noise generator (TNG) to simultaneously predict the posterior noise and truncated noise images. After such processing, the model of the present invention can simultaneously generate a conditional noise ∈ t and the rough SR image This avoids the low-quality results seen in traditional dual-output networks.

[0028] In order to improve the overall operation results of the diffusion model and enable the model to be used in the system, the time of sampling images must be saved. Referring to the idea of ​​truncated network, the present invention designs a new truncation method. For the truncated noise image, the time step t can be used to r Start denoising, where t r It is significantly smaller than the original denoising time step T (T ≥ 1000) in the ordinary diffusion model. Taking advantage of this, the model can reduce the sampling step from T to the truncation step t r (t r =100). In addition, the diffusion model redesigned by the present invention reduces the model parameters while maintaining high performance. Finally, compared with the large-rate super-resolution method based solely on the diffusion model, TNG saves nearly 90% of the sampling time and 50% of the parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural block diagram of the conditional truncation noise generator in the present invention;

[0030] Figure 2 This is an example diagram of Position-wise up and down sampling in the present invention;

[0031] Figure 3 Schematic diagram of the structure of the novel Fourier cross attention module in the present invention;

[0032] Figure 4 Schematic diagram of the structure of the cross self-attention module in the present invention;

[0033] Figure 5 This is the reverse sampling process in the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention proposes a remote sensing image super-resolution reconstruction method, such as Figure 1 As shown, the method includes the following contents:

[0036] The low-resolution remote sensing image to be processed is obtained, and the noisy image and the low-resolution remote sensing image to be processed are input into the trained dual-output truncated diffusion model to obtain a truncated image and conditional noise; the truncated image is denoised according to the conditional noise to obtain a high-resolution image.

[0037] The training process of the dual-output truncated diffusion model includes:

[0038] S1: Collect high-quality and high-resolution remote sensing image datasets, crop and downsample the remote sensing images to obtain low-resolution images.

[0039] Collect high-quality and high-resolution remote sensing image datasets, such as UC Merced Land Use Dataset and Remote Sensing Scene Classification. Crop remote sensing images to a certain size, for example, 128×128, 256×256, or 512×512.

[0040] The cropped image is sampled, and the specific downsampling needs to be specified according to the magnification factor (for example, if the magnification factor is 8, the 128×128 image needs to be downsampled to 16×16, and the sampling methods of other magnification factors are similar). Preferably, bicubic sampling is used to obtain low-resolution images of different sizes. The paired high-resolution images and low-resolution images are divided into training sets and test sets in a ratio of 8:2.

[0041] S2: randomly generate noise and generate a noisy image based on the noise; use a conditional truncated noise generator to process the low-resolution image and the noisy image to obtain a truncated image and conditional noise.

[0042] Generate a noisy image and complete the step-by-step denoising of the noisy image through the following denoising formula:

[0043]

[0044] Among them, y t-1 represents the noise image at time t-1, y t represents the noise image at time t, α t is a hyperparameter that increases with time t, ∈ is a real random noise at time t.

[0045] Inspired by a network called SwinIR, this paper cancels the use of the traditional UNet model and proposes a conditional truncated noise generator TNG as a denoising network. Different from the traditional method, the conditional truncated noise generator of this invention first converts the low-resolution image The shape and size are reconstructed to the same as the real high-resolution image The present invention achieves this effect using a position-wise approach.

[0046] like Figure 2 As shown in the figure, the downsampling and upsampling of the position-wise sampling method are calculated as follows, assuming that A and B are two feature maps of different sizes. They both contain three dimensions (channel, height, and width). A(B) is the result of position-wise downsampling (position-wise upsampling) on ​​B(A). Note that c, i, and j are the indices of the generated feature maps, representing channels, height, and width, respectively:

[0047]

[0048] Use a simple 3×3 depth convolution to model low-level features, that is, convolve the low-resolution image and the noisy image to obtain low-level features (convoluted low-resolution features and convoluted noise features);

[0049] Afterwards, the present invention carefully designs a high-level feature extractor (composed of multiple FCA modules) with residual connection, which introduces a novel Fourier cross attention (FCA), and inputs low-level features, namely convolution low-resolution features and convolution noise features, into the high-level feature extractor, which are processed by multiple FCA modules in turn in the high-level feature extractor to obtain Fourier low-resolution features and Fourier noise features.

[0050] Too few stacked FCA modules will lead to poor results, while too many will not improve performance but will lead to excess model parameters. Preferably, after multiple experiments, combined with comprehensive considerations of performance and quality, the present invention selects 12 stacked FCA modules.

[0051] The FCA module processes the convolution low-resolution features and convolution noise features by: performing discrete Fourier transform on the convolution low-resolution features and convolution noise features respectively; multiplying the features after discrete Fourier transform with two learnable filters respectively to obtain frequency domain low-resolution features and frequency domain noise features; inputting the frequency domain low-resolution features and frequency domain noise features into two cross-self-attention modules (CA) for processing respectively to obtain cross-self-attention low-resolution features and cross-self-attention noise features; performing inverse discrete Fourier transform on the cross-self-attention low-resolution features and cross-self-attention noise features, and then inputting them into the GELU activation function, LayerNorm layer and FFN layer for processing in turn to obtain two outputs of the FCA module. Among them, the cross-self-attention module includes three linear layers and a logmax function.

[0052] like Figure 3 As shown, in FCA, the input low-resolution feature f is first convolved t and the convolution noise feature f lr Perform discrete Fourier transform (DFT) to enter the Fourier domain. Next, convolute the low-resolution feature f t and f lr and the corresponding learnable filter ω t and ω n Multiply them together to collect frequency perception clues. Figure 4 As shown, next, cross attention is performed on the two features as follows:

[0053] f lr =logmax(Qf lr )·K(f t ) T )Vf t ),

[0054] Among them, Q, K, and V are linear layers designed to learn labels in the Fourier domain. Afterwards, the logmax function is calculated as follows to alleviate numerical instability:

[0055]

[0056] At the end of the FCA module, the features extracted in the Fourier domain are mapped back to the spatial domain through the inverse discrete Fourier transform IDFT and input into a sequential GELU activation function, LayerNorm and FFN.

[0057] In FCA, the features extracted by DFT are input into the cross-attention CA module and mapped back to the spatial domain by IDFT. The complexity of DFT and IDFT is nlog(n). Due to the symmetry of the real image DFT, the feature length of the frequency domain (1 / 2)n is half of that of the spatial domain (n), and the complexity of CA in the Fourier domain is reduced to (1 / 2)O(n2 ). Therefore, the time complexity of FCA is 2nlog(n)+(1 / 2)O(n 2 ), which is better than the ordinary self-attention layer O(n 2 ) is much smaller.

[0058] The convolution low-resolution features and the Fourier low-resolution features are concatenated to obtain low-resolution concatenated features; the convolution noise features and the Fourier noise features are concatenated to obtain noise concatenated features.

[0059] At the end of TNG, an encoder (Decoder network) is used to process the low-resolution splicing features and the noise splicing features to obtain deep image features and deep noise features. In the present invention, the Decoder network is mainly composed of a convolutional network and various activation functions stacked into a network layer.

[0060] The depth image features and depth noise features are processed by position-wise upsampling to obtain truncated images and conditional noise.

[0061] In TNG, the present invention stacks 12 FCA modules as advanced feature extractors. Compared with the UNet model in the original framework, the model of the present invention has only 1 / 3 of the computational cost and can simultaneously estimate the conditional noise ∈ t and the rough SR image Among them, the truncation step t set in the present invention r The smaller it is, the shorter the sampling time is. Different from the existing accelerated versions of diffusion models (such as DDIM, LDM), TNG applies a Fourier-based network to effectively learn conditional noise.

[0062] S3: De-noise the truncated image according to the conditional noise to obtain a high-resolution image.

[0063] According to the denoising formula in S2, the truncated image is denoised using conditional noise to obtain a high-resolution image.

[0064] S4: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained dual-output truncated diffusion model.

[0065] Calculate the total model loss and continuously update the model parameters according to the gradient direction. When the loss function converges or reaches the preset number of iterations, save the optimal model parameters and obtain

[0066] The present invention designs a truncation loss (T loss) to reduce the truncated image and the real noisy image The gap between:

[0067]

[0068] Among them, T loss represents the truncation loss, t r is expressed as the number of truncation steps, Represents a truncated image. represents a noisy image, It means that the next step of truncated image is calculated by the denoising formula in S2. Represents the true noisy image for the next step.

[0069] The total loss of the model of the present invention is the weighted sum of truncation loss, classical diffusion loss and cross entropy loss. The total loss function of the model is:

[0070] TC loss =D loss +P loss +α·T loss

[0071] Among them, TC loss Represents the total loss of the model, D loss represents the classical diffusion loss; P loss represents the cross entropy loss, which aims to reduce the predicted rough high-resolution image The difference between the original high-resolution image Y0; α represents the weight, T loss represents the truncation loss.

[0072] Where D loss and P loss They are respectively the following formulas, where x is the input low-resolution image:

[0073]

[0074] in, If p=1, it represents L1 loss.

[0075] like Figure 5 As shown in the figure, when the model is tested with the test set, the low-resolution image in the test set is input into the network model. The model first samples a random noise, which is a noise image of T steps. Then, according to the set truncation step (100 steps), a truncated image is generated in one step. This truncated image is then iteratively denoised to gradually generate a high-resolution remote sensing image without noise. During model training, it is recommended to use a graphics card with a higher configuration for training and use the Adam optimizer for optimizing the network. A linear sampling method from 1e-5 to 1e-2 is used, and the iteration step of TNG is set to 1000 steps during training. Due to the cross-multiple loss functions between modules in the conditional truncated noise generator, a specific trade-off design is required to integrate them.

[0076] After the model training is completed, the low-resolution remote sensing image to be processed is obtained, and the noisy image and the low-resolution remote sensing image to be processed are input into the trained dual-output truncated diffusion model to obtain a truncated image and conditional noise; the truncated image is denoised according to the conditional noise to obtain a high-resolution image.

[0077] In summary, the present invention aims to solve the problems of low quality and lack of authenticity in large-rate super-resolution in the prior art, as well as the problem of long sampling time in super-resolution tasks using ordinary diffusion-based models. First, the present invention utilizes the characteristics of Fourier transform to couple the characteristics of noise and truncated noise images into the Fourier domain, and designs a new conditional truncated noise generator to simultaneously predict the posterior noise and truncated noise images. This not only greatly improves the quality of image generation, but also greatly reduces the time used for sampling images and the number of model parameters. The newly designed model structure of the present invention also enables it to demonstrate performance far superior to traditional methods in large-rate super-resolution tasks. At the same time, the present invention designs a loss function combination that is more suitable for the present invention, with better training effects and higher model accuracy.

[0078] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image super-resolution reconstruction method, characterized in that: include: Obtain a low-resolution remote sensing image to be processed, input the noise image and the low-resolution remote sensing image to be processed into a trained dual-output truncated diffusion model to obtain a truncated image and conditional noise; De-noising the truncated image according to the conditional noise to obtain a high-resolution image; The training process of the dual-output truncated diffusion model includes: S1: Collect high-quality and high-resolution remote sensing image datasets, crop and downsample the remote sensing images to obtain low-resolution images; S2: randomly generate noise and generate a noise image based on the noise; use a conditional truncated noise generator to process the low-resolution image and the noise image to obtain a truncated image and conditional noise; S3: De-noising the truncated image according to the conditional noise to obtain a high-resolution image; S4: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain the trained dual-output truncated diffusion model.

2. A remote sensing image super-resolution reconstruction method according to claim 1, characterized in that: The process of processing low-resolution images and noisy images by the conditional truncated noise generator includes: Perform position-wise downsampling on the low-resolution image and the noisy image to obtain the sampled low-resolution image and the noisy image; Perform convolution processing on the low-resolution image and the noise image to obtain convolution low-resolution features and convolution noise features; The convolution low-resolution features and convolution noise features are processed by multiple FCA modules in sequence to obtain Fourier low-resolution features and Fourier noise features; Concatenate the convolution low-resolution features and the Fourier low-resolution features to obtain low-resolution concatenated features; concatenate the convolution noise features and the Fourier noise features to obtain noise concatenated features; An encoder is used to process low-resolution splicing features and noise splicing features to obtain deep image features and deep noise features; The depth image features and depth noise features are processed by position-wise upsampling to obtain truncated images and conditional noise.

3. A remote sensing image super-resolution reconstruction method according to claim 2, characterized in that: The number of FCA modules is 12.

4. The remote sensing image super-resolution reconstruction method according to claim 2, characterized in that: The convolution low-resolution features and convolution noise features are discrete Fourier transformed respectively; the features after discrete Fourier transformation are multiplied by two learnable filters respectively to obtain frequency domain low-resolution features and frequency domain noise features; the frequency domain low-resolution features and frequency domain noise features are input into two cross-self-attention modules for processing respectively to obtain cross-self-attention low-resolution features and cross-self-attention noise features; the cross-self-attention low-resolution features and cross-self-attention noise features are discretely inverse transformed, and then they are input into the GELU activation function, LayerNorm layer and FFN layer for processing in turn to obtain the two outputs of the FCA module.

5. A remote sensing image super-resolution reconstruction method according to claim 4, characterized in that: The criss-cross self-attention module consists of three linear layers and a logmax function.

6. A remote sensing image super-resolution reconstruction method according to claim 1, characterized in that: The total loss of the model is the weighted sum of truncation loss, classical diffusion loss and cross entropy loss.

7. A remote sensing image super-resolution reconstruction method according to claim 6, characterized in that: The calculation formula of truncation loss is: Among them, T loss represents the truncation loss, t r is expressed as the number of truncation steps, Represents a truncated image. represents a noisy image, It means that the next truncated image is calculated by the classic denoising formula in the diffusion model. Represents the true noisy image for the next step.

Images