Super-resolution remote sensing image acquisition method and device

Abstract

The invention discloses a super-resolution remote sensing image acquisition method and system, a medium and equipment, relates to the technical field of image restoration, and is realized through an enhanced network of a multi-scale parameter-free attention mechanism, the network extracts shallow layer features of a low-resolution remote sensing image by using a convolutional layer, and solves the problem that the super-resolution network of the remote sensing image is low in cost. Shallow layer features are input into a proposed multi-scale parameter-free attention enhancement module, the module uses parallel connection and combination of convolution layers of multiple convolution kernels of different sizes to refine extraction of multi-scale features, and under a parameter-free attention mechanism, feature information with high contribution is enhanced through a symmetric activation function, and redundant information is suppressed. And a final reconstructed image is generated by a reconstruction module after passing through a multi-scale parameter-free attention enhancement module with six residual errors connected. A large number of experimental results show that compared with an existing representative method, the network has remarkable reconstruction advantages in the aspects of performance indexes and visual effects, and both the PSNR and the SSIM are superior to those of other comparison methods.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a method and device for acquiring a super-resolution remote sensing image. Background Art

[0002] Image super-resolution (SR) technology is an important technology in computer vision to improve image resolution, and restore high-resolution (HR) images from degraded low-resolution (LR) images. In recent years, remote sensing images have been widely used in environmental monitoring, target detection, scene classification, resource exploration, and land cover classification. However, due to the influence of hardware and environmental factors, the acquired remote sensing images have low-resolution problems such as small amount of information and low clarity. Therefore, how to improve the resolution and image quality of remote sensing images has become a research hotspot for many researchers.

[0003] With the rapid development of deep learning technology, existing image SR methods often use single image super-resolution (SISR) technology based on convolutional neural network (CNN) to learn mapping relationships from large-scale LR-HR image pairs, thereby achieving more accurate super-resolution reconstruction effects. Among them, the SRCNN proposed by Dong et al. uses a three-layer convolutional neural network structure to extract deep features from the original image, thereby gradually restoring super-resolution details and texture information. EDSR improves performance by eliminating unnecessary modules in traditional residual networks. RCAN proposed by Zhang et al. improves the accuracy of SR reconstruction by adopting a channel attention mechanism. DBPN uses iterative up- and down-sampling to provide an error feedback mechanism for super-resolution, so that the features in the up- and down-sampling stages are connected to improve the SR results. In order to obtain better super-resolution reconstruction effects, the second-order attention network (SAN) uses an attention mechanism to focus on different regions and channels of the image. The Transformer-based SwinIR effectively improves the image's detail recovery ability and SR reconstruction quality by learning long-range dependencies and regional correlations in the image.

[0004] In the field of remote sensing, although the reconstruction effect of the above-mentioned image super-resolution network is good, the current SR network with increasing depth often introduces feature information redundancy, which in turn increases the number of parameters and computational complexity of the network model. In addition, due to the complex factors of remote sensing image degradation and content diversity, multi-scale information extraction of remote sensing images is the key to remote sensing image SR. Among them, the attention mechanism significantly improves the performance of image super-resolution, but it also leads to problems such as the complexity of the network structure, the increase of parameters, slow speed and high computational cost. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art that the remote sensing image super-resolution network structure is complex, has too many parameters and a large model scale, the main purpose of the present invention is to provide a super-resolution remote sensing image acquisition method.

[0006] To achieve the above object, the present invention adopts the following technical solution, a method for acquiring super-resolution remote sensing images, comprising the following steps:

[0007] Low-resolution remote sensing image I obtained through multiple convolutional layers LR Perform shallow feature extraction to obtain shallow features;

[0008] A multi-scale parameter-free attention enhancement module is constructed by connecting multiple convolutional layers with convolutional kernels of different sizes in parallel, and multiple multi-scale parameter-free attention enhancement modules are residually connected to form a multi-scale parameter-free attention enhancement block, and shallow features are input into the multi-scale parameter-free attention enhancement block to obtain multi-scale features;

[0009] Based on the parameter-free attention mechanism, the multi-scale features are extracted in deep layers and fused with shallow features to obtain a super-resolution remote sensing image I SR .

[0010] The super-resolution remote sensing image I SR , expressed as:

[0011] I SR =F(I LR ,θ) (1)

[0012] Among them, F(·) represents the mapping function and θ is the network parameter.

[0013] The low-resolution remote sensing image I LR The original high-resolution remote sensing image I HR Obtained via bicubic downscaling.

[0014] The network parameter θ is expressed as:

[0015]

[0016] in, It is represented as the i-th remote sensing low-resolution image in the training set, and the total number of images in the training set is N; It represents a high-resolution remote sensing image reconstructed from a low-resolution image through a mapping function; For the corresponding The i-th original high-resolution remote sensing image; is the loss function.

[0017] The multi-scale parameter-free attention enhancement module consists of three convolutional layers with convolution kernels of different sizes.

[0018] The enhanced network of the multi-scale parameter-free attention mechanism adopts the L1 loss function, which is expressed as:

[0019]

[0020] Among them, ||·||1 is the L1 loss function.

[0021] A system for obtaining a super-resolution remote sensing image, comprising:

[0022] The resolution reconstruction system constructs a multi-scale parameter-free attention mechanism enhanced network, and the multi-scale parameter-free attention mechanism enhanced network reconstructs the super-resolution of remote sensing images. The resolution reconstruction includes:

[0023] Shallow feature extraction module, through multiple convolutional layers to obtain low-resolution remote sensing images I LR Perform shallow feature extraction to obtain shallow features;

[0024] An enhancement module, which constructs a multi-scale parameter-free attention enhancement module by connecting multiple convolution layers with convolution kernels of different sizes in parallel, and multiple multi-scale parameter-free attention enhancement modules are residually connected to form a multi-scale parameter-free attention enhancement block;

[0025] A multi-scale feature acquisition module, which inputs shallow features into the multi-scale parameter-free attention enhancement block to obtain multi-scale features;

[0026] The high-definition image acquisition module extracts deep multi-scale features based on the parameter-free attention mechanism and fuses them with shallow features to obtain super-resolution remote sensing images I SR .

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1) A multi-scale parameter-free attention enhancement network (MPAEN) is proposed to solve the remote sensing image super-resolution problem, which simplifies the network structure without reducing the accuracy and effectively improves the speed and performance of the network. Its subjective and objective comparisons are better than other existing representative SR methods.

[0029] 2) Design a multi-scale parameter-free attention enhancement module (MPAEM). This module not only adaptively detects multi-scale detail information of remote sensing images, but also uses a parameter-free attention mechanism to quickly extract feature information from shallow to deep layers. In efficient multi-scale feature extraction, this module ensures that the number of model parameters is reduced while also improving the efficiency of super-resolution reconstruction of remote sensing images. In order to prevent gradient explosion and improve the stability of network training, residual connections are used between each module to perform network feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0031] Figure 1 It is the overall network framework diagram of MPAEN of the present invention;

[0032] Figure 2 is a schematic diagram of subjective comparison of reconstruction results of different methods on the PatternNet dataset in an embodiment of the present invention;

[0033] Figure 3 3 is a schematic diagram of subjective comparison of reconstruction results of different methods on the NWPUR-ESISC45 dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0035] In the field of remote sensing, although the reconstruction effect of the current image super-resolution network is good, the current SR network with increasing depth often introduces feature information redundancy, which in turn increases the number of parameters and computational complexity of the network model. In addition, due to the complex factors of remote sensing image degradation and content diversity, multi-scale information extraction of remote sensing images is the key to remote sensing image SR. Among them, the attention mechanism significantly improves the performance of image super-resolution, but it also leads to problems such as the complexity of the network structure, the increase of parameters, slow speed and high computational cost.

[0036] In order to overcome the shortcomings of the above-mentioned prior art that the remote sensing image super-resolution network structure is complex, has too many parameters and a large model scale, the main purpose of the present invention is to provide a super-resolution remote sensing image acquisition method.

[0037] The overall network framework of this application is as follows Figure 1 As shown in Figure 2, the entire network consists of shallow feature extraction, deep multi-scale feature extraction and image reconstruction. In the multi-scale parameter-free attention mechanism enhanced network MPAEN network framework, the low-resolution remote sensing image I is first LR The shallow feature extraction is performed, and then multiple continuous residual connected MPAEM blocks are used to capture the key information in the image faster and better under the parameter-free attention mechanism, and deep multi-scale feature extraction and fusion are performed to improve the reconstructed image I SR The quality and efficiency of the entire network can be expressed as:

[0038] I SR =F(I LR ,θ) (1)

[0039] Where F(·) represents the mapping function of the MPAEN network, and θ is the network parameter. In the experimental data set of this application, the original high-resolution remote sensing image I HR , Generate low-resolution remote sensing image I through bicubic downscaling (BD) LR . And learning I LR and I HR The end-to-end mapping function F between them is the ultimate goal of this application study. H and W represent the height and width of the remote sensing image, respectively. C represents the number of channels of the remote sensing image. r represents the size of the magnification. The training dataset is represented as The entire network update parameters can be expressed as:

[0040]

[0041] Although the mean square loss function and the L2 norm loss function obtain higher objective evaluation indicators in image reconstruction, remote sensing images reconstructed using these loss functions in recent years often have overly smooth texture problems. Experiments have found that these loss functions have a subtle improvement in the performance of remote sensing image reconstruction. In order to improve the prediction accuracy and operation rate of the model, the MPAEN of this application finally adopts the L1 loss function to reduce network calculations. Therefore, the loss function of this application is defined as:

[0042]

[0043] In order to efficiently obtain multi-scale feature information of remote sensing images, this application redesigns the basic convolutional block composition used for remote sensing image feature extraction in SISR tasks, and adds a parameter-free attention mechanism by design to enhance useful feature information and suppress redundant feature information. Figure 2 As shown in the figure, the MPAEM block consists of three convolutional layers with convolutional kernels of different scales. The feature information extracted by different convolutional kernels is shared to obtain multi-scale feature information. Then, the multi-scale feature information F i With input from the MPAEM block O i-1 Perform residual connection to form the pre-attention feature map U of this module i At the same time, the multi-scale feature information F i Through an activation function σ symmetric about the origin a (·) is passed to obtain the attention map V i The final pre-attention feature map U i and attention map V i The final output O of the MPAEM block of the multi-scale parameter-free attention mechanism is generated by multi-pixel multiplication. i =U i ⊙V i , where ⊙ represents the multiplication on the unit pixel. Then the process of the MPAEM block can be expressed as:

[0044]

[0045] Among them, F i j represents the feature of the jth convolutional layer in the multi-scale convolutional layer of the i-th MPAEM block; σ(·) represents the activation function after the convolutional layer; Concat(·) represents the multi-scale fusion layer; Represented as addition on unit pixels.

[0046] For the design between MPAEM blocks, this application uses a local residual learning strategy to improve the reconstruction performance while ensuring the reduction of computational cost. In summary, the MPAEM block not only utilizes the multi-scale features of remote sensing images to enhance the reconstruction ability of the network, but also enhances the information with high contribution and suppresses redundant information through symmetric activation functions and residual connections under the parameter-free attention mechanism, thereby achieving the goal of remote sensing image super-resolution.

[0047] for Figure 2 , Figure 3 Supplementary explanation of the English abbreviation:

[0048] Bicubic interpolation algorithm (Bicubic); Deep Back-Projection Network (DBPN); Residual Channel Attention Network (RCAN); Residual Dense Network (RDN); Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution (SAFMN); Mixed High-Order Attention Network (MHAN), Second-Order Attention Network (SAN); Enhanced Deep Super-Resolution Network (EDSR); Swin Transformer image restoration (Swin Transformer-based Image Restoration, SwinIR).

[0049] Example:

[0050] In order to verify the effectiveness of the method proposed in this application, a large number of experiments were conducted on the PatternNet and NWPU-RESISC45 datasets. The PatternNet remote sensing dataset images are collected from Google Earth images and through the Google Map API of some US cities, including 38 categories of scenes, each with 800 remote sensing images of size 256×256 pixels. In this experiment, 50 and 2 non-repeated remote sensing images are randomly selected in each type of scene as the training set and test set, respectively. Therefore, there are a total of 1,900 images in the training set and 76 images in the test set. The size of the HR remote sensing images in the training set and the test set is 256×256 pixels, and the corresponding LR remote sensing images are generated by bicubic downscaling BD of the reference HR remote sensing images, and the image size is 64×64 pixels.

[0051] The NWPU-RESISC45 remote sensing dataset contains 45 types of remote sensing scene data, totaling 31,500 256×256 pixel images. Except for islands, lakes, mountains and other low spatial resolutions, the spatial resolution of most scene categories is between 0.2 and 30m. At the same time, the dataset takes into account the influence of natural conditions such as different weather, seasons, and lighting, and has rich image changes in background, occlusion, etc. For the 45 types of scenes contained in this dataset, 50 and 2 non-repeated remote sensing images are randomly selected from each type of scene as the training set and test set, respectively. The LR remote sensing images are also generated by bicubic downscaling BD of HR remote sensing images. Therefore, there are a total of 2,250 images in the training set and 90 images in the test set. Like most remote sensing image SR tasks, the experimental objective evaluation indicators are tested on the Y channel.

[0052] The experimental implementation of this application is based on BasicSR. In the training phase of the MPAEN network, 16×16 pixel LR image patches are first randomly extracted, and then the training data is expanded using data augmentation.

[0053] For the design of the network, this experiment is deployed in the PyTorch framework. In the experimental details, the learning rate is initialized to 1×10 -4 , and every 1×10 4 The number of iterations is halved, for a total of 10 6 The training is repeated for 10 iterations, and the L1 norm loss function is minimized by using the Adam optimizer. This experiment selects four objective evaluation indicators, PSNR, SSIM, NIQE, and VIF, to evaluate the SR performance of the MPAEN network. These indicators are widely used in image quality assessment.

[0054] In order to verify the superiority of the proposed method, this application selects 8 advanced representative SR reconstruction methods, including DBPN, RDN, RCAN, SAN, MHAN, EDSR, SwinIR, and SAFMN, for experimental comparison. Tables 1 and 2 show the quantitative comparison of PSNR, SSIM, NIQE, and VIF indicators of ×4SR on the PatternNet dataset and the NWPU-RESISC45 dataset, respectively, as well as the average time for running the test 10 times on the dataset. Compared with the above SR methods, the method proposed in this application outperforms other methods by at least 0.1865 and 0.1409 dB on the PatternNet dataset and the NWPU-RESISC45 dataset, respectively. This proves the superiority of the MPAEM module proposed in this application.

[0055] The qualitative results of this method on the PatternNet dataset and the NWPU-RESISC45 dataset are shown as follows: Figure 2-3As shown in the figure. From the subjective remote sensing image, it can be seen that although some texture details can be inferred from the reconstruction results of other SR methods, the reconstructed image is blurred due to insufficient feature utilization. On the other hand, in the reconstruction effect, the reconstruction result of the method proposed in this application is clearer in details. And in terms of objective evaluation, all indicators are better than other SR methods.

[0056] Table 1 Comparison of objective evaluation indicators of reconstruction results of different methods on PatternNet dataset

[0057] Method Runtime / ms PNSR / dB↑ SSIM↑ NIQE↓ VIF↑ Bicubic - 24.6517 0.6786 9.5012 0.2491 DBP 75.32 25.6999 0.7622 9.1287 0.3170 RCAN 118.07 25.8211 0.7366 8.9969 0.3329 RDN 130.29 25.6505 0.7324 9.1222 0.3286 EDSR 211.36 25.9187 0.7408 8.7950 0.3317 SAFMN 72.06 25.8857 0.7399 9.2657 0.2984 MHAN 152.13 25.7419 0.7636 9.1135 0.3002 SAN 126.72 25.4496 0.7241 9.2342 0.2953 SwinIR 60.86 25.9543 0.7423 8.9779 0.3395 Ours 20.40 26.1408 0.7697 8.8042 0.3458

[0058] Table 2 Comparison of objective evaluation indicators of reconstruction results of different methods on the NWPUR-ESISC45 dataset

[0059]

[0060]

[0061] In this embodiment, an efficient parameter-free attention mechanism multi-scale feature extraction module: MPAEM is proposed. This module extracts multi-scale features of remote sensing images by combining multiple convolutional layers with different convolution kernel sizes, and enhances meaningful feature information and suppresses redundant feature information under the parameter-free attention mechanism, making full use of potential features.

[0062] This ablation experiment designs a set of experiments on the impact of different numbers of MPAEM modules on the network. The experimental results are shown in Table 3. When the number of MPAEM increases from 1 to 6, the performance of the proposed MPAEN network is enhanced. However, when the number of MPAEM is 6, as the SR network deepens, the reconstruction performance of the network begins to show a performance decline trend. The main reason is that the accumulation of multiple MPAEM modules leads to feature redundancy in remote sensing images and overfitting of the network during training, thereby reducing the reconstruction performance of the SR network.

[0063] Therefore, the network of this application uses 6 MPAEM modules to extract multi-scale feature information from remote sensing images while considering the computational cost and network reconstruction performance to achieve better remote sensing image reconstruction effect.

[0064] Table 3 Comparison of reconstruction performance of different numbers of MPAEM on the NWPUR-ESISC45 dataset

[0065]

[0066] The basic principle of a super-resolution remote sensing image acquisition method is as follows: a novel remote sensing image SR reconstruction network is proposed, which is composed of 6 residual-connected MPAEM modules to reconstruct remote sensing images with clear textures. In each MPAEM block, by combining multiple convolution kernels of different sizes side by side, the parallel connection of different convolutions is realized, and the local multi-scale information is fully utilized. At the same time, under the action of the parameter-free attention mechanism, meaningful feature information is enhanced and redundant feature information is suppressed, making full use of potential feature information. In the entire network, the residual-connected MPAEM module is used to extract fine texture detail information, and combined with the remote sensing feature information extracted by its shallow feature, the effective fusion of local and global spatial features is realized, and the utilization rate of multi-scale feature information is improved.

[0067] A large number of experiments have shown that, compared with the current advanced and representative SR methods, the proposed method has achieved the highest PSNR value and reconstructed the clearest remote sensing SR image, thus verifying the effectiveness and superiority of the proposed method. In future work, super-resolution reconstruction will be performed on remote sensing images of scenes in other fields to further improve the generalization ability of the reconstruction method.

[0068] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0069] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. A method for acquiring super-resolution remote sensing images, characterized in that: The following steps are involved: Constructing an enhanced network of a multi-scale parameter-free attention mechanism, including connecting multiple convolution layers with convolution kernels of different sizes in parallel to construct a multi-scale parameter-free attention enhancement module, and performing residual connection on multiple multi-scale parameter-free attention enhancement modules to obtain a multi-scale parameter-free attention enhancement network; Through multiple convolutional layers, the low-resolution remote sensing image I is processed LR Perform shallow feature extraction to obtain shallow features; Inputting the shallow features into the multi-scale parameter-free attention enhancement network to obtain multi-scale features; Inputting shallow features into the multi-scale parameter-free attention enhancement network to obtain multi-scale features; Based on the parameter-free attention mechanism, the multi-scale features are extracted in deep layers and fused with shallow features to obtain a super-resolution remote sensing image I SR .

2. The method for acquiring super-resolution remote sensing images according to claim 1, characterized in that: The super-resolution remote sensing image I SR , expressed as: I SR =F(I LR ,θ) (1) Among them, F(·) represents the mapping function and θ is the network parameter.

3. The method for acquiring super-resolution remote sensing images according to claim 1, characterized in that: The low-resolution remote sensing image I LR The original high-resolution remote sensing image I HR Obtained via bicubic downscaling.

4. The method for acquiring super-resolution remote sensing images according to claim 1, wherein: The network parameters are expressed as: in, It is represented as the i-th remote sensing low-resolution image in the training set, and the total number of images in the training set is N; It represents a high-resolution remote sensing image reconstructed from a low-resolution image through a mapping function; For the corresponding The i-th original high-resolution remote sensing image; is the loss function.

5. The method for acquiring super-resolution remote sensing images according to claim 1, characterized in that: The multi-scale parameter-free attention enhancement module includes three convolutional layers with convolution kernels of different sizes.

6. The method for acquiring super-resolution remote sensing images according to claim 1, characterized in that: The loss function used by the enhanced network of the multi-scale parameter-free attention mechanism is expressed as follows: Among them, ||·||1 is the L1 loss function, is the loss function and F(·) represents the mapping function.

7. A system for obtaining a super-resolution remote sensing image, characterized in that: include: The feature extraction module uses multiple convolutional layers to obtain low-resolution remote sensing images I LR Perform shallow feature extraction to obtain shallow features; Constructing an enhancement processing module, using multiple convolution layers with different sizes of convolution kernels to connect in parallel to construct a multi-scale parameter-free attention enhancement module, and performing residual connection on multiple multi-scale parameter-free attention enhancement modules to form a multi-scale parameter-free attention enhancement block; A multi-scale feature acquisition module, which inputs shallow features into the multi-scale parameter-free attention enhancement block to obtain multi-scale features; The super-resolution image acquisition module extracts deep multi-scale features from the multi-scale features based on the parameter-free attention mechanism and fuses them with the dual shallow features to obtain a super-resolution remote sensing image I SR .

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images