A hyperspectral image super-resolution reconstruction method based on multiple attention mechanisms

Abstract

The application provides a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism, and mainly comprises the following contents: 1) collecting hyperspectral image data and obtaining a training set and a test set through a data enhancement technology; 2) constructing a shallow feature extraction module and an image reconstruction module mainly based on a pixel attention mechanism; 3) constructing a deep feature extraction module mainly based on a transformer self-attention mechanism; 4) connecting the shallow attention mechanism module and the deep attention mechanism module through an ECA channel attention mechanism module to simultaneously explore multi-channel information; 5) adopting a context feature fusion module to fuse multi-band information; and 6) adopting the training set to fine-tune the hyperspectral image super-resolution reconstruction model parameters, and adopting the test set to test the hyperspectral image super-resolution reconstruction model. Experiments show that the method can effectively improve the resolution of the hyperspectral image, and has universal applicability and can be effectively applied to hyperspectral images in multiple scenes.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing image quality enhancement technology, and in particular to a method for super-resolution reconstruction of hyperspectral images based on a multiple attention mechanism. Background Technology

[0002] Hyperspectral images, as a data cube, contain hundreds of consecutive band images covering a wide spectral range. For any point in space, a hyperspectral image can reconstruct the corresponding material region and obtain its spatial and material properties through its continuous and detailed spectral curves. With the rapid development of aerospace and remote sensing technologies, hyperspectral images based on remote sensing satellites are widely used in land detection, urban planning, road network layout, agricultural yield estimation, disaster prevention, and other fields. Each spectral image in a hyperspectral imaging system corresponds to a very narrow spectral window, divided into tens to hundreds of consecutive spectral windows within a limited spectral range. However, achieving a good signal-to-noise ratio for spectral images requires a large instantaneous field of view and long exposure times to collect sufficient photons. Therefore, the spatial resolution of hyperspectral images is lower than that of natural and multispectral images. This affects the practical application of hyperspectral images. To more accurately describe images, hyperspectral image super-resolution algorithms have been proposed, which infer high-resolution hyperspectral images from a series of consecutive low-resolution hyperspectral images.

[0003] In recent years, with the superior performance of deep learning in computer vision tasks, SRCNN (Reference: C. Dong, CCLoy, K. He, and X. Tang. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, vol.38, no.2, pp.295–307.), VDSR (Reference: Kim, J.; Lee, JK; Lee, K. M. Accurate image super-resolution using very deep convolutional networks[J]. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp.1646–1654.), EDSR (Reference: Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M. Enhanced Deep Residual Networks for Single Image) have emerged. Super-Resolution[J].In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 12–26 July 2017; pp.1132–1140.), deep learning networks such as RDN (Reference: Y. Zhang, Y. Tian, ​​Y. Kong, B. Zhong, and Y. Fu[J]. Residual dense network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp.2472–2481.) have been proposed to address the image super-resolution problem.Despite achieving good results, these methods are designed for three-channel natural images. Directly applying them to hyperspectral networks ignores the spectral correlations between hyperspectral images, hindering the network's representational power. Furthermore, directly applying such parameter-growing networks to tens or even hundreds of hyperspectral images leads to a dramatic increase in parameters, making training difficult. Therefore, more and more super-resolution methods for single hyperspectral images have been proposed. Hu et al. proposed a deep spectral difference convolutional neural network combined with a spatial error correction model, achieving good spatial information enhancement and spectral information preservation (Reference: Hu, Y. Li, and W. Xie. Hyperspectral image super-resolution by spectral difference learning-ing and spatial error correction[J]. IEEE Geoscience and Remote Sensing Letters, 2017, vol. 14, no. 10, pp. 1825-1829.). Li et al. proposed a grouped deep recursive residual network by designing a grouped recursive module and embedding it into the global residual structure (Reference: Y.Li, L.Zhang, C.Dingl, W.Wei, and Y.Zhang. Single hyperspectral image super-resolution with grouped deeprecursive residual network[J].in Proc.IEEE 4th Int.Conf.Mul-timedia Big Data (BigMM), Sep.2018,pp.1–4.). Jiang et al. proposed a deep neural network based on a spatial spectral prior network, which fully utilizes the spatial information and spectral correlation between hyperspectral data (Reference: J.Jiang, H.Sun, X.Liu and J.Ma.Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery[J].IEEE Transactions on Computational Imaging,2020, vol.6,pp.1082-1096.). However, the above methods have limited improvement in spectral and spatial resolution.

[0004] This invention proposes a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism. Addressing the issue of insufficient spatial resolution in hyperspectral images, the feature extraction capabilities of different attention mechanisms in deep learning networks are comprehensively analyzed. A hierarchical multiple attention extraction network and a hyperspectral image fusion network are constructed, ultimately proposing a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention aims to overcome the shortcomings of existing technologies and propose a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism. This method includes acquiring hyperspectral image data, constructing a shallow feature extraction module and an image reconstruction module, constructing a deep feature extraction module, constructing an ECA channel attention mechanism module, constructing a context feature fusion module, fine-tuning the model parameters using a training set, and testing the hyperspectral image super-resolution reconstruction model using a test set.

[0006] To achieve the above objectives, this invention proposes a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism, comprising the following steps:

[0007] S1: Collect hyperspectral image data and augment the dataset using data augmentation methods such as rotation, flipping, and cropping to obtain the training and test sets.

[0008] S2: Construct a shallow feature extraction module and an image reconstruction module with pixel attention as the core.

[0009] S3: Construct a deep feature extraction module with the transformer self-attention mechanism as its core.

[0010] S4: The ECA channel attention mechanism module is used to connect the shallow attention mechanism module and the deep attention mechanism module to explore multi-channel information simultaneously.

[0011] S5: Uses a context feature fusion module to fuse multi-band information.

[0012] S6: Fine-tune the parameters of the hyperspectral image super-resolution reconstruction model using the training set, and test the hyperspectral image super-resolution reconstruction model using the test set.

[0013] Preferably, the hyperspectral image data in step S1 includes the Cave indoor hyperspectral image dataset, Pavia airborne remote sensing imagery, and Chikusei recorded airborne remote sensing imagery. The data augmentation methods include image cropping and image flipping, as well as image rotations at angles of 90°, 120°, and 270°.

[0014] Preferably, the construction of the shallow feature extraction module and image reconstruction module with pixel attention as the core in step S2 mainly includes the following methods and steps:

[0015] (1) Shallow Feature Attention Module. The shallow attention module consists of multiple transitive large-kernel pixel attention modules. The large-kernel pixel attention module first increases the channel dimension to 64 using a 1×1 2D convolution, then divides into a pixel attention branch and a main branch. The pixel attention branch uses two 5×5 convolutions with 3×3 dilated kernels to extract shallow image features. The large kernel convolution allows for a larger receptive field in the pixel attention branch. The pixel attention branch and the main branch are connected via multiplication. The connected attention branch integrates features through a 3×3 2D convolution and then uses residual connection to input features into the module. To alleviate training instability caused by the large-kernel pixel attention, a pixel normalization layer is added at the end of the module.

[0016] (2) Image Reconstruction Module. In order to perform super-resolution reconstruction of the feature image, a reconstruction module is constructed, which mainly includes image upsampling layers and bicubic interpolation method.

[0017] Preferably, the deep feature extraction module constructed in step S3, which is based on the transformer self-attention mechanism, mainly includes several progressively forward-propagating ETL modules. These modules include a transformer layer and an MLP layer, which are connected by skip connections. The transformer layer contains the transformer self-attention mechanism, which explores the similarity between each pixel and other pixels by querying the key matrix Q, the key matrix K, and the value matrix V. To reduce the number of parameters in the transformer layer, this method divides the image into different pixel blocks and uses a stripe shift window mechanism to explore the autocorrelation between different image blocks from both the H and W directions. The MLP layer integrates the transformed features through a fully connected perceptron layer.

[0018] Preferably, in step S4, an ECA channel attention mechanism module is used to connect the shallow attention mechanism module and the deep attention mechanism module to simultaneously explore multi-channel information. The shallow feature extraction module and the deep feature extraction module focus on feature extraction between the current channels. This method captures features across multiple channels by constructing a channel attention module, placing greater emphasis on channels containing more information.

[0019] Preferably, in step S5, a context feature fusion module is used to fuse multi-band information. Hyperspectral images contain dozens or hundreds of bands, and the information between each band can complement each other. This method adds a context feature fusion module to explore the spectral information of hyperspectral images based on the characteristics of hyperspectral imaging. It fuses multi-band information band by band using a progressive method between upper and lower bands.

[0020] Preferably, in step S6, the parameters of the hyperspectral image super-resolution reconstruction model are fine-tuned using the training set, and the model is tested using the test set. Specifically, this includes optimizing the designed model using the ADAM optimizer (β1 = 0.9, β2 = 0.999), with an initial learning rate of 10 for all layers. -4 The loss function is halved every 35 epochs. The network training is guided by a combination of L1 loss and gradient loss (GV), where the L1 loss calculates the absolute error between the target value and the model output estimate, as shown in the following equation.

[0021]

[0022] Gradient loss uses the variance of the image gradient map as a metric for image edge detail recovery, and calculates the gradients G in the x and y directions of the high-resolution and low-resolution images using the Sobel operator. SRx G SRy G LRx G LRy To better calculate the image gradient variance, the image is divided into n×n non-overlapping image blocks, and the average gradient value v of each block along the y-direction is calculated. i As shown in Equation 8. Finally, the gradient loss GV is obtained by calculating the mean values ​​of the high-resolution and low-resolution images along the x and y directions, as shown in the following equation.

[0023]

[0024] The overall loss is calculated as shown in the following formula.

[0025] Loss = L GV +L1

[0026] Therefore, the hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism of the present invention improves upon existing hyperspectral image super-resolution algorithms by introducing a multiple attention mechanism to extract hyperspectral image features at multiple levels. Simultaneously, it effectively fuses multi-band image features, improving spatial resolution while preserving the spectral resolution of the original hyperspectral image during image reconstruction. The super-resolution reconstructed hyperspectral image can be effectively applied to fields such as land detection, urban planning, road network layout, agricultural yield estimation, and disaster prevention. Attached Figure Description

[0027] The description of the present invention will become apparent and readily understood in conjunction with the following drawings, wherein:

[0028] Figure 1 This invention presents a flowchart of a hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism;

[0029] Figure 2 This is a technical flowchart of the algorithm model of this patent;

[0030] Figure 3 This is a detailed diagram of the shallow feature extraction module;

[0031] Figure 4 Detailed diagram of the deep feature extraction module;

[0032] Figure 5 This is a detailed diagram of the context feature fusion module. Detailed Implementation

[0033] according to Figure 1 The steps shown below provide a detailed description of the hyperspectral image super-resolution reconstruction method based on the multiple attention mechanism of the present invention.

[0034] Step S1: The hyperspectral image data includes the Cave indoor hyperspectral image dataset, Pavia airborne remote sensing imagery, and Chikusei recorded airborne remote sensing imagery. Data augmentation methods include image cropping and image flipping, as well as image rotation at angles of 90°, 120°, and 270°.

[0035] Step S2: Construct a shallow feature extraction module and an image reconstruction module with pixel attention as the core, mainly including the following methods and steps:

[0036] (1) Shallow Feature Attention Module. The shallow attention module consists of multiple transitive large-kernel pixel attention modules. The large-kernel pixel attention module first increases the channel dimension to 64 using a 1×1 2D convolution, then divides into a pixel attention branch and a main branch. The pixel attention branch uses two 5×5 convolutions with 3×3 dilated kernels to extract shallow image features. The large kernel convolution allows for a larger receptive field in the pixel attention branch. The pixel attention branch and the main branch are connected via multiplication. The connected attention branch integrates features through a 3×3 2D convolution and then uses residual connection to input features into the module. To alleviate the training instability caused by the large-kernel pixel attention, a pixel normalization layer is added at the end of the module, such as... Figure 3 As shown;

[0037] (2) Image Reconstruction Module. In order to perform super-resolution reconstruction of the feature image, a reconstruction module is constructed, which mainly includes image upsampling layers and bicubic interpolation method.

[0038] Step S3: Constructing a deep feature extraction module based on the transformer self-attention mechanism mainly includes several progressively forward-propagating ETL modules. These modules include a transformer layer and an MLP layer, connected by skip connections. The transformer layer contains the transformer self-attention mechanism, which explores the similarity between each pixel and other pixels by querying the key matrix Q, the key matrix K, and the value matrix V. To reduce the number of parameters in the transformer layer, this method divides the image into different pixel blocks and employs a stripe shift window mechanism to explore the autocorrelation between different image blocks from both the H and W directions. The MLP layer integrates the transformed features through a fully connected perceptron layer, such as... Figure 4 As shown.

[0039] Step S4: Construct an ECA channel attention mechanism module to connect the shallow attention mechanism module and the deep attention mechanism module, simultaneously exploring multi-channel information. While the shallow and deep feature extraction modules focus on feature extraction between the current channels, this method captures features across multiple channels by constructing a channel attention module, placing greater emphasis on channels containing more information.

[0040] Step S5: Construct a context feature fusion module to fuse multi-band information. Hyperspectral images contain dozens or hundreds of bands, and the information between each band can complement each other. This method adds a context feature fusion module to explore the spectral information of hyperspectral images, taking advantage of the characteristics of hyperspectral imaging. It fuses multi-band information band by band using a progressive method between upper and lower bands, such as... Figure 5 As shown.

[0041] Step S6: Train the model built in steps S1 to S5 on the training set. The complete model structure is as follows: Figure 2 As shown. During training, we feed the low-resolution hyperspectral image (64×64 pixels) cropped in step S1 into the multi-attention mechanism hyperspectral image super-resolution network for training. The learning rate during training is 10. -4 The ADAM optimizer (β1 = 0.9, β2 = 0.999) was trained for a total of 100 epochs. During testing, several 64×64 images were input. The images first underwent shallow feature extraction, followed by channel attention to explore inter-channel relationships. Deep features were then extracted using a transformer self-attention module, and contextual feature fusion was used to explore inter-band feature information. Finally, a reconstruction module obtained a high-resolution hyperspectral image. The PSNR, SSIM, and SAM values ​​of the super-resolution hyperspectral image and the original image were calculated to assess the model's performance. Finally, compared with other state-of-the-art methods, the hyperspectral image super-resolution reconstruction method based on a multi-attention mechanism presented in this invention exhibits excellent performance.

[0042] The above description is merely a specific embodiment of the present invention, and the scope of protection of the present invention is not limited thereto. Any transformations or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for super-resolution reconstruction of hyperspectral images based on a multiple attention mechanism, characterized in that, Includes the following steps: S1: Collect hyperspectral image data, and then expand the dataset through data augmentation methods such as rotation, flipping, and cropping to obtain the training set and test set; S2: Construct a shallow feature extraction module and an image reconstruction module based on pixel attention; including the following steps: (1) Shallow feature attention module; The shallow attention module consists of multiple transitive large kernel pixel attention modules. The large kernel pixel attention module first increases the channel dimension to 64 through a 2D convolution with a 1×1 kernel, and then divides into a pixel attention branch and a main branch. The pixel attention branch uses two 5×5 convolution kernels and a 3×3 dilated convolution to extract shallow features of the image. The large kernel convolution can obtain a larger receptive field in the pixel attention branch. The pixel attention branch and the main branch are connected by multiplication. The connected attention branch integrates features through a 2D convolution with a 3×3 kernel and then inputs features into the residual connection module. A pixel normalization layer is added at the end of the module. (2) Image reconstruction module; including an image upsampling layer, which uses bicubic interpolation for upsampling; S3: Construct a deep feature extraction module with the transformer self-attention mechanism as its core; including several progressive forward propagation ETL modules, including transformer layers and MLP layers, which are connected by skip connections; the transformer layer contains the transformer self-attention mechanism, which explores the similarity between each pixel and other pixels by querying the query matrix Q, key matrix K and value matrix V, dividing the image into different pixel blocks, and at the same time using a stripe shift window mechanism to explore the autocorrelation between different image blocks from the H and W directions respectively; the MLP layer integrates the transformed features through a fully connected perceptual layer; S4: Employ the ECA channel attention mechanism module to connect the shallow attention mechanism module and the deep attention mechanism module to explore multi-channel information simultaneously; S5: Multi-band information is fused using a context feature fusion module; S6: Fine-tune the parameters of the hyperspectral image super-resolution reconstruction model using the training set, and test the hyperspectral image super-resolution reconstruction model using the test set.

2. The hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism according to claim 1, characterized in that, The hyperspectral image data in step S1 includes the Cave indoor hyperspectral image dataset, Pavia airborne remote sensing imagery, and Chikusei airborne remote sensing imagery. The data augmentation methods include image cropping and image flipping, as well as image rotation at angles of 90°, 120°, and 270°.

3. The hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism according to claim 1, characterized in that, The step S3 described above constructs a deep feature extraction module based on the transformer self-attention mechanism, which includes several progressively forward-propagating ETL modules. These modules include transformer layers and MLP layers, which are connected by skip connections. The transformer layer contains the transformer self-attention mechanism, which explores the similarity between each pixel and other pixels by querying the query matrix Q, the key matrix K, and the value matrix V, dividing the image into different pixel blocks. At the same time, it adopts a stripe shift window mechanism to explore the autocorrelation between different image blocks from both the H and W directions. The MLP layer integrates the transformed features through a fully connected perceptual layer.

4. The hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism according to claim 1, characterized in that, In step S4, the ECA channel attention mechanism module connects the shallow attention mechanism module and the deep attention mechanism module to explore multi-channel information simultaneously, while the shallow feature extraction module and the deep feature extraction module focus on feature extraction between the current channels.

5. The hyperspectral image super-resolution reconstruction method based on a multiple attention mechanism according to claim 1, characterized in that, In step S5, a context feature fusion module is used to fuse multi-band information. The information between each band can complement each other. In view of the characteristics of hyperspectral images, a context feature fusion module is added to explore the spectral information of hyperspectral images. It fuses multi-band information band by band through a progressive method between upper and lower bands.

Images