Hyperspectral super-resolution reconstruction method based on second-order spectral space attention

Abstract

The invention relates to the technical field of image restoration, in particular to a hyperspectral super-resolution reconstruction method based on second-order spectral space attention. The method comprises the steps that Swin Transform is combined with a convolutional neural network, and global-local context space spectral information is captured; constructing a space-spectrum attention mechanism of second-order covariance statistics, and obtaining more context features; performing up-sampling by using a deconvolution operation, and reconstructing a final hyperspectral super-resolution image; obtaining a trained hyperspectral super-resolution reconstruction model by using the training data set; and inputting the current test sample set into the trained model to obtain a hyperspectral super-resolution image. The method has an excellent reconstruction result, and the hyperspectral super-resolution reconstruction performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a high-spectral super-resolution reconstruction method based on second-order spectral spatial attention. Background Art

[0002] Hyperspectral images (HSI) generally continuously collect spectral information of dozens or even hundreds of bands in the electromagnetic spectrum, with high spectral resolution. Different spectral features can be used to reflect the properties of different materials, which makes hyperspectral images widely used in material observation and analysis tasks. However, in real scenes, hyperspectral images are limited by the imaging mechanism, and the spatial resolution of the collected hyperspectral images is usually not high, which in turn restricts the application of hyperspectral images in other fields. Therefore, it is of great significance to study the efficient, stable and accurate reconstruction algorithm of hyperspectral images.

[0003] Image super-resolution reconstruction aims to use one or more low-resolution images to reconstruct a high-resolution image with rich detail information through software. In the early days, traditional models were usually used for super-resolution reconstruction of single hyperspectral images, but this would cause information loss, accuracy limitations, and high requirements for input data.

[0004] In recent years, with the continuous development of deep learning technology, researchers have begun to try to apply neural networks to image super-resolution tasks, achieving better results than traditional methods. However, due to the large differences between HSI spectral bands and high dimensionality, there are few training samples and the difficulty of training, applying neural networks to image super-resolution tasks is still a challenging hot issue. Summary of the invention

[0005] In order to solve the technical problem of low performance of hyperspectral super-resolution, the purpose of the present invention is to provide a hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention. The technical solution adopted is as follows:

[0006] Combined with the convolutional network, a shallow feature extraction module is constructed to preliminarily extract the shallow spatial spectral feature information in the original image;

[0007] A deep feature extraction module (RSTAB) is constructed using Swin Transformer and second-order spectral spatial attention to aggregate feature information at different levels;

[0008] A feature aggregation and reconstruction module is constructed using three convolutional layers and second-order spatial spectral attention to reconstruct the final hyperspectral super-resolution image;

[0009] Using the training data set to obtain a trained hyperspectral super-resolution reconstruction model;

[0010] The current test sample set is input into the trained hyperspectral super-resolution reconstruction model to obtain a hyperspectral super-resolution image.

[0011] Furthermore, the shallow feature extraction module constructed by combining the convolutional network includes:

[0012] F0=H shallow (y); where H shallow (·) indicates that the convolution structure uses a 3×3 convolution kernel to extract shallow spatial-spectral features, y represents the input original image, and F0 represents the processing result of the shallow feature extraction module.

[0013] Furthermore, the deep feature extraction module includes:

[0014] Swin Transformer is combined with Convolutional Neural Network (Conv) to capture global-local contextual spatial-spectral information; specifically: residual Swin Transformer attention module (STB), convolutional layers, residual connections, second-order spatial-spectral attention module (SOAB) and Leaky ReLU activation function;

[0015] For the residual Swin Transformer attention module, the Swin Transformer layer can be regarded as a specific instance of spatially varying convolution;

[0016] For the convolutional layer, a three-layer convolutional layer of 1×1, 3×3, and 1×1 is used;

[0017] Residual connections provide short identity mapping connections to aggregate feature information at different levels;

[0018] For the activation function, the Leaky Relu activation function is used to enhance the linear expression ability of the model;

[0019] The second-order spatial-spectral attention module highlights the high-frequency information of the features and maintains the integrity of the spectrum; for the second-order spatial-spectral attention module, the second-order spatial-spectral attention module combines the spectral attention of the second-order covariance statistics and the spatial attention of the second-order covariance statistics to capture the overall feature paired spectral information and spatial position correlation.

[0020] Furthermore, the residual Swin Transformer attention module (STB), convolutional layer, residual connection, second-order spatial-spectral attention module (SOAB) and Leaky ReLU activation function include:

[0021] For the Swin Transformer attention module, conventional window partitioning and moving window partitioning are used alternately to make the windows interact with each other;

[0022] For the convolution layer, a three-layer convolution layer of 1×1, 3×3, and 1×1 is used to further extract feature information;

[0023] Residual connections provide short identity mapping connections to aggregate feature information at different levels;

[0024] For the second-order spectral-spatial attention module, the spectral attention module of the second-order covariance statistics is combined with the spectral attention module of the second-order covariance statistics to construct the spatial-spectral attention mechanism of the second-order covariance statistics to obtain more contextual features.

[0025] Furthermore, the calculation formula of the conventional window partition is:

[0026] X1=FFN(LN(WMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, WMSA represents the conventional window partition operation, FFN represents the feedforward layer, and X1 represents the result after conventional window processing.

[0027] Furthermore, the calculation formula of the moving window partition is:

[0028] X2=FFN(LN(SWMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, SWMSA represents the moving window partition operation, FFN represents the feedforward layer, and X2 represents the result after moving window processing.

[0029] Furthermore, the second-order spatial-spectral attention module comprises:

[0030] The spectral attention module of the second-order covariance statistics combines the squeezing and dilation algorithms to generate spectral descriptors and capture spectral correlations, respectively;

[0031] The spatial attention module of the second-order covariance statistic combines average pooling, depthwise separable convolution and anti-average pooling.

[0032] Furthermore, the feature aggregation and reconstruction module includes:

[0033] The convolutional layer consists of three convolutional layers: 1×1, 3×3, and 1×1;

[0034] Deconvolution performs upsampling operations.

[0035] Furthermore, deconvolution includes:

[0036] I SR=H recontruction (G); where G is the fusion spatial spectrum feature, H recontruction (·) represents the deconvolution operation, which is the inverse operation of convolution. SR Represents the final result of hyperspectral image reconstruction.

[0037] Furthermore, various training parameters of the prediction model are set, and the various training parameters include neuron activation function, fully connected layer activation function, optimization function, loss function, number of network iterations, batch size, and learning rate.

[0038] The embodiments of the present invention have at least the following beneficial effects:

[0039] The present invention provides a hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention. Firstly, in order to extract shallow spatial-spectral features containing rich low-frequency information, a 3×3 convolution kernel is used in combination with a convolutional layer; in order to improve feature utilization and make the network pay more attention to high-frequency information and maintain spectral consistency, a deep feature extraction module is constructed by combining Swin Transformer with second-order spectral spatial attention; three convolutional layers and second-order spatial spectral attention are used to construct a feature aggregation and reconstruction module to reconstruct the final hyperspectral super-resolution image; three public data set training samples are used to obtain a trained prediction model; the trained model is superior to traditional models and other deep learning models on the test samples of the three data sets, which is beneficial to improving the hyperspectral super-resolution reconstruction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A method flow chart of a high-spectral super-resolution reconstruction method based on second-order spectral spatial attention provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the STSOSR model structure provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a Swin Transformer network structure diagram provided by an embodiment of the present invention;

[0044] Figure 4A schematic diagram of a residual Swin Transformer attention module (RSTAB) provided by one embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a covariance spectral-spatial attention module (SOAB) provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, characteristics and effects of a high-spectral super-resolution reconstruction method based on second-order spectral spatial attention proposed by the present invention are described in detail as follows in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The following is a detailed description of a high-spectral super-resolution reconstruction method based on second-order spectral spatial attention provided by the present invention in conjunction with the accompanying drawings.

[0049] See also Figure 1 , which shows a flowchart of a method for high-spectral super-resolution reconstruction based on second-order spectral spatial attention provided by an embodiment of the present invention, the method comprising the following steps:

[0050] Step S100, constructing a shallow feature extraction module in combination with a convolutional network to preliminarily extract shallow spatial spectral feature information in the original image.

[0051] First, low-level features are very important for hyperspectral image reconstruction. A convolution layer with a convolution kernel of 3×3 is used to extract shallow spatial-spectral features that contain rich low-frequency information. Convolution is better at initial image visual processing and is conducive to network stability and optimization.

[0052] As a preferred embodiment of the present invention, the expression of the shallow feature extraction algorithm can be:

[0053] F0=H shallow (y); where y represents the original input image, H shallow (·) indicates that the convolution structure uses a 3×3 convolution kernel to extract shallow spatial-spectral features, and F0 represents the processing result of the shallow feature extraction module.

[0054] In step S200, a deep feature extraction module (RSTAB) is constructed using Swin Transformer and second-order spectral spatial attention to aggregate feature information at different levels.

[0055] In an embodiment of the present invention, in order to obtain multi-scale extracted feature information, make the network pay more attention to high-frequency information and maintain spectral consistency, Swin Transformer is combined with attention to help the model understand and reconstruct the complex structure and content of hyperspectral images, and use residual connections to obtain the dependencies between long-range pixels; in order to enhance the discriminative representation and statistical modeling capabilities, the spectral attention of the second-order covariance statistic and the spectral attention of the second-order covariance statistic are combined to enhance the nonlinear modeling capabilities of the model. Among them, the deep feature extraction module includes: Swin Transformer combined with a convolutional neural network (Conv) to capture global-local contextual spatial spectral information; specifically: residual Swin Transformer attention module (STB), convolutional layer, residual connection, second-order spatial-spectral attention module (SOAB) and Leaky ReLU activation function. Please refer to Figure 3 , Figure 3 This is a schematic diagram of the Swin Transformer network structure; please refer to Figure 4 , Figure 4 Schematic diagram of the residual Swin Transformer attention module (RSTAB), Figure 4 In the figure, Feed Forward is the feedforward layer; Layer Norm is the layer normalization, W-MSA is the conventional window partition operation, SW-MSA is the moving window partition operation, and Input Embedding is the input of the encoder.

[0056] In some embodiments of the present invention, a deep feature extraction module is constructed by using Swin Transformer and second-order spectral spatial attention to aggregate feature information at different levels, which can be achieved by following the steps S21-S22:

[0057] S21 uses residual Swin Transformer attention to help the model improve feature extraction, multi-scale information integration and long-range dependency modeling capabilities.

[0058] In an embodiment of the present invention, a hyperspectral image contains a large amount of spatial and spectral information, and SwinTransformer can accurately capture this information to achieve precise super-resolution reconstruction; the attention mechanism can help the model focus on the features of the image including spatial and spectral information at different scales; the residual connection captures the dependencies between long-range pixels, improves the accuracy and coherence of super-resolution image reconstruction, and in order to highlight the high-frequency information of the features and maintain the integrity of the spectrum, a spatial-spectral attention module of the second-order covariance statistic is used.

[0059] In some embodiments of the present invention, residual Swin Transformer attention is used to help the model improve feature extraction, multi-scale information integration and long-range dependency modeling capabilities, which can be achieved by following steps S211-S214:

[0060] In step S211, for the residual Swin Transformer attention module, the Swin Transformer layer is used, which can be regarded as a specific instance of spatially varying convolution.

[0061] Step S212, for the convolution layer, a three-layer convolution layer of 1×1, 3×3, and 1×1 is used.

[0062] The convolutional layer enables the network to learn features more effectively and enhances the translation invariance of the model.

[0063] Step S213: The residual connection provides a short identity mapping connection to aggregate feature information at different levels.

[0064] Step S214: For the activation function, a Leaky Relu activation function is used to enhance the linear expression capability of the model.

[0065] As an embodiment of the present invention, the calculation formula of the conventional window partition is:

[0066] X1=FFN(LN(WMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, WMSA represents the conventional window partition operation, FFN represents the feedforward layer, and X1 represents the result after conventional window processing.

[0067] As an embodiment of the present invention, the calculation formula of the moving window partition includes:

[0068] X2=FFN(LN(SWMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, SWMSA represents the moving window partition operation, FFN represents the feedforward layer, and X2 represents the result after moving window processing.

[0069] Step S22, for the second-order spatial-spectral attention module, the high-frequency information of the feature is highlighted and the integrity of the spectrum is maintained; for the second-order spatial-spectral attention module, the second-order spatial-spectral attention module combines the spectral attention module of the second-order covariance statistic and the spatial attention module of the second-order covariance statistic to capture the overall feature paired spectral information and spatial position correlation. Among them, the second-order spatial-spectral attention module is also the spatial-spectral attention module of the second-order covariance statistic, that is, the second-order covariance spectral-spatial attention module (SOAM). Please refer to Figure 5 , Figure 5 Schematic diagram of the covariance spectral-spatial attention module (SOAM), Figure 5 In it, Batch Norm is batch normalization, AvgPool is average pooling, Depth Conv is depth-separable convolution, UnAvgPool is reverse average pooling, R-Conv is squeezing, E-Conv is expansion, Conv is convolutional neural network, and Sigmoid is the fully connected layer activation function.

[0070] In some embodiments of the present invention, step S22 may be implemented by following steps S221-S222:

[0071] Step S221, the spectral attention module of the second-order covariance statistic combines the R-Vonv and E-Vonv algorithms to generate spectral descriptors and capture spectral correlations respectively.

[0072] In the embodiment of the present invention, the second-order covariance statistic reveals the relationship between spectral bands, and has strong fitting ability and strong discriminant characterization. The squeeze algorithm performs a deep separable convolution (DepthConv) operation on the spectral covariance matrix to generate a spectral descriptor. The second-order spatial-spectral attention module highlights the high-frequency information of the feature and maintains the integrity of the spectrum.

[0073] In some embodiments of the present invention, the expression of the squeezing algorithm may be:

[0074] d spctral =DepthConv(cov spctral ), where d spctral represents each spectral band descriptor, DepthConv(·) represents the deep convolution operation, which can reduce the amount of model calculation, cov spctral Next, a gating mechanism is applied in the excitation stage, including two convolutional layers with ReLU activation function between layers, to capture the dependency of the bands and generate the final spectral weight matrix w spctral represents the spectral weight matrix, where c represents the number of channels.

[0075] As an embodiment of the present invention, the expression of the expansion algorithm may be:

[0076] w spctral =Conv E (δ(Conv R (d spctral ))); where δ represents the ReLU activation function, Conv R Represents the dimension reduction layer, Conv E Represents a dimensionality increase layer, and the dimensionality reduction factor is usually set to 16.

[0077] In step S222, the spatial attention module of the second-order covariance statistic combines batch normalization (Batch Norm), average pooling (AvgPool), depth-separable convolution (Depth Conv) and unaverage pooling (UnAvgPool).

[0078] In order to distinguish valuable high-frequency spatial details from a large amount of low-frequency information, the spatial attention module of the second-order covariance statistics is used to enhance the model's spatial reconstruction ability.

[0079] The spatial attention module of the second-order covariance statistic captures the spatial position correlation of the overall feature pair and performs feature scaling on the position. The average pooling operation is used for downsampling to reduce the spatial dimension to a fixed size to reduce the computational cost. The covariance matrix performs inter-row convolution operations. Since each band has its own feature details, a filter size of h is used between the convolution layers. ′ w ′ ×1 depth convolution, where h ′ and w ′ Represents the reduced spatial height and width, and can adaptively find the appropriate spatial descriptor for each feature in each band. Each row of the covariance matrix represents the statistical correlation between a specific band and all spectral bands. The deep convolution operation encodes the nonlinear dual dependency between features at all positions in each band, which can reduce the amount of model calculation.

[0080] As an embodiment of the present invention, the expression of the average pooling algorithm is:

[0081] cov spatial =COV(AvgPool(F)), where F represents the extraction of feature information, AvgPool(·) represents the average pooling downsampling operation, and COV(·) represents the calculation of the pairwise spatial position covariance. spatial represents the covariance matrix.

[0082] Each row of the covariance represents the statistical association of each feature with all features.

[0083] As an embodiment of the present invention, the expression of the depthwise separable convolution algorithm is:

[0084] d ′ spatial =DepthConv(cov spctral ), where DepthConv(·) represents the deep convolution operation. The weight matrix is ​​upsampled to h×w to generate the final spatial attention weight matrix It encodes the nonlinear dual dependencies between features at all positions in each band.

[0085] As an embodiment of the present invention, the expression of the inverse average pooling algorithm is:

[0086] w spatial =UnAvgPool(d ′ spatial ), where UnAvgPool(·) represents the unpooling operation, d ′ spatial Represents each spatial descriptor, w spatial represents the spatial weight matrix.

[0087] Step S300, constructing a feature aggregation reconstruction module using three convolutional layers and second-order spatial spectral attention to reconstruct the final hyperspectral super-resolution image.

[0088] In order to reconstruct the final hyperspectral super-resolution image, a combination of multi-layer convolutional layers and second-order spatial-spectral attention is used to adaptively fuse and correct the extracted deep features in the spatial and spectral dimensions to generate the final deep spatial-spectral feature information. The convolutional layer consists of three convolutional layers of 1×1, 3×3, and 1×1.

[0089] As an embodiment of the present invention, the expression of the feature aggregation reconstruction algorithm is:

[0090] I SR =H recontruction (H fuse (F)+F0); where F represents the extracted feature information, H fuse (·) represents convolution and attention operations, as well as long residual connection addition operations, F0 represents shallow spatial-spectral feature information, H recontruction (·) represents the deconvolution operation, I SR Represents the final result of hyperspectral image reconstruction.

[0091] A reconstruction model for hyperspectral super-resolution was obtained.

[0092] Step S400: obtaining a trained hyperspectral super-resolution reconstruction model using a training data set.

[0093] In this embodiment, various training parameters of the prediction model are set during the training process. The various training parameters include a neuron activation function, a fully connected layer activation function, an optimization function, a loss function, a number of network iterations, a batch size, and a learning rate. The configuration of each training parameter is shown in Table 1:

[0094]

[0095]

[0096] Table 1

[0097] In Table 1, (1) the neuron activation function is Leaky Relu, and the fully connected layer activation function is Sigmoid; (2) the network training optimization function is Adam, and the loss function is BCE function; (3) the number of network iterations epochs = 75, and 16 samples are used in each batch to update the weights, that is, batch_size is 16, and the learning rate is le-4.

[0098] It should be noted that the training process of the prediction model is an existing technology and will not be elaborated in detail here.

[0099] At this point, a trained reconstruction model is obtained.

[0100] Step S500: input the current test sample set into the trained hyperspectral super-resolution reconstruction model to obtain a hyperspectral super-resolution image.

[0101] In one embodiment of the present invention, the test samples of the three data sets in the current period are input into the trained reconstruction model to reconstruct the super-resolution of the test image and obtain a hyperspectral super-resolution image. In other embodiments, the test sample of the data set in the current period can be set to 1, and the implementer can set the test samples of different data sets according to the actual situation, which is not specifically limited here.

[0102] Thus, the embodiment of the present invention achieves more accurate hyperspectral super-resolution reconstruction.

[0103] In order to verify the implementation effect of the embodiment of the present invention and better illustrate the advantages of the embodiment of the present invention, the following experiment is disclosed.

[0104] 1. Determine the evaluation indicators.

[0105] The embodiment of the present invention uses three objective evaluation indicators, namely, spectral mapping (SAM), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) super-resolution reconstruction, to quantitatively analyze the reconstructed images of each model, which evaluate the similarity between the reconstructed image and the real HSI from the pixel, spatial and spectral dimensions respectively. Higher PSNR and SSIM values ​​mean better reconstruction effect and higher spatial similarity, and a smaller SAM value means less spectral distortion.

[0106] 2. Prepare the dataset.

[0107] In the embodiments of the present invention, the data sets are derived from three public hyperspectral data sets.

[0108] (1) The Pavia Centre dataset was taken over the Pavia Centre region in northern Italy in 2001. It contains 102 spectral bands and the size of the meaningful area scene is 1096×715 pixels. In order to evaluate the proposed second-order spectral spatial attention method (STSOSR) based on Swin Transformer, the central area of ​​the image is cropped to obtain a regional image of 1095×715×102 pixels. Four non-overlapping HSI blocks of 256×256 pixels are cropped from the left area of ​​the image as test data, and then overlapping blocks are randomly cropped from the remaining area as training data sets. The cropped image blocks are generated accordingly using bicubic downsampling and Gaussian blur operations.

[0109] (2) The CAVE dataset contains 32 HSIs captured under illumination conditions, with a spatial size of 512×512, 31 bands, and a spectral range from 400nm to 700nm. The embodiment of the present invention randomly selects 20 images as the training set (10% of the samples are randomly selected for the validation set), and the remaining 11 images are used for the test set. The cropped image blocks are processed in the same way as the Pavia Centre dataset, and the corresponding datasets are generated using bicubic downsampling and Gaussian blur operations.

[0110] 3. Set up the experiment.

[0111] In the embodiment of the present invention, the experiment was trained on Python 3.7.11 and Pytorch 1.10.0 and Windows operating system, and was equipped with an NVIDIA GPU GeForce GTX 3080 graphics card.

[0112] 4. Comparative experiment.

[0113] In the embodiment of the present invention, the traditional model and the deep learning model are used as comparative experiments. When setting the experiment, bicubic interpolation (Bicubic) is the traditional model, and three-dimensional convolutional neural network (3DFCNN), image denoising recurrent neural network (GDRNN), random forest super-resolution (RFSR) and adaptive sparse representation super-resolution (SSPSR) are used as comparative experiments. Among them, Bicubic means calculating new pixel values ​​through known values ​​around pixels, 3DFCNN means using the three-dimensional convolution structure in deep learning, GDRNN means using the recurrent neural network structure, RFSR means using the random forest model for high-spectral super-resolution reconstruction, and SSPSR means using the super-resolution reconstruction idea of ​​sparse representation theory.

[0114] On the Pavia dataset, the comparative experimental model evaluation results are shown in Table 2:

[0115]

[0116]

[0117] Table 2

[0118] In Table 2, (1) the average performance of the three evaluation indicators at reconstruction coefficients of ×4 and ×8. It is observed that the proposed STSOSR method is significantly better than other algorithms in all objective evaluation indicators; (2) the PSNR value of the STSOSR method is 0.16dB higher than that of the SSPSR (Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery) method at reconstruction coefficients of ×4 and 0.05dB higher at reconstruction coefficients of ×8; (3) Similarly, the SAM evaluation indicator value is also significantly lower than that of the SSPSR method at ×4 and ×8 upsampling coefficients by 0.17 and 0.37 respectively. Please refer to Figure 2 , Figure 2 This is a schematic diagram of the STSOSR model structure. Figure 2 Here y represents the original input image, I represents the final hyperspectral image reconstruction result, Conv represents convolutional neural network, SOAB represents the second-order spatial-spectral attention module, and RSTAB represents the residual Swin Transformer attention module.

[0119] On the Pavia dataset reconstruction coefficient ×4, the evaluation results of the specified test image comparison experimental model are shown in Table 3:

[0120]

[0121] Table 3

[0122] On the Pavia dataset reconstruction coefficient ×8, the evaluation results of the specified test image comparison experimental model are shown in Table 4:

[0123]

[0124] Table 4

[0125] From the quantitative comparison results of the comparison methods on the reconstruction scales of ×4 and ×8 on two randomly selected test images in Tables 3 and 4, it can be seen that the proposed STSOSR method has better results than other methods. The method provided by the embodiment of the present invention improves the PSNR value of ×4 by 0.16dB and 0.13dB for the specified two test images, which is higher than that of the SSPSR method. The SAM values ​​can achieve quite satisfactory results.

[0126] On the CAVE dataset, the comparative experimental model evaluation results are shown in Table 5:

[0127]

[0128]

[0129] Table 5

[0130] In Table 5, (1) all indicators of the STSOSR method are higher than those of other algorithms; (2) the PSNR value of the STSOSR method is improved by 0.5784dB and 1.5440dB respectively compared with the suboptimal algorithm; (3) the SAM index is reduced by 0.1186 and 0.0727 respectively.

[0131] On the CAVE dataset reconstruction coefficient ×4, the evaluation results of the specified test image comparison experimental model are shown in Table 6:

[0132]

[0133] Table 6

[0134] On the CAVE dataset reconstruction coefficient ×8, the evaluation results of the specified test image comparison experimental model are shown in Table 7:

[0135]

[0136]

[0137] Table 7

[0138] In Tables 6 and 7, STSOSR provides the objective evaluation index results of the proposed STSOSR method on two randomly selected test images at reconstruction scales ×4 and ×8 coefficients, and it can be seen that the results are better than all methods.

[0139] On the Chikusei dataset, the comparative experimental model evaluation results are shown in Table 8:

[0140]

[0141]

[0142] Table 8

[0143] In Table 8, (1) the average objective evaluation index values ​​of STSOSR reconstruction results at reconstruction coefficients ×4 and ×8 are higher than those of the comparison methods; (2) the PSNR value of the STSOSR method is improved by 0.3873dB and 0.088dB at ×4 and ×8 respectively compared with the second best result; (3) SSIM is improved by 0.0107 and 0.0014 respectively; (4) SAM is reduced by 0.0968 and 0.1018 respectively.

[0144] On the Chikusei dataset reconstruction coefficient ×4, the evaluation results of the specified test image comparison experimental model are shown in Table 9:

[0145]

[0146] Table 9

[0147] On the Chikusei dataset reconstruction coefficient ×8, the evaluation results of the specified test image comparison experimental model are shown in Table 10:

[0148]

[0149]

[0150] Table 10

[0151] In Tables 9 and 10, the quantitative comparison results of STSOSR and various comparison methods are made on randomly selected test images to reconstruct ×4 and ×8. The STSOSR method can achieve quite satisfactory results in PSNR, SSIM and SAM, which proves the superiority of the STSOSR algorithm.

[0152] Therefore, it can be concluded from Tables 8, 9 and 10 that the STSOSR method can better restore image information and reconstruct with stronger spectral consistency.

[0153] 5. Ablation experiment.

[0154] It should be noted that in order to explore the contribution of each component of the STSOSR model to the hyperspectral super-resolution reconstruction performance, an ablation experiment was carried out. The STSOSR model is mainly composed of the spectral and spatial attention of the second-order covariance statistics. These components will be gradually removed to evaluate their impact on the model performance.

[0155] (1) Without attention (w / o CA / SA): The spectral spatial attention module with the second-order covariance statistic removed is built on the residual Swin Transformer attention framework.

[0156] (2) Only spectral attention (w / o CA): The spatial attention module with the second-order covariance statistic removed is added to the residual Swin Transformer attention framework.

[0157] (3) Only spatial attention (w / o SA): The spectral attention module with the second-order covariance statistics removed is built on the residual Swin Transformer attention framework.

[0158] (4) Complete STSOSR model (w): contains spectral attention of second-order covariance statistics and spatial attention of second-order covariance statistics.

[0159] The evaluation results of the ablation experiment model are shown in Table 11:

[0160]

[0161] Table 11

[0162] It can be seen from Table 11 that: (1) the STSOSR method without attention shows a significant decline in hyperspectral reconstruction performance, indicating the importance of the attention module for image reconstruction; (2) the STSOSR method that only contains the second-order covariance spectral attention has improved performance in hyperspectral super-resolution reconstruction, but the reconstruction effect is poor; (3) the STSOSR method that only contains the second-order covariance spatial attention has improved the accuracy of hyperspectral super-resolution reconstruction, but the effect is not good; (4) the SOSTSR method that adds the second-order covariance spectral attention and spatial attention has achieved the best performance, which proves that adding spatial attention on the basis of spectral attention is more conducive to the representation of the correlation between spectral features and spatial positions, and verifies the importance of the second-order covariance statistic spectral spatial attention module in generalization ability.

[0163] In summary, firstly, the Swin Transformer is used as a framework, and residual connections are used to fuse the original features with the deep features; then, the spectral spatial attention module of the second-order covariance statistic is introduced to construct a deep network module, and the feature aggregation reconstruction module is constructed by deconvolution; finally, the STSOSR model is trained, and the well-trained model is selected for reconstruction to obtain the reconstruction result. The present invention obtains a more accurate reconstruction model by improving the Swin Transformer model, and improves the accuracy and reliability of hyperspectral super-resolution based on the reconstruction model.

[0164] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0165] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A high-spectral super-resolution reconstruction method based on second-order spectral spatial attention, characterized in that: The method comprises the following steps: Combined with the convolutional network, a shallow feature extraction module is constructed to preliminarily extract the shallow spatial spectral feature information in the original image; A deep feature extraction module (RSTAB) is constructed using Swin Transformer and second-order spectral spatial attention to aggregate feature information at different levels; A feature aggregation and reconstruction module is constructed using three convolutional layers and second-order spatial spectral attention to reconstruct the final hyperspectral super-resolution image; Using the training data set to obtain a trained hyperspectral super-resolution reconstruction model; The current test sample set is input into the trained hyperspectral super-resolution reconstruction model to obtain a hyperspectral super-resolution image.

2. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 1 is characterized in that: The shallow feature extraction module constructed by combining the convolutional network includes: F0=H shallow (y); where H shallow (·) indicates that the convolution structure uses a 3×3 convolution kernel to extract shallow spatial-spectral features, y represents the input original image, and F0 represents the processing result of the shallow feature extraction module.

3. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 1 is characterized in that: The deep feature extraction module comprises: Swin Transformer is combined with Convolutional Neural Network (Conv) to capture global-local contextual spatial-spectral information; specifically: residual Swin Transformer attention module (STB), convolutional layers, residual connections, second-order spatial-spectral attention module (SOAB) and Leaky ReLU activation function; For the residual Swin Transformer attention module, the Swin Transformer layer can be regarded as a specific instance of spatially varying convolution; For the convolutional layer, a three-layer convolutional layer of 1×1, 3×3, and 1×1 is used; Residual connections provide short identity mapping connections to aggregate feature information at different levels; For the activation function, the Leaky Relu activation function is used to enhance the linear expression ability of the model; The second-order spatial-spectral attention module highlights the high-frequency information of the features and maintains the integrity of the spectrum; for the second-order spatial-spectral attention module, the second-order spatial-spectral attention module combines the spectral attention of the second-order covariance statistics and the spatial attention of the second-order covariance statistics to capture the overall feature paired spectral information and spatial position correlation.

4. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 3 is characterized in that: The residual Swin Transformer attention module (STB), convolutional layer, residual connection, second-order spatial-spectral attention module (SOAB) and Leaky ReLU activation function include: For the Swin Transformer attention module, conventional window partitioning and moving window partitioning are used alternately to make the windows interact with each other; For the convolution layer, a three-layer convolution layer of 1×1, 3×3, and 1×1 is used to further extract feature information; Residual connections provide short identity mapping connections to aggregate feature information at different levels; For the second-order spectral-spatial attention module, the spectral attention module of the second-order covariance statistics is combined with the spectral attention module of the second-order covariance statistics to construct the spatial-spectral attention mechanism of the second-order covariance statistics to obtain more contextual features.

5. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 4 is characterized in that: The calculation formula of the conventional window partition is: X1=FFN(LN(WMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, WMSA represents the conventional window partition operation, FFN represents the feedforward layer, and X1 represents the result after conventional window processing.

6. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 4 is characterized in that: The calculation formula of the moving window partition is: X2=FFN(LN(SWMSA(LN(X))+X))+X, where X is the local window feature, LN is the layer normalization, SWMSA represents the moving window partition operation, FFN represents the feedforward layer, and X2 represents the result after moving window processing.

7. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 2 is characterized in that: The second-order spatial-spectral attention module comprises: The spectral attention module of the second-order covariance statistics combines the squeezing and dilation algorithms to generate spectral descriptors and capture spectral correlations, respectively; The spatial attention module of the second-order covariance statistic combines average pooling, depthwise separable convolution and anti-average pooling.

8. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 1 is characterized in that: The feature aggregation and reconstruction module includes: The convolutional layer consists of three convolutional layers: 1×1, 3×3, and 1×1; Deconvolution performs upsampling operations.

9. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 8 is characterized in that: Deconvolution, including: I SR =H recontruction (G); where G is the fusion spatial spectrum feature, H recontruction (·) represents the deconvolution operation, which is the inverse operation of convolution. SR Represents the final result of hyperspectral image reconstruction.

10. The hyperspectral super-resolution reconstruction method based on second-order spectral spatial attention according to claim 1, characterized in that: The training parameters of the prediction model are set, including neuron activation function, fully connected layer activation function, optimization function, loss function, number of network iterations, batch size, and learning rate.

Images