Hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perception and attention

Abstract

The present invention discloses a hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perception attention. The hyperspectral image is input at the input end and preprocessed to obtain the total training set and total test set of the hyperspectral image and the multispectral image; the spatial attention extraction module and the spectral attention extraction module are trained with the total training set to extract the spatial attention and spectral attention of the total training set respectively; the full-scale link U-shaped fusion module is trained with the total training set to output a feature map; the network parameters are updated using the Adam optimizer and the network total loss function to obtain a trained network model; the trained network model is tested with the total test set to calculate the peak signal-to-noise ratio, and it is iterated until the best network model is obtained; the total test set is input into the best network model to obtain a hyperspectral image with super-resolution reconstruction. Experimental results show that the super-resolution effect of the present invention reaches a high level.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral image super-resolution reconstruction, and in particular to a hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perceptual attention. Background Art

[0002] The imaging spectrometer of hyperspectral remote sensing technology simultaneously images dozens or hundreds of bands of the ground object, which can obtain the continuous spectral information of the ground object and realize the simultaneous extraction of the spatial information, radiation information and spectral information of the ground object. Therefore, hyperspectral imaging (HSI) has been applied to change detection, target detection and ground object classification. However, due to hardware limitations, although hyperspectral images have high spectral resolution, their spatial resolution is usually low, which greatly affects the practical application effect of hyperspectral images. In this context, how to enhance the spatial resolution of hyperspectral images (HSI) through some auxiliary multispectral image (MSI) information to achieve super-resolution reconstruction of hyperspectral images has become a problem of particular concern.

[0003] In recent years, more and more scholars have studied the fusion tasks of low spatial resolution hyperspectral images (LR-HSI) and high spatial resolution multispectral images (HR-MSI). Among these fusion methods, some deep learning methods based on convolutional neural networks have achieved good results. For example, the spatial spectral reconstruction network (SSR-NET) designed a new loss function to constrain spatial and spectral patterns, achieving good results. However, this LR-HSI and HR-MSI fusion method ignores hierarchical information, the preservation of spectral information in HSI and the preservation of spatial information in MSI, so that in the fusion process, the spectral information of HSI and the spatial information of MSI are difficult to be continuously transmitted to the fused image, and the quality of the fused image needs to be improved. Summary of the invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perceptual attention.

[0005] The technical solution of the present invention is: a hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perceptual attention, which is carried out in the following steps:

[0006] Step 1. Input the hyperspectral image Y∈R P×Q×C , preprocess Y to obtain the total training set and total test set of hyperspectral images and multispectral images, where P×Q is the spatial size of each band image of Y, and C is the number of bands of Y;

[0007] Step 2. Build the best network model

[0008] Step 2.1 Use the total training set to train the spatial spectral attention extraction module, and extract the spatial attention Spa of the total training set respectively. i and spectral attention Spe i , i=1, 2, 3;

[0009] Step 2.2 uses the total training set to train the full-scale linked U-shaped fusion module and output feature maps F1, F2, F3, F2' and F3';

[0010] Step 2.3 Using the Adam optimizer and the network total loss function L t Update network parameters to obtain a trained network model;

[0011] Step 2.4: Test the trained network model with the total test set, calculate the peak signal-to-noise ratio, and iterate until the best network model is obtained.

[0012] Step 3. Input the total test set into the optimal network model to obtain the reconstructed super-resolution hyperspectral image.

[0013] The step 1 is specifically as follows:

[0014] Step 1.1 Normalize the element values ​​in the hyperspectral image Y to 0-255 to obtain the normalized hyperspectral image Y′∈R P×Q×C ;

[0015] Step 1.2: Get the hyperspectral image Y′∈R P×Q×C The edge of the image is clipped to obtain the clipped hyperspectral image Y1∈R P′×Q′×C ; The P′×Q′ is the spatial size of each band image of the cropped hyperspectral image Y1, P′=P+w_edge, Q′=Q+h_edge; the w_edge=P / / 4*4-P, h_edge=Q / / 4*4-Q, where “ / / ” is an integer division operation, and the integer part of the quotient is returned;

[0016] Step 1.3 divides the work area, wherein the work area starting column w_str = (P′-A) / / 2, the work area starting row h_str = (Q′-B) / / 2, the work area ending column w_end = w_str+A, and the work area ending row h_end = h_str+B, wherein A and B are the width and height of the multispectral image to be obtained respectively;

[0017] Step 1.4 Divide the test data

[0018] First, we intercept the hyperspectral image Y1 from the w_str column to the w_end column and the h_str row to the h_end row to obtain the hyperspectral image test_ref∈R A×B×C ;

[0019] Perform Gaussian blur on the hyperspectral image test_ref, and pass in parameters ksize=(5,5), sigmaX=2 to obtain the hyperspectral image test_lr′∈R A×B×C ;

[0020] Downsample the hyperspectral image test_lr′ by 4 times, pass in the parameter dsize=(A / / 4, B / / 4), where “ / / ” is an integer division operation, and returns the integer part of the quotient to obtain the test set hyperspectral image test_lr∈R a×b×C , where a×b is the spatial size of each band image of the hyperspectral image test_lr obtained after downsampling, and a=A / / 4, b=B / / 4;

[0021] Then, c channels are extracted from the hyperspectral image test_ref to obtain the test set multispectral image test_hr∈R A ×B×c , the sampling method is equidistant sampling, where c is the number of channels of the multispectral image to be obtained;

[0022] Step 1.5 Divide the training data

[0023] First, the working area of ​​the hyperspectral image Y1 is assigned to 0, that is, the hyperspectral image y1∈R P′×Q′×C The values ​​of the w_str column to the w_end column and the h_str row to the h_end row are all assigned to 0, and the hyperspectral image train_ref∈R is obtained. P ′×Q′×C ;

[0024] Then, Gaussian blur is performed on the hyperspectral image train_ref, and parameters ksize = (5, 5), sigmaX = 2 are passed in to obtain the hyperspectral image train_lr′∈R P′×Q′×C ;

[0025] The hyperspectral image train_lr′ is downsampled by 4 times, and the parameter dsize=(P′ / / 4, Q′ / / 4) is passed in, where “ / / ” is an integer division operation, and the integer part of the quotient is returned to obtain the training set hyperspectral image train_lr∈R e×f×C , where e×f is the spatial size of each band image of the hyperspectral image train_lr obtained after downsampling, and e=P′ / / 4, f=Q′ / / 4;

[0026] Finally, c channels are extracted from the hyperspectral image train_ref to obtain the training set multispectral image trainhr∈R P′×Q′×c , the sampling method is equidistant sampling, and the P′×Q′ is the spatial size of each band image of the obtained multispectral image train_hr;

[0027] Step 1.6 From the hyperspectral image train_ref∈R P′×Q′×C Randomly cut out a hyperspectral image test_ref∈R A×B×C Hyperspectral images of the same size image_ref∈R A×B×C ;

[0028] Downsample the hyperspectral image image_ref, and take the spatial size multiplier as 1 / 4 to obtain the downsampled hyperspectral image As the training hyperspectral image Lr;

[0029] Finally, from the training set multispectral image train_hr, the multispectral image image_hr∈R corresponding to the hyperspectral image image_ref in the hyperspectral image train_ref is extracted. A×B×c , as the training multispectral image Hr.

[0030] The step 2 is specifically as follows:

[0031] Set the overall neural network iteration number T, the initial optimal peak signal-to-noise ratio psnr_b = 0, set epoch = 1, and start building the optimal network model;

[0032] The step 2.1 trains the spatial attention extraction module to obtain the spatial attention Spa of the total training set i The details are as follows:

[0033] Step 2.1.1.1 Use a set of convolutional layers Conv2_0 to extract features from the multispectral image Hr and obtain the feature map Hr1∈R H×W×C ; Where H×W is the spatial size of each band of Hr, and C is the number of bands of Lr;

[0034] Conv2_0 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0035] Step 2.1.1.2 Build the spatial attention submodule SpaAM. The execution steps are as follows:

[0036] First, call the encapsulated torch.mean function to return the result of averaging the input feature map by channel dimension to obtain the feature map avg_sa;

[0037] Then call the encapsulated torch.max function to return the result of finding the maximum value of the input feature map according to the channel dimension, and get the feature map max_sa;

[0038] Then call the encapsulated torch.cat function to connect the feature map avg_sa and the feature map max_sa according to the channel dimension to obtain the feature map avg_max;

[0039] Finally, a set of convolutional layers Conv2_1 are used to extract features from the feature map avg_max to obtain spatial attention;

[0040] Conv2_1 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 1 convolution kernel of size 7×7, performs a border padding operation of Padding=3, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function Sigmoid as an activation function for the operation;

[0041] Step 2.1.1.3 Use the spatial attention submodule SpaAM to extract features from the feature map Hr1 and obtain the spatial attention Spa1∈R H×W×1 ;

[0042] Step 2.1.1.4 Use discrete wavelet transform DWT to downsample the feature map Hr1 to obtain the feature map

[0043] Step 2.1.1.5 Use a set of convolutional layers Conv22 to transform the feature map Hr 1D Perform feature extraction and obtain feature map

[0044] Conv2_2 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 2C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0045] Step 2.1.1.6 Use the spatial attention submodule SpaAM to extract features from the feature map Hr2 and obtain the spatial attention

[0046] Step 2.1.1.7 Use discrete wavelet transform DWT to downsample the feature map Hr2 to obtain the feature map

[0047] Step 2.1.1.8 uses a set of convolutional layers Conv23 to transform the feature map Hr 2D Perform feature extraction and obtain feature map

[0048] Conv2_3 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0049] Step 2.1.1.9 Use the spatial attention submodule SpaAM to extract features from the feature map Hr3 and obtain the spatial attention

[0050] The step 2.1 trains the spectral attention extraction module to obtain the spectral attention Spe of the total training set i , the specific steps are as follows:

[0051] Step 2.1.2.1 Perform a 4-fold upsampling operation on the hyperspectral image Lr to obtain the feature map Lr1∈R H×W×C ;

[0052] Step 2.1.2.2 Build the spectral attention submodule SpeAM. The execution steps are as follows:

[0053] First, call the encapsulated adaptive average pooling method torch.nn.AdaptiveAvgPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum to obtain the feature map avg_pool∈R 1×1×C ;

[0054] Use a set of convolutional layers Conv2_4 to extract features from the feature map avg_pool to obtain the feature map avg_mid;

[0055] Conv2_4 includes 1 convolution operation and 1 activation operation, where the convolution layer contains in_planes / / 16 convolution kernels of size 1×1, each convolution kernel performs convolution operation with a step size of 1 pixel, and uses the nonlinear activation function ReLU as the activation function for operation;

[0056] Use a set of convolutional layers Conv2_5 to extract features from the feature map avg_mid and obtain the feature map avg_out∈R 1×1×C ;

[0057] Conv2_5 contains one convolution operation, where the convolution layer contains in_planes convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0058] Then call the encapsulated adaptive maximum pooling method torch.nn.AdaptiveMaxPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum to obtain the feature map max_pool∈R 1×1×C ;

[0059] Use the convolution layer Conv2_4 to extract features from the feature map max_pool to obtain the feature map max_mid;

[0060] Use the convolutional layer Conv2_5 to extract the feature map max_mid and obtain the feature map max_out∈R 1×1×C ;

[0061] Add the feature map avg_out and the feature map max_out to get the feature map out∈R 1×1×C ;

[0062] Finally, the nonlinear activation function Sigmoid is called to activate the feature map out, and the feature map obtained after activation is returned;

[0063] Step 2.1.2.3 Use the spectral attention submodule SpeAM to extract features from the feature map Lr1, pass the parameter in_planes = C to the spectral attention submodule, and obtain the spectral attention Spe1∈R 1×1×C ;

[0064] Step 2.1.2.4 Use discrete wavelet transform DWT to downsample the feature map Lr1 to obtain the feature map

[0065] Step 2.1.2.5 Use the convolutional layer Conv22 to transform the feature map Lr 1D Perform feature extraction and obtain feature map

[0066] Step 2.1.2.6 Use the spectral attention submodule SpeAM to extract features from the feature map Lr2, pass the parameter in_planes = 2C to the spectral attention submodule, and obtain the spectral attention Spe2∈R 1×1×2C ;

[0067] Step 2.1.2.7 Use discrete wavelet transform DWT to downsample the feature map Lr2 to obtain the feature map

[0068] Step 2.1.2.8 uses the convolutional layer Conv2_3 to transform the feature map Lr 2D Perform feature extraction and obtain feature map

[0069] Step 2.1.2.9 Use the spectral attention submodule SpeAM to extract features from the feature map Lr3, pass the parameter in_planes=8C to the spectral attention submodule, and obtain the spectral attention Spe3∈R 1×1×8C ;

[0070] The step 2.2 is specifically as follows:

[0071] Coding process:

[0072] Step 2.2.1.1 Call the encapsulated torch.cat function to connect the feature map Hr1 and the feature map Lr1 according to the channel dimension to obtain the feature map X 1C ;

[0073] Step 2.2.1.2 Use a set of convolutional layers Conv2_6 to transform the feature map X 1C Perform feature extraction and obtain feature map X 1D ∈R H×W×C ;

[0074] The Conv26 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0075] Step 2.2.1.3 Build the spatial-spectral joint perception attention module SSJPA. The execution steps are as follows:

[0076] Definition S i For the feature map to be enhanced, Spa i and Spe i For the incoming spatial attention and spectral attention, the feature map after information enhancement is expressed as:

[0077]

[0078] in represents matrix multiplication, i=1, 2, 3;

[0079] Step 2.2.1.4 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 1D Perform information enhancement, pass in spatial attention Spa1 and spectral attention Spe1, and finally obtain the feature map X1∈R after information enhancement H×W×C ;

[0080] Step 2.2.1.5 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map

[0081] Step 2.2.1.6 Use the convolutional layer Conv2_2 to transform the feature map X 2D Perform feature extraction and obtain feature map

[0082] Step 2.2.1.7 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 2C Perform information enhancement, pass in spectral attention Spe2 and spatial attention Spa2, and finally obtain the feature map after information enhancement

[0083] Step 2.2.1.8 Use discrete wavelet transform DWT to downsample the feature map X2 to obtain the feature map

[0084] Step 2.2.1.9 uses the convolutional layer Conv2_3 to transform the feature map X 3D Perform feature extraction and obtain feature map

[0085] Step 2.2.1.10 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 3C Perform information enhancement, pass in spectral attention Spe3 and spatial attention Spa3, and finally obtain the feature map after information enhancement

[0086] Step 2.2.1.11 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map Then use discrete wavelet transform DWT to transform the feature map X 1D ′ is downsampled to obtain the feature map

[0087] Step 2.2.1.12 uses a set of convolutional layers Conv27 to transform the feature map X 1DD ′ Perform feature extraction and obtain feature map

[0088] Conv2_7 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0089] Step 2.2.1.13 Use the packaged torch.cat function to connect the feature map X3 and the feature map X1′ according to the spectral dimension to obtain the feature map

[0090] Step 2.2.1.14 uses a set of convolutional layers Conv2_8 to transform the feature map F3D Perform feature extraction and obtain feature map

[0091] The Conv28 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 16C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0092] Step 2.2.1.15 Use the spectral + spatial attention module CBAM to the feature map F 3C Perform feature enhancement to obtain the feature map after feature enhancement

[0093] The spectral + spatial attention module CBAM performs the following steps:

[0094] Definition G v is the feature map to be enhanced;

[0095] First, the spectral attention submodule SpeAM is used to i Perform feature extraction and pass the feature map G to the spectral attention submodule v The number of spectra, get the spectrum attention Spe G ;

[0096] Then use the spectral attention Spe G For the feature map G v Enhance the spectral information and obtain the feature map G spe ;

[0097] Then use the spatial attention submodule SpaAM to the feature map G Spe Perform feature extraction to obtain spatial attention Spa G ;

[0098] Finally, use the space to pay attention to the spa G For the feature map G spe Perform spatial information enhancement to obtain the output feature map G O ;

[0099] Step 2.2.1.16 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map

[0100] Step 2.2.1.17 uses the convolutional layer Conv2_2 to transform the feature map X 1D ″Perform feature extraction and obtain feature map

[0101] Decoding process;

[0102] Step 2.2.2.1 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map

[0103] Step 2.2.2.2 Use inverse wavelet transform IWT to transform the feature map F 3I ″Upsample to obtain the feature map F 3II ″∈R H ×W×C ;

[0104] Step 2.2.2.3 Use a set of convolutional layers Conv2_9 to transform the feature map F 3II ″ performs feature extraction and obtains feature map F3″∈R H×W×C ;

[0105] The Conv2_9 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0106] Step 2.2.2.4 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map

[0107] Step 2.2.2.5 Use a set of convolutional layers Conv2_10 to transform the feature map Y 2I Perform feature extraction and obtain feature map

[0108] The Conv2_10 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0109] Step 2.2.2.6 Use the packaged torch.cat function to connect the feature map X1″ and the feature map Y2 according to the spectral dimension, and then connect it with the feature map X2 according to the spectral dimension to finally obtain the feature map

[0110] Step 2.2.2.7 Use a set of convolutional layers Conv2_11 to transform the feature map F 2D Perform feature extraction and obtain feature map

[0111] The Conv2_11 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0112] Step 2.2.2.8 Use the spectral + spatial attention module CBAM to the feature map F 2C Perform feature enhancement to obtain the feature map after feature enhancement

[0113] Step 2.2.2.9 Use the inverse wavelet transform IWT to upsample the feature map F2 to obtain the feature map Y 1I ∈R H×W×C ;

[0114] Step 2.2.2.10 uses the convolutional layer Conv2_9 to transform the feature map Y 1I Perform feature extraction and obtain the feature map Y1∈R H×W×C ;

[0115] Step 2.2.2.11 Use a set of convolutional layers Conv2_12 to extract features from feature map X2 and obtain feature map

[0116] The Conv2_12 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0117] Step 2.2.2.12 Use inverse wavelet transform IWT to transform the feature map X 2C ′ is upsampled to obtain the feature map X 2I ′∈R H ×W×C ;

[0118] Step 2.2.2.13. Use the convolutional layer Conv2_9 to obtain the feature map X 2I ′ performs feature extraction and obtains the feature map X2′∈R H×W×C ;

[0119] Output process:

[0120] Step 2.2.3.1. Use the packaged torch.cat function to connect the feature map X1 and the feature map X2′ according to the spectral dimension, then connect it with the feature map F3″ according to the spectral dimension, and then connect it with the feature map Y1 according to the spectral dimension, and finally get the feature map F 1D ∈RH ×W×4C ;

[0121] Step 2.2.3.2. Use a set of convolutional layers Conv2_13 to transform the feature map F 1D Perform feature extraction and obtain the feature map F1∈R H×W×C ;

[0122] The Conv2_13 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0123] Step 2.2.3.3. Use discrete wavelet transform DWT to downsample the feature map F1 to obtain the feature map

[0124] Step 2.2.3.4 uses the convolutional layer Conv2_10 to transform the feature map F 2D ′ Perform feature extraction and obtain feature map

[0125] Step 2.2.3.5 Use discrete wavelet transform DWT to downsample the feature map F2′ to obtain the feature map

[0126] Step 2.2.3.6. Use the convolutional layer Conv2_8 to transform the feature map F 3D ′ Perform feature extraction and obtain feature map

[0127] Step 2.2.3.7 outputs the feature map F1, the feature map F2, the feature map F3, the feature map F2′ and the feature map F3′ in sequence;

[0128] The total network loss L in step 2.3 t Follow the steps below to obtain:

[0129] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F1 and the corresponding position of the hyperspectral image image_ref mse1 ;

[0130] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F2′ and the corresponding position of the output feature map F2 mse2 ;

[0131] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F3′ and the corresponding position of the output feature map F3 mse3 ;

[0132] Weight the three loss functions to obtain the final network total loss L t :

[0133] L t =0.6*L mse1 +0.4*L mse2 +0.4*L mse3 ;

[0134] The step 2.4 is as follows:

[0135] Step 2.4.1 Input the obtained test set hyperspectral image test_lr and test set multispectral image test_hr as the hyperspectral image Lr and multispectral image Hr respectively into the trained network model to obtain the output feature map F 1t , F 2t , F 3t , F 2t ′,F 3t ′;

[0136] Step 2.4.2 calculates the peak signal-to-noise ratio psnr;

[0137] First, calculate the hyperspectral image test_ref and the output feature map F according to the following formula 1t The mean square error between:

[0138] mse=mean((test_ref-F 1t ) 2 )

[0139] Where mean(·) represents the packaged averaging function np.mean;

[0140] Then, call the packaged np.max function to obtain the maximum element max of the hyperspectral image test_ref;

[0141] Finally, the peak signal-to-noise ratio psnr is calculated according to the following formula:

[0142] psnr = 10 * log10 (max 2 / mse)

[0143] Where log10(·) represents the encapsulated function np.log10 that calculates the logarithm with base 10;

[0144] Step 2.4.3: Determine the relationship between the obtained peak signal-to-noise ratio psnr and the optimal peak signal-to-noise ratio psnr_b. If psnr_b<psnr, then take psnr_b←psnr, and save the current network model as the optimal network model; otherwise, do not make any changes;

[0145] Step 2.4.4 epoch←epoch+1, if epoch>T, the construction of the best network model has been completed, and the best network model is saved, otherwise return to step 2.1.1.1.

[0146] The present invention adopts a multi-level feature extraction strategy, uses denser full-scale jump connections, explores feature information within a fine-grained range, and realizes a flexible combination of multi-scale and multi-path features; at the same time, the present invention applies a spatial-spectral joint perception attention module, which can make full use of the attention map calculated by the module, effectively embed spatial information and spectral information into the fused image, and realize uninterrupted transmission and aggregation of information. The present invention fully taps deep-level features to improve the accuracy of reconstructed images, effectively improves the spatial resolution of hyperspectral images, and keeps their spectral resolution as undistorted as possible. Experimental results on the public Pavia Center dataset and Washington DC Mall dataset show that the super-resolution effect of the present invention can reach a high level. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] Figure 1 This is a network structure diagram of the spatial spectral attention extraction module according to an embodiment of the present invention.

[0148] Figure 2 This is a structural diagram of a full-scale linked U-shaped fusion module according to an embodiment of the present invention.

[0149] Figure 3 This is a structural diagram of the spatial-spectral joint perception attention extraction module of an embodiment of the present invention.

[0150] Figure 4 The figure is a schematic diagram showing the comparison of the results of the Pavia Center dataset between the embodiment of the present invention and the prior art.

[0151] Figure 5 4 is a schematic diagram comparing the results of the Washington DC Mall data set between the embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0152] A hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perceptual attention of the present invention is carried out in the following steps:

[0153] Step 1. Input the hyperspectral image Y∈R P×Q×C, preprocess Y to obtain the total training set and total test set of hyperspectral images and multispectral images, where P×Q is the spatial size of each band image of Y, and C is the number of bands of Y;

[0154] Step 1.1 Normalize the element values ​​in the hyperspectral image Y to 0-255 to obtain the normalized hyperspectral image Y′∈R P×Q×C , the normalization process is as follows:

[0155] Call the packaged numpy.max function to return the maximum element value Y_Max in Y, and call the packaged numpy.min function to return the minimum element value Y_Min in Y; obtain the normalized image Y′ of Y according to the following formula:

[0156] Y′=255*((Y-Y_Min) / (Y_Max-Y_Min));

[0157] Step 1.2: Get the hyperspectral image Y′∈R P×Q×C The edge of the image is clipped. First, the hyperspectral image width clipping value w_edge=P / / 4*4-P is obtained; where “ / / ” is an integer division operation, and the integer part of the quotient is returned.

[0158] Then the hyperspectral image height clipping value h_edge=Q / / 4*4-Q is obtained;

[0159] If w_edge=0, then reassign w_edge to -1, otherwise do not modify; if h_edge=0, then reassign h_edge to -1, otherwise do not modify;

[0160] Finally, the 0th to P+w_edgeth rows and the 0th to Q+h_edgeth columns of the hyperspectral image Y′ are cut as the edge-cropped new hyperspectral image Y1∈R P′×Q′×C ; The P′×Q′ is the spatial size of each band image of the cropped hyperspectral image Y1, P′=P+w_edge, Q′=Q+h_edge;

[0161] Step 1.3 divides the work area, the work area starting column w_str = (P′-A) / / 2, the work area starting row h_str = (Q′-B) / / 2, the work area ending column w_end = w_str+A, the work area ending row h_end = h_str+B, wherein A and B are the width and height of the multispectral image to be obtained, respectively, and in this embodiment, A=B=128;

[0162] Step 1.4 Divide the test data

[0163] First, we intercept the hyperspectral image Y1 from the w_str column to the w_end column and the h_str row to the h_end row to obtain the hyperspectral image test_ref∈R A×B×C ;

[0164] Call the packaged cv2.GaussianBlur method to perform Gaussian blur on the hyperspectral image test_ref, and pass in parameters ksize=(5,5), sigmaX=2 to obtain the hyperspectral image test_lr′∈R A×B×C ;

[0165] Call the packaged cv2.resize method to downsample the hyperspectral image test_lr′ by 4 times, pass in the parameter dsize=(A / / 4, B / / 4), where “ / / ” is an integer division operation, and returns the integer part of the quotient to obtain the test set hyperspectral image test_lr∈R a×b×C , where a×b is the spatial size of each band image of the hyperspectral image test_lr obtained after downsampling, and a=A / / 4, b=B / / 4;

[0166] Then, c channels are extracted from the hyperspectral image test_ref to obtain the test set multispectral image test_hr∈R A ×B×c , the sampling method is equidistant sampling, where c is the number of channels of the multispectral image to be obtained, and in this embodiment, c=5;

[0167] Step 1.5 Divide the training data

[0168] First, the working area of ​​the hyperspectral image Y1 is assigned to 0, that is, the hyperspectral image Y1∈R P′×Q′×C The values ​​of the w_str column to the w_end column and the h_str row to the h_end row are all assigned to 0, and the hyperspectral image train_ref∈R is obtained. P ′×Q′×C ;

[0169] Then call the good cv2.GaussianBlur method to perform Gaussian blur on the hyperspectral image train_ref, and pass in the parameters ksize=(5,5), sigmaX=2, and get the hyperspectral image train_lr′∈R P′×Q×C ;

[0170] Call the packaged cv2.resize method to downsample the hyperspectral image train_lr′ by 4 times, pass in the parameter dsize=(P′ / / 4, Q′ / / 4), where “ / / ” is an integer division operation, and returns the integer part of the quotient to obtain the training set hyperspectral image train_lr∈R e×f×C, where e×f is the spatial size of each band image of the hyperspectral image train_lr obtained after downsampling, and e=P′ / / 4, f=Q′ / / 4;

[0171] Finally, c channels are extracted from the hyperspectral image train_ref to obtain the training set multispectral image train_hr∈R P′×Q′×c , the sampling method is equidistant sampling, and the P′×Q′ is the spatial size of each band image of the obtained multispectral image train_hr;

[0172] Step 1.6 From the hyperspectral image train_ref∈R P′×Q′×C Randomly cut out a hyperspectral image test_ref∈R A×B×C Hyperspectral images of the same size image_ref∈R A×B×C ;

[0173] Downsample the hyperspectral image image_ref, and take the spatial size multiplier as 1 / 4 to obtain the downsampled hyperspectral image As the training hyperspectral image Lr;

[0174] Finally, from the training set multispectral image train_hr, the multispectral image image_hr∈R corresponding to the hyperspectral image image_ref in the hyperspectral image train_ref is extracted. A×B×c , as the training multispectral image Hr.

[0175] Step 2. Build the best network model

[0176] Step 2.1 Figure 1 As shown, the total training set is used to train the spatial spectral attention extraction module, and the spatial attention Spa of the total training set is extracted respectively. i and spectral attention Spe i , i=1, 2, 3;

[0177] Set the overall neural network iteration number T, the initial optimal peak signal-to-noise ratio psnr_b = 0, set epoch = 1, and start building the optimal network model;

[0178] The training spatial attention extraction module obtains the spatial attention Spa of the total training set i The details are as follows:

[0179] Step 2.1.1.1 Use a set of convolutional layers Conv2_0 to extract features from the multispectral image Hr and obtain the feature map Hr1∈R H×W×C ; Where H×W is the spatial size of each band of Hr, and C is the number of bands of Lr;

[0180] Conv2_0 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0181] Step 2.1.1.2 Build the spatial attention submodule SpaAM. The execution steps are as follows:

[0182] First, call the encapsulated torch.mean function to return the result of averaging the input feature map by channel dimension to obtain the feature map avg_sa;

[0183] Then call the encapsulated torch.max function to return the result of finding the maximum value of the input feature map according to the channel dimension, and get the feature map max_sa;

[0184] Then call the encapsulated torch.cat function to connect the feature map avg_sa and the feature map max_sa according to the channel dimension to obtain the feature map avg_max;

[0185] Finally, a set of convolutional layers Conv2_1 are used to extract features from the feature map avg_max to obtain spatial attention;

[0186] Conv2_1 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 1 convolution kernel of size 7×7, performs a border padding operation of Padding=3, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function Sigmoid as an activation function for the operation;

[0187] Step 2.1.1.3 Use the spatial attention submodule SpaAM to extract features from the feature map Hr1 and obtain the spatial attention Spa1∈R H×W×1 ;

[0188] Step 2.1.1.4 Use discrete wavelet transform DWT to downsample the feature map Hr1 to obtain the feature map

[0189] The specific implementation of the discrete wavelet transform DWT is as follows:

[0190] First, we start from row 0 of the input feature map and intercept it row by row with a step size of 2. Then we divide all element values ​​in the feature map by 2 to get the feature map x. 01 ;

[0191] Then, starting from the first row of the input feature map, the feature map is intercepted row by row with a step size of 2, and all element values ​​in the obtained feature map are divided by 2 to obtain the feature map x02 ;

[0192] Then take 2 as the step size from the feature map x 01 Starting from the 0th column, we intercept each column and get the feature map x1;

[0193] Then take 2 as the step size from the feature map x 02 Starting from the 0th column, we intercept each column and get the feature map x2;

[0194] Then take 2 as the step size from the feature map x 01 Start from the first column and intercept column by column to get the feature map x3;

[0195] Then take 2 as the step size from the feature map x 02 Start from the first column and intercept column by column to get the feature map x4;

[0196] Then, the characteristic graphs x1, x2, x3, x4 are calculated according to formulas (1)-(4) to obtain characteristic graphs x_LL, x_HL, x_LH, x_HH. Formulas (1)-(4) are as follows:

[0197] x_LL=x1+x2+x3+x4 (1)

[0198] x_HL=-x1-x2+x3+x4 (2)

[0199] x_LH=-x1+x2-x3+x4 (3)

[0200] x_HH=x1-x2-x3+x4 (4)

[0201] Finally, call the encapsulated torch.cat function to connect the feature maps x_LL, x_HL, x_LH, x_HH according to the channel dimension, and return the connection result to complete the discrete wavelet transform DWT;

[0202] Step 2.1.1.5 Use a set of convolutional layers Conv2_2 to transform the feature map Hr 1D Perform feature extraction and obtain feature map

[0203] Conv2_2 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 2C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0204] Step 2.1.1.6 Use the spatial attention submodule SpaAM to extract features from the feature map Hr2 and obtain the spatial attention

[0205] Step 2.1.1.7 Use discrete wavelet transform DWT to downsample the feature map Hr2 to obtain the feature map

[0206] Step 2.1.1.8 uses a set of convolutional layers Conv2_3 to transform the feature map Hr 2D Perform feature extraction and obtain feature map

[0207] Conv2_3 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0208] Step 2.1.1.9 Use the spatial attention submodule SpaAM to extract features from the feature map Hr3 and obtain the spatial attention

[0209] The training spectral attention extraction module obtains the spectral attention Spe of the total training set i , the specific steps are as follows:

[0210] Step 2.1.2.1 Perform a 4-fold upsampling operation on the hyperspectral image Lr to obtain the feature map Lr1∈R H×W×C ;

[0211] Step 2.1.2.2 Build the spectral attention submodule SpeAM. The execution steps are as follows:

[0212] First, call the encapsulated adaptive average pooling method torch.nn.AdaptiveAvgPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum to obtain the feature map avg_pool∈R 1×1×C ;

[0213] Use a set of convolutional layers Conv2_4 to extract features from the feature map avg_pool to obtain the feature map avg_mid;

[0214] Conv2_4 includes 1 convolution operation and 1 activation operation, where the convolution layer contains in_planes / / 16 convolution kernels of size 1×1, each convolution kernel performs convolution operation with a step size of 1 pixel, and uses the nonlinear activation function ReLU as the activation function for operation;

[0215] Use a set of convolutional layers Conv2_5 to extract features from the feature map avg_mid and obtain the feature map avg_out∈R 1×1×C ;

[0216] The Conv25 contains 1 convolution operation, where the convolution layer contains in_planes convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0217] Then call the encapsulated adaptive maximum pooling method torch.nn.AdaptiveMaxPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum to obtain the feature map max_pool∈R 1×1×C ;

[0218] Use the convolution layer Conv2_4 to extract features from the feature map max_pool to obtain the feature map max_mid;

[0219] Use the convolutional layer Conv2_5 to extract the feature map max_mid and obtain the feature map max_out∈R 1×1×C :

[0220] Add the feature map avg_out and the feature map max_out to get the feature map out∈R 1×1×C ;

[0221] Finally, the nonlinear activation function Sigmoid is called to activate the feature map out, and the feature map obtained after activation is returned;

[0222] Step 2.1.2.3 Use the spectral attention submodule SpeAM to extract features from the feature map Lr1, pass the parameter in_planes = C to the spectral attention submodule, and obtain the spectral attention Spe1∈R 1×1×C ;

[0223] Step 2.1.2.4 Use discrete wavelet transform DWT to downsample the feature map Lr1 to obtain the feature map

[0224] Step 2.1.2.5 uses the convolutional layer Conv2_2 to transform the feature map Lr 1D Perform feature extraction and obtain feature map

[0225] Step 2.1.2.6 Use the spectral attention submodule SpeAM to extract features from the feature map Lr2, pass the parameter in_planes = 2C to the spectral attention submodule, and obtain the spectral attention Spe2∈R 1×1×2C ;

[0226] Step 2.1.2.7 Use discrete wavelet transform DWT to downsample the feature map Lr2 to obtain the feature map

[0227] Step 2.1.2.8 uses the convolutional layer Conv2_3 to transform the feature map Lr 2D Perform feature extraction and obtain feature map

[0228] Step 2.1.2.9 Use the spectral attention submodule SpeAM to extract features from the feature map Lr3, pass the parameter in_planes=8C to the spectral attention submodule, and obtain the spectral attention Spe3∈R 1×1×8C ;

[0229] Step 2.2 Figure 2 As shown, the full-scale linked U-shaped fusion module is trained with the total training set, and the feature maps F1, F2, F3, F2' and F3' are output;

[0230] Coding process:

[0231] Step 2.2.1.1 Call the encapsulated torch.cat function to connect the feature map Hr1 and the feature map Lr1 according to the channel dimension to obtain the feature map X 1C ;

[0232] Step 2.2.1.2 Use a set of convolutional layers Conv2_6 to transform the feature map X 1C Perform feature extraction and obtain feature map X 1D ∈R H×W×C ;

[0233] The Conv26 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0234] Step 2.2.1.3 Construct the spatial-spectral joint perception attention module SSJPA as follows Figure 3 As shown, the execution steps are as follows:

[0235] Definition S i For the feature map to be enhanced, Spa i and Spe i For the incoming spatial attention and spectral attention, the feature map after information enhancement is expressed as:

[0236]

[0237] in represents matrix multiplication, i=1, 2, 3;

[0238] Step 2.2.1.4 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 1DPerform information enhancement, pass in spatial attention Spa1 and spectral attention Spe1, and finally obtain the feature map X1∈R after information enhancement H×W×C ;

[0239] Step 2.2.1.5 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map

[0240] Step 2.2.1.6 Use the convolutional layer Conv2_2 to transform the feature map X 2D Perform feature extraction and obtain feature map

[0241] Step 2.2.1.7 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 2C Perform information enhancement, pass in spectral attention Spe2 and spatial attention Spa2, and finally obtain the feature map after information enhancement

[0242] Step 2.2.1.8 Use discrete wavelet transform DWT to downsample the feature map X2 to obtain the feature map

[0243] Step 2.2.1.9 uses the convolutional layer Conv2_3 to transform the feature map X 3D Perform feature extraction and obtain feature map

[0244] Step 2.2.1.10 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 3C Perform information enhancement, pass in spectral attention Spe3 and spatial attention Spa3, and finally obtain the feature map after information enhancement

[0245] Step 2.2.1.11 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map Then use discrete wavelet transform DWT to transform the feature map X 1D ′ is downsampled to obtain the feature map

[0246] Step 2.2.1.12 uses a set of convolutional layers Conv2_7 to transform the feature map X 1DD ′ Perform feature extraction and obtain feature map

[0247] The Conv27 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0248] Step 2.2.1.13 Use the packaged torch.cat function to connect the feature map X3 and the feature map X1′ according to the spectral dimension to obtain the feature map

[0249] Step 2.2.1.14 uses a set of convolutional layers Conv2_8 to transform the feature map F 3D Perform feature extraction and obtain feature map

[0250] The Conv2_8 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 16C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0251] Step 2.2.1.15 Use the spectral + spatial attention module CBAM to the feature map F 3C Perform feature enhancement to obtain the feature map after feature enhancement

[0252] The spectral + spatial attention module CBAM performs the following steps:

[0253] Definition G v is the feature map to be enhanced;

[0254] First, the spectral attention submodule SpeAM is used to i Perform feature extraction and pass the feature map G to the spectral attention submodule v The number of spectra, get the spectrum attention Spe G ;

[0255] Then use the spectral attention Spe G For the feature map G v Enhance the spectral information and obtain the feature map G spe ;

[0256] Then use the spatial attention submodule SpaAM to the feature map G spe Perform feature extraction to obtain spatial attention Spa G ;

[0257] Finally, use the space to pay attention to the spa G For the feature map GSpe Perform spatial information enhancement to obtain the output feature map G o ;

[0258] Step 2.2.1.16 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map

[0259] Step 2.2.1.17 uses the convolutional layer Conv2_2 to transform the feature map X 1D ″Perform feature extraction and obtain feature map

[0260] Decoding process;

[0261] Step 2.2.2.1 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map

[0262] Step 2.2.2.2 Use inverse wavelet transform IWT to transform the feature map F 3I ″Upsample to obtain the feature map F 3II ″∈R H ×W×C ;

[0263] The specific implementation of the inverse wavelet transform IWT is as follows:

[0264] Define the feature map I∈R to be inverse wavelet transformed Ih×Iw×Ic First, the output size after inverse wavelet transform is obtained, that is, the output feature map height Ih′=2×Ih, width Iw′=2×Iw, and spectrum number Ic′=Ic×4;

[0265] Then, the spectrum from 0 to Ic of the characteristic graph I is intercepted, and all element values ​​in the intercepted characteristic graph are divided by 2 to obtain the characteristic graph x 01 ′;

[0266] Then intercept the spectrum from Ic to 2Ic of the characteristic graph I, and divide all the element values ​​in the intercepted characteristic graph by 2 to obtain the characteristic graph x 02 ′;

[0267] Then intercept the 2Ic to 3Ic spectra of characteristic graph I, and divide all element values ​​in the intercepted characteristic graph by 2 to obtain characteristic graph x 03 ′;

[0268] Then intercept the 3Ic to 4Ic spectra of characteristic graph I, and divide all element values ​​in the intercepted characteristic graph by 2 to obtain characteristic graph x 04 ′;

[0269] Then, call the encapsulated torch.zeros function to construct the all-zero tensor I′∈RIh′×Iw′×Ic′ ; and according to formulas (7)-(10), the feature map x 01 ′,x 02 ′,x 03 ′,x 04 ' is calculated to obtain the feature graphs x1', x2', x3', x4'. Formulas (5)-(8) are as follows:

[0270] x1′=x 01 ′-x 02 ′-x 03 ′+x 04 ′ (5)

[0271] x2′=x 01 ′-x 02 ′+x 03 ′-x 04 ′ (6)

[0272] x3′=x 01 ′+x 02 ′-x 03 ′-x 04 ′ (7)

[0273] x4′=x 01 ′+x 02 ′+x 03 ′+x 04 ′ (8)

[0274] Finally, fill x1′ to the even columns of the even rows of I′, fill x2′ to the even columns of the odd rows of I′, fill x3′ to the odd columns of the even rows of I′, and fill x4′ to the odd columns of the odd rows of I′ to obtain the upsampling result of the inverse wavelet transform I′∈R Ih′×Iw′×Ic′ ;

[0275] Step 2.2.2.3 Use a set of convolutional layers Conv2_9 to transform the feature map F 3II ″ performs feature extraction and obtains feature map F3″∈R H×W×C ;

[0276] The Conv2_9 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0277] Step 2.2.2.4 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map

[0278] Step 2.2.2.5 Use a set of convolutional layers Conv2_10 to transform the feature map Y 2I Perform feature extraction and obtain feature map

[0279] The Conv2_10 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0280] Step 2.2.2.6 Use the packaged torch.cat function to connect the feature map X1″ and the feature map Y2 according to the spectral dimension, and then connect it with the feature map X2 according to the spectral dimension to finally obtain the feature map

[0281] Step 2.2.2.7 Use a set of convolutional layers Conv2_11 to transform the feature map F 2D Perform feature extraction and obtain feature map

[0282] The Conv2_11 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0283] Step 2.2.2.8 Use the spectral + spatial attention module CBAM to the feature map F 2C Perform feature enhancement to obtain the feature map after feature enhancement

[0284] Step 2.2.2.9 Use the inverse wavelet transform IWT to upsample the feature map F2 to obtain the feature map Y 1I ∈R H×W×C ;

[0285] Step 2.2.2.10 uses the convolutional layer Conv2_9 to transform the feature map Y 1I Perform feature extraction and obtain the feature map Y1∈R H×W×C ;

[0286] Step 2.2.2.11 Use a set of convolutional layers Conv2_12 to extract features from feature map X2 and obtain feature map

[0287] The Conv2_12 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 4C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation;

[0288] Step 2.2.2.12 Use inverse wavelet transform IWT to transform the feature map X 2C ′ is upsampled to obtain the feature map X 2I ′∈R H ×W×C ;

[0289] Step 2.2.2.13. Use the convolutional layer Conv2_9 to obtain the feature map X 2I ′ performs feature extraction and obtains the feature map X2′∈R H×W×C ;

[0290] Output process:

[0291] Step 2.2.3.1. Use the packaged torch.cat function to connect the feature map X1 and the feature map X2′ according to the spectral dimension, then connect it with the feature map F3″ according to the spectral dimension, and then connect it with the feature map Y1 according to the spectral dimension, and finally get the feature map F 1D ∈R H ×W×4C ;

[0292] Step 2.2.3.2. Use a set of convolutional layers Conv2_13 to transform the feature map F 1D Perform feature extraction and obtain the feature map F1∈R H×w×C ;

[0293] The Conv213 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation;

[0294] Step 2.2.3.3. Use discrete wavelet transform DWT to downsample the feature map F1 to obtain the feature map

[0295] Step 2.2.3.4 uses the convolutional layer Conv2_10 to transform the feature map F 2D ′ Perform feature extraction and obtain feature map

[0296] Step 2.2.3.5 Use discrete wavelet transform DWT to downsample the feature map F2′ to obtain the feature map

[0297] Step 2.2.3.6. Use the convolutional layer Conv2_8 to transform the feature map F 3D ′ Perform feature extraction and obtain feature map

[0298] Step 2.2.3.7 outputs the feature map F1, feature map F2, feature map F3, feature map F2 and feature map F3′ in sequence;

[0299] Step 2.3 Using the Adam optimizer and the network total loss function L t Update network parameters to obtain a trained network model;

[0300] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F1 and the corresponding position of the hyperspectral image image_ref mse1 ;

[0301] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F2′ and the corresponding position of the output feature map F2 mse2 ;

[0302] Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F3′ and the corresponding position of the output feature map F3 mse3 ;

[0303] Weight the three loss functions to obtain the final network total loss function L t :

[0304] L t =0.6*L mse1 +0.4*L mse2 +0.4*L mse3 ;

[0305] Step 2.4: Test the trained network model with the total test set, calculate the peak signal-to-noise ratio, and iterate until the best network model is obtained.

[0306] Step 2.4.1: Replace the hyperspectral image Lr and multispectral image Hr used in training with the obtained test set hyperspectral image test_lr and test set multispectral image test_hr, respectively, and input them into the trained network model (repeat steps 2.1.1.1-2.2.3.7), and obtain the output feature map F 1t , F 2t , F 3t , F 2t ′,F 3t ′;

[0307] Step 2.4.2 calculates the peak signal-to-noise ratio psnr;

[0308] First, calculate the hyperspectral image test_ref and the output feature map F according to the following formula 1t The mean square error between:

[0309] mse=mean((test_ref-F 1t ) 2 )

[0310] Where mean(·) represents the packaged averaging function np.mean;

[0311] Then, call the packaged np.max function to obtain the maximum element max of the hyperspectral image test_ref;

[0312] Finally, the peak signal-to-noise ratio psnr is calculated according to the following formula:

[0313] psnr = 10 * log10 (max 2 / mse)

[0314] Where log10(·) represents the encapsulated function np.log10 that calculates the logarithm with base 10;

[0315] Step 2.4.3: Determine the relationship between the obtained peak signal-to-noise ratio psnr and the optimal peak signal-to-noise ratio psnr_b. If psnr_b<psnr, then take psnr_b←psnr, and save the current network model as the optimal network model; otherwise, do not make any changes;

[0316] Step 2.4.4epoch←epoch+1, if epoch>T, the construction of the best network model has been completed, and the best network model is saved, otherwise return to step 2.1.1.1 and start training again.

[0317] Step 3. Input the total test set into the best network model, and the output feature map F1 is the best fusion result, thus obtaining a reconstructed super-resolution hyperspectral image.

[0318] To verify the effectiveness of the present invention, the root mean square error (RMSE), peak signal-to-noise ratio (PSNR), relative dimensionless global error (ERGAS) and spectral angle similarity (SAM) were used as objective indicators to evaluate the results, and the evaluation results of the present invention were compared with the MSDCNN method, TFNET method, ResTFNet method, SSFCNN method, ConSSFCNN method and SSR-NET method on the Pavia Center and Washington DC Mall datasets.

[0319] Table 1 and Figure 4 The evaluation indicators and pseudo-color RGB images of the reconstruction results of all methods on the Pavia Center dataset are shown respectively. From the experimental results, it can be seen that the present invention achieves the best results in all four evaluation indicators on the Pavia Center dataset. Compared with the second-ranked algorithm SSR-NET in reconstruction quality, the PSNR of the present invention is improved by 5.65%, and the RMSE, ERGAS and SAM are reduced by 21.65%, 21.90% and 9.90% respectively. These data fully illustrate the effectiveness of the present invention on the Pavia Center dataset, proving that it has a good ability to preserve spatial-spectral information.

[0320] Table 1 Performance comparison of different fusion algorithms on the Pavia Center dataset

[0321]

[0322] Figure 4 This is a schematic diagram comparing the results of the Washington DC Mall data set between the embodiment of the present invention and the MSDCNN method, TFNET method, ResTFNet method, SSFCNN method, ConSSFCNN method, and SSR-NET method, wherein the first row shows the fused pseudo-color RGB result image, and the second row is the differential image after pseudo-color processing.

[0323] Table 2 and Figure 5 The evaluation indicators and pseudo-color RGB images of the reconstruction results of all methods on the Washington DC Mall dataset are shown respectively. The experimental results show that the present invention achieves the best in all four indicators on the Washington DC Mall dataset. Compared with ResTFNet, which ranks second in reconstruction quality, the PSNR of the present invention is improved by 59.52%, and the RMSE, ERGAS and SAM are reduced by 16.80%, 59.93% and 58.62%, respectively. It can be seen that the performance of the present invention on the Washington DCMall dataset has made a huge leap compared with other methods. Compared with other datasets, the Washington DCMall dataset has more bands, which shows that the present invention still has good performance on datasets with higher spectral resolution.

[0324] Table 2 Performance comparison of different fusion algorithms on the Washington DC Mall dataset

[0325]

[0326] Figure 5This is a schematic diagram comparing the results of the Washington DC Mall data set using the examples of the present invention, the MSDCNN method, the TFNET method, the ResTFNet method, the SSFCNN method, the ConSSFCNN method, and the SSR-NET method, wherein the first row shows the fused pseudo-color RGB result image, and the second row is the differential image after pseudo-color processing.

Claims

1. A hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perception and attention, characterized in that Follow the steps below: Step 1. Input the hyperspectral image Y∈R P×Q×C , preprocess Y to obtain the total training set and total test set of hyperspectral images and multispectral images, where P×Q is the spatial size of each band image of Y, and C is the number of bands of Y; Step 2. Build the best network model Step 2.1 Use the total training set to train the spatial spectral attention extraction module, and extract the spatial attention Spa of the total training set respectively. i and spectral attention Spe i , i=1,2,3; Step 2.2 Use the total training set to train the full-scale link U-type fusion module and output the feature map F1∈R H×W×C , feature map Feature Map Feature Map And feature map Step 2.3 Using the Adam optimizer and the network total loss function L t Update network parameters to obtain a trained network model; Step 2.4: Test the trained network model with the total test set, calculate the peak signal-to-noise ratio, and iterate until the best network model is obtained. Step 3. Input the total test set into the optimal network model to obtain the reconstructed super-resolution hyperspectral image.

2. The hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perception attention according to claim 1 is characterized in that The step 1 is specifically as follows: Step 1.1 Normalize the element values ​​in the hyperspectral image Y to 0-255 to obtain the normalized hyperspectral image Y'∈R P ×Q×C ; Step 1.2: Get the hyperspectral image Y'∈R P×Q×C The edge of the image is clipped to obtain the clipped hyperspectral image Y1∈R P'×Q'×C ; The P'×Q' is the spatial size of each band image of the cropped hyperspectral image Y1, P'=P+w_edge, Q'=Q+h_edge; the w_edge=P / / 4*4-P, h_edge=Q / / 4*4-Q, where " / / " is an integer division operation, and the integer part of the quotient is returned; Step 1.3 divides the work area, wherein the work area starting column w_str=(P'-A) / / 2, the work area starting row h_str=(Q'-B) / / 2, the work area ending column w_end=w_str+A, the work area ending row h_end=h_str+B, wherein A and B are the width and height of the multispectral image to be obtained respectively; Step 1.4 Divide the test data First, we intercept the hyperspectral image Y1 from the w_str column to the w_end column and the h_str row to the h_end row to obtain the hyperspectral image test_ref∈R A×B×C ; Perform Gaussian blur on the hyperspectral image test_ref, and pass in parameters ksize=(5,5), sigmaX=2 to obtain the hyperspectral image test_lr'∈R A×B×C ; Downsample the hyperspectral image test_lr' by 4 times, pass in the parameter dsize = (A / / 4, B / / 4), where " / / " is an integer division operation, and returns the integer part of the quotient to obtain the test set hyperspectral image test_lr∈R a×b×C , where a×b is the spatial size of each band image of the hyperspectral image test_lr obtained after downsampling, and a=A / / 4, b=B / / 4; Then, c channels are extracted from the hyperspectral image test_ref to obtain the test set multispectral image test_hr∈R A×B×c , the sampling method is equidistant sampling, where c is the number of channels of the multispectral image to be obtained; Step 1.5 Divide the training data First, the working area of ​​the hyperspectral image Y1 is assigned to 0, that is, the hyperspectral image Y1∈R P'×Q'×C The values ​​of the w_str column to the w_end column and the h_str row to the h_end row are all assigned to 0, and the hyperspectral image train_ref∈R is obtained. P′×Q′×C ; Then, Gaussian blur is performed on the hyperspectral image train_ref, and parameters ksize=(5,5) and sigmaX=2 are passed in to obtain the hyperspectral image train_lr'∈R P'×Q'×C ; Downsample the hyperspectral image train_lr' by 4 times, pass in the parameter dsize = (P' / / 4, Q' / / 4), where " / / " is an integer division operation, and returns the integer part of the quotient to obtain the training set hyperspectral image train_lr∈R e×f×C , where e×f is the spatial size of each band image of the hyperspectral image train_lr obtained after downsampling, and e=P' / / 4, f=Q' / / 4; Finally, c channels are extracted from the hyperspectral image train_ref to obtain the training set multispectral image train_hr∈R P '×Q'×c , the sampling method is equidistant sampling, and the P'×Q' is the spatial size of each band image of the obtained multispectral image train_hr; Step 1.6 From the hyperspectral image train_ref∈R P'×Q'×C Randomly cut out a hyperspectral image test_ref∈R A×B×C Hyperspectral images of the same size image_ref∈R A×B×C ; Downsample the hyperspectral image image_ref, and take the spatial size multiplier as 1 / 4 to obtain the downsampled hyperspectral image As the training hyperspectral image Lr; Finally, from the training set multispectral image train_hr, the multispectral image image_hr∈R corresponding to the hyperspectral image image_ref in the hyperspectral image train_ref is extracted. A×B×c , as the training multispectral image Hr.

3. The hyperspectral image super-resolution reconstruction method based on spatial-spectral joint perception attention according to claim 2 is characterized in that The step 2 is specifically as follows: Set the overall neural network iteration number T, the initial optimal peak signal-to-noise ratio psnr_b = 0, set epoch = 1, and start building the optimal network model; The step 2.1 trains the spatial attention extraction module to obtain the spatial attention Spa of the total training set i The details are as follows: Step 2.1.1.1 Use a set of convolutional layers Conv2_0 to extract features from the multispectral image Hr and obtain the feature map Hr1∈R H×W×C ; Where H×W is the spatial size of each band of Hr, and C is the number of bands of the hyperspectral image Lr; Conv2_0 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation; Step 2.1.1.2 Build the spatial attention submodule SpaAM. The execution steps are as follows: First, call the encapsulated torch.mean function to return the result of averaging the input feature map by channel dimension to obtain the feature map avg_sa; Then call the encapsulated torch.max function to return the result of finding the maximum value of the input feature map according to the channel dimension, and get the feature map max_sa; Then call the encapsulated torch.cat function to connect the feature map avg_sa and the feature map max_sa according to the channel dimension to obtain the feature map avg_max; Finally, a set of convolutional layers Conv2_1 are used to extract features from the feature map avg_max to obtain spatial attention; Conv2_1 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 1 convolution kernel of size 7×7, performs a border padding operation of Padding=3, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function Sigmoid as an activation function for the operation; Step 2.1.1.3 Use the spatial attention submodule SpaAM to extract features from the feature map Hr1 and obtain the spatial attention Spa1∈R H×W×1 ; Step 2.1.1.4 Use discrete wavelet transform DWT to downsample the feature map Hr1 to obtain the feature map Step 2.1.1.5 Use a set of convolutional layers Conv2_2 to transform the feature map Hr 1D Perform feature extraction and obtain feature map Conv2_2 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 2C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation; Step 2.1.1.6 Use the spatial attention submodule SpaAM to extract features from the feature map Hr2 and obtain the spatial attention Step 2.1.1.7 Use discrete wavelet transform DWT to downsample the feature map Hr2 to obtain the feature map Step 2.1.1.8 uses a set of convolutional layers Conv2_3 to transform the feature map Hr 2D Perform feature extraction and obtain feature map Conv2_3 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation; Step 2.1.1.9 Use the spatial attention submodule SpaAM to extract features from the feature map Hr3 and obtain the spatial attention The step 2.1 trains the spectral attention extraction module to obtain the spectral attention Spe of the total training set i , the specific steps are as follows: Step 2.1.2.1 Perform a 4-fold upsampling operation on the hyperspectral image Lr to obtain the feature map Lr1∈R H×W×C ; Step 2.1.2.2 Build the spectral attention submodule SpeAM. The execution steps are as follows: First, call the encapsulated adaptive average pooling method torch.nn.AdaptiveAvgPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum, and get the feature map avg_pool∈R 1×1×C ; Use a set of convolutional layers Conv2_4 to extract features from the feature map avg_pool to obtain the feature map avg_mid; Conv2_4 includes 1 convolution operation and 1 activation operation, where the convolution layer contains in_planes / / 16 convolution kernels of size 1×1, each convolution kernel performs convolution operation with a step size of 1 pixel, and uses the nonlinear activation function ReLU as the activation function for operation; Use a set of convolutional layers Conv2_5 to extract features from the feature map avg_mid and obtain the feature map avg_out∈R 1 ×1×C ; Conv2_5 contains one convolution operation, where the convolution layer contains in_planes convolution kernels of size 1×1, and each convolution kernel performs convolution operation with a step size of 1 pixel; Then call the encapsulated adaptive maximum pooling method torch.nn.AdaptiveMaxPool2d to calculate the eigenvalues ​​of the module input feature map spectrum by spectrum to obtain the feature map max_pool∈R 1×1×C ; Use the convolution layer Conv2_4 to extract features from the feature map max_pool to obtain the feature map max_mid; Use the convolutional layer Conv2_5 to extract the feature map max_mid and obtain the feature map max_out∈R 1 ×1×C ; Add the feature map avg_out and the feature map max_out to get the feature map out∈R 1×1×C ; Finally, the nonlinear activation function Sigmoid is called to activate the feature map out, and the feature map obtained after activation is returned; Step 2.1.2.3 Use the spectral attention submodule SpeAM to extract features from the feature map Lr1, pass the parameter in_planes = C to the spectral attention submodule, and obtain the spectral attention Spe1∈R 1×1×C ; Step 2.1.2.4 Use discrete wavelet transform DWT to downsample the feature map Lr1 to obtain the feature map Step 2.1.2.5 uses the convolutional layer Conv2_2 to transform the feature map Lr 1D Perform feature extraction and obtain feature map Step 2.1.2.6 Use the spectral attention submodule SpeAM to extract features from the feature map Lr2, pass the parameter in_planes = 2C to the spectral attention submodule, and obtain the spectral attention Spe2∈R 1×1×2C ; Step 2.1.2.7 Use discrete wavelet transform DWT to downsample the feature map Lr2 to obtain the feature map Step 2.1.2.8 uses the convolutional layer Conv2_3 to transform the feature map Lr 2D Perform feature extraction and obtain feature map Step 2.1.2.9 Use the spectral attention submodule SpeAM to extract features from the feature map Lr3, pass the parameter in_planes=8C to the spectral attention submodule, and obtain the spectral attention Spe3∈R 1×1×8c ; The step 2.2 is specifically as follows: Coding process: Step 2.2.1.1 Call the encapsulated torch.cat function to connect the feature map Hr1 and the feature map Lr1 according to the channel dimension to obtain the feature map X 1C ; Step 2.2.1.2 Use a set of convolutional layers Conv2_6 to transform the feature map X 1c Perform feature extraction and obtain feature map X 1D ∈R H ×W×C ; Conv2_6 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation; Step 2.2.1.3 Build the spatial-spectral joint perception attention module SSJPA. The execution steps are as follows: Definition S i For the feature map to be enhanced, Spa i and Spe i For the incoming spatial attention and spectral attention, the feature map after information enhancement is expressed as: in Indicates matrix multiplication, i = 1, 2, 3; Step 2.2.1.4 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 1D Perform information enhancement, pass in spatial attention Spa1 and spectral attention Spe1, and finally obtain the feature map X1∈R after information enhancement H×W×C ; Step 2.2.1.5 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map Step 2.2.1.6 Use the convolutional layer Conv2_2 to transform the feature map X 2D Perform feature extraction and obtain feature map Step 2.2.1.7 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 2C Perform information enhancement, pass in spectral attention Spe2 and spatial attention Spa2, and finally obtain the feature map after information enhancement Step 2.2.1.8 Use discrete wavelet transform DWT to downsample the feature map X2 to obtain the feature map Step 2.2.1.9 uses the convolutional layer Conv2_3 to transform the feature map X 3D Perform feature extraction and obtain feature map Step 2.2.1.10 Use the spatial-spectral joint perception attention module SSJPA to the feature map X 3C Perform information enhancement, pass in spectral attention Spe3 and spatial attention Spa3, and finally obtain the feature map after information enhancement Step 2.2.1.11 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map Then use discrete wavelet transform DWT to transform the feature map X 1D 'Downsample to get the feature map Step 2.2.1.12 uses a set of convolutional layers Conv2_7 to transform the feature map X 1DD 'Extract features and get feature maps Conv2_7 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 8C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation; Step 2.2.1.13 Use the packaged torch.cat function to connect the feature map X3 and the feature map X1' according to the spectral dimension to obtain the feature map Step 2.2.1.14 uses a set of convolutional layers Conv2_8 to transform the feature map F 3D Perform feature extraction and obtain feature map The Conv2_8 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains 16C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and uses a nonlinear activation function ReLU as an activation function for the operation; Step 2.2.1.15 Use the spectral + spatial attention module CBAM to the feature map F 3C Perform feature enhancement to obtain the feature map after feature enhancement The spectral + spatial attention module CBAM performs the following steps: Definition G v is the feature map to be enhanced; First, the spectral attention submodule SpeAM is used to i Perform feature extraction and pass the feature map G to the spectral attention submodule v The spectrum number is obtained, and the spectrum attention Spe G ; Then use the spectral attention Spe G For the feature map G v Enhance the spectral information and obtain the feature map G Spe ; Then use the spatial attention submodule SpaAM to the feature map G Spe Perform feature extraction to obtain spatial attention Spa G ; Finally, use the space to pay attention to the spa G For the feature map G Spe Perform spatial information enhancement to obtain the output feature map G o ; Step 2.2.1.16 Use discrete wavelet transform DWT to downsample the feature map X1 to obtain the feature map Step 2.2.1.17 uses the convolutional layer Conv2_2 to transform the feature map X 1D "Extract features and get feature maps Decoding process; Step 2.2.2.1 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map Step 2.2.2.2 Use inverse wavelet transform IWT to transform the feature map F 3I "Upsample to get the feature map F 3II ”∈R H×W×C ; Step 2.2.2.3 Use a set of convolutional layers Conv2_9 to transform the feature map F 3II "Extract features and get feature map F3"∈R H ×W×C ; The Conv2_9 includes 1 convolution operation and 1 activation operation, wherein the convolution layer contains C convolution kernels of size 3×3, performs a border padding operation of Padding=1, each convolution kernel performs a convolution operation with a step size of 1 pixel, and selects a nonlinear activation function ReLU as an activation function for operation; Step 2.2.2.4 Use the inverse wavelet transform IWT to upsample the feature map F3 to obtain the feature map Step 2.2.2.5 Use a set of convolutional layers Conv2_10 to transform the feature map Y 2I Perform feature extraction and obtain feature map The Conv2_10 includes 1 layer of convolution operation and 1 layer of activation operation. The convolution layer contains 4C convolution kernels of size 3×3, performs boundary padding operation with Padding = 1, each convolution kernel performs convolution operation with a stride of 1 pixel, and the non-linear activation function ReLU is selected as the activation function for operation; Step 2.2.2.6 Use the packaged torch.cat function to connect the feature map X1" and the feature map Y2 according to the spectral dimension, and then connect it with the feature map X2 according to the spectral dimension to finally obtain the feature map Step 2.2.2.7 Use a set of convolutional layers Conv2_11 to transform the feature map F 2D Perform feature extraction and obtain feature map The Conv2_11 includes 1 layer of convolution operation and 1 layer of activation operation. The convolution layer contains 4C convolution kernels of size 3×3, performs boundary padding operation with Padding = 1, each convolution kernel performs convolution operation with a stride of 1 pixel, and the non-linear activation function ReLU is selected as the activation function for operation; Step 2.2.2.8 Use the spectral + spatial attention module CBAM to the feature map F 2C Perform feature enhancement to obtain the feature map after feature enhancement Step 2.2.2.9 Use the inverse wavelet transform IWT to upsample the feature map F2 to obtain the feature map Y 1I ∈R H×W×C ; Step 2.2.2.10 uses the convolutional layer Conv2_9 to transform the feature map Y 1I Perform feature extraction and obtain the feature map Y1∈R H ×W×C ; Step 2.2.2.11 Use a set of convolutional layers Conv2_12 to extract features from feature map X2 and obtain feature map The Conv2_12 includes 1 layer of convolution operation and 1 layer of activation operation. The convolution layer contains 4C convolution kernels of size 3×3, performs boundary padding operation with Padding = 1, each convolution kernel performs convolution operation with a stride of 1 pixel, and the non-linear activation function ReLU is selected as the activation function for operation; Step 2.2.2.12 Use inverse wavelet transform IWT to transform the feature map X 2C 'Upsample to get feature map X 2I '∈R H×W×C ; Step 2.2.2.

13. Use the convolutional layer Conv2_9 to obtain the feature map X 2I 'Perform feature extraction and obtain feature map X2'∈R H×W×C ; Output process: Step 2.2.3.

1. Use the packaged torch.cat function to connect the feature map X1 and the feature map X2' according to the spectral dimension, then connect it with the feature map F3' according to the spectral dimension, and then connect it with the feature map Y1 according to the spectral dimension, and finally get the feature map F 1D ∈R H×W×4C ; Step 2.2.3.

2. Use a set of convolutional layers Conv2_13 to transform the feature map F 1D Perform feature extraction and obtain the feature map F1∈R H ×W×C ; The Conv2_13 includes 1 layer of convolution operation and 1 layer of activation operation. The convolution layer contains C convolution kernels of size 3×3, performs boundary padding operation with Padding = 1, each convolution kernel performs convolution operation with a stride of 1 pixel, and the non-linear activation function ReLU is selected as the activation function for operation; Step 2.2.3.

3. Use discrete wavelet transform DWT to downsample the feature map F1 to obtain the feature map Step 2.2.3.4 uses the convolutional layer Conv2_10 to transform the feature map F 2D 'Extract features and get feature maps Step 2.2.3.5 Use discrete wavelet transform DWT to downsample the feature map F2' to obtain the feature map Step 2.2.3.

6. Use the convolutional layer Conv2_8 to transform the feature map F 3D 'Extract features and get feature maps In step 2.2.3.7, feature maps F1, F2, F3, F2' and F3' are output in sequence; The total network loss L in step 2.3 t Follow the steps below to obtain: Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F1 and the corresponding position of the hyperspectral image image_ref mse1 ; Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F2' and the corresponding position of the output feature map F2 mse2 ; Call the packaged mean square error loss function nn.MSELoss to obtain the mean L of the square difference between the output feature map F3' and the corresponding position of the output feature map F3 mse3 ; Weight the three loss functions to obtain the final network total loss L t : L t =0.6*L mse1 +0.4*L mse2 +0.4*L mse3 ; The specific content of step 2.4 is as follows: Step 2.4.1 Input the obtained test set hyperspectral image test_lr and test set multispectral image test_hr as the hyperspectral image Lr and multispectral image Hr respectively into the trained network model to obtain the output feature map F 1t , F 2t , F 3t , F 2t ', F 3t '; In step 2.4.2, calculate the peak signal-to-noise ratio psnr; First, calculate the hyperspectral image test_ref and the output feature map F according to the following formula 1t The mean square error between: mse=mean((test_ref-F 1t ) 2 ) where mean(·) represents the encapsulated function np.mean for calculating the average value; Then, call the encapsulated np.max function to obtain the maximum element max of the hyperspectral image test_ref; Finally, calculate the peak signal-to-noise ratio psnr according to the following formula: psnr=10*log10(max 2 / mse) where log10(·) represents the encapsulated function np.log10 for taking the logarithm with base 10; In step 2.4.3, judge the size relationship between the obtained peak signal-to-noise ratio psnr and the best peak signal-to-noise ratio psnr_b. If psnr_b < psnr, then let psnr_b ← psnr and save the current network model as the best network model; otherwise, do not make any changes; In step 2.4.4, epoch ← epoch + 1. If epoch > T, the construction of the best network model is completed, and save the best network model; otherwise, return to step 2.1.1.1.

Images