Hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection

Abstract

This paper discloses a hyperspectral super-resolution method using asymmetric attention and wavelet subband injection. The method mainly includes: extracting multispectral, multiresolution, high-frequency detail features using discrete wavelet transform; constructing a detail extraction network encoder; constructing a detail extraction network decoder; preprocessing low-resolution hyperspectral images; constructing a spatial-spectral fusion network encoder; constructing an asymmetric feature selection attention module; constructing a spatial-spectral fusion network encoder; and training the network using the L1 loss function. The method extracts deep, multiresolution details from the multispectral image through a wavelet network and injects them into a U-Net, resulting in a learned hyperspectral image with better spatial-spectral detail information.

Description

Technical Field

[0001] The present invention relates to a hyperspectral super-resolution method, in particular to a hyperspectral super-resolution method utilizing asymmetric attention and wavelet sub-band injection. Background Art

[0002] Hyperspectral images contain dozens to hundreds of bands, allowing for clearer identification of ground features than multispectral images. However, obtaining high spatial resolution HS images is difficult due to hardware and budget constraints. Therefore, a common approach is to fuse LR-HS images with HR-MS images to create HR-HS images.

[0003] With the development of deep learning, an increasing number of deep learning models have been applied to hyperspectral super-resolution, such as the Spatial-Spectral Fusion Network (SSF), the Deep Two-branch Convolutional Neural Network (Two-CNN), the Deep Recurrent Network (DRN), the Progressive Zero-Centered Residual Network (PZRes-Net), and the Deep Spatial-Spectral Attention Convolutional Neural Network (DSSA-CNN). Some methods, such as Deep Hyperspectral Sharpening (DHSIS) and Deep Blind Source Hyperspectral Fusion (DBIN), utilize deep learning networks to learn degradation models to reconstruct HS images. To improve the interpretability of deep learning, researchers have proposed deep unfolding networks to address the HS fusion problem, such as the Multispectral Hyperspectral Fusion Network (MHF-Net), the Fusion Variational Network (VaFuNet), and the Model-Oriented Hyperspectral Image Super-Resolution (MOG-DCN). These deep learning methods simply use the concatenation of HS and MS images along the spectral channels as the network input, without fully considering the underlying multi-scale spatial information.

[0004] Some researchers have proposed fusing HS and MS images at different scales. For example, Zhou et al. [Zhou, Feng, et al. "Pyramid fully convolutional network for hyperspectral and multispectral image fusion." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12.5(2019):1549-1558.] proposed a PFCN network to address the fusion of MSI and HSI. This network consists of two subnetworks: the first extracts spectral information from LR-HS images using convolutional kernels and encodes it into deep features. The second integrates the HR-MS image pyramid with the encoded deep features to obtain the HR-HS image. Xu et al. [Xu, Shuang, et al. "HAM-MFN: Hyperspectral and multispectral image multiscale fusion network with RAP loss." IEEE Transactions on Geoscience and Remote Sensing 58.7(2020):4618-4628.] proposed a HAM-MFN network to address the fusion of MS and HS images. This network first gradually amplifies the deep features of the LR-HS image using deconvolution, then fuses the deep features of the LR-HS image and the MS image at different scales to finally obtain the HR-HS image. However, this structure ignores the basic and shallow features of the MSI. Therefore, Xiao et al. [Xiao J, Li J, Yuan Q, et al. "ADual-UNet with Multistage Details Injection for Hyperspectral Image Fusion," IEEE Transactions on Geoscience and Remote Sensing, 2021.] proposed a dual U-Net fusion method, which first uses an encoder-decoder network to extract multispectral spatial features at different scales. These scale features are then injected into a U-Net network to reconstruct the HS image. However, these networks simply use convolution to extract the multiscale deep features of the MS image and inject them into the HS image to enhance resolution. The purpose of fusion should be to inject the spatial detail features of the MS into the HS image. Simply using convolution to extract the multiscale features of the MS image cannot obtain detailed features in a targeted manner. Summary of the Invention

[0005] The present invention aims to provide a hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection.

[0006] The technical solution to realize the present invention is: a hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection, comprising the following steps:

[0007] The first step is to use discrete wavelet transform to extract multi-spectral and multi-resolution high-frequency detail features;

[0008] The second step is to build a detail extraction network encoder based on the high-frequency details extracted by discrete wavelet transform and convolution operation;

[0009] The third step is to build a detail extraction network decoder through the deep features and deconvolution operation of the detail extraction encoder;

[0010] The fourth step is to preprocess the low-resolution hyperspectral image, that is, upsample the low-resolution hyperspectral image to the same spatial size as the multispectral image;

[0011] The fifth step is to construct a spatial-spectral fusion network encoder using the upsampled hyperspectral image, high-resolution multispectral image and deep features extracted by the detail extraction network encoder;

[0012] In the sixth step, the asymmetric feature selection attention module is used to select the deep multi-resolution features output by the spatial-spectral fusion network encoder;

[0013] Step 7: Use the deep features output by the asymmetric feature selection attention module, the spatial-spectral fusion network encoder, and the detail extraction network decoder to construct the spatial-spectral fusion network decoder, and use the ReLU activation function to obtain the fusion result;

[0014] Step 8: Train the network using the L1 loss function.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection is implemented.

[0016] The method of the present invention has a simple structure. Compared with the existing technology, its significant features are: (1) firstly, multi-resolution wavelet detail features are extracted by discrete wavelet transform, and then these detail features are converted into deep detail features by encoder-decoder and injected into the HS image to enhance the resolution; (2) an asymmetric feature selection attention module is used to select features of different resolutions; (3) the network structure is simple and the computational complexity is low.

[0017] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural diagram of the method of the present invention.

[0019] Figure 2 This is the structural diagram of the asymmetric feature selection attention module.

[0020] Figure 3 The pseudo-color composite images and corresponding mean square error images (bands 30, 20, and 10) of the fusion results of different methods on real and fake apples data are shown in the present invention. (a) CNMF, (b) HySure, (c) ICCV, (d) DHSIS, (e) DBIN, (f) CNN-Fus, (g) MOG-DCN, and (h) DMW-UNet.

[0021] Figure 4 This is a comparison of the MPSNR curves of each band of the fusion results of different methods of the present invention on real and fake apples data.

[0022] Figure 5 This is the comparison of MSSIM curves of each band of the fusion results of different methods of the present invention on real and fake apples data.

[0023] Figure 6 The pseudo-color composite images and corresponding mean square error images (bands 30, 20, and 10) of the fusion results of different methods of the present invention on the img1 data are shown. (a) CNMF, (b) HySure, (c) ICCV, (d) DHSIS, (e) DBIN, (f) CNN-Fus, (g) MOG-DCN, and (h) DMW-UNet.

[0024] Figure 7 It is a comparison of the MPSNR curves of each band of the fusion results of different methods on the img1 data using the method of the present invention.

[0025] Figure 8 This is a comparison of the MSSIM curves of each band of the fusion results of different methods on the img1 data using the method of the present invention. DETAILED DESCRIPTION

[0026] In response to the problems existing in the prior art, the present invention first uses discrete wavelet transform to extract the multi-scale detail features of MS, then uses convolution to learn these detail features to obtain multi-scale depth detail features, and finally injects these features into U-Net to enhance the resolution of HS images. The present invention has the ability to extract the spatial-spectral features of hyperspectral images in both Euclidean space and topological space, and has excellent performance in hyperspectral image super-resolution. The experimental results show that the average peak signal-to-noise ratio (MPSNR) on the CAVE dataset can reach 47.0245, and the average peak signal-to-noise ratio (MPSNR) on the Harvard dataset can reach 45.8118. The experimental results show that the network structure proposed in the present invention can well restore the spatial and spectral information of HS images.

[0027] Combine Figure 1 , the implementation process of the present invention is described in detail, the method steps of the present invention are as follows:

[0028] The first step is to use discrete wavelet transform to extract multi-resolution high-frequency detail features of multi-spectrum. Represents a high-resolution multispectral image, where H, W, and b represent the height, width, and number of bands of the high-resolution multispectral image, respectively. This paper uses Haar discrete wavelet transform (filter bank is "db1") to extract multi-resolution high-frequency detail information of the multispectral image, where the low-pass filter is represented by Φ and the high-pass filter bank is represented by Ψ. The low-frequency subband image with all the low-pass filters in the row at the dth scale is represented as

[0029]

[0030] The high-frequency sub-band images in three directions are represented as

[0031]

[0032] in, and denote the conjugates of Φ and Ψ, respectively. Represents the d-th scale C and separable filter Convolution, C d-1 represents the low-pass subband image at the d-1th scale, and Respectively represent the high-frequency sub-band images in the horizontal, vertical and diagonal directions at the dth scale. The multispectral image is transformed by discrete wavelet to obtain the low-frequency sub-band image C1 and the high-frequency sub-band image W1 1 、W1 2 and W1 3 , and concatenate these three high-frequency sub-band images in the channel dimension to obtain the high-frequency image W1, which is expressed as

[0033] W1=Concat(W1 1 ,W12 ,W1 3 )

[0034] Among them, Concat(.) represents the channel-dimensional concatenation operation. Since the low-frequency sub-band image C1 also contains high-frequency information, the low-frequency sub-band image of the multispectral image is transformed using discrete wavelet transform to obtain the low-frequency sub-band image C2 and the high-frequency sub-band image and And these three high-frequency sub-band images are spliced ​​in the channel dimension to obtain the high-frequency image W2, which is expressed as

[0035]

[0036] The second step is to build a detail extraction network encoder based on the high-frequency image extracted by discrete wavelet transform and convolution operation. First, for the high-frequency image W1, a 3×3 convolution is used to obtain the deep features of the first output of the detail extraction encoder. This process is expressed as

[0037] WE1=Conv 3×3 (W1)

[0038] Among them, Conv 3×3 (.) represents a convolution operation with a 3×3 kernel. Because direct pooling will cause a large loss of high-frequency information, a discrete wavelet transform is used instead of a downsampling operation. A discrete wavelet transform is performed on WE1 to obtain depth detail information, and then concatenated in the channel dimension to obtain W3. The high-frequency images W2 and W3 are then concatenated in the channel dimension and a 3×3 convolution is performed to obtain the depth features of the second output of the detail extraction encoder. This process is represented as

[0039] WE2=Conv 3×3 (Concat(W2,W3))

[0040] The third step is to construct a detail extraction network decoder through the deep features extracted by the encoder and the deconvolution operation. The detail extraction network decoder is a supplement to the information of the spatial-spectral joint network decoder. The decoder consists of a deconvolution with a stride of 2 and a 3×3 convolution. The deep features of the three outputs of the decoder are expressed as

[0041]

[0042] Among them, Deconv(.) represents deconvolution, WD n Represents the deep features output by the decoder of the detail extraction network at the nth stage.

[0043] The fourth step is to preprocess the low-resolution hyperspectral image, that is, upsample the low-resolution hyperspectral image to the same spatial size as the multispectral image. The transformation process is:

[0044]

[0045] Here, Up(.) represents a spatial upsampling operation, and the upsampling method is bilinear interpolation with a scale factor of 8.

[0046] In the fifth step, we use the upsampled hyperspectral image, high-resolution multispectral image, and the deep features extracted by the detail extraction network encoder to construct a spatial-spectral fusion network encoder. This section uses residual blocks to extract deep features, so we will briefly introduce residual blocks below. The residual block consists of two 3×3 convolutions and a ReLU activation function. This process is expressed as

[0047] RB(X in )=Conv 3×3 (Relu(Conv 3×3 (X in )))+X in

[0048] Among them, X in Represents the input feature, Relu(.) represents the ReLU function, and RB(.) represents the residual block operation. The first output feature of the encoder is the concatenation of the upsampled HS image and the MS image in the channel dimension as the original input, and then the 3×3 convolution and residual block are used to learn the deep features after the concatenation, and finally the deep features are obtained. Next, the maximum pooling feature of the encoder n-1 order output is concatenated with the deep detail feature, and the 3×3 convolution and residual block are used to learn the deep features after the concatenation, to obtain the n-order output feature of the encoder. This process is expressed as

[0049]

[0050] Among them, MUe n represents the output features of the encoder at the nth stage, RB(·) represents the residual operation, and maxpool(.) represents the maximum pooling operation.

[0051] Step 6: Figure 2 As shown in Figure 1, the asymmetric feature selection attention module is used to select the features output by the spatial-spectral fusion network encoder. The following is an example of the asymmetric feature selection attention module based on MUe0. First, the spatial size and channel number of the input deep features are made the same as those of MUe0 by using deconvolution and convolution. The process is expressed as

[0052]

[0053] Then the three obtained depth features are added element by element to obtain the depth feature Su. This process is expressed as

[0054]

[0055] Next, we will use the spatial and channel attention mechanisms to select important features for multi-resolution features.

[0056] In order to obtain the channel attention coefficient of the deep feature Su, we first use the global average pooling operation to extract the global receptive field of Su, and abstract each feature channel into a feature point. This process is expressed as

[0057] Cs=SAvgpool(Su)

[0058] Among them, SAvgpool(·) represents the spatial dimension average pooling operation. Then, a two-layer multi-layer perceptron network is used to perform nonlinear feature transformation to construct the correlation between feature maps. This process is expressed as

[0059] Cz=fc(Relu(fc(Cs)))

[0060] Among them, fc(.) represents the fully connected layer.

[0061] In order to obtain the spatial attention coefficient of the deep feature, we first perform average pooling and maximum pooling on the Su channel dimension to obtain two deep features with the same spatial dimension as the deep feature Su and a channel dimension of 1. Then we concatenate the two deep features on the channel dimension. This process is expressed as

[0062] Ss=Concat(CAvgpool(Su),CMaxpool(Su))

[0063] Among them, CAvgpool(.) represents the channel-dimensional average pooling operation, and CMaxpool(.) represents the channel-dimensional maximum pooling operation. Then, Ss is calculated using 7×7 convolution to obtain the spatial attention coefficient. This process is expressed as

[0064] Sz=Conv 7×7 (Ss)

[0065] Multiply the obtained spatial and channel attention coefficients to obtain the spatial spectral attention coefficient, which is expressed as

[0066] Sc=Cz*Sz

[0067] Three 1×1 convolutions are used to obtain three spatial spectral attention coefficients. This process is expressed as

[0068] SC i =Conv 1×1 (Sc),i∈{0,1,2}

[0069] Among them, Conv 1×1 Represents 1×1 convolution. Then the softmax function is applied to the three attention coefficients to obtain Sa+Sb+Sc=1, which is expressed as

[0070]

[0071] Among them, Sa, Sb and Sc represent the attention coefficients of Su0, Su1 and Su2 respectively. The spatial spectrum attention coefficient is multiplied by the deep feature to obtain the attention module, and the three attention modules are added pixel by pixel to obtain the output of the module. This process is expressed as

[0072] AFSSC1=Sa·Su0+Sb·Su1+Sc·Su2

[0073] AFSSC1 uses MUe0 as the benchmark, and uses deconvolution and convolution operations to make the spatial size and number of channels of MUe1 and MUe2 the same as MUe0, and then adds them to obtain Su. AFSSC0 uses MUe1 as the benchmark, and uses deconvolution or pooling and convolution operations to make the spatial size and number of channels of MUe0 and MUe2 the same as MUe1, and then adds them to obtain Su.

[0074] In the seventh step, the deep features of the asymmetric feature selection attention module, the spatial-spectral fusion network encoder and the detail extraction network decoder are used to construct the spatial-spectral fusion network decoder, and the fusion result is obtained using the ReLU activation function. The deep features of MUe2 are extracted using convolution, residual blocks and deconvolution to obtain the first output of the decoder. The output of the asymmetric feature selection attention module AFSSC is then used. n-1 , the output MUd of the spatial-spectral fusion network encoder n-1 And the detail extraction network decoder outputs WD n-1 Then use 3×3 convolution and residual block, and finally use deconvolution to get the nth output of the spatial spectrum fusion module decoder. This process is expressed as

[0075]

[0076] Among them, MUd n Represents the decoder at stage n.

[0077] The decoder output MUd2 and the detail extracted decoder WD2 are spliced ​​in the channel dimension, and 3×3 convolution is used to extract features. Then the upsampled HS image and the extracted features are added pixel by pixel, and finally the ReLU activation function is used on the addition result to obtain the fused image. This process is represented as

[0078]

[0079] The eighth step is to train the network using the L1 loss function.

[0080]

[0081] in, represents the HR-HSI reconstructed by the network, and Z represents the reference HR-HSI.

[0082] The effect of the present invention can be further illustrated by the following simulation experiments:

[0083] Simulation conditions

[0084] The simulation experiments used two sets of real hyperspectral data: the CAVE dataset and the Harvard dataset. The CAVE dataset consists of 32 indoor hyperspectral images with a spatial size of 512×512, a band range of 400nm-700nm, a spectral resolution of 10nm, and 31 bands. The Harvard dataset consists of 50 indoor and outdoor hyperspectral images with a spatial size of 1040×1392, a band range of 420-720nm, a spectral resolution of 10nm, and 31 bands. To facilitate the acquisition of training data blocks in the experiment, the present invention cuts off a 1024×1024 spatial size from the upper left corner of the image as the training and test sets.

[0085] LR-HS images are generated by applying an 8×8 Gaussian filter (mean 0, standard deviation 2) and downsampling each band of the reference image by 8 pixels vertically and horizontally, i.e., a decimation factor of 8×8. HR-MS images of the same scene are simulated and downsampled to the HR-HS images. The spectral downsampling matrix R uses the response function of a Nikon D700 camera. For the CAVE dataset, the first 20 hyperspectral images are used for training, and the last 12 hyperspectral images are used for testing. For the Harvard dataset, the first 30 hyperspectral images are used for training, and the last 20 hyperspectral images are used for testing. Because deep learning requires a large amount of data as a training set, patches of these training hyperspectral images are used as the training set for deep learning. Each LR-HSI patch is 4×4×31, each HR-MSI patch is 32×32×3, and each HR-HSI patch is 32×32×31.

[0086] The simulation experiments were conducted using Python 3.8 and Pytorch 1.10 under the Windows 10 operating system. Furthermore, the activation function used in this experiment was unified as Relu, and the network was trained using the ADAM optimizer. The learning rate was set to 0.0001, and the number of iterations was fixed at 1000. Other network structure hyperparameter configurations are summarized in Table 1.

[0087] Analysis of simulation experiment results

[0088] Table 2 shows the super-resolution results of the CAVE dataset by the method of the present invention and its comparative method, and Table 3 shows the super-resolution results of the Harvard dataset by the method of the present invention and its comparative method.

[0089] Table 1 Network hyperparameter configuration

[0090]

[0091]

[0092] Table 2 Super-resolution results of the CAVE dataset by the method of the present invention and its comparative method

[0093] Method mpsnr rmse ergas sam uiqi mssim CNMF 34.3027 5.4723 2.6006 7.8920 0.7710 0.9388 HySure 34.7822 5.3232 2.4181 11.5451 0.8043 0.9107 ICCV15 35.6888 4.7694 2.2064 7.8787 0.7986 0.9531 DHSIS 46.2977 1.4653 0.6641 3.8452 0.9242 0.9904 DBIN 45.7831 1.5233 0.6781 3.6035 0.9260 0.9925 CNN-Fus 44.5789 1.8994 0.8689 5.4241 0.8735 0.9850 MOG-DCN 46.3167 1.4217 0.6450 3.5967 0.9278 0.9924 DMW-UNet 47.0245 1.3564 0.6091 3.4703 0.9336 0.9927

[0094] Table 3 Super-resolution results of the method of the present invention and its comparative method on the Harvard dataset

[0095]

[0096]

[0097] Table 2 shows the super-resolution results of the CAVE dataset by the method of the present invention and its comparative method. From Table 2, it can be seen that DMW-UNet has the largest values ​​in the indicators mpsnr, uiqi and mssim, and the smallest values ​​in RMSE, ERGAS and SAM. Figure 3 It can be shown that the reconstruction error of the method of the present invention is the smallest. Figure 4 and Figure 5 It can be seen that the MPSNR and MSSIM values ​​of the method of the present invention are the highest in each band.

[0098] Table 3 shows the super-resolution results of the Harvard dataset by the method of the present invention and its comparative method. From Table 2, it can be seen that DMW-UNet has the largest values ​​in mpsnr, uiqi and mssim, and the smallest values ​​in RMSE, ERGAS and SAM. Figure 6 It can be shown that the reconstruction error of the method of the present invention is the smallest. Figure 7 and Figure 8 It can be seen that the MPSNR and MSSIM values ​​of the method of the present invention are the highest in each band.

[0099] From the above experimental results, it can be seen that the method of the present invention has good spatial and spectral reconstruction capabilities.

Claims

1. A hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection, characterized in that: The following steps are involved: The first step is to use discrete wavelet transform to extract multi-spectral and multi-resolution high-frequency detail features; In the second step, for the high-frequency image W1, a 3×3 convolution is used to obtain the deep features of the first output of the detail extraction encoder. This process is expressed as WE1=Conv 3×3 (W1) Among them, Conv 3×3 (.) represents a convolution operation with a convolution kernel of 3×3; discrete wavelet transform is performed on WE1 to obtain depth detail information, and then concatenated in the channel dimension to obtain W3; high-frequency images W2 and W3 are concatenated in the channel dimension, and a 3×3 convolution is performed to obtain the depth features of the second output of the detail extraction encoder. This process is represented as WE2=Conv 3×3 (Concat(W2,W3)) In the third step, the detail extraction network decoder consists of a deconvolution with a stride of 2 and a 3×3 convolution; the deep features of the three outputs of the decoder are expressed as Among them, Deconv(.) represents deconvolution, WD n Represents the deep features output by the n-th stage detail extraction network decoder; The fourth step is to preprocess the low-resolution hyperspectral image, that is, upsample the low-resolution hyperspectral image to the same spatial size as the multispectral image; The fifth step is to use the residual block to extract the deep features. The residual block consists of two 3×3 convolutions and a ReLU activation function. This process is expressed as RB(X in )=Conv 3×3 (Release(Conv 3×3 (X in )))+X in Among them, X in Represents the input feature, Relu(.) represents the ReLU function, and RB(.) represents the residual block operation; the first output feature of the encoder is the concatenation of the upsampled HS image and the MS image in the channel dimension as the original input, and then the 3×3 convolution and residual block are used to learn the depth feature of the concatenation, and finally the depth feature is obtained; next, the maximum pooling feature of the encoder n-1 order output is concatenated with the depth detail feature, and the 3×3 convolution and residual block are used to learn the depth feature of the concatenation, and the n-order output feature of the encoder is obtained; this process is expressed as Among them, MUe n represents the output features of the encoder at the nth stage, RB(·) represents the residual operation, and maxpool(.) represents the maximum pooling operation; In the sixth step, the asymmetric feature selection attention module is used to select the deep multi-resolution features output by the spatial-spectral fusion network encoder; In the seventh step, convolution, residual block and deconvolution are used to extract deep features from MUe2 to obtain the first output of the decoder; then the output of the asymmetric feature selection attention module AFSSC is used n-1 , the output MUd of the spatial-spectral fusion network encoder n-1 And the detail extraction network decoder outputs WD n-1 Splicing, then using 3×3 convolution and residual blocks, and finally using deconvolution to get the nth output of the spatial spectrum fusion module decoder; this process is expressed as Among them, MUd n Represents the decoder of the nth stage; then the decoder output MUd2 and the detail extracted decoder WD2 are spliced ​​in the channel dimension, and 3×3 convolution is used to extract features. Then the upsampled HS image and the extracted features are added pixel by pixel, and finally the ReLU activation function is used on the addition result to obtain the fused image This process is represented as Step 8: Train the network using the L1 loss function.

2. The hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection according to claim 1, characterized in that: The first step is to use discrete wavelet transform to extract multi-resolution high-frequency detail features of multi-spectrum, as follows: remember Represents a high-resolution multispectral image, where H, W, and b represent the height, width, and number of bands of the high-resolution multispectral image, respectively. Haar discrete wavelet transform is used to extract multi-resolution high-frequency detail information of the multispectral image, where the low-pass filter is represented by Φ and the high-pass filter group is represented by Ψ. The low-frequency subband image with all the low-pass filters in the dth scale row is represented as The high-frequency sub-band images in three directions are represented as in, and denote the conjugates of Φ and Ψ, respectively. Represents the d-th scale C and separable filter Convolution, C d-1 represents the low-pass subband image at the d-1th scale, and Represents the high-frequency sub-band images in the horizontal, vertical and diagonal directions at the dth scale respectively; the multispectral image is subjected to discrete wavelet transform to obtain the low-frequency sub-band image C1 and the high-frequency sub-band image and And these three high-frequency sub-band images are spliced ​​in the channel dimension to obtain the high-frequency image W1, which is expressed as Concat(.) represents the channel-dimensional concatenation operation; the low-frequency sub-band image C1 of the multispectral image is transformed using discrete wavelet transform to obtain the low-frequency sub-band image C2 and the high-frequency sub-band image C2. and And these three high-frequency sub-band images are spliced ​​in the channel dimension to obtain the high-frequency image W2, which is expressed as 3. The hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection according to claim 1, characterized in that: The fourth step is to preprocess the low-resolution hyperspectral image, that is, upsample the low-resolution hyperspectral image to the same spatial size as the multispectral image. The transformation process is: Here, Up(.) represents a spatial upsampling operation, and the upsampling method is bilinear interpolation with a scale factor of 8.

4. The hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection according to claim 1, characterized in that: In the sixth step, the asymmetric feature selection attention module is used to select features output by the spatial-spectral fusion network encoder; For the asymmetric feature selection attention module based on MUe0, first, deconvolution and convolution are used to make the spatial size and channel number of the input deep features the same as MUe0. The process is expressed as Then the three obtained depth features are added element by element to obtain the depth feature Su. This process is expressed as In order to obtain the channel attention coefficient of the deep feature Su, we first use the global average pooling operation to extract the global receptive field of Su, and abstract each feature channel into a feature point. This process is expressed as Cs=SAvgpool(Su) Among them, SAvgpool(·) represents the spatial dimension average pooling operation; then the two-layer multi-layer perceptron network is used to perform nonlinear feature transformation to construct the correlation between feature maps. This process is expressed as Cz=fc(Relu(fc(Cs))) Among them, fc(.) represents the fully connected layer; In order to obtain the spatial attention coefficient of the deep feature, we first perform average pooling and maximum pooling on the Su channel dimension to obtain two deep features with the same spatial dimension as the deep feature Su and a channel dimension of 1; then we concatenate the two deep features on the channel dimension. This process is expressed as Ss=Concat(CAvgpool(Su),CMaxpool(Su)) Among them, CAvgpool(.) represents the channel-dimensional average pooling operation, and CMaxpool(.) represents the channel-dimensional maximum pooling operation; then Ss is calculated using 7×7 convolution to obtain the spatial attention coefficient. This process is expressed as Sz=Conv 7×7 (Ss) Multiply the obtained spatial and channel attention coefficients to obtain the spatial spectral attention coefficient, which is expressed as Sc=Cz*Sz Three 1×1 convolutions are used to obtain three spatial spectral attention coefficients. This process is expressed as SC i =Conv 1×1 (Sc),i∈{0,1,2} Among them, Conv 1×1 represents a 1×1 convolution; then the softmax function is applied to the three attention coefficients to obtain Sa+Sb+Sc=1, which is expressed as Among them, Sa, Sb and Sc represent the attention coefficients of Su0, Su1 and Su2 respectively; the obtained spatial spectrum attention coefficient is multiplied by the deep feature to obtain the attention module, and the three attention modules are added pixel by pixel to obtain the output of the module. This process is expressed as AFSSC1=Sa·Su0+Sb·Su1+Sc·Su2 Among them, AFSSC1 is based on MUe0, and uses deconvolution and convolution operations to make the spatial size and number of channels of MUe1 and MUe2 the same as MUe0, and then adds them to obtain Su; while AFSSC0 is based on MUe1, and uses deconvolution or pooling and convolution operations to make the spatial size and number of channels of MUe0 and MUe2 the same as MUe1, and then adds them to obtain Su.

5. The hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection according to claim 1, characterized in that: The eighth step is to train the network using the L1 loss function, specifically in, represents the HR-HSI reconstructed by the network, and Z represents the reference HR-HSI.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection as described in any one of claims 1 to 5 is implemented.

Images