Progressive hyperspectral image super-resolution method based on grouping strategy

Abstract

The present application discloses a progressive hyperspectral image super-resolution method based on a grouping strategy, which relates to the field of new generation information technology. The progressive hyperspectral image super-resolution method based on a grouping strategy described in the present application realizes a progressive improvement in the spatial resolution of the feature map through the fusion of three stages, and fully fuses the spatial and spectral information in the multispectral image and the spatial and spectral information in the hyperspectral image in each stage to obtain a reconstructed high-resolution hyperspectral image. Compared with the prior art, the reconstructed high-resolution hyperspectral image obtained by the hyperspectral image super-resolution method described in the present application contains more spatial texture information and spectral information, and the reconstructed high-resolution hyperspectral image is closer to the true value image.

Description

Technical Field

[0001] The invention belongs to the field of new generation information technology and relates to a progressive hyperspectral image super-resolution method based on a grouping strategy. Background Art

[0002] Hyperspectral image super-resolution technology has important application value in the fields of remote sensing monitoring, precision agriculture and environmental monitoring. Hyperspectral images have rich spectral information and can accurately depict material properties, but their spatial resolution is low; while multispectral images have high spatial resolution, their spectral resolution is limited. Existing hyperspectral image super-resolution methods based on deep learning can fuse the complementary information of hyperspectral images and multispectral images, but the fusion of spatial information and spectral information is not sufficient, and the reconstructed high-resolution hyperspectral image is not close to the true image. To address this problem, this application proposes a progressive hyperspectral image super-resolution method based on a grouping strategy. Summary of the invention

[0003] In view of the defects and shortcomings in the above-mentioned prior art, the present application proposes a progressive hyperspectral image super-resolution method based on a grouping strategy.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] A progressive hyperspectral image super-resolution method based on a grouping strategy comprises the following steps:

[0006] S1, obtain a test image and a series of images for training;

[0007] S2, constructing a progressive hyperspectral image super-resolution network based on a grouping strategy; the progressive hyperspectral image super-resolution network based on a grouping strategy includes a fusion module I, an upsampling module I, a first Add layer, a convolution layer I, a fusion module II, an upsampling module II, a second Add layer connected in sequence Layer, convolution layer II, fusion module III; fusion module I is used to fuse the spatial information and spectral information of the 2-fold upsampled hyperspectral image and the 4-fold downsampled multispectral image, and obtain a feature map with rich spatial information and spectral information of the same spatial resolution as the 2-fold upsampled hyperspectral image. This is the initial fusion of the first stage; upsampling module I is used to upsample the feature map output by fusion module I and increase its spatial scale by 2 times to obtain a feature map with the same resolution as the 4-fold upsampled hyperspectral image and spectral information; the first Add layer is used to add the features of the 4-fold upsampled hyperspectral image and the feature map output by upsampling module I to obtain a feature map with enhanced spectral information. Convolution layer I is used to perform a convolution operation on the feature map output by the first Add layer to obtain a feature map with spectral information integrated in the 4-fold upsampled hyperspectral image; fusion module II performs a convolution operation on the feature map output by convolution layer I The output feature map and the spatial information and spectral information in the 2-fold downsampled multispectral image are fused in the second stage to obtain a feature map with enhanced feature information; the upsampling module II is used to upsample the feature map output by the fusion module II and increase its spatial scale by 2 times to obtain a feature map with the same resolution as the 8-fold upsampled hyperspectral image and spectral information; the second Add layer is used to add the features of the 8-fold upsampled hyperspectral image and the feature map output by the upsampling module II to obtain a feature map with further enhanced spectral information; the convolution layer II is used to perform a convolution operation on the feature map output by the second Add layer to obtain a feature map with spectral information integrated in the 8-fold upsampled hyperspectral image; the fusion module III performs a third stage fusion on the feature map output by the convolution layer II and the spatial information and spectral information in the multispectral image used to input the fusion module III to obtain a reconstructed high-resolution hyperspectral image;

[0008] S3. Total loss of progressive hyperspectral image super-resolution network using grouping strategy and training a progressive hyperspectral image super-resolution network based on a grouping strategy using a series of images for training, thereby obtaining a progressive hyperspectral image super-resolution network model based on a grouping strategy;

[0009] S4. The hyperspectral image to be processed and the multispectral image to be processed are input into the fusion module I in the progressive hyperspectral image super-resolution network model based on the grouping strategy, and forward propagated once along the progressive hyperspectral image super-resolution network model based on the grouping strategy to obtain the hyperspectral image after super-resolution. The hyperspectral image after super-resolution is the reconstructed high-resolution hyperspectral image.

[0010] Preferably, in step S2, the structures of fusion module II and fusion module III are the same as those of fusion module I, the structures of convolution layer I and convolution layer II are the same, and the structures of upsampling module I and upsampling module II are the same.

[0011] Preferably, in step S2, fusion module I includes a Concat layer, a spatial-spectral information fusion module and an Add layer connected in parallel, the input end of the Concat layer is also connected to the input end of the multi-scale feature extraction module, and the output end of the Concat layer is sequentially connected to the convolution layer III (the convolution kernel size is 3×3 and the step size is 1), the spatial-spectral information fusion module, the spatial information enhancement module, the convolution layer IV (the convolution kernel size is 3×3 and the step size is 1) and the Add layer, wherein the output end of the multi-scale feature extraction module is also connected to the input end of the spatial information enhancement module.

[0012] Preferably, in step S2, the spatial spectrum information fusion module in the fusion module I includes a grouped convolution layer I, a grouped convolution layer II, a grouped convolution layer III, a Transformer module I, a Transformer module II, a Transformer module III, a Concat layer and a convolution layer, the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are connected in parallel, and the output ends of the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are connected to the Transformer module I, the Transformer module II and the Transformer module III respectively, and the Transformer module I is connected to the Transformer module II. The output ends of the mer module I, the Transformer module II and the Transformer module III are all connected to the Concat layer, and the Concat layer is connected to the convolution layer (the step size is 1 and the convolution kernel size is 3×3); in this application, the input end of the Transformer module I, the input end of the Transformer module II and the input end of the Transformer module III are all connected to the output end of the convolution layer III in the fusion module I; the input ends of the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are all connected to the input end of the Concat layer in the fusion module I.

[0013] Preferably, in step S2, the number of groups in the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are 2, 4 and 8 respectively, and the convolution kernel sizes of the convolution layers in the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are all 3×3.

[0014] Preferably, in step S2, the network structures of Transformer module I, Transformer module II and Transformer module III are the same.

[0015] Preferably, in step S2, the Transformer I module includes three Reshape layers, the input end of the first Reshape layer is connected to the output end of the grouped convolution layer I, the spectral feature map output by the grouped convolution layer I is input to the first Reshape layer, the input end of the second Reshape layer is connected to the output end of the convolution layer III in the fusion module I, and the spatial feature map output by the convolution layer III is input to the second Reshape layer; the first Reshape layer is respectively connected to the Value linear layer I, the Key linear layer I and the Query linear layer I, the second Reshape layer is respectively connected to the Value linear layer II, the Key linear layer II and the Query linear layer II, the Query linear layer I and the Query linear layer II are both connected to the first Add layer, The first Add layer and Key linear layer I are both connected to the first matrix multiplication unit, the first Add layer and Key linear layer II are both connected to the second matrix multiplication unit, the first matrix multiplication unit is connected to the first Softmax layer, the second matrix multiplication unit is connected to the second Softmax layer, the first Softmax layer and Value linear layer I are both connected to the third matrix multiplication unit, the second Softmax layer and Value linear layer II are both connected to the fourth matrix multiplication unit, the third matrix multiplication unit and the fourth matrix multiplication unit are both connected to the second Add layer, the second Add layer is connected to the normalization layer, the feedforward network, the third Add layer and the third Reshape layer in sequence, and the output of the second Add layer is also connected to the input of the third Add layer.

[0016] Preferably, in step S2, the multi-scale feature extraction module includes a dilated convolution layer I, a dilated convolution layer II, a dilated convolution layer III, a dilated convolution layer IV and a spatial attention module in parallel, and the dilated convolution layer I, the dilated convolution layer II, the dilated convolution layer III, the dilated convolution layer IV and the spatial attention module are all connected to the Concat layer, and the Concat layer is sequentially connected to the channel attention module and the convolution layer (the convolution kernel size is 3×3).

[0017] Preferably, in step S2, the dilated convolution layer I, the dilated convolution layer II, the dilated convolution layer III and the dilated convolution layer IV are convolution layers with a convolution kernel of 3×3 and a dilation rate of 1, 4, 8 and 12 respectively.

[0018] Preferably, in step S2, the spatial information enhancement module includes a first element-by-element multiplication module and a first Add layer connected in parallel and a second element-by-element multiplication module and a second Add layer connected in parallel, the input end of the first element-by-element multiplication module and the input end of the second element-by-element multiplication module are both connected to the input end of the Concat layer, the output end of the Concat layer is sequentially connected to the average pooling layer, the first convolution layer (the convolution kernel size is 3×3), the second convolution layer (the convolution kernel size is 3×3) and the first Sigmoid activation layer, the output end of the first Sigmoid activation layer is respectively connected to the first element-by-element multiplication module and the second element-by-element multiplication module, the output end of the first element-by-element multiplication module is connected to the input end of the first Add layer, and the output end of the second element-by-element multiplication module is connected to the input end of the first Add layer. The output ends of the two element-by-element multiplication modules are connected to the input end of the second Add layer, the output end of the second Add layer is connected to the third convolution layer (the convolution kernel size is 3×3), the third convolution layer is respectively connected to the average pooling layer and the first convolution unit, the output end of the average pooling layer is connected to the second convolution unit, the output ends of the first convolution unit and the second convolution unit are both connected to the third element-by-element multiplication module, the output end of the third element-by-element multiplication module is connected to the second Sigmoid activation layer, the output end of the second Sigmoid activation layer and the output end of the first Add layer are both connected to the fourth element-by-element multiplication module, the output end of the first Add layer and the output end of the fourth element-by-element multiplication module are also connected to the input end of the third Add layer.

[0019] Preferably, in step S2, the first convolution unit and the second convolution unit have the same structure, and the first convolution unit includes two sequentially connected convolution layers (the convolution kernel size is 3×3).

[0020] Preferably, step S3 specifically includes the following steps: inputting the series of images for training into the progressive hyperspectral image super-resolution network based on the grouping strategy, performing forward propagation, and calculating the total loss of the progressive hyperspectral image super-resolution network based on the grouping strategy. , and in the total loss Back propagation is carried out under the guidance of , the weight parameters of the progressive hyperspectral image super-resolution network based on grouping strategy are updated, and the training process of one epoch is completed. After iterating the training process of 500 epochs, the training of the progressive hyperspectral image super-resolution network based on grouping strategy is completed, and the progressive hyperspectral image super-resolution network model based on grouping strategy is obtained.

[0021] Preferably, in step S3, the loss of the progressive hyperspectral image super-resolution network based on the grouping strategy is Predicted image output by the progressive hyperspectral image super-resolution network with grouping strategy With the true value image The loss between the two, the loss of the progressive hyperspectral image super-resolution network based on the grouping strategy The calculation method is as shown in formula (1):

[0022] (1)

[0023] In formula (1), N is the number of image pairs used for training.

[0024] Compared with the prior art, the beneficial technologies of this application are:

[0025] This application realizes a progressive improvement in the spatial resolution of the feature map through the fusion of three stages, and fully fuses the spatial and spectral information in the multispectral image and the spatial and spectral information in the hyperspectral image at each stage to obtain a reconstructed high-resolution hyperspectral image. Compared with the prior art, the reconstructed high-resolution hyperspectral image obtained by the hyperspectral image super-resolution method described in this application contains more spatial texture information and spectral information, and the reconstructed high-resolution hyperspectral image is closer to the true value image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the network structure of progressive hyperspectral image super-resolution based on grouping strategy in this application;

[0027] Figure 2 for Figure 1 Schematic diagram of the network structure of the fusion module I;

[0028] Figure 3 for Figure 2 Schematic diagram of the network structure of the hollow spectrum information fusion module;

[0029] Figure 4 for Figure 3 Schematic diagram of the network structure of Transformer module I;

[0030] Figure 5 for Figure 2 Schematic diagram of the network structure of the multi-scale feature extraction module;

[0031] Figure 6 for Figure 2 Schematic diagram of the network structure of the spatial information enhancement module;

[0032] Figure 7 The images obtained by testing the SSRNet method, DSPNet method, MoG-DCN method, and 3DTNet method in this application; Figure 7In the figure, the first row is the pseudo-color image of the CAVE dataset, the second row is the error image of the CAVE dataset, the third row is the pseudo-color image of the Chikusei dataset, and the fourth row is the error image of the Chikusei dataset;

[0033] Figure 8 The image obtained by testing the method described in this application and the existing super-resolution method such as the MIMO-SST method, GT represents the true value image; Figure 8 In the figure, the first row is the pseudo color image of the CAVE dataset, the second row is the error image of the CAVE dataset, the third row is the pseudo color image of the Chikusei dataset, and the fourth row is the error image of the Chikusei dataset. DETAILED DESCRIPTION

[0034] A method for hyperspectral image super-resolution based on deep learning, comprising the following steps:

[0035] S1, obtaining a test image and a series of images for training; wherein step S1 specifically includes the following specific steps:

[0036] S1-1, obtain CAVE dataset and Chikusei dataset;

[0037] The CAVE dataset in this application is an existing dataset, and its acquisition URL is: https: / / www.cs.columbia.edu / CAVE / databases / multispectral / , where the CAVE dataset contains 32 multispectral images of different scenes, and the size of each image is 512×512×31; in the field of hyperspectral image super-resolution research, the multispectral images in the CAVE dataset are usually regarded as hyperspectral images, for example, the paper "Learning a 3D-CNN and Transformer prior for hyperspectral image super-resolution" and the paper "Fusformer: A Transformer-Based Fusion Network for Hyperspectral Image Super-Resolution" regard the multispectral images in the CAVE dataset as hyperspectral images; therefore, the multispectral images in the CAVE dataset are also regarded as hyperspectral images in this application;

[0038] The Chikusei dataset is an existing dataset, which can be obtained from the URL: https: / / naotoyokoya.com / Download.html. The Chikusei dataset contains a hyperspectral image with a size of 2517×2335×128.

[0039] S1-2, respectively, performs data simulation processing on the hyperspectral images in the CAVE dataset and the Chikusei dataset to obtain test images and a series of images for training, which specifically includes the following steps:

[0040] (1) The hyperspectral images in the CAVE dataset and the Chikusei dataset are subjected to spectral degradation processing to obtain high-resolution multispectral images;

[0041] (2) The hyperspectral images in the CAVE dataset and the Chikusei dataset are subjected to spatial degradation processing respectively to obtain low-resolution hyperspectral images; wherein, the high-resolution multispectral image and the low-resolution hyperspectral image obtained by performing spectral degradation processing and spatial degradation processing on each hyperspectral image in the CAVE dataset are a pair of preprocessed images; the high-resolution multispectral image and the low-resolution hyperspectral image obtained by performing spectral degradation processing and spatial degradation processing on the hyperspectral images in the Chikusei dataset are a pair of preprocessed images;

[0042] (3) The first 20 pairs of preprocessed images obtained based on the CAVE dataset are used as training images, and the remaining 12 pairs of preprocessed images are used as test images;

[0043] Three image blocks are cut out from the corresponding positions of a pair of preprocessed images obtained based on the Chikusei dataset (the pair of preprocessed images includes a multispectral image and a corresponding hyperspectral image) as test images, and the remaining parts of the pair of preprocessed images obtained based on the Chikusei dataset are used as training images; wherein, the image block size cut out of the multispectral image in the pair of preprocessed images obtained based on the Chikusei dataset is 512×512, and the image block size cut out of the hyperspectral image is 64×64;

[0044] The multispectral image and the hyperspectral image in the above training image are segmented to obtain a hyperspectral training sub-image of size 12×12 and a multispectral training sub-image of size 96×96; the multispectral training sub-image is downsampled to obtain a 2-fold downsampled multispectral image and a 4-fold downsampled multispectral image, respectively. The bicubic interpolation algorithm is used for downsampling in this application; the hyperspectral training sub-image is upsampled to obtain a 2-fold upsampled hyperspectral image, a 4-fold upsampled hyperspectral image and an 8-fold upsampled hyperspectral image, respectively. The bicubic interpolation algorithm is used for upsampling in this application;

[0045] The above-mentioned 2x upsampled hyperspectral image, 4x upsampled hyperspectral image, 8x upsampled hyperspectral image, multispectral training sub-image, 2x downsampled multispectral image and 4x downsampled multispectral image constitute a series of images for training.

[0046] Spectral degradation refers to the application of spectral response function to reduce the channel dimension of hyperspectral images. The spectral response function of CAVE dataset can be obtained from: https: / / maxmax.com / nikon_d700_study.htm, and the spectral response function of Chikusei dataset can be obtained from: https: / / www.satimagingcorp.com / satellite-sensors / ikonos / ;

[0047] Spatial degradation means that the hyperspectral image is first blurred using the cv2.GaussianBlur() function in the OpenCV library, where the Gaussian blur kernel size of the cv2.GaussianBlur() function is 8×8 and the standard deviation of the cv2.GaussianBlur() function is 2; then, the bicubic interpolation algorithm is used for downsampling at an 8-fold ratio.

[0048] S2, construct a progressive hyperspectral image super-resolution network based on grouping strategy;

[0049] The structure of the progressive hyperspectral image super-resolution network based on the grouping strategy is as follows: Figure 1 As shown, Figure 1 middle, Denotes the Add layer. The progressive hyperspectral image super-resolution network based on the grouping strategy includes a fusion module I, an upsampling module I, a first Add layer, a convolution layer I, a fusion module II, an upsampling module II, a second Add layer, a convolution layer II, and a fusion module III connected in sequence; the structures of the fusion module II and the fusion module III are the same as those of the fusion module I, and the functions of the fusion module II and the fusion module III are the same as those of the fusion module I; the structures of the convolution layer I and the convolution layer II are the same, the functions of the convolution layer I and the convolution layer II are the same, and the convolution kernel of the convolution layer I is 3×3 and the step size is 1.

[0050] The fusion module I is used to fuse the spatial information and spectral information of the input 2x upsampled hyperspectral image and 4x downsampled multispectral image, and obtain a feature map with rich spatial information and spectral information of the same spatial resolution as the 2x upsampled hyperspectral image, which is the initial fusion of the first stage; the upsampling module I is used to upsample the feature map output by the fusion module I and increase its spatial scale by 2 times, so as to obtain a feature map with the same resolution as the 4x upsampled hyperspectral image and spectral information; the first Add layer is used to add the features of the 4x upsampled hyperspectral image and the feature map output by the upsampling module I to obtain a feature map with enhanced spectral information, and the convolution layer I is used to perform a convolution operation on the feature map output by the first Add layer to obtain a feature map with spectral information integrated in the 4x upsampled hyperspectral image; the fusion module II performs the second stage fusion of the spatial information and spectral information in the feature map output by the convolution layer I and the 2x downsampled multispectral image. Specifically, the fusion module II further fuses the spatial information and spectral information of the input 2x downsampled multispectral image and the feature map output by the convolution layer I to obtain a feature map with enhanced feature information; the upsampling module II is used to perform a convolution operation on the feature map output by the first Add layer to obtain a feature map with spectral information integrated in the 4x upsampled hyperspectral image. The feature map output by the fusion module II is upsampled and its spatial scale is increased by 2 times to obtain a feature map with the same resolution as the 8-fold upsampled hyperspectral image and spectral information. The second Add layer is used to add the input 8-fold upsampled hyperspectral image and the feature map output by the upsampling module II to obtain a feature map with further enhanced spectral information. The convolution layer II is used to perform a convolution operation on the feature map output by the second Add layer to obtain a feature map with spectral information integrated in the 8-fold upsampled hyperspectral image. The fusion module III performs a third-stage fusion of the spatial information and spectral information in the feature map output by the convolution layer II and the multispectral image input to the fusion module III. Specifically, the fusion module III further fuses the spatial information and spectral information of the input multispectral image and the feature map output by the convolution layer II to obtain a reconstructed high-resolution hyperspectral image. The present application realizes a progressive improvement in the spatial resolution of the feature map through the fusion of three stages, and fuses the spatial information and spectral information in the multispectral image and the spatial information and spectral information in the hyperspectral image in each fusion stage, and finally obtains a reconstructed high-resolution hyperspectral image, achieving a progressive optimization effect.

[0051] In the present application, the upsampling module I and the upsampling module II have the same structure and function. The upsampling module I has the same structure and function as the sub-pixel convolution layer module shown in the paper "Real-Time Single Image and Video Super-Resolution Using an EfficientSub-Pixel Convolutional Neural Network".

[0052] The network structure of the fusion module I in this application is as follows: Figure 2 As shown, Figure 2 middle, Represents the Concat layer, Represents the Add layer. Fusion module I includes a parallel Concat layer, a spatial-spectral information fusion module, and an Add layer, that is, the input end of the Concat layer, the input end of the spatial-spectral information fusion module, and the input end of the Add layer are connected, the 2-fold upsampled hyperspectral image is the image input to the Concat layer, the spatial-spectral information fusion module, and the Add layer, the input end of the Concat layer is also connected to the input end of the multi-scale feature extraction module, the 4-fold downsampled multispectral image is the image input to the Concat layer and the multi-scale feature extraction module, the output end of the Concat layer is sequentially connected to the convolution layer III (the convolution kernel size is 3×3, the step size is 1), the spatial-spectral information fusion module, the spatial information enhancement module, the convolution layer IV (the convolution kernel size is 3×3, the step size is 1), and the Add layer, among which the output end of the multi-scale feature extraction module is also connected to the input end of the spatial information enhancement module;

[0053] The Concat layer is used to concatenate the features of the 2-fold upsampled hyperspectral image and the 4-fold downsampled multispectral image to obtain a feature map that preliminarily integrates the hyperspectral and multispectral image information. The convolution layer III is used to extract spatial information from the feature map output by the Concat layer to obtain a spatial feature map with rich spatial details. The spatial-spectral information fusion module is used to use a grouping strategy to extract the features of adjacent spectral bands from the spatial feature map output by the convolution layer III and the 2-fold upsampled hyperspectral image, and to fuse the sufficient spatial information and spectral information to obtain a fused feature map with rich spatial and spectral information. The multi-scale feature extraction module is used to extract local and global spatial detail information from the 4-fold downsampled multispectral image to obtain a fused feature map with multi-scale spatial information. The spatial information enhancement module is used to fuse and enhance the feature map output by the spatial-spectral information fusion module and the feature map output by the multi-scale feature extraction module, so that the multi-scale spatial information in the multispectral image is injected into the fused features to obtain the feature map with enhanced representation of spatial detail information; the convolution layer IV is used to adjust the number of channels of the feature map output by the spatial information enhancement module to be consistent with the number of channels of the 2-fold upsampled hyperspectral image; the Add layer is used to add the features of the 2-fold upsampled hyperspectral image and the feature map output by the convolution layer IV to better retain the spectral information of the 2-fold upsampled hyperspectral image, and obtain a feature map with rich spatial and spectral information with the same spatial resolution as the 2-fold upsampled hyperspectral image.

[0054] The network structure of the spatial spectrum information fusion module in the fusion module I is as follows: Figure 3 As shown, Figure 3 middle, In the present application, the spatial-spectral information fusion module includes a grouped convolution layer I, a grouped convolution layer II, a grouped convolution layer III, a Transformer module I, a Transformer module II, a Transformer module III, a Concat layer and a convolution layer. The grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are connected in parallel, that is, the input ends of the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are connected, and the output ends of the grouped convolution layer I, the output ends of the grouped convolution layer II and the output ends of the grouped convolution layer III are connected to the Transformer module I, the Transformer module II and the Transformer module III respectively. The Transformer module I, the Transformer module II and the Transformer module III are connected to the Transformer module I, the Transformer module II and the Transformer module III respectively. The output of Transformer module Ⅲ is connected to the Concat layer, and the Concat layer is connected to the convolution layer (with a step size of 1 and a convolution kernel size of 3×3); in this application, the input of Transformer module Ⅰ, the input of Transformer module Ⅱ and the input of Transformer module Ⅲ are all connected to the output of convolution layer Ⅲ in fusion module Ⅰ; in this application, the input of grouped convolution layer Ⅰ, grouped convolution layer Ⅱ and grouped convolution layer Ⅲ are all connected to the input of the Concat layer in fusion module Ⅰ, for inputting a 2x upsampled hyperspectral image; in this application, the number of groups in grouped convolution layer Ⅰ, grouped convolution layer Ⅱ and grouped convolution layer Ⅲ are 2, 4 and 8 respectively, and the convolution kernel size of the convolution layer in grouped convolution layer Ⅰ, grouped convolution layer Ⅱ and grouped convolution layer Ⅲ is 3×3. Transformer module Ⅰ, Transformer module Ⅱ and Transformer module Ⅲ have the same network structure and function.

[0055] In the present application, the network structure of the spatial-spectral information fusion module is designed based on the grouping strategy. Specifically, in the present application, the double-upsampled hyperspectral image is parallelly extracted through the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III to extract the features of adjacent spectral bands, and obtain the feature maps with spectral features respectively. In the present application, by setting the number of groups of the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III to 2, 4 and 8 respectively, the multi-scale extraction of spectral features is effectively realized; the feature maps with spectral features obtained by the grouped convolution layer I, the grouped convolution layer II and the grouped convolution layer III are respectively input into the Transformer module I, the Transformer module II and the Transformer module III, and the Transformer module I, the Transformer module II and the Transformer module III are respectively input into the Transformer module I, the Transformer module II and the Transformer module III. The former module III also inputs the spatial feature map output by the convolution layer III in the fusion module I, and the Transformer module I, Transformer module II and Transformer module III respectively fuse the global spatial information and spectral information in the input feature map; then, the feature maps output by the Transformer module I, Transformer module II and Transformer module III are spliced ​​in the channel dimension through the Concat layer, and the global spectral features and global spatial features extracted from different groups are integrated to obtain a feature map with rich global spectral features and global spatial features; the convolution layer is used to fuse the spatial spectral information of different groups in the feature map output by the Concat layer to obtain a fused feature map with rich spatial and spectral information.

[0056] In this application, the network structure of the TransformerⅠ module is as follows: Figure 4 As shown; Figure 4 middle, Indicates the Add layer, Indicates matrix multiplication. Transformer module I, Transformer module II and Transformer module have the same structure and function. This application takes Transformer I module as an example to introduce the network structure and function of Transformer I module, as follows:

[0057] The Transformer Ⅰ module includes three Reshape layers. The input of the first Reshape layer is connected to the output of the grouped convolution layer Ⅰ, and the spectral feature map output by the grouped convolution layer Ⅰ is input to the first Reshape layer. The input of the second Reshape layer is connected to the output of the convolution layer Ⅲ in the fusion module Ⅰ, and the spatial feature map output by the convolution layer Ⅲ is input to the second Reshape layer. The first Reshape layer is connected to the Value linear layer Ⅰ, the Key linear layer Ⅰ and the Query linear layer Ⅰ respectively, and the second Reshape layer is connected to the Value linear layer Ⅱ, the Key linear layer Ⅱ and the Query linear layer Ⅱ respectively. The Query linear layer Ⅰ and the Query linear layer Ⅱ are both connected to the first Add layer, and the first Ad The d layer and Key linear layer I are both connected to the first matrix multiplication unit, the first Add layer and Key linear layer II are both connected to the second matrix multiplication unit, the first matrix multiplication unit is connected to the first Softmax layer, the second matrix multiplication unit is connected to the second Softmax layer, the first Softmax layer and Value linear layer I are both connected to the third matrix multiplication unit, the second Softmax layer and Value linear layer II are both connected to the fourth matrix multiplication unit, the third matrix multiplication unit and the fourth matrix multiplication unit are both connected to the second Add layer, the second Add layer is connected to the normalization layer, the feedforward network, the third Add layer and the third Reshape layer in sequence, and the output of the second Add layer is also connected to the input of the third Add layer.

[0058] For the TransformerⅡ module, the input of the first Reshape layer in the TransformerⅡ module is connected to the output of the grouped convolutional layer Ⅱ, the spectral feature map output by the grouped convolutional layer Ⅱ is input to the first Reshape layer, the input of the second Reshape layer is connected to the output of the convolutional layer Ⅲ in the fusion module Ⅰ, and the spatial feature map output by the convolutional layer Ⅲ is input to the second Reshape layer;

[0059] For the Transformer III module, the input of the first Reshape layer in the Transformer III module is connected to the output of the grouped convolutional layer III, the spectral feature map output by the grouped convolutional layer III is input to the first Reshape layer, the input of the second Reshape layer is connected to the output of the convolutional layer III in the fusion module I, and the spatial feature map output by the convolutional layer III is input to the second Reshape layer.

[0060] The functions of the TransformerⅠ module are as follows:

[0061] The first Reshape layer reshapes the spectral feature map output by the grouped convolution layer I to calculate the self-attention in the channel dimension. The Query Linear Layer I, Key Linear Layer I and Value Linear Layer I respectively perform linear transformation on the feature map output by the first Reshape layer to generate the query matrix, key matrix and value matrix of the spectral information respectively. The second Reshape layer reshapes the spatial feature map output by the convolution layer III in the fusion module I to calculate the self-attention in the channel dimension. The Query Linear Layer II, Key Linear Layer II and Value Linear Layer II respectively perform linear transformation on the feature map output by the second Reshape layer. The query matrix, key matrix and value matrix of spatial information are generated by the transformation of the property, respectively; the first Add layer is used to add the query matrix output by the Query linear layer I and the query matrix output by the Query linear layer II element by element, which is helpful to integrate the query information of spectral features and spatial features, and obtain the joint query matrix of spatial information and spectral information; the first matrix multiplication unit performs matrix multiplication on the joint query matrix output by the first Add layer and the key matrix output by the Key linear layer I, and the second matrix multiplication unit performs matrix multiplication on the joint query matrix output by the first Add layer and the key matrix output by the Key linear layer II. The first Softmax layer and the second Softmax layer are respectively used to perform matrix multiplication on the joint query matrix output by the first Add layer and the key matrix output by the Key linear layer II. The matrices output by the first matrix multiplication unit and the second matrix multiplication unit are processed separately. The third matrix multiplication unit performs matrix multiplication on the matrix output by the first Softmax layer and the value matrix output by the Value layer I to calculate the attention weights, and obtains a matrix with a global spatial position relationship. The fourth matrix multiplication unit performs matrix multiplication on the matrix output by the second Softmax layer and the value matrix output by the Value layer II to calculate the attention weights, and obtains a matrix with global spectral information. The second Add layer adds the matrix output by the third matrix multiplication unit and the matrix output by the fourth matrix multiplication unit to fuse the global spatial information and spectral information to obtain a matrix with global spectral information. There is a feature map with global spatial position relationship and global spectral information; the normalization layer is used to perform layer normalization operation on the feature map output by the second Add layer, the feedforward network is used to perform nonlinear transformation operation on the feature map output by the normalization layer, the third Add layer is used to perform feature addition operation on the feature map output by the second Add layer and the feature map output by the feedforward network, and the third Reshape layer is used to perform Reshape operation on the feature map output by the third Add layer to obtain a feature map with the same dimension as the spectral feature map input by Transformer module I (i.e., the spectral feature map output by the grouped convolution layer I) and the spatial feature map (i.e., the spectral feature map output by the convolution layer of the fusion module I).In this application, the feedforward network has the same structure and function as the MLP module disclosed in the paper "Multiscale spatial-spectral transformer network for hyperspectral and multispectral image fusion".

[0062] In this application, the network structure of the multi-scale feature extraction module is as follows: Figure 5 As shown, Figure 5 middle, Denotes the Concat layer. The multi-scale feature extraction module includes the parallel-connected dilated convolution layer I, dilated convolution layer II, dilated convolution layer III, dilated convolution layer IV and spatial attention module. The dilated convolution layer I, dilated convolution layer II, dilated convolution layer III, dilated convolution layer IV and spatial attention module are all connected to the Concat layer, which is sequentially connected to the channel attention module and the convolution layer (the convolution kernel size is 3×3); among them, the dilated convolution layer I, dilated convolution layer II, dilated convolution layer III and dilated convolution layer IV are convolution layers with a convolution kernel size of 3×3 and a dilation rate of 1, 4, 8 and 12 respectively.

[0063] In the present application, the atrous convolution layer I, atrous convolution layer II, atrous convolution layer III, and atrous convolution layer IV are arranged in parallel to capture the feature information of the multispectral image under different receptive fields for 4 times downsampled, thereby realizing multi-scale feature extraction; and the spatial attention module arranged in parallel with the atrous convolution layer I, atrous convolution layer II, atrous convolution layer III, and atrous convolution layer IV effectively improves the feature representation capability of the multispectral image by enhancing the feature response of important spatial positions; the Concat layer is used to perform feature concatenation on the feature maps output by the atrous convolution layer I, atrous convolution layer II, atrous convolution layer III, atrous convolution layer IV, and the spatial attention module to obtain a feature map with multi-scale spatial information; the channel attention module is used to perform weighted operations on the feature maps output by the Concat layer to highlight important channel features and obtain a feature map with enhanced channel information; the convolution layer is used to perform smoothing operations on the feature maps output by the channel attention module to obtain a feature map with rich multi-scale spatial information. In this application, the spatial attention module has the same structure and function as the Spatial Attention Module module shown in the paper "Cbam: Convolutional block attention module"; the channel attention module has the same structure and function as the ChannelAttention Module module shown in the paper "Cbam: Convolutional block attention module".

[0064] This application Figure 2 The network structure of the spatial information enhancement module in Figure 6 As shown, Figure 6 middle, Represents the Concat layer, represents the element-wise multiplication module, The spatial information enhancement module includes the first parallel element-by-element multiplication module and the first Add layer, and the second parallel element-by-element multiplication module and the second Add layer. The first parallel element-by-element multiplication module and the first Add layer input the feature map F output by the spatial-spectral information fusion module. fus The second parallel element-by-element multiplication module and the second Add layer input the feature map F output by the multi-scale feature extraction module spa , the input of the first element-by-element multiplication module and the input of the second element-by-element multiplication module are connected to the input of the Concat layer, the output of the Concat layer is connected to the average pooling layer, the first convolution layer (the convolution kernel size is 3×3), the second convolution layer (the convolution kernel size is 3×3) and the first Sigmoid activation layer in sequence, the output of the first Sigmoid activation layer is connected to the first element-by-element multiplication module and the second element-by-element multiplication module respectively, the output of the first element-by-element multiplication module is connected to the input of the first Add layer, the output of the second element-by-element multiplication module is connected to the input of the second Add layer, the output of the second Add layer is connected to the third convolution layer (the convolution kernel size is 3×3), and the third The convolution layer is connected to the average pooling layer and the first convolution unit respectively. The output of the average pooling layer is connected to the second convolution unit. The first convolution unit and the second convolution unit have the same structure and function. The first convolution unit includes two convolution layers connected in sequence (the convolution kernel size is 3×3). The output ends of the first convolution unit and the second convolution unit are both connected to the third element-by-element multiplication module. The output end of the third element-by-element multiplication module is connected to the second Sigmoid activation layer. The output end of the second Sigmoid activation layer and the output end of the first Add layer are both connected to the fourth element-by-element multiplication module. The output end of the first Add layer and the output end of the fourth element-by-element multiplication module are also connected to the input end of the third Add layer.

[0065] In the spatial information enhancement module, the feature map F output by the spatial-spectral information fusion module fus And the feature map F output by the multi-scale feature extraction module spa After concatenation by the Concat layer, the average pooling layer, the first convolution layer, the second convolution layer and the first Sigmoid activation layer are processed to obtain the attention map in the channel direction; the first element-by-element multiplication module combines the attention map output by the first Sigmoid activation layer and the feature map F output by the spatial-spectral information fusion modulefus Perform element-by-element multiplication. The second element-by-element multiplication module combines the attention map output by the first Sigmoid activation layer and the feature map F output by the multi-scale feature extraction module spa Perform element-by-element multiplication. The first Add layer outputs the feature map F of the spatial spectrum information fusion module. fus The feature map output by the first element-by-element multiplication module is added, and the second Add layer adds the feature map F output by the multi-scale feature extraction module. spa The feature map output by the second element-by-element multiplication module is added, and the third convolution layer performs a convolution operation on the feature map output by the second Add layer. The feature map output by the third convolution layer enters the first convolution unit for local feature extraction operation to obtain a feature map with rich local spatial information. At the same time, the feature map output by the third convolution layer also enters the average pooling layer for average pooling operation to obtain a feature map represented by global features. The feature map output by the average pooling layer enters the second convolution unit for global feature extraction operation to obtain a feature map with rich global spatial information. The third element-by-element multiplication module is used to perform a convolution operation on the feature map output by the first convolution unit. The feature map output by the second convolution unit is multiplied element by element to obtain a weight map with local spatial information and global spatial information. The second Sigmoid activation layer activates the weight map output by the third element-by-element multiplication module, and then the fourth element-by-element multiplication module performs element-by-element multiplication on the weight map output by the second Sigmoid activation layer and the feature map output by the first Add layer to obtain a feature map with enhanced spatial information. The third Add layer is used to perform feature addition on the feature map output by the fourth element-by-element multiplication module and the feature map output by the first Add layer to obtain a feature map with enhanced spectral information and spatial information.

[0066] S3. Total loss of progressive hyperspectral image super-resolution network using grouping strategy And a series of images used for training are used to train a progressive hyperspectral image super-resolution network based on a grouping strategy, and a progressive hyperspectral image super-resolution network model based on a grouping strategy is obtained; specifically, the following steps are included:

[0067] In this application, the loss of a progressive hyperspectral image super-resolution network based on a grouping strategy Predicted image output by the progressive hyperspectral image super-resolution network based on grouping strategy With the true value image The loss between the two, the loss of the progressive hyperspectral image super-resolution network based on the grouping strategy The calculation method is as shown in formula (1):

[0068] (1)

[0069] In formula (1), N is the number of image pairs used for training.

[0070] Input the series of images used for training into the progressive hyperspectral image super-resolution network based on grouping strategy, perform forward propagation, and calculate the total loss of the progressive hyperspectral image super-resolution network based on grouping strategy , and in the total loss Back propagation is carried out under the guidance of , the weight parameters of the progressive hyperspectral image super-resolution network based on grouping strategy are updated, and the training process of one epoch is completed. After iterating the training process of 500 epochs, the training of the progressive hyperspectral image super-resolution network based on grouping strategy is completed, and the progressive hyperspectral image super-resolution network model based on grouping strategy is obtained.

[0071] In this embodiment, the training process of the progressive hyperspectral image super-resolution network based on the grouping strategy is implemented based on the NVIDIA 2080Ti GPU chip, and the Adam optimizer is used to optimize the loss gradient and back propagate. During the training process, the batch size is set to 4 and the learning rate is set to 1×10 -4 , the attenuation coefficient gamma is set to 0.996.

[0072] S4. The hyperspectral image to be processed and the multispectral image to be processed are input into the fusion module I in the trained progressive hyperspectral image super-resolution network model based on grouping strategy, and forward propagated once along the progressive hyperspectral image super-resolution network model based on grouping strategy to obtain the hyperspectral image after super-resolution. The hyperspectral image after super-resolution is the reconstructed high-resolution hyperspectral image.

[0073] test:

[0074] In order to compare the effect of the progressive hyperspectral image super-resolution method based on the grouping strategy described in this application, this application tests and compares the five existing hyperspectral image super-resolution methods with the hyperspectral image super-resolution method described in this application based on the test image described in this application, wherein the five existing hyperspectral image super-resolution methods include: SSRNet method (from "SSR-NET: Spatial-Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion"), DSPNet method (from "DualSpatial-Spectral Pyramid Network with Transformer for Hyperspectral ImageFusion"), MoG-DCN method (from "Model-Guided Deep Hyperspectral Image Super-Resolution"), 3DT-Net method (from "Learning a 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution"), MIMO-SST method (from "MIMO-SST: Multi-Input Multi-Output Spatial-Spectral Transformer for Hyperspectral and Multispectral Image Fusion》). The test strategy adopted in this application is: directly use the test images described in this application to test the progressive hyperspectral image super-resolution method described in this application and the above-mentioned five existing hyperspectral image super-resolution methods. The test results are shown in Table 1, Figure 7 and Figure 8 The progressive hyperspectral image super-resolution method described in this application is represented by Ours method in Table 1.

[0075] Table 1 shows the test results obtained by different hyperspectral image super-resolution methods

[0076]

[0077] The evaluation indicators used in this application include PSNR, SAM, ERGAS, SSIM and RMSE, where PSNR stands for Peak Signal-to-Noise Ratio, which is used to measure the quality difference between the reconstructed image and the reference image (i.e., the true value image); SAM (Spectral Angle Mapper) is used to evaluate spectral similarity, and the similarity between different pixels is determined by calculating the spectral angle; ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse) is a dimensionless global error indicator that reflects the overall quality of the reconstructed image; SSIM (Structural Similarity Index) evaluates the visual quality of the image by comparing the brightness, contrast and structural information of the image; RMSE (Root Mean Square Error) is used to measure the average error between the reconstructed image and the reference image. Among the above five evaluation indicators, the larger the values ​​of PSNR and SSIM, the better, and the smaller the values ​​of the remaining three evaluation indicators SAM, ERGAS and RMSE, the better.

[0078] It can be seen from Table 1 that when the hyperspectral image super-resolution method described in the present application is tested on the test image obtained based on the CAVE data set, the hyperspectral image super-resolution method described in the present application achieves good results in terms of the three indicators of PSNR, SSIM, and RMSE. When the test image is tested on the Chikusei data set, the super-resolution method described in the present application achieves good results in terms of the five indicators of PSNR, SAM, ERGAS, SSIM, and RMSE. Specifically:

[0079] When testing the test images obtained based on the CAVE dataset, the PSNR value obtained by the super-resolution method described in the present application is 0.13% higher than that of the 3DT-Net method in the prior art (the method with the highest PSNR test value in the prior art); this shows that compared with the 3DT-Net method, the super-resolution method described in the present application has better performance in detail preservation and image clarity;

[0080] The SAM value obtained by the super-resolution method described in this application is relatively low, reaching 2.1333, which shows that the super-resolution method described in this application is superior in restoring the spectral details of the image, the error of the spectral angle is small, and the spectral consistency of the hyperspectral image is better maintained;

[0081] The ERGAS value obtained by the super-resolution method described in this application is relatively low, reaching 0.4896, which shows that the super-resolution method described in this application performs better in maintaining the spectral consistency and overall quality of the image, the spectral error is small, and the overall quality of the reconstructed high-resolution hyperspectral image is better;

[0082] The SSIM value obtained by the super-resolution method described in this application is relatively high, reaching 0.9955, which shows that the super-resolution method described in this application performs better in terms of image structure fidelity and can better maintain the details and structural consistency of the image;

[0083] The RMSE value obtained by the super-resolution method described in the present application is 2.44% lower than that of the 3DT-Net method in the prior art (the method with the lowest RMSE test value in the above prior art); this shows that compared with the 3DT-Net method, the super-resolution method described in the present application is more advantageous in reducing the overall image reconstruction error, and the reconstructed high-resolution hyperspectral image is closer to the true image.

[0084] When testing the test image obtained based on the Chikusei dataset, the PSNR value obtained by the super-resolution method described in the present application is 2.31% higher than the MIMO-SST method in the prior art (the method with the highest PSNR test value in the above prior art); this shows that compared with the MIMO-SST method, the super-resolution method described in the present application has better performance in detail preservation and image clarity;

[0085] The SAM value obtained by the super-resolution method described in the present application is 7.62% lower than that of the MoG-DCN method in the prior art (the method with the lowest SAM test value in the above prior art); this shows that compared with the MoG-DCN method, the super-resolution method described in the present application is more superior in restoring the spectral details of the image, significantly reduces the error of the spectral angle, and better maintains the spectral consistency of the hyperspectral image;

[0086] The ERGAS value obtained by the super-resolution method described in the present application is 4.31% lower than that of the MIMO-SST method in the prior art (the method with the lowest ERGAS test value in the above prior art); this shows that compared with the MIMO-SST method, the super-resolution method described in the present application has more advantages in maintaining the spectral consistency and overall quality of the image, with smaller spectral errors and better overall quality of the reconstructed high-resolution hyperspectral image;

[0087] The SSIM value obtained by the super-resolution method described in the present application is slightly higher than that of the MoG-DCN method in the prior art (the method with the lowest SSIM test value in the prior art) by 0.07%. This shows that compared with the MoG-DCN method, the super-resolution method described in the present application has a better performance in image structure fidelity and can better maintain the details and structural consistency of the image.

[0088] The RMSE value obtained by the super-resolution method described in the present application is 13.79% lower than that of the MIMO-SST method in the prior art (the method with the lowest RMSE test value in the above prior art); this shows that compared with the MIMO-SST method, the super-resolution method described in the present application is more advantageous in reducing the overall image reconstruction error, and the reconstructed high-resolution hyperspectral image is closer to the true image.

[0089] In addition, from Figure 7 and Figure 8 It can be seen that the hyperspectral image super-resolution method described in this application can better retain texture detail information and better preserve spectral information.

[0090] In summary, the reconstructed hyperspectral image obtained by the hyperspectral image super-resolution method described in the present application contains more texture detail information and spectral information, and is closer to the true value image.

Claims

1. A progressive hyperspectral image super-resolution method based on a grouping strategy, characterized by: The steps include: S1, obtain a test image and a series of images for training; S2. Construct a progressive hyperspectral image super-resolution network based on a grouping strategy; the progressive hyperspectral image super-resolution network based on a grouping strategy includes a fusion module I, an upsampling module I, a first Add layer, a convolution layer I, a fusion module II, an upsampling module II, a second Add layer, a convolution layer II, and a fusion module III connected in sequence; the fusion module I is used to perform a first-stage fusion of spatial information and spectral information, the upsampling module I is used to perform upsampling and increase the spatial scale, the first Add layer is used to perform feature addition, the convolution layer I is used to perform a convolution operation, and the fusion module II performs a second-stage fusion of spatial information and spectral information; The upsampling module II is used to perform upsampling and increase the spatial scale. The second Add layer is used to perform feature addition. The convolution layer II is used to perform convolution operations. The fusion module III is used to perform the third stage of fusion of spatial information and spectral information to obtain a reconstructed high-resolution hyperspectral image. S3. Total loss of progressive hyperspectral image super-resolution network using grouping strategy and training a progressive hyperspectral image super-resolution network based on a grouping strategy using a series of images for training, thereby obtaining a progressive hyperspectral image super-resolution network model based on a grouping strategy; S4. The hyperspectral image to be processed and the multispectral image to be processed are input into the fusion module I in the progressive hyperspectral image super-resolution network model based on the grouping strategy, and then forward propagated once along the progressive hyperspectral image super-resolution network model based on the grouping strategy to obtain the reconstructed high-resolution hyperspectral image.

2. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 1, characterized in that: In step S2, the structures of fusion module II and fusion module III are the same as those of fusion module I, the structures of convolution layer I and convolution layer II are the same, and the structures of upsampling module I and upsampling module II are the same.

3. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 1, characterized in that: In step S2, the fusion module I includes a Concat layer, a spatial-spectral information fusion module and an Add layer connected in parallel. The input end of the Concat layer is also connected to the input end of the multi-scale feature extraction module. The output end of the Concat layer is sequentially connected to the convolution layer III, the spatial-spectral information fusion module, the spatial information enhancement module, the convolution layer IV and the Add layer, wherein the output end of the multi-scale feature extraction module is also connected to the input end of the spatial information enhancement module.

4. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 3 is characterized by: In step S2, the spatial-spectral information fusion module includes a grouped convolutional layer I, a grouped convolutional layer II, a grouped convolutional layer III, a Transformer module I, a Transformer module II, a Transformer module III, a Concat layer and a convolutional layer. The grouped convolutional layer I, the grouped convolutional layer II and the grouped convolutional layer III are connected in parallel. The output ends of the grouped convolutional layer I, the grouped convolutional layer II and the grouped convolutional layer III are respectively connected to the Transformer module I, the Transformer module II and the Transformer module III. The output ends of the Transformer module I, the Transformer module II and the Transformer module III are all connected to the Concat layer, and the Concat layer is connected to the convolutional layer; wherein, the input ends of the Transformer module I, the Transformer module II and the Transformer module III are all connected to the output end of the convolutional layer III in the fusion module I, and the input ends of the grouped convolutional layer I, the grouped convolutional layer II and the grouped convolutional layer III are all connected to the input end of the Concat layer in the fusion module I.

5. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 4 is characterized in that: In step S2, the network structures of Transformer module I, Transformer module II and Transformer module III are the same.

6. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 4 is characterized in that: In step S2, the Transformer I module includes three Reshape layers, the input end of the first Reshape layer is connected to the output end of the grouped convolution layer I, and the input end of the second Reshape layer is connected to the output end of the convolution layer III in the fusion module I; the first Reshape layer is connected to the Value linear layer I, the Key linear layer I and the Query linear layer I, respectively, the second Reshape layer is connected to the Value linear layer II, the Key linear layer II and the Query linear layer II, respectively, the Query linear layer I and the Query linear layer II are both connected to the first Add layer, and the first Add layer and the Key linear layer I are both connected to the first matrix multiplication unit. The first Add layer and Key linear layer II are both connected to the second matrix multiplication unit, the first matrix multiplication unit is connected to the first Softmax layer, the second matrix multiplication unit is connected to the second Softmax layer, the first Softmax layer and Value linear layer I are both connected to the third matrix multiplication unit, the second Softmax layer and Value linear layer II are both connected to the fourth matrix multiplication unit, the third matrix multiplication unit and the fourth matrix multiplication unit are both connected to the second Add layer, the second Add layer is sequentially connected to the normalization layer, the feedforward network, the third Add layer and the third Reshape layer, and the output of the second Add layer is also connected to the input of the third Add layer.

7. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 3 is characterized by: In step S2, the multi-scale feature extraction module includes a parallel-connected atrous convolution layer I, atrous convolution layer II, atrous convolution layer III, atrous convolution layer IV and a spatial attention module. The atrous convolution layer I, atrous convolution layer II, atrous convolution layer III, atrous convolution layer IV and the spatial attention module are all connected to the Concat layer, and the Concat layer is sequentially connected to the channel attention module and the convolution layer.

8. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 3 is characterized by: In step S2, the spatial information enhancement module includes a first element-by-element multiplication module and a first Add layer in parallel and a second element-by-element multiplication module and a second Add layer in parallel. The input ends of the first element-by-element multiplication module and the second element-by-element multiplication module are connected to the input end of the Concat layer. The output end of the Concat layer is sequentially connected to the average pooling layer, the first convolution layer, the second convolution layer and the first Sigmoid activation layer. The output end of the first Sigmoid activation layer is respectively connected to the first element-by-element multiplication module and the second element-by-element multiplication module. The output end of the first element-by-element multiplication module is connected to the input end of the first Add layer. The second element-by-element multiplication module The output end of the network is connected to the input end of the second Add layer, the output end of the second Add layer is connected to the third convolution layer, the third convolution layer is respectively connected to the average pooling layer and the first convolution unit, the output end of the average pooling layer is connected to the second convolution unit, the output ends of the first convolution unit and the second convolution unit are both connected to the third element-by-element multiplication module, the output end of the third element-by-element multiplication module is connected to the second Sigmoid activation layer, the output end of the second Sigmoid activation layer and the output end of the first Add layer are both connected to the fourth element-by-element multiplication module, and the output end of the first Add layer and the output end of the fourth element-by-element multiplication module are also connected to the input end of the third Add layer.

9. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 8, characterized in that: In step S2, the first convolution unit and the second convolution unit have the same structure, and the first convolution unit includes two convolution layers connected in sequence.

10. The progressive hyperspectral image super-resolution method based on grouping strategy according to claim 1, characterized in that: Step S3 specifically includes the following steps: inputting the series of images for training into the progressive hyperspectral image super-resolution network based on the grouping strategy, performing forward propagation, and calculating the total loss of the progressive hyperspectral image super-resolution network based on the grouping strategy. , and in the total loss Back propagation is carried out under the guidance of , the weight parameters of the progressive hyperspectral image super-resolution network based on grouping strategy are updated, and the training process of one epoch is completed. After iterating the training process of 500 epochs, the training of the progressive hyperspectral image super-resolution network based on grouping strategy is completed, and the progressive hyperspectral image super-resolution network model based on grouping strategy is obtained.

Images