Single hyperspectral image super-resolution method based on multi-scale cross-spectrum Transform network

Abstract

The invention discloses a single hyperspectral image super-resolution method based on a multi-scale cross-spectrum Transform network, and belongs to the technical field of hyperspectral image processing, and the method comprises the steps: providing a multi-scale cross-spectrum Transform network composed of a branch network and a backbone network; the method comprises the following steps: dividing low-resolution input into overlapped spectrum groups, extracting multi-scale spatial spectrum features through a branch network, performing up-sampling, and splicing to obtain local features; the backbone network expands and deepens features through a cross-channel Transform module, and obtains global features in combination with self-attention and convolution; and adding local and global feature residuals, performing up-sampling, fusing with a bicubic up-sampling result, and finally performing convolution to generate a super-resolution image. According to the single hyperspectral image super-resolution method based on the multi-scale cross-spectrum Transform network provided by the invention, the spatial resolution of the hyperspectral image is remarkably improved while the spectral consistency is ensured.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image processing technology, and in particular to a method for super-resolution of a single hyperspectral image based on a multi-scale transspectral Transformer network. Background Technology

[0002] Hyperspectral images (HSIs) can record the spectral reflectance information of the same scene across dozens to hundreds of consecutive narrow bands. Compared to traditional RGB or multispectral images, they contain richer spectral features and have significant application value in fields such as target recognition, environmental monitoring, agricultural analysis, and medical diagnosis. However, due to limitations in sensor hardware performance, imaging system design, and acquisition costs, the spatial resolution of acquired HSIs is usually low, limiting their application in downstream refined analysis tasks.

[0003] To address this issue, hyperspectral image super-resolution (HSI-SR) technology has been proposed. Among them, the single hyperspectral image super-resolution (SHSR) method has become a research focus because it relies solely on a single low-resolution HSI image for reconstruction, possesses high independence and application flexibility. Existing SHSR methods can be divided into traditional methods (low-rank constraints, sparse representation, dictionary learning, etc.) and deep learning-based methods. Traditional methods rely on manually set prior assumptions or sparse constraints, which are difficult to fully represent complex spatial-spectral distribution features, thus limiting reconstruction performance. Deep learning-based methods mostly use 2D / 3D convolution. 2D convolution does not consider the spectral correlation between bands, while 3D convolution alleviates this problem but introduces a large number of parameters and high computational cost. Moreover, these methods are mostly limited to local spatial-spectral feature modeling and are difficult to fully characterize global spectral dependence across bands. Even if some studies introduce grouped convolution, multi-branch structures, or Transformer hybrid frameworks, there are still problems such as limited local modeling, insufficient representation of cross-band dependence, large parameter cost, and structural redundancy, which cannot efficiently balance local detail reconstruction and global spectral consistency. Summary of the Invention

[0004] The purpose of this invention is to provide a single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network to solve the problems existing in the background technology.

[0005] To achieve the above objectives, this invention provides a single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network. Let the input low-resolution hyperspectral image LR-HSI be... The corresponding high-resolution hyperspectral image HR-HSI is ,in It is the spatial resolution of HSI. It is the number of spectral bands. It is a scaling factor, and its specific contents include: Will If divided into group G, then Each group contains c bands, and there is an overlap of o bands between adjacent groups; A multi-scale transspectral Transformer network consisting of a branch network and a backbone network is proposed. The branch network incorporates a multi-scale attention fusion module (MAFM) to extract deep multi-scale spatial spectral features within each group. A progressive upsampling strategy is then employed to obtain upsampled feature maps. Finally, the upsampled feature maps from all groups are concatenated to obtain fused local features. ; The backbone network incorporates a cross-channel spectral Transformer module (CSTM) to fuse local features. Channel expansion is performed to obtain local features By combining N CSTM modules with self-attention mechanisms and depthwise separable convolutions, global deep spatial-spectral features are obtained. ; By using residual connections, local features are... With global deep spatial spectral features The addition yields enhanced deep spatial spectral features. and to Upsampling is performed in the spatial dimension to obtain reconstructed features. ; After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. ; Through a 1×1 convolutional layer Further adjust the channel dimensions to generate the final SR reconstruction result predicted super-resolution image. .

[0006] Preferably, the specific details of introducing the multi-scale attention fusion module (MAFM) into the branch network are as follows: MAFM is introduced to extract multi-scale spatial-spectral features within a group. MAFM fully utilizes spatial location information and captures the differences between adjacent spectral bands, thereby achieving more detailed joint spatial-spectral modeling. This process can be represented as: ; in, This represents the input of the g-th spectral group; This represents the MAFM feature extraction function; The deep multi-scale spatial spectrum features obtained from the g-th group; After obtaining the deep features within the group, in order to reduce the computational burden caused by direct upsampling, a progressive upsampling strategy is adopted. Specifically, PixelShuffle

[41] upsampling is used to obtain a higher resolution feature map. The corresponding process is defined as follows: ; in This indicates the PixelShuffle operation; This represents the obtained upsampled feature map; Finally, the feature maps of all groups are concatenated to obtain the overall output of the branch network. This operation can be formally represented as: ; in, This indicates a splicing operation in the spectral dimension.

[0007] Preferably, the specific content of MAFM is as follows: The input features are first divided into multiple subgroups. Within each subgroup, three different dilation rates are used for position-aware dilated convolutions to capture spatial context information at different scales. Compared to ordinary convolutions, dilated convolutions can expand the receptive field without significantly increasing parameters or computational cost, thereby improving the ability to model multi-scale structures. Exponential relative position encoding (ERPE) is introduced into each dilated convolution to explicitly inject spatial position information. ERPE first generates normalized two-dimensional relative coordinates, and then generates adaptive position biases through a learnable MLP. This MLP consists of two convolutional layers and a ReLU activation function. For the first Group input, extracting multi-scale spatial features Represented as: ; ; ; in, The convolution function represents the expansion ratio of 1, 2, and 3; The exponential relative position encoding function is used; the convolutional features at three scales are activated by ReLU and then concatenated along the channel dimension, and then... 1. Convolutional integration yields intra-group fusion features. The process is defined as follows: ; in, This indicates a channel splicing operation; subsequently, features are merged. The residual spatial block and residual channel attention block (RCAB) are fed into the residual spatial block and residual channel attention block respectively to extract spatial and spectral features in parallel. This operation can be defined as follows: ; ; in, Represents the residual space block function. This represents the residual channel attention block function. and These represent multi-scale spatial and spectral features, respectively. The residual spatial block contains two 3×3 convolutional layers and an intermediate ReLU activation layer, with residual connections used to maintain feature stability. RCAB builds upon this by introducing a channel attention (CA) mechanism. Through global average pooling, two 1×1 convolutional layers, and a sigmoid activation function, it achieves adaptive recalibration of spectral channel weights, thereby highlighting key information channels.

[0008] To further integrate spatial and spectral features, a fusion module based on depthwise convolution and pointwise convolution is designed. The spatial branch uses depthwise convolution to enhance the representation of local details, while the spectral branch uses pointwise convolution to achieve efficient interaction and information aggregation between different spectral channels. Finally, the output features of the two branches are added element-wise to obtain the fusion result. This process can be formally represented as: ; in, This represents depthwise separable convolution; This represents point convolution.

[0009] In summary, the MAFM proposed in this invention can efficiently capture the spatial and spectral features of HSI at multiple scales. By combining explicit positional encoding with multi-scale convolutional feature modeling, it enhances the model's ability to perceive spatial structures at different scales and improves the expression and fusion of spectral features, thereby providing a robust feature foundation for high-fidelity reconstruction of the overall network.

[0010] Preferably, a cross-channel spectral Transformer module (CSTM) is introduced into the backbone network to obtain the feature fusion result. The content is as follows: First, the local features obtained from the previous stage fusion are processed by 1×1 convolution. Channel expansion is performed to increase the feature representation dimension and achieve cross-spatial-spectral domain information mapping, resulting in features. This step is represented as: ; Subsequently, The input is sequentially passed through N CSTM modules. Each module combines a self-attention mechanism with depthwise separable convolutions to fully exploit the spatial context and spectral dependencies of the feature maps, thereby obtaining global deep spatial-spectral features. This process can be represented as: ; By using residual connections, local features are... With global features By combining local details with global dependencies, enhanced deep spatial-spectral features are obtained. The corresponding operation is represented as: ; Next, use PixelShuffle to... Spatial upsampling is performed to improve feature resolution and obtain reconstructed features. The process can be represented as follows: ; at the same time, After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. This operation can be formally represented as: .

[0011] Preferably, the specific content of CSTM is as follows: Multi-scale local spatial spectral features extracted by the branching network First, it is projected to a higher-dimensional space through a convolutional layer to obtain... Subsequently, The data is sequentially fed into several cascaded CSTM modules to fully extract global spectral features across spectral bands. Each CSTM contains a single normalization layer, a multi-head depthwise convolution-based self-attention (MDSA) module, and a feedforward network. Let... This represents the input feature of the nth CSTM. Let represent the output feature obtained after the nth CSTM. Then, its forward propagation process can be formally represented as: ; ; in, This represents the LayerNorm layer. This represents the multi-head depthwise separable convolutional spectral self-attention function. Indicates intermediate features.

[0012] Unlike traditional Transformers that primarily focus on the spatial domain, MDSA computes self-attention in the spectral dimension, thereby effectively learning global dependencies between bands. Input Features First, pointwise convolutions and depthwise convolutions are used to extract local spatial-spectral features. These features are then used to generate the query Q and key K, while the value V is obtained through standard convolutions. Subsequently, Q, K, and V are reshaped into the tensor shape required for multi-head attention, and the self-attention matrix is ​​computed along the spectral dimension. This process is defined as follows: ; ; ; ; in, and represents point convolution and depthwise separable convolution, respectively, where d is the feature dimension.

[0013] CSTM achieves efficient fusion of multi-scale spatial-spectral features and global spectral dependencies by combining CNN local feature extraction with cross-spectral self-attention mechanisms. MDSA focuses on long-range correlations between spectral bands, effectively characterizing the complex spectral structure in hyperspectral images; simultaneously, convolutional operations further enhance the representational power of local features while maintaining good computational efficiency. This design balances network expressiveness and scalability, providing more robust feature support for subsequent SR reconstruction.

[0014] Preferably, the final SR reconstruction result predicts the super-resolution image. The content is as follows: Through a 1×1 convolutional layer Further adjust the channel dimensions to generate the final SR reconstruction result. The correspondence is as follows: ; in, , and These represent the 1×1 convolution operations in each step. This represents the high-resolution image obtained after bicubic interpolation.

[0015] Preferably, the loss function is as follows: To simultaneously maintain spatial detail and spectral consistency, we designed the overall loss function as a weighted composite of three parts: the basic L1 loss, the Spectral Angle Mapper (SAM) loss, and the spatial-spectral gradient loss. This combination aims to balance pixel accuracy, spectral morphology fidelity, and texture detail recovery in the reconstruction.

[0016] First, L1 loss, as a fundamental term, is used to measure the pixel-by-pixel difference between the reconstructed image and the real HIS. Compared to L2 loss, L1 loss is more robust in handling outlier pixels, and it is defined as follows: ; Where N represents the number of samples in the batch. and These represent the nth real image and the reconstructed image, respectively.

[0017] Secondly, to enhance spectral consistency and preserve inter-band correlation, a SAM loss is introduced. This term constrains the directional consistency of the model in the spectral domain by calculating the angle between the reconstructed and true spectral vectors of each pixel. Its expression is: ; Furthermore, to preserve edge and texture information during high-resolution reconstruction, a gradient loss is introduced. This term enhances the model's learning of structural details and edge features by comparing the gradient differences between the reconstructed and ground images in the spatial and spectral domains. Its definition is as follows: ; in, This represents the pixel gradient along the horizontal, vertical, and spectral dimensions. In summary, the final loss function is defined as: ; in, and The weighting coefficients for spectral loss and gradient loss are empirically set to 0.5 and 0.1, respectively. This combined strategy allows the model to simultaneously focus on spectral fidelity and spatial texture reconstruction during optimization, thereby achieving a balance between visual quality and spectral consistency.

[0018] Therefore, the present invention adopts the above-mentioned single hyperspectral image super-resolution method based on multi-scale transspectral Transformer network, which effectively combines the advantages of CNN (convolutional neural networks) in local feature extraction with the ability of Transformer in long-range dependency modeling, so as to simultaneously improve the spatial clarity and spectral consistency of the reconstructed image.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a flowchart of the single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network according to the present invention; Figure 2 This is a schematic diagram of the overall framework of MCTNet in an embodiment of the present invention; Figure 3 This is a schematic diagram of the MAFM structure in an embodiment of the present invention, wherein (a) is a schematic diagram of the MAFM architecture; and (b) is a schematic diagram of the ERPE architecture. Figure 4 This is a schematic diagram of the CSTM structure in an embodiment of the present invention; Figure 5 This is a visualization of the image reconstruction results of different methods on the Pavia Center dataset when the scale factor is 4 in the embodiments of the present invention. Figure 6 This is a visualization of the image reconstruction results of different methods on the Chikusei dataset when the scale factor is 4 in the embodiments of the present invention; Figure 7 This is a visualization of the image reconstruction results of different methods on the CAVE dataset when the scale factor is 4 in the embodiments of the present invention; Figure 8 This is a visualization of the image reconstruction results of different methods on the XiongAn dataset when the scale factor is 4 in the embodiments of the present invention. Figure 9 This is an error graph showing the image reconstruction results of different methods on the Pavia Centre dataset when the scale factor is 4 in this embodiment of the invention. Figure 10 This is an error graph showing the image reconstruction results of different methods on the Chikusei dataset when the scale factor is 4 in this embodiment of the invention; Figure 11 This is an error graph showing the image reconstruction results of different methods on the CAVE dataset when the scale factor is 4 in an embodiment of the present invention. Figure 12 This is an error graph showing the image reconstruction results of different methods on the XiongAn dataset when the scale factor is 4 in this embodiment of the invention. Figure 13 The figures show the average spectral difference curves of test images on the Pavia Centre dataset for different methods in the embodiments of the present invention, where (a) is the average spectral difference curve when the scale factor is 4; and (b) is the average spectral difference curve when the scale factor is 8. Figure 14 The figures show the average spectral difference curves of test images on the Chikusei dataset for different methods in the embodiments of the present invention, where (a) is the average spectral difference curve when the scale factor is 4; and (b) is the average spectral difference curve when the scale factor is 8. Figure 15 The figures show the average spectral difference curves of test images on the CAVE dataset for different methods in the embodiments of the present invention, wherein (a) is the average spectral difference curve when the scale factor is 4; and (b) is the average spectral difference curve when the scale factor is 8. Figure 16 The figures show the average spectral difference curves of test images on the XiongAn dataset for different methods in the embodiments of the present invention, where (a) is the average spectral difference curve when the scale factor is 4; and (b) is the average spectral difference curve when the scale factor is 8. Detailed Implementation

[0021] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] Please see Figures 1-4 A single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network is proposed, assuming the input low-resolution hyperspectral image LR-HSI is... The corresponding high-resolution hyperspectral image HR-HSI is ,in It is the spatial resolution of HSI. It is the number of spectral bands. It is a scaling factor, the purpose of which is to predict the corresponding SR image from a single LR-HSI image through a designed network. The overall mapping process can be formally defined as: ; in, This represents the proposed MCTNet mapping function.

[0023] The specific content includes: To fully utilize the correlation between adjacent bands while also considering computational efficiency, If divided into group G, then Each group contains c bands, and there is an overlap of o bands between adjacent groups to ensure the continuity of spectral information between groups.

[0024] A multi-scale transspectral Transformer network consisting of a branch network and a backbone network is proposed. The branch network incorporates a multi-scale attention fusion module (MAFM) to extract deep multi-scale spatial spectral features within each group. A progressive upsampling strategy is then employed to obtain upsampled feature maps. Finally, the upsampled feature maps from all groups are concatenated to obtain fused local features. .

[0025] MAFM is introduced into the branch network to extract multi-scale spatial-spectral features within the group. MAFM makes full use of spatial location information and captures the differences between adjacent spectral bands, thereby achieving more detailed joint spatial-spectral modeling. This process can be represented as: ; in, Indicates the first One spectral group input; This represents the MAFM feature extraction function; For from the first The group obtained deep multi-scale spatial spectral features; After obtaining the deep features within the group, in order to reduce the computational burden caused by direct upsampling, a progressive upsampling strategy is adopted. Specifically, PixelShuffle

[41] upsampling is used to obtain a higher resolution feature map. The corresponding process is defined as follows: ; in This indicates the PixelShuffle operation; This represents the obtained upsampled feature map; Finally, the feature maps of all groups are concatenated to obtain the overall output of the branch network. This operation can be formally represented as: ; in, This indicates a splicing operation in the spectral dimension.

[0026] To fully exploit the complex spatial-spectral information in hyperspectral images, a MAFM based on explicit position information encoding is proposed. This module enhances the model's ability to perceive spatial details and spectral consistency while maintaining low computational complexity by introducing position-aware and multi-scale modeling mechanisms into the convolutional structure.

[0027] The input features are first divided into multiple subgroups. Within each subgroup, three different dilation rates are used for position-aware dilated convolutions to capture spatial context information at different scales. Compared to ordinary convolutions, dilated convolutions can expand the receptive field without significantly increasing parameters or computational cost, thereby improving the ability to model multi-scale structures. Exponential relative position encoding (ERPE) is introduced into each dilated convolution to explicitly inject spatial position information. ERPE first generates normalized two-dimensional relative coordinates, and then generates adaptive position biases through a learnable MLP. This MLP consists of two convolutional layers and a ReLU activation function. For the first Group input, extracting multi-scale spatial features Represented as: ; ; ; in, The convolution function represents the expansion ratio of 1, 2, and 3; The exponential relative position encoding function is used; the convolutional features at three scales are activated by ReLU and then concatenated along the channel dimension, and then... 1. Convolutional integration yields intra-group fusion features. The process is defined as follows: ; in, This indicates a channel splicing operation; subsequently, features are merged. The residual spatial block and residual channel attention block (RCAB) are fed into the residual spatial block and residual channel attention block respectively to extract spatial and spectral features in parallel. This operation can be defined as follows: ; ; in, Represents the residual space block function. This represents the residual channel attention block function. and These represent multi-scale spatial and spectral features, respectively. The residual spatial block contains two 3×3 convolutional layers and an intermediate ReLU activation layer, with residual connections used to maintain feature stability. RCAB builds upon this by introducing a channel attention (CA) mechanism. Through global average pooling, two 1×1 convolutional layers, and a sigmoid activation function, it achieves adaptive recalibration of spectral channel weights, thereby highlighting key information channels.

[0028] To further integrate spatial and spectral features, a fusion module based on depthwise convolution and pointwise convolution is designed. The spatial branch uses depthwise convolution to enhance the representation of local details, while the spectral branch uses pointwise convolution to achieve efficient interaction and information aggregation between different spectral channels. Finally, the output features of the two branches are added element-wise to obtain the fusion result. This process can be formally represented as: ; In summary, the MAFM proposed in this invention can efficiently capture the spatial and spectral features of HSI at multiple scales. By combining explicit positional encoding with multi-scale convolutional feature modeling, it enhances the model's ability to perceive spatial structures at different scales and improves the expression and fusion of spectral features, thereby providing a robust feature foundation for high-fidelity reconstruction of the overall network.

[0029] The backbone network incorporates a cross-channel spectral Transformer module (CSTM) to fuse local features. Channel expansion is performed to obtain local features By combining N CSTM modules with self-attention mechanisms and depthwise separable convolutions, global deep spatial-spectral features are obtained. ; By using residual connections, local features are... With global deep spatial spectral features The addition yields enhanced deep spatial spectral features. and to Upsampling is performed in the spatial dimension to obtain reconstructed features. .

[0030] After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. .

[0031] To further enhance the model's ability to model long-range spectral correlations, a cross-channel spectral Transformer module (CSTM) is designed in the backbone network. Specifically, the local features obtained from the previous stage fusion are first processed by 1×1 convolutions. Channel expansion is performed to increase the feature representation dimension and achieve cross-spatial-spectral domain information mapping, resulting in features. This step is represented as: ; Subsequently, The input is sequentially passed through N CSTM modules. Each module combines a self-attention mechanism with depthwise separable convolutions to fully exploit the spatial context and spectral dependencies of the feature maps, thereby obtaining global deep spatial-spectral features. This process can be represented as: ; By using residual connections, local features are... With global features By combining local details with global dependencies, enhanced deep spatial-spectral features are obtained. The corresponding operation is represented as: ; Next, use PixelShuffle to... Spatial upsampling is performed to improve feature resolution and obtain reconstructed features. The process can be represented as follows: ; at the same time, After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. This operation can be formally represented as: .

[0032] Long-range spatial-spectral dependence plays a crucial role in SHSR missions, and we propose CSTM to efficiently capture long-range spectral correlations across the entire spectrum.

[0033] Multi-scale local spatial spectral features extracted by the branching network First, it is projected to a higher-dimensional space through a convolutional layer to obtain... Subsequently, The data is sequentially fed into several cascaded CSTM modules to fully extract global spectral features across spectral bands. Each CSTM contains a single normalization layer, a multi-head depthwise convolution-based self-attention (MDSA) module, and a feedforward network. Let... This represents the input feature of the nth CSTM. Let represent the output feature obtained after the nth CSTM. Then, its forward propagation process can be formally represented as: ; ; in, This represents the LayerNorm layer. This represents the multi-head depthwise separable convolutional spectral self-attention function. Indicates intermediate features.

[0034] Unlike traditional Transformers that primarily focus on the spatial domain, MDSA computes self-attention in the spectral dimension, thereby effectively learning global dependencies between bands. Input Features First, pointwise convolutions and depthwise convolutions are used to extract local spatial-spectral features. These features are then used to generate the query Q and key K, while the value V is obtained through standard convolutions. Subsequently, Q, K, and V are reshaped into the tensor shape required for multi-head attention, and the self-attention matrix is ​​computed along the spectral dimension. This process is defined as follows: ; ; ; ; in, and represents point convolution and depthwise separable convolution, respectively, where d is the feature dimension.

[0035] CSTM achieves efficient fusion of multi-scale spatial-spectral features and global spectral dependencies by combining CNN local feature extraction with cross-spectral self-attention mechanisms. MDSA focuses on long-range correlations between spectral bands, effectively characterizing the complex spectral structure in hyperspectral images; simultaneously, convolutional operations further enhance the representational power of local features while maintaining good computational efficiency. This design balances network expressiveness and scalability, providing more robust feature support for subsequent SR reconstruction.

[0036] Through a 1×1 convolutional layer Further adjust the channel dimensions to generate the final SR reconstruction result. The correspondence is as follows: ; in, , and These represent the 1×1 convolution operations in each step. This represents the high-resolution image obtained after bicubic interpolation.

[0037] The loss function directly affects the optimization objective of the model and the quality of the reconstruction results. To simultaneously maintain spatial detail and spectral consistency, we designed the overall loss function as a weighted composite of three parts: the basic L1 loss, the Spectral Angle Mapper (SAM) loss, and the spatial-spectral gradient loss. This combination aims to balance pixel accuracy, spectral morphology fidelity, and texture detail restoration in the reconstruction.

[0038] First, L1 loss, as a fundamental term, is used to measure the pixel-by-pixel difference between the reconstructed image and the real HIS. Compared to L2 loss, L1 loss is more robust in handling outlier pixels, and it is defined as follows: ; Where N represents the number of samples in the batch. and These represent the nth real image and the reconstructed image, respectively.

[0039] Secondly, to enhance spectral consistency and preserve inter-band correlation, a SAM loss is introduced. This term constrains the directional consistency of the model in the spectral domain by calculating the angle between the reconstructed and true spectral vectors of each pixel. Its expression is: ; Furthermore, to preserve edge and texture information during high-resolution reconstruction, a gradient loss is introduced. This term enhances the model's learning of structural details and edge features by comparing the gradient differences between the reconstructed and ground images in the spatial and spectral domains. Its definition is as follows: ; in, This represents the pixel gradient along the horizontal, vertical, and spectral dimensions. In summary, the final loss function is defined as: ; in, and The weighting coefficients for spectral loss and gradient loss are empirically set to 0.5 and 0.1, respectively. This combined strategy allows the model to simultaneously focus on spectral fidelity and spatial texture reconstruction during optimization, thereby achieving a balance between visual quality and spectral consistency.

[0040] To comprehensively evaluate the effectiveness of the proposed method, this embodiment conducts experiments on four representative and widely used HSI benchmark datasets: Pavia Centre, Chikusei, CAVE, and XiongA. These datasets cover various typical scenarios, including urban, agricultural, and indoor environments, and possess different spectral bandwidths and spatial resolutions, providing a foundation for comprehensively validating the model's cross-scenario generalization performance. A systematic comparison is also performed with several representative HSI-SR methods. The comparison methods include Bicubic, GDRRN, MCNet, SSPSR, GELIN, PDE-Net, MSDformer, and SNLSR. To ensure fairness, all comparison methods are retrained based on their official public code under the same training strategy and experimental environment. To comprehensively evaluate reconstruction performance, the experiment used the following five commonly used metrics: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), spectral angle mapper (SAM), relative global root mean square error (ERGAS), and cross correlation (CC). The experimental results are as follows: (1) Quantitative comparison results (bold indicates best, underline indicates second best) Table 1 shows the quantitative comparison results on the Pavia_Centre dataset with scale factors of 4 and 8.

[0041] Table 2 shows the quantitative comparison results on the Chikusei dataset with scale factors of 4 and 8.

[0042] Table 3. Quantitative comparison results of different methods on the CAVE dataset with scale factors of 4 and 8.

[0043] Table 4 shows the quantitative comparison results of different methods on the XiongAn dataset with scale factors of 4 and 8.

[0044] (2) Visualization of reconstruction results With a scale factor of 4, the visualization results of image reconstruction using different methods on the PaviaCenter, Chikusei, CAVE, and XiongAn datasets are shown below. Figures 5-8 As shown.

[0045] With a scale factor of 4, the image reconstruction error graphs of different methods were tested on the PaviaCenter, Chikusei, CAVE, and XiongAn datasets, respectively. Figures 9-12 As shown.

[0046] The mean spectral difference curves of test images from different methods on the PaviaCenter, Chikusei, CAVE, and XiongAn datasets at different scale factors are shown below. Figures 13-16 As shown.

[0047] Experiments on four benchmark datasets—Pavia Centre, CAVE, Chikusei, and XiongAn—demonstrate that this proposed method outperforms mainstream methods in multiple evaluation metrics, including PSNR, SSIM, and SAM. Visualization results and error plots show that, compared to other comparative methods, the proposed method restores clearer local details while maintaining the overall structure, exhibiting the smallest error distribution. The spectral error plots demonstrate that the proposed method maintains the lowest spectral error across all bands, reflecting its advantages in maintaining spectral consistency and reducing band distortion.

[0048] The experimental environment for this embodiment is as follows: (1) Hardware environment: The experiment is based on NVIDIA RTX A6000 GPU and implemented using the PyTorch framework.

[0049] (2) Dataset processing: Four public datasets, Pavia Center, CAVE, Chikusei, and XiongAn, were used. The specific processing method was as follows: ①Pavia Center: The cropped image size of this dataset is 1096×715×102. Data splitting is set according to different scale factors. With a scale factor of 4, the test set consists of two 256×256 images from the first 256 columns; the validation set consists of 64×64 image patches cropped from columns 256-320; and the training set consists of 64×64 image patches after column 320 with a 32-pixel overlap. With a scale factor of 8, the validation set consists of 128×128 image patches between columns 256-384; and the training set consists of 128×128 image patches after column 384 with a 64-pixel overlap. Low-resolution samples are obtained by bicubic interpolation downsampling from the corresponding high-resolution images.

[0050] ②Chikusei: On the Chikusei dataset, a sub-image of size 2304×2048 was cropped from the central region of the original image. The first 256 columns were divided into nine 256×256 image patches as the test set, columns 256-384 as the validation set, and the remaining portion was used for training. At different scale factors, the validation and training sets were split according to the following rules: at a scale factor of 4, a 64×64 image patch with 32 pixels of overlap was cropped from the corresponding region; at a scale factor of 8, a 128×128 image patch with 64 pixels of overlap was cropped. The low-resolution hyperspectral image was obtained by bicubic interpolation downsampling from the corresponding high-resolution image.

[0051] ③CAVE: The CAVE dataset contains 32 512×512 hyperspectral images of indoor scenes, covering various materials and lighting types. In the experiment, 80% of the images were used for training, with the test and validation sets each accounting for 10%. At a scale factor of 4, the test set consisted of 12 non-overlapping 256×256 image patches, the validation set consisted of several 64×64 non-overlapping samples, and the training set consisted of 64×64 samples, with a 32-pixel overlap between training samples. At a scale factor of 8, the validation set contained several 128×128 non-overlapping samples, and the training set consisted of several 128×128 samples, with a 64-pixel overlap between training samples. The low-resolution hyperspectral images were obtained by bicubic interpolation downsampling from the corresponding high-resolution images.

[0052] ④XiongAn: This dataset produces images with a size of 1580×3750×256 after processing, exhibiting high spatial resolution and rich spectral levels, thus placing high demands on the robustness and generalization ability of the algorithm. The data partitioning method is consistent with the PaviaCentre dataset: the test set consists of several 256×256 regions from the first 512 columns of the image, the validation set is taken from columns 512-768, and the remaining portion serves as the training set. When the scale factor is 4, 64×64 image patches are cropped from the corresponding regions as both the validation and training sets, with a 32-pixel overlap between training samples. When the scale factor is 8, 128×128 image patches are cropped from the corresponding regions as both the validation and training sets, with a 64-pixel overlap between training samples. Low-resolution hyperspectral images are obtained by bicubic interpolation downsampling from the corresponding high-resolution images.

[0053] (3) Network parameter settings: In the proposed method, the input low-resolution image is divided into G groups along the spectral dimension, each group containing c=8 spectral bands, and there are o=2 overlapping bands between adjacent groups to ensure the continuity of information between bands. The above settings are consistent across all datasets. The channel compression ratio of CA is set to 16, the backbone network contains 3 CSTM modules, and the number of channels C of the intermediate feature map is set to 180 to achieve a balance between feature representation capability and computational cost. In the upsampling stage, the network adopts PixelShuffle combined with a progressive magnification strategy. When the spatial magnification factor is ×4, the branch network and the backbone network each perform a 2x upsampling operation. When the magnification factor is ×8, the branch network performs a 4x upsampling operation, while the backbone network performs a 2x upsampling operation, thereby gradually improving the spatial resolution and reducing reconstruction artifacts.

[0054] (4) Training parameter settings: During training, the Adam optimizer was used to update the parameters. The initial learning rate was 0.0001, and it was decayed to 10% of the original rate after every 30 epochs. The model training lasted for a total of 150 epochs, and the batch size was 16.

[0055] Therefore, this invention adopts the above-mentioned single hyperspectral image super-resolution method based on multi-scale transspectral Transformer network, which fully integrates the advantages of CNN in local feature extraction and the ability of Transformer in long-range dependency modeling, so as to significantly improve the spatial resolution of hyperspectral images while ensuring spectral consistency.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for super-resolution of a single hyperspectral image based on a multi-scale transspectral Transformer network, characterized in that, Let the input low-resolution hyperspectral image LR-HSI be... The corresponding high-resolution hyperspectral image HR-HSI is ,in It is the spatial resolution of HSI. It is the number of spectral bands. It is a scaling factor, and its specific contents include: Will If divided into group G, then Each group contains c bands, and there is an overlap of o bands between adjacent groups; A multi-scale transspectral Transformer network consisting of a branch network and a backbone network is proposed. The branch network incorporates a multi-scale attention fusion module (MAFM) to extract deep multi-scale spatial spectral features within each group. A progressive upsampling strategy is then employed to obtain upsampled feature maps. Finally, the upsampled feature maps from all groups are concatenated to obtain fused local features. ; The backbone network incorporates a cross-channel spectral Transformer module (CSTM) to fuse local features. Channel expansion is performed to obtain local features By combining N CSTM modules with self-attention mechanisms and depthwise separable convolutions, global deep spatial-spectral features are obtained. ; By using residual connections, local features are... With global deep spatial spectral features The addition yields enhanced deep spatial spectral features. and to Upsampling is performed in the spatial dimension to obtain reconstructed features. ; After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. ; Through a 1×1 convolutional layer Further adjust the channel dimensions to generate the final SR reconstruction result predicted super-resolution image. .

2. The method for super-resolution of a single hyperspectral image based on a multi-scale transspectral Transformer network according to claim 1, characterized in that, The specific details of introducing the Multi-Scale Attention Fusion (MAFM) module into the branch network are as follows: MAFM is introduced to extract intragroup multi-scale spatial-spectral features for joint spatial-spectral modeling, as follows: ； in, This represents the input of the g-th spectral group; This represents the MAFM feature extraction function; The deep multi-scale spatial spectrum features obtained from the g-th group; After obtaining the deep features within the group, a progressive upsampling strategy is adopted, using PixelShuffle upsampling to obtain a high-resolution feature map. The corresponding process is defined as follows: ； in This indicates the PixelShuffle operation; This represents the obtained upsampled feature map; Finally, the feature maps of all groups are concatenated to obtain the overall output of the branch network. , represented as: ； in, This indicates a splicing operation in the spectral dimension.

3. The single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network according to claim 2, characterized in that, The specific content of MAFM is as follows: The input features are first divided into multiple subgroups. Within each subgroup, three different dilation rates are used for position-aware dilated convolutions. An exponential relative position encoding (ERPE) is introduced into each dilated convolution. The ERPE first generates normalized two-dimensional relative coordinates, and then an adaptive position bias is generated through a learnable multilayer dynamic programming (MLP). The MLP consists of two convolutional layers and a ReLU activation function. For the first Group input, extracting multi-scale spatial features Represented as: ； ； ； in, The convolution function represents the expansion ratio of 1, 2, and 3; The exponential relative position encoding function is used; the convolutional features at three scales are activated by ReLU and then concatenated along the channel dimension, and then...

1. Convolutional integration yields intra-group fusion features. The process is defined as follows: ； in, This indicates a channel splicing operation; Subsequently, the fusion features The residual spatial block and residual channel attention block (RCAB) are fed separately to extract spatial and spectral features in parallel, defined as follows: ； ； in, Represents the residual space block function. This represents the residual channel attention block function. and These are multi-scale spatial features and spectral features, respectively. Building upon this, RCAB introduces a channel attention CA mechanism, which uses global average pooling, two 1×1 convolutional layers, and a Sigmoid activation function to achieve adaptive recalibration of spectral channel weights. To further integrate spatial and spectral features, a fusion module based on depthwise convolution and pointwise convolution is designed. The spatial branch uses depthwise convolution to enhance the representation of local details, while the spectral branch uses pointwise convolution to achieve efficient interaction and information aggregation between different spectral channels. The output features of the two branches are added element-wise to obtain the fusion result, represented as: ； in, This represents depthwise separable convolution; This represents point convolution.

4. The single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network according to claim 3, characterized in that, Introducing a cross-channel spectral Transformer module (CSTM) into the backbone network to obtain the feature fusion result. The content is as follows: First, the local features obtained from the previous stage fusion are processed by 1×1 convolution. Channel expansion is performed to increase the feature representation dimension and achieve cross-spatial-spectral domain information mapping, resulting in features. , represented as: ； Subsequently, The input is sequentially passed through N CSTM modules; each module combines a self-attention mechanism with depthwise separable convolution to obtain global deep spatial-spectral features. , represented as: ； By using residual connections, local features are... With global features By combining local details and global dependencies, enhanced deep spatial-spectral features are obtained. , represented as: ； Use PixelShuffle Upsampling is performed in the spatial dimension to obtain the reconstructed features. , represented as: ； at the same time, After bicubic interpolation upsampling to the desired space size, we obtain After channel alignment using 1×1 convolution, and with Add them together to obtain the feature fusion result. , represented as: 。 5. The single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network according to claim 4, characterized in that, The specific details of CSTM are as follows: Multi-scale local spatial spectral features extracted by the branching network First, it is projected to a higher-dimensional space through a convolutional layer to obtain... ; Subsequently, Several cascaded CSTM modules are fed in sequence. Each CSTM contains a layer normalization layer, a multi-head depthwise separable convolutional spectral self-attention module (MDSA), and a feedforward network. make This represents the input feature of the nth CSTM. Let the output feature obtained after the nth CSTM be represented as: Then the forward propagation process can be expressed as: ； ； in, This represents the LayerNorm layer. This represents the multi-head depthwise separable convolutional spectral self-attention function; Indicates intermediate features; Input features First, pointwise convolution and depthwise convolution are used to extract local spatial-spectral features. These features are then used to generate the query Q and key K, while the value V is obtained through standard convolution. Subsequently, Q, K, and V are reshaped into the tensor shape required for multi-head attention, and the self-attention matrix is ​​computed along the spectral dimension, defined as follows: ； ； ； ； in, and represents point convolution and depthwise separable convolution, respectively, where d is the feature dimension.

6. The single hyperspectral image super-resolution method based on a multi-scale transspectral Transformer network according to claim 5, characterized in that, Generate the final SR reconstruction result and predict the super-resolution image. The content is as follows: Through a 1×1 convolutional layer Further adjust the channel dimensions to generate the final SR reconstruction result. The correspondence is as follows: ； in, , and These represent the 1×1 convolution operations in each step. This represents the high-resolution image obtained after bicubic interpolation.

7. The method for super-resolution of a single hyperspectral image based on a multi-scale transspectral Transformer network according to claim 6, characterized in that, It also includes the loss function, the contents of which are as follows: To maintain both spatial detail and spectral consistency, the overall loss function is designed as a weighted composite of three parts: the basic L1 loss, the spectral angle mapping (SAM) loss, and the spatial-spectral gradient loss. L1 loss, as a fundamental term, is used to measure the pixel-by-pixel difference between the reconstructed image and the real HIS, and is defined as follows: ； Where N represents the number of samples in the batch; and These represent the nth real image and the reconstructed image, respectively. By introducing SAM loss, the model's directional consistency in the spectral domain is constrained by calculating the angle between the reconstructed and true spectral vectors for each pixel. The expression is as follows: ； A gradient loss is introduced, which enhances the model's learning of structural details and edge features by comparing the gradient differences between the reconstructed and real images in the spatial and spectral domains. The loss is defined as follows: ； in, Represents the pixel gradient along the horizontal, vertical, and spectral dimensions; The final loss function is defined as: ； in, and The weighting coefficients represent the spectral loss and gradient loss.

Images