A Hyperspectral Super-Resolution Reconstruction Method Based on Transformer and RNN

Abstract

This application relates to the fields of computer vision and image processing, specifically disclosing a hyperspectral super-resolution reconstruction method based on Transformer and RNN. First, the initial dataset is preprocessed and mean-normalized to obtain a normalized feature tensor. Then, a dimension is added to the first dimension of the normalized feature tensor. Next, an encoding module performs convolution operations on the spatial and spectral dimensions of the processed data and performs weight normalization to obtain an encoded feature tensor. Then, four spectral-spatial residual modules extract deeper features from the encoded feature tensor to obtain the reconstructed image. Finally, inverse 3D convolution is used to decode the reconstructed image. This application achieves efficient and accurate hyperspectral image reconstruction by using a concatenation of spatial and channel mixing and an improved linear attention mechanism instead of the traditional transform.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing, and more specifically, to a hyperspectral super-resolution reconstruction method based on Transformer and RNN. Background Technology

[0002] Hyperspectral imaging (HSI) captures the continuous spectral response curve of each pixel, providing rich spectral information suitable for target detection, recognition, and classification. However, HSI typically has low spatial resolution, making it unsuitable for small target detection and detail capture. In contrast, multispectral imaging (MSI), while having fewer bands, offers high spatial resolution and provides clear spatial details. Super-resolution (SR) techniques aim to convert low-resolution images into high-resolution images by fusing MSI and HSI to generate HSI with high spatial resolution.

[0003] Feature extraction is a prerequisite for feature fusion. Traditional CNN methods have limitations in feature extraction, such as limited convolutional window size and neglect of long-range dependencies, leading to poor reconstruction results. Transformer models, due to their self-attention mechanism, have achieved significant performance improvements in deep learning, particularly in natural language processing and computer vision tasks. The self-attention mechanism can capture global dependencies between different positions in the input sequence, effectively handling long-range dependencies, which is especially important for global image reconstruction. The MCT-Net algorithm proposed by Guo Bo et al., through multi-level cross-transformer operations, achieves cross-modal information fusion of spatial and spectral modes in hyperspectral and multispectral images, improving the preservation of spatial-spectral information in the fused image.

[0004] The self-attention mechanism is highly effective in deep learning, particularly in natural language processing and computer vision, due to its ability to capture long-range dependencies. This mechanism enables models to perform well when processing global information and is crucial for understanding complex contexts and extracting long-range features.

[0005] However, the computational complexity of the self-attention mechanism increases quadratically with the input length, leading to a sharp increase in computational and memory requirements when processing high-resolution images or long sequences of data, which limits its application.

[0006] Therefore, a hyperspectral super-resolution reconstruction method based on Transformer and RNN is provided. Summary of the Invention

[0007] This application is made in order to solve the above-mentioned technical problems.

[0008] Specifically, according to one aspect of this application, a hyperspectral super-resolution reconstruction method based on Transformer and RNN is provided, comprising:

[0009] S1. Collect images of different types as the initial dataset, with each type containing several images, and use MATLAB software to fuse the images of each type into a hyperspectral image;

[0010] S2. The hyperspectral image is subjected to mean normalization to obtain a normalized feature tensor.

[0011] S3. Add a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B,1,C,H,W];

[0012] S4. The normalized feature tensor after the dimension increase is subjected to 3D convolution and weight normalization by the encoding module to obtain the encoded feature tensor.

[0013] S5. Deep feature extraction is performed on the encoded feature tensor through the Spectral-Spatial Residual Module (SSRM).

[0014] S6. Repeat step S5 multiple times to obtain the reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module and used as the input of the next SSRM module.

[0015] S7. Use inverse 3D convolution to decode the reconstructed image.

[0016] Optionally, in S1, the initial dataset contains 31 images for each category, in PNG format, representing 31 bands of hyperspectral imagery. The 31 images are then fused into a hyperspectral image in .mat format using MATLAB software. The hyperspectral image contains two key values: Low Spatial Resolution (LR) and High Resolution Multispectral (HR), where the size of the LR image is half that of the HR image.

[0017] Optionally, step S5 includes: S51, sequentially feeding the encoded feature tensor into three dual 3D convolution modules, adding the data processed by the dual 3D convolution modules to the result of the previous dual 3D convolution as the input of the next dual 3D convolution module, repeating this process twice to form three residual shapes, wherein the outputs of the three dual 3D convolution modules are the first residual feature tensor, the second residual feature tensor, and the third residual feature tensor, respectively; S52, converting the third residual feature tensor into a 4D tensor, performing deep feature extraction on the converted 4D tensor through three spatial and channel cascaded hybrid modules to obtain a spatial-channel hybrid feature tensor, and restoring it to a 5D shape; S53, multiplying the first and second residual feature tensors by super-learning parameters respectively, and then superimposing them with the spatial-channel hybrid feature tensor after restoring its 5D shape; S54, after feature extraction of the superimposed data through basic 3D convolution, applying the ReLU activation function to introduce nonlinearity, and inputting the processed data into a dual 3D convolution for feature refinement.

[0018] S51 includes: performing convolutional encoding on the encoded feature tensor through a first dual 3D convolution module to obtain a first 3D convolutional feature tensor; adding the first 3D convolutional feature tensor to the encoded feature tensor to obtain a first residual feature tensor; performing convolutional encoding on the first residual feature tensor through a second dual 3D convolution module to obtain a second 3D convolutional feature tensor; adding the second 3D convolutional feature tensor to the first residual feature tensor to obtain a second residual feature tensor; performing convolutional encoding on the second residual feature tensor through a third dual 3D convolution module to obtain a third 3D convolutional feature tensor; adding the third 3D convolutional feature tensor to the second residual feature tensor to obtain a third residual feature tensor.

[0019] The 4D tensor has a data shape of [(B*C),C+1,H,W], which is then transformed into [(B*C),H,W,C+1] to reduce memory jumps.

[0020] The spatial mixing module includes:

[0021] R = OmniShift(x)W R K = OmniShift(x)W K V = OmniShift(x)W V

[0022]

[0023] sR=σ(R)

[0024] X'=sR⊙X

[0025] Where x is the input to the spatial mixing module, OmniShift is the data shift, R is the query matrix, and W... R K is the weight matrix of the query, and W is the key matrix. K V is the weight matrix of the keys, V is the value matrix, and W is the weight matrix of the keys. V X is the weight matrix of values, X is the global attention result, RUN_CUDA is the execution flag of CUDA operation, B is the batch size, T is the time step, C is the number of channels, spatialdecayj is the spatial decay factor, spatialfirstj is the spatial initialization parameter, sR is the data of query matrix R after passing through the sigmoid activation function, σ represents the sigmoid function, X' is the output of the spatial mixing module, and ⊙ represents element-wise multiplication.

[0026] The channel mixing module includes:

[0027]

[0028] KV = W KV ·K1+b KV

[0029] R1 = OmniShift(x1)W R1

[0030] Output = σ(R1)⊙KV

[0031] Where b is the weight matrix of the linear layer. KV It is the bias term for the linear layer.

[0032] Optionally, the overall loss of hyperspectral super-resolution reconstruction is:

[0033] L_total=L1+λL_HFL

[0034] L1 = nn.L1Loss(SR, label)

[0035] Where L_total is the total loss, L1 is the mean absolute error, λ is the hyperparameter, L_HFL is the high-frequency loss in the frequency domain, nn.L1Loss is the loss function used in PyTorch to calculate the mean absolute error, SR is the reconstructed image, and label is the label of the real image.

[0036] Optionally, the criteria for measuring reconstruction are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0037] The peak signal-to-noise ratio is:

[0038]

[0039] where PNSR is the peak signal-to-noise ratio, MAX l is the maximum value of the image color, MSE is the mean square error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i,j) is the pixel value of the original image at position (i,j), K(i,j) is the pixel value of the reconstructed or predicted image at position (i,j), i is the row index with a value range of 0 ≤ i < m, and j is the column index with a value range of 0 ≤ j < n;

[0040] The structural similarity is as follows:

[0041] SSIM(X,Y) = l(X,Y)·c(X,Y)·s(X,Y)

[0042]

[0043] where SSIM is the structural similarity, l(X,Y) is the luminance contrast term, μ X 、μ Y respectively represent the means of images X and Y, c(X,Y) is the contrast term, σ X 、σ Y respectively represent the variances of images X and Y, s(X,Y) is the structural similarity term, C1, C2, and C3 are constants respectively, σ XY represents the covariance of images X and Y, X(i,j) is the pixel value at the i-th row and j-th column in image X, Y(i,j) is the pixel value at the i-th row and j-th column in image Y, H is the height (number of rows) of the image, and W is the width (number of columns) of the image.

[0044] Specifically, according to another aspect of the present application, a hyperspectral super-resolution reconstruction system based on Transformer and RNN is provided, which includes: an encoding module, a spectral-spatial residual module (SSRM, Spectral-Spatial Residual Module), and a decoding module;

[0045] The encoding module includes a 3D convolution and a weight-normalized hidden function, where the 3D convolution kernel can perform convolution operations in three dimensions, namely the spatial dimensions (height, width) and the spectrum;

[0046] The spectral-spatial residual module includes a double 3D convolution module and a spatial and channel concatenated hybrid module. Among them, the double 3D convolution module contains two basic 3D convolutions, and the basic 3D convolution contains a 3D convolution, a normalization processing hidden function, and a ReLu activation function; the spatial and channel concatenated hybrid module is in the form of a series connection of a spatial hybrid module and a channel hybrid module;

[0047] The decoding module is used to decode the reconstructed image using inverse 3D convolution.

[0048] Compared with existing technologies, this application provides a hyperspectral super-resolution reconstruction method based on Transformer and RNN. This method reduces data bias in certain dimensions by implementing mean normalization. During the encoding stage, it utilizes 3D convolution, normalization functions, and activation functions to capture deep-level feature information. Furthermore, it combines linear attention mechanisms and CUDA parallel computing through four SSRM modules integrating spatial and channel fusion, significantly reducing the model's computational complexity and enabling efficient processing of high-resolution data. In addition, through dedicated CUDA optimization, the algorithm's computational speed and resource utilization efficiency are improved, allowing it to quickly handle global dependencies in high-resolution images and long-sequence data, ultimately achieving fast and accurate reconstruction of hyperspectral images. Attached Figure Description

[0049] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0050] Figure 1 The illustration shows a flowchart of a hyperspectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of this application.

[0051] Figure 2 The illustration shows the SSRM structure of the hyperspectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of this application.

[0052] Figure 3 The figure illustrates a spatial and channel cascaded hybrid module structure of a hyperspectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of this application.

[0053] Figure 4 The figure illustrates the overall architecture of a hyperspectral super-resolution reconstruction system based on Transformer and RNN according to an embodiment of this application. Detailed Implementation

[0054] The embodiments according to this application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0055] Example 1:

[0056] like Figure 1 As shown, the hyperspectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of this application includes: S1, collecting images of different types as an initial dataset, wherein each type contains several images, and using MATLAB software to fuse the several images of each type into a hyperspectral image; S2, performing mean normalization processing on the hyperspectral images to obtain a normalized feature tensor; S3, adding a dimension to the first dimension of the normalized feature tensor, so that the data shape is adjusted to [B,1,C,H,W]; S4, performing 3D convolution and weight normalization on the normalized feature tensor after adding dimensions through an encoding module to obtain an encoded feature tensor; S5, performing deep feature extraction on the encoded feature tensor through a Spectral-Spatial Residual Module (SSRM); S6, repeating step S5 multiple times to obtain a reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module as the input of the next SSRM module; S7, decoding the reconstructed image using inverse 3D convolution.

[0057] In this embodiment, step S1 involves collecting images of different types as an initial dataset, where each type contains several images. MATLAB software is then used to fuse these images of each type into a hyperspectral image. Specifically, the initial dataset contains 31 images of each type in PNG format, representing 31 bands of the hyperspectral spectrum. MATLAB software is used to fuse these 31 images into a hyperspectral image in .mat format. At this point, the hyperspectral image contains two key values: Low Spatial Resolution (LR) and High Resolution Multispectral (HR), where the size of LR is half that of HR. Hyperspectral super-resolution technology aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images. This method can improve the detail clarity of the image while maintaining the integrity of the spectral information.

[0058] In this embodiment, step S2 involves performing mean normalization on the hyperspectral image to obtain a normalized feature tensor. It should be understood that because hyperspectral images are large in size, contain many objects, and have an uneven distribution, mean normalization is performed on the preprocessed data to prevent significant bias in certain dimensions from affecting network training. This makes the distribution of the input data closer to zero mean, thereby improving the super-resolution reconstruction effect. Specifically, mean normalization is performed on the hyperspectral image to obtain a normalized feature tensor, wherein the data shape of the normalized feature tensor is (batch size, channel, height, weight)(12, 31, 32, 32).

[0059] In this embodiment, step S3 involves adding a dimension to the first dimension of the normalized feature tensor, adjusting the data shape to [B, 1, C, H, W]. It should be understood that adding a dimension during batch processing maintains dimensional consistency in the data. Even when processing a single sample, this dimensional consistency helps simplify code implementation, avoiding conditional checks and adjustments to the data dimensions under different circumstances. Therefore, by further adding a dimension to the first dimension of the normalized feature tensor, the data shape becomes [B, 1, C, H, W], i.e., (batch size, 1, channel, height, weight) (12, 1, 31, 32, 32).

[0060] In this embodiment, step S4 involves performing 3D convolution and weight normalization on the normalized feature tensor after adding dimensions using the encoding module to obtain the encoded feature tensor. It should be understood that 3D convolution is a powerful feature extraction tool capable of simultaneously processing spatial and spectral information of an image, while weight normalization is a method for reparameterizing the weights of a neural network. It adjusts the weights of convolutional layers to ensure that the activation outputs of different channels have similar scales. This helps accelerate network convergence and improve training stability. Combining the two helps extract and fuse the spatial-spectral features of hyperspectral images, improving the training efficiency and stability of the model, and enhancing its generalization ability. Therefore, the normalized feature tensor after adding dimensions is further subjected to 3D convolution and weight normalization operations using the encoding module.

[0061] Specifically, the normalized feature tensor after adding dimensions is fed into a 3D convolution. The 3D convolution kernel performs convolution operations on the normalized feature tensor in both the spatial dimension (height, width) and the spectral dimension, thereby learning complex cross-dimensional correlations, compressing redundant information, reducing the number of network parameters, and improving computational efficiency. The data shape after convolution becomes (batch size, 1+channel, channel, height, weight)(12, 32, 31, 32, 32). Then, the features extracted by the 3D convolution are weighted and normalized to obtain the encoded feature tensor.

[0062] In this embodiment of the application, step S5 involves deep feature extraction of the encoded feature tensor using a Spectral-Spatial Residual Module (SSRM). Specifically, as shown... Figure 2 As shown, step S5 includes: S51, sequentially feeding the encoded feature tensor into three dual 3D convolution modules, adding the data processed by the dual 3D convolution modules to the result of the previous dual 3D convolution as the input of the next dual 3D convolution module, repeating this process twice to form three residual shapes, wherein the outputs of the three dual 3D convolution modules are the first residual feature tensor, the second residual feature tensor, and the third residual feature tensor, respectively; S52, converting the third residual feature tensor into a 4D tensor, performing deep feature extraction on the converted 4D tensor through three spatial and channel cascaded hybrid modules to obtain a spatial-channel hybrid feature tensor, and restoring it to a 5D shape; S53, multiplying the first and second residual feature tensors by the super-learning parameters respectively, and then superimposing them with the spatial-channel hybrid feature tensor after restoring the 5D shape; S54, after feature extraction of the superimposed data through basic 3D convolution, applying the ReLU activation function to introduce nonlinearity, and inputting the processed data into a dual 3D convolution for feature refinement.

[0063] It is understandable that processing with dual 3D convolutional modules can capture features at different scales, which is crucial for understanding the local and global structure of images. Each dual 3D convolutional module can be viewed as extracting features at different scales, helping the model learn richer spatial and spectral information. Furthermore, utilizing the idea of ​​ResNet (Residual Network), by adding the data processed by dual 3D convolution to the result of the previous dual 3D convolution, the network depth can be increased without causing the gradient vanishing problem, while enhancing feature transfer and making deep network training possible. Therefore, to better improve the super-resolution reconstruction quality of hyperspectral images, multi-level, multi-scale feature extraction and residual connections are performed on the encoded feature tensor. This method better preserves the image's detailed information while reducing noise and artifacts, thereby improving the accuracy and interpretability of the reconstructed image.

[0064] Specifically, S51 includes: performing convolutional encoding on the encoded feature tensor through a first dual 3D convolution module to obtain a first 3D convolutional feature tensor; adding the first 3D convolutional feature tensor to the encoded feature tensor to obtain a first residual feature tensor; performing convolutional encoding on the first residual feature tensor through a second dual 3D convolution module to obtain a second 3D convolutional feature tensor; adding the second 3D convolutional feature tensor to the first residual feature tensor to obtain a second residual feature tensor; performing convolutional encoding on the second residual feature tensor through a third dual 3D convolution module to obtain a third 3D convolutional feature tensor; adding the third 3D convolutional feature tensor to the second residual feature tensor to obtain a third residual feature tensor.

[0065] Next, it should be understood that the spatial and channel concatenated hybrid module can simultaneously perform feature fusion in both spatial and channel dimensions. This hybridization enhances the model's understanding of spatial structure and inter-channel relationships, thereby extracting richer feature representations. Therefore, the spatial and channel concatenated hybrid module is used to perform deeper feature extraction on the third residual feature tensor. Furthermore, considering that 4D tensors (typically batch size, number of channels, depth, height, and width) are a common data format in deep learning, especially when processing image or hyperspectral data, allowing the model to operate simultaneously in both spatial (depth, height, and width) and channel dimensions, the third residual feature tensor is first converted to a 4D tensor before feature extraction to better suit the processing requirements of the spatial and channel concatenated hybrid module. In addition, maintaining dimensional consistency of data is crucial in deep learning models, ensuring that data can flow and be processed correctly between different parts of the network. Therefore, it is also necessary to restore the data after feature extraction to its original 5D shape.

[0066] Specifically, in step S52, the data shape of the 4D tensor is [(B*C),C+1,H,W], which is (batchsize*channel, channel+1, height, weight)(12*31, 32, 32, 32). At this point, the number of channels is channel+1. By increasing the number of channels, the network has more capacity to represent more detailed feature levels. Furthermore, considering that subsequent modules use GPU computation, modern accelerated hardware typically optimizes the processing of spatial dimension data during continuous memory access. Based on this, the data shape at this point is changed to [(B*C),H,W,C+1], which is (batch size*channel, height, weight, channel+1)(12*31, 32, 32, 32).

[0067] In particular, such as Figure 3 As shown, the spatial blending module is implemented as follows: First, the input data is shifted (similar to relative position encoding in Transformer) and fed into three parallel linear layers, similar to the self-attention module of Transformer, resulting in three key values ​​(i.e., matrices R, K, and V). K and V calculate the global attention result through a linear complexity bidirectional attention mechanism. The calculated global attention result is multiplied by σ(R), which controls the output probability, to adjust the intensity of attention. The sigmoid function compresses the value of R to between 0 and 1, serving as a control factor for the output probability. The adjusted attention result is linearly projected to further process the features. After the output linear projection, layer normalization is used to stabilize the features and reduce internal covariate shifts. Then, element-wise multiplication (⊙) is used to multiply the normalized features with the original input features to enhance important features and suppress unimportant features. Finally, the features processed by element-wise multiplication are input into a linear layer to obtain the final output of the spatial blending module. The specific formula is as follows:

[0068] R = OmniShift(x)W R K = OmniShift(x)W K V = OmniShift(x)W V

[0069]

[0070] sR=σ(R)

[0071] X'=sR⊙X

[0072] Where x is the input to the spatial mixing module, R is the query matrix used in conjunction with the key K to calculate the attention weights, and W... RK is the query weight matrix, K is the key matrix, representing the query features of the input features, used as key features in the self-attention mechanism, and W is the query weight matrix. K V is the weight matrix for the key, used to generate the key matrix from the input features. V is the value matrix, representing the value features in the self-attention mechanism, containing basic information for the output. W... V `x` is the weight matrix for values, used to generate a value matrix from the input. `X` is the global attention result. `RUN_CUDA` is the execution flag for CUDA operations, used to control whether the code runs on the GPU. The `RUN_CUDA` function is typically used for some complex operations in neural networks. By running custom operations on the GPU, it can accelerate computation. `B` is the batch size, `T` is the time step, used for the time dimension in sequence data or time series analysis, `C` is the number of channels in the input features, representing the feature depth, `spatialdecayj` is the spatial decay factor, which may control the decay in the spatial dimension, used to weaken the feature response far from the center, `spatialfirstj` is the spatial initial factor, setting the initial feature distribution in the space, `sR` is the data of the query matrix `R` after passing through the sigmoid activation function, where σ represents the sigmoid function, `X'` is the output of the spatial mixing module, and `⊙` represents element-wise multiplication.

[0073] Here, OmniShift is a data shift mechanism that spatially shifts the input data x to generate different feature arrangements. The structure consists of three 2D convolutions with kernel windows of 1x1, 3x3, and 5x5, respectively. Data is fed into each of the three convolutions, and the output is multiplied by a hyperparameter for each input data. After the input data is shifted, it enters a bidirectional attention mechanism along with the keys (K) and values ​​(V) generated from different features. This helps the model use multiple spatial feature arrangements in computation, avoiding reliance on features at fixed locations. This feature diversity helps the model capture complex spatial structures and contextual information.

[0074] In particular, such as Figure 3 As shown, the implementation process of the channel mixing module is as follows: The output data of the spatial mixing module is used as the input data. The input data is shifted, and the shifted data is processed by the ReLU activation function. Then, the square root of the activated data is taken to obtain K1, and K1 is fed into the linear layer to obtain the matrix KV. The specific formula is as follows:

[0075]

[0076] KV = W KV ·K1+b KV

[0077] R1 = OmniShift(x1)W R1

[0078] Output = σ(R1)⊙KV

[0079] Among them, W KV It is the weight matrix of the linear layer, b KV It is the bias term for the linear layer.

[0080] Furthermore, it should be understood that the super-learning parameters enable the model to automatically optimize feature combinations based on training data, improving model performance and generalization ability. Based on this, the first and second residual feature tensors are multiplied by the super-learning parameters to obtain weighted feature tensors (these tensors reflect the importance of their respective features and their contribution to the final task). In this way, the model can dynamically adjust the importance of different feature tensors, making more important features contribute more to the final result. Further superimposing these weighted feature tensors with the spatial channel hybrid feature tensor after restoring the 5D shape not only integrates feature information from different sources to obtain a more comprehensive image representation, but also promotes gradient flow, improves training stability, and reduces the gradient vanishing problem in deep networks, similar to skip connections or residual connections in ResNet. Therefore, to enhance the model's ability to learn and represent features and improve the performance and efficiency of hyperspectral image super-resolution reconstruction, the first and second residual feature tensors are multiplied by the super-learning parameters and then superimposed with the spatial channel hybrid feature tensor after restoring the 5D shape.

[0081] Specifically, the implementation process of S53 is as follows:

[0082] 1. Multiply by super-learning parameters: Multiply the first residual feature tensor and the second residual feature tensor by their respective super-learning parameters to obtain the first weighted feature tensor and the second weighted feature tensor. The super-learning parameters are learned during model training and are used to adjust the contribution or weight of the feature tensors in the final superposition.

[0083] 2. Superimposing Feature Tensors: The first weighted feature tensor, the second weighted feature tensor, and the spatial channel hybrid feature tensor after restoring the 5D shape are superimposed. This step can be achieved by adding element-wise (Add operation).

[0084] Through this process, the model can integrate feature information from different stages and adjust the weights of each feature through super-learning parameters in order to obtain better feature representations, thereby improving the performance of hyperspectral image super-resolution reconstruction.

[0085] Then, considering that different feature tensors may contain redundant information or conflict in certain regions, direct superposition can easily lead to information confusion and may lack deep features extracted from the data. Furthermore, deep learning models require nonlinearity to learn and simulate complex functions. If the superimposed data is not processed by a nonlinear activation function, the model may only capture linear relationships, limiting its expressive power and ability to handle complex tasks. Therefore, to capture deeper features and enable the model to better learn complex patterns and relationships in the data, the superimposed data is processed by feature extraction using basic 3D convolution, followed by the application of the ReLU activation function to introduce nonlinearity. The processed data is then input into a dual 3D convolution for feature refinement. This helps improve the model's feature extraction capability, nonlinear mapping capability, and feature fusion effect, thereby improving the performance of hyperspectral image super-resolution reconstruction.

[0086] In this embodiment, step S6 involves repeatedly performing step S5 to obtain a reconstructed image. The data processed by the SSRM module is fused with the output of the previous SSRM module and used as the input to the next SSRM module. It should be understood that sequentially feeding feature tensor information into four SSRM modules, with each feature tensor processed by an SSRM module fused with the initial feature tensor, provides better supplementary cues for learning. SSRM extracts high-frequency features, while the original features provide low-frequency features, enhancing the model's understanding of image content and thus achieving high-quality hyperspectral image reconstruction. Therefore, to better complete the image reconstruction, the feature tensor is processed continuously through four SSRM modules, and after each processing, the resulting output feature tensor is merged with the initial input feature tensor and used as the input to the next SSRM module.

[0087] Specifically, the implementation process of S6 is as follows:

[0088] 1. Initialize input: Pass the initial overlay data as input to the first SSRM module.

[0089] 2. First SSRM processing: The data is processed through the first SSRM module, which extracts and fuses spatial-spectral features through its internal 3D convolutional layers, activation functions and other feature extraction mechanisms.

[0090] 3. Feature Fusion: The output of the first SSRM module is fused with the output of the previous module (which will be the initial input data in the first iteration). This fusion can be achieved through simple addition, more complex fusion mechanisms, or attention mechanisms, with the aim of integrating feature information at different levels.

[0091] 4. Subsequent SSRM iterations: The fused feature data is passed as input to the next SSRM module. This process is repeated four times, with each iteration further refining and improving the feature representation, gradually improving the quality of the reconstructed image.

[0092] 5. Feature Iteration Refinement: Each iteration allows the model to mine and utilize the spectral-spatial information in the data more deeply, thereby improving the accuracy and quality of the reconstructed image.

[0093] 6. Final Reconstruction: After four iterations, the final output will be a reconstructed hyperspectral image with improved spectral and spatial resolution.

[0094] In this embodiment, step S7 involves decoding the reconstructed image using inverse 3D convolution. It should be understood that inverse 3D convolution is an inverse operation that can quickly reconstruct images and restore their detailed information. Since step S2 performed mean normalization, inverse 3D convolution is used to add back the mean subtracted in step S2, thereby restoring the data to its initial state and ensuring the accuracy of the reconstructed image.

[0095] Specifically, the overall loss of the hyperspectral super-resolution reconstruction method based on Transformer and RNN consists of two parts: mean absolute error (L1) and frequency domain high-frequency loss (L_HFL), and a hyperparameter λ used to balance these two losses. The specific formula is expressed as:

[0096] L_total=L1+λL_HFL

[0097] L1 = nn.L1Loss(SR, label)

[0098] Where L_total is the total loss; L1 is the mean absolute error, which measures the pixel-level difference between the reconstructed image (SR) and the ground truth image (label); L_HFL is the high-frequency loss in the frequency domain, which focuses on the high-frequency components of the image in the frequency domain, which typically contain edge and detail information of the image; λ is a hyperparameter used to balance the weights between the L1 loss and the L_HFL loss; nn.L1Loss is the loss function in PyTorch used to calculate the mean absolute error, SR is the reconstructed image, and label is the label of the ground truth image.

[0099] It is worth mentioning that the criteria for measuring the reconstruction are the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM).

[0100] Among them, the peak signal-to-noise ratio (PSNR) is an index for measuring image quality, which measures the reconstruction quality of an image based on the mean squared error (MSE). The higher the PSNR value, the better the image quality. The specific formula is expressed as:

[0101]

[0102] where PNSR is the peak signal-to-noise ratio, MAX l is the maximum value of the image color, MSE is the mean squared error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i, j) is the pixel value of the original image at position (i, j), K(i, j) is the pixel value of the reconstructed or predicted image at position (i, j), i is the row index, with the value range of 0 ≤ i < m, j is the column index, with the value range of 0 ≤ j < n;

[0103] Among them, the structural similarity (SSIM) is used to evaluate the perceptual similarity of images, paying particular attention to the similarity of brightness, contrast, and structural information of images. The closer the SSIM value is to 1, the more similar the two images are in structure and the better the image quality. The specific formula is expressed as:

[0104] SSIM(X, Y) = l(X, Y)·c(X, Y)·s(X, Y)

[0105]

[0106] where l(X, Y) is the luminance contrast term, μ X 、μ Y respectively represent the means of images X and Y, c(X, Y) is the contrast term, σ X 、σ Y respectively represent the variances of images X and Y, s(X, Y) is the structural similarity term, C1, C2, and C3 are constants respectively, σ XY represents the covariance of images X and Y, X(i, j) is the pixel value at the i-th row and j-th column in image X, Y(i, j) is the pixel value at the i-th row and j-th column in image Y, H is the height (number of rows) of the image, and W is the width (number of columns) of the image.

[0107] In summary, the hyperspectral super-resolution reconstruction method based on Transformer and RNN according to the embodiments of this application is clarified. It first preprocesses and normalizes the initial dataset to obtain a normalized feature tensor, and then adds a dimension to the first dimension of the normalized feature tensor. Next, the processed data is convolved with spatial and spectral dimensions and weighted and normalized by an encoding module to obtain an encoded feature tensor. Then, four spectral-spatial residual modules are used to extract deeper features from the encoded feature tensor to obtain the reconstructed image. Finally, inverse 3D convolution is used to decode the reconstructed image. This application achieves efficient and accurate reconstruction of hyperspectral images by using a concatenation of spatial and channel mixing and an improved linear attention mechanism instead of the traditional transform.

[0108] Example 2:

[0109] Figure 4 The figure illustrates the overall architecture of a hyperspectral super-resolution reconstruction system based on Transformer and RNN according to an embodiment of this application, as shown below. Figure 4 As shown, the hyperspectral super-resolution reconstruction system based on Transformer and RNN according to the embodiments of this application includes: an encoding module, four spectral-spatial residual modules (SSRMs) and a decoding module;

[0110] The encoding module includes a 3D convolution and a weighted normalized hidden function, wherein the 3D convolution kernel can perform convolution operations in three dimensions, namely spatial dimension (height, width) and spectrum;

[0111] The spectral-spatial residual module includes four dual 3D convolution modules, one basic 3D convolution, one ReLU activation function, and three spatial and channel concatenated hybridization modules. The dual 3D convolution modules contain two basic 3D convolutions, each of which contains a 3D convolution, a normalization hidden function, and a ReLU activation function.

[0112] The space and channel cascaded hybrid module is a combination of a space hybrid module and a channel hybrid module connected in series.

[0113] The decoding module is used to decode the reconstructed image using inverse 3D convolution.

[0114] Here, those skilled in the art will understand that the specific functions and operations of each step in the above-described hyperspectral super-resolution reconstruction system based on Transformer and RNN have been referenced above. Figure 1The description of the hyperspectral super-resolution reconstruction method based on Transformer and RNN has been detailed, and therefore, its repeated description will be omitted.

[0115] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit of the technical solutions of the present invention.

Claims

1. A hyperspectral super-resolution reconstruction method based on Transformer and RNN, characterized in that, include: S1. Collect images of different types as the initial dataset, with each type containing several images, and use MATLAB software to fuse the images of each type into a hyperspectral image; S2. The hyperspectral image is subjected to mean normalization to obtain a normalized feature tensor. S3. Add a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B,1,C,H,W]; B is the batch size, C is the number of channels, H is the image height, and W is the image width; S4. The normalized feature tensor after the dimension increase is subjected to 3D convolution and weight normalization by the encoding module to obtain the encoded feature tensor. S5. Deep feature extraction is performed on the encoded feature tensor using the Spectral-Spatial Residual Module (SSRM). S6. Repeat step S5 multiple times to obtain the reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module and used as the input of the next SSRM module. S7. Use inverse 3D convolution to decode the reconstructed image; Wherein, S5 includes: S51. The encoded feature tensor is sequentially fed into three dual 3D convolution modules. The data processed by the dual 3D convolution modules is added to the result of the previous dual 3D convolution as the input of the next dual 3D convolution module. This process is repeated twice to form three residual shapes. The outputs of the three dual 3D convolution modules are the first residual feature tensor, the second residual feature tensor, and the third residual feature tensor, respectively. S52. Convert the third residual feature tensor into a 4D tensor. Use three spatial and channel cascaded hybrid modules to perform deep feature extraction on the converted 4D tensor to obtain a spatial-channel hybrid feature tensor and restore it to a 5D shape. S53. Multiply the first residual feature tensor and the second residual feature tensor by the super-learning parameters respectively, and then superimpose them with the spatial channel hybrid feature tensor after recovering the 5D shape. S54. After extracting features from the superimposed data through basic 3D convolution, the ReLU activation function is applied to introduce non-linearity, and the processed data is input into a dual 3D convolution for feature refinement. In S52, the spatial and channel cascaded mixing module includes a spatial mixing module and a channel mixing module. The spatial mixing module includes: R=OmniShift(x)W R ,K=OmniShift(x)W K ,V=OmniShift(x)W V sR=σ(R) X'=sR⊙X Where x is the input to the spatial mixing module, OmniShift is the data shift, R is the query matrix, and W... R K is the weight matrix of the query, and W is the key matrix. K V is the weight matrix of the keys, V is the value matrix, and W is the weight matrix of the keys. V X is the weight matrix of values, X is the global attention result, RUN_CUDA is the execution flag of CUDA operation, B is the batch size, T is the time step, C is the number of channels, spatialdecayj is the spatial decay factor, spatialfirstj is the spatial initialization parameter, sR is the data of query matrix R after passing through the sigmoid activation function, σ represents the sigmoid function, X' is the output of the spatial mixing module, and ⊙ represents element-wise multiplication; In S52, the channel mixing module includes: KV=W KV ·K1+b KV R1=OmniShift(x1)W R1 Output = σ(R1)⊙KV Among them, W KV It is the weight matrix of the linear layer, b KV It is the bias term for the line layer.

2. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 1, characterized in that, In S1, the initial dataset contains 31 images for each category, in PNG format, representing 31 bands of hyperspectral imagery. The 31 images are then fused into a hyperspectral image in .mat format using MATLAB software. The hyperspectral image contains two key values: low spatial resolution (LR) and high resolution multispectral (HR). The size of the LR image is half that of the HR image.

3. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 2, characterized in that, S51 includes: The encoded feature tensor is convolutionally encoded through the first dual 3D convolution module to obtain the first 3D convolutional feature tensor. The first 3D convolutional feature tensor is added to the encoded feature tensor to obtain the first residual feature tensor. The first residual feature tensor is convolutionally encoded through the second dual 3D convolution module to obtain the second 3D convolution feature tensor. The second 3D convolution feature tensor is added to the first residual feature tensor to obtain the second residual feature tensor. The second residual feature tensor is convolutionally encoded through a third dual 3D convolution module to obtain the third 3D convolutional feature tensor. The third 3D convolutional feature tensor is then added to the second residual feature tensor to obtain the third residual feature tensor.

4. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 3, characterized in that, In S52, the data shape of the 4D tensor is [(B*C),C+1,H,W], and it is transformed into [(B*C),H,W,C+1] to reduce memory jumps.

5. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 1, characterized in that, The overall loss of hyperspectral super-resolution reconstruction is: L_total=L1+λL_HFL L1 = nn.L1 Loss(SR, label) Where L_totalL is the total loss, L1 is the mean absolute error, λ is the hyperparameter, L_HFL is the high-frequency loss in the frequency domain, nn.L1 Loss is the loss function in PyTorch used to calculate the mean absolute error, SR is the reconstructed image, and label is the label of the real image.

6. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 5, characterized in that, The criteria for measuring reconstruction are peak signal-to-noise ratio and structural similarity (SSIM). The peak signal-to-noise ratio is: where PNSR is the peak signal-to-noise ratio, MAX l is the maximum value of the image color, MSE is the mean square error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i, j) is the pixel value of the original image at position (i, j), K(i, j) is the pixel value of the reconstructed or predicted image at position (i, j), i is the row index with a value range of 0 ≤ i < m, and j is the column index with a value range of 0 ≤ j < n; The structural similarity is as follows: SSIM(X,Y)=l(X,Y)·c(X,Y)·s(X,Y) Where SSIM is structural similarity, l(X,Y) is the brightness contrast term, and μ X μ Y Let X and Y represent the means of the image, respectively, and c(X,Y) be the contrast term. X σ Y Let s(X,Y) represent the variances of images X and Y, respectively. s(X,Y) is the structural similarity term, and C1, C2, and C3 are constants, respectively. XY Let X(i,j) represent the covariance of images X and Y, where X(i,j) is the pixel value in the i-th row and j-th column of image X, and Y(i,j) is the pixel value in the i-th row and j-th column of image Y. H is the height of the image, and W is the width of the image.

7. A hyperspectral super-resolution reconstruction system based on Transformer and RNN, wherein the system is applied to the method of claim 1, characterized in that, include: The module includes an encoding module, a spectral-spatial residual module (SSRM), and a decoding module. The encoding module includes a 3D convolution and a weighted normalized hidden function, wherein the 3D convolution kernel can perform convolution operations in three dimensions, namely spatial dimension and spectral dimension; The spectral-spatial residual module includes a dual 3D convolution module and a spatial and channel concatenated hybrid module. The dual 3D convolution module contains two basic 3D convolutions, each containing a 3D convolution, a normalization hidden function, and a ReLU activation function. The space and channel cascaded hybrid module is a combination of a space hybrid module and a channel hybrid module connected in series. The decoding module is used to decode the reconstructed image using inverse 3D convolution.

Images