Hyperspectral super-resolution reconstruction method based on Transform and RNN (Recurrent Neural Network)

Abstract

The invention relates to the field of computer vision and image processing, and particularly discloses a hyperspectral super-resolution reconstruction method based on Transformer and RNN, which comprises the following steps: firstly, carrying out preprocessing and mean normalization on an initial data set to obtain a normalized feature tensor, adding one dimension to the first dimension of the normalized feature tensor, and carrying out mean normalization on the normalized feature tensor to obtain a normalized feature tensor; the processed data is subjected to convolution operation of spatial dimension and spectral dimension and weight normalization through an encoding module to obtain an encoding feature tensor, deeper feature extraction is performed on the encoding feature tensor through four spectrum-spatial residual modules to obtain a reconstructed image, and finally, the reconstructed image is decoded by using anti-3D convolution. According to the invention, through serial connection of space mixing and channel mixing, an improved linear attention mechanism is adopted to replace a traditional transform, and efficient and accurate reconstruction of the hyperspectral image is realized.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and image processing, and more specifically, to a high-spectral super-resolution reconstruction method based on Transformer and RNN. Background Art

[0002] Hyperspectral imaging (HSI) technology can capture the continuous spectral response curve of each pixel, providing rich spectral information, which is suitable for target detection, recognition and classification. However, the spatial resolution of HSI is usually low, which is not conducive to small target detection and detail capture. In contrast, although multispectral image (MSI) has fewer bands, it has high spatial resolution and can provide clear spatial details. Super-resolution technology (SR) aims to convert low-resolution images into high-resolution images, and generates HSI with high spatial resolution by fusing MSI and HSI.

[0003] Feature extraction is the premise of feature fusion. Traditional CNN methods have limitations in feature extraction, such as limited convolution window range, ignoring long-distance dependencies, and resulting in poor reconstruction effects. The Transformer model has achieved significant improvement in the field of deep learning, especially in natural language processing and computer vision tasks, due to its self-attention mechanism. The self-attention mechanism can capture the global dependencies between different positions in the input sequence and effectively handle long-distance dependencies, which is particularly important for global image reconstruction. The MCT-Net algorithm proposed by Guo Bo et al. realizes the cross-modal information fusion of spatial and spectral modalities of hyperspectral images and multispectral images through multi-level cross Transformers, and improves the spatial-spectral information retention of the fused image.

[0004] The self-attention mechanism is effective in deep learning, especially in natural language processing and computer vision, because it can capture long-distance dependencies. This mechanism enables the model to perform well in processing global information and is crucial for understanding complex contexts and extracting long-range features.

[0005] However, the computational complexity of the self-attention mechanism grows quadratically with the input length, resulting in a sharp increase in computational and memory requirements when processing high-resolution images or long sequence data, limiting its application.

[0006] Therefore, a high-spectral super-resolution reconstruction method based on Transformer and RNN is provided. Summary of the invention

[0007] In order to solve the above technical problems, this application is proposed.

[0008] Specifically, according to one aspect of the present application, a hyperspectral super-resolution reconstruction method based on Transformer and RNN is provided, which includes:

[0009] S1. Collect different types of images as the initial data set, where each type contains several images, and use MATLAB software to fuse several images of each type into a hyperspectral image;

[0010] S2, performing mean normalization processing on the hyperspectral image to obtain a normalized feature tensor;

[0011] S3. Add a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B, 1, C, H, W];

[0012] S4, performing 3D convolution and weight normalization on the normalized feature tensor after the dimension increase through the encoding module to obtain an encoded feature tensor;

[0013] S5, performing deep feature extraction on the encoded feature tensor through the spectral-spatial residual module (SSRM);

[0014] S6, repeating step S5 multiple times to obtain a reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module as the input of the next SSRM module;

[0015] S7. Decode the reconstructed image using inverse 3D convolution.

[0016] Optionally, in the S1, there are 31 images of each type in the initial data set, in PNG format, representing 31 bands of the hyperspectral spectrum, and the 31 images are fused into a hyperspectral image in .mat format using MATLAB software, wherein the hyperspectral image contains two key values, namely low spatial resolution (LR) and high resolution multispectral (HR), wherein the size of the LR image is half of that of the HR image.

[0017] Optionally, the S5 includes: S51, sending the encoded feature tensor to three dual 3D convolution modules in sequence, adding the data processed by the dual 3D convolution module to the result obtained by the previous dual 3D convolution as the input of the next dual 3D convolution module, repeating twice to form three residual shapes, wherein the outputs of the three dual 3D convolution modules are the first residual feature tensor, the second residual feature tensor and the third residual feature tensor respectively; S52, converting the third residual feature tensor into a 4D tensor, performing deep feature extraction on the converted 4D tensor through three spatial and channel series mixing modules to obtain a spatial channel mixed feature tensor, and restoring it to a 5D shape; S53, multiplying the first residual feature tensor and the second residual feature tensor by the super learning parameters respectively, and superimposing them with the spatial channel mixed feature tensor after restoring the 5D shape; S54, after extracting features from the superimposed data through the basic 3D convolution, applying the ReLU activation function to introduce nonlinearity, and inputting the processed data into a dual 3D convolution for feature refinement.

[0018] Among them, the S51 includes: convolution encoding the encoded feature tensor through a first dual 3D convolution module to obtain a first 3D convolution feature tensor, adding the first 3D convolution feature tensor to the encoded feature tensor to obtain a first residual feature tensor; convolution encoding the first residual feature tensor through a second dual 3D convolution module to obtain a second 3D convolution feature tensor, adding the second 3D convolution feature tensor to the first residual feature tensor to obtain a second residual feature tensor; convolution encoding the second residual feature tensor through a third dual 3D convolution module to obtain a third 3D convolution feature tensor, and adding the third 3D convolution feature tensor to the second residual feature tensor to obtain a third residual feature tensor.

[0019] The data shape of the 4D tensor is [(B*C), C+1, H, W], and it is changed to [(B*C), H, W, C+1] to reduce memory jumps.

[0020] Among them, the spatial mixing module includes:

[0021] R = OmniShift(x)W R , K = OmniShift(x)W K , V = OmniShift(x)W V

[0022]

[0023] sR=σ(R)

[0024] X'=sR⊙X

[0025] Where x is the input to the spatial blending module, OmniShift is the data shift, R is the query matrix, and W R is the query weight matrix, K is the key matrix, W K is the key weight matrix, V is the value matrix, W V is the weight matrix of the value, X is the global attention result, RUN_CUDA is the execution flag of the CUDA operation, B is the batch size, T is the time step, C is the number of channels, spatialdecayj is the spatial decay factor, spatialfirstj is the spatial initial parameter, sR is the data of the query matrix R after the sigmoid activation function, σ represents the sigmoid function, X' is the output of the spatial mixing module, and ⊙ represents element-by-element multiplication.

[0026] Among them, the channel mixing module includes:

[0027]

[0028] KV=W KV K1+b KV

[0029] R1=OmniShift(x1)W R1

[0030] Output=σ(R1)⊙KV

[0031] Among them, is the weight matrix of the linear layer, b KV is the bias term of the linear layer.

[0032] Optionally, the overall loss for hyperspectral super-resolution reconstruction is:

[0033] L_total=L1+λL_HFL

[0034] L1 = nn.L1Loss(SR, label)

[0035] Among them, L_total is the overall loss, L1 is the mean absolute error, λ is a hyperparameter, L_HFL is the high-frequency loss in the frequency domain, nn.L1Loss is the loss function used to calculate the mean absolute error in PyTorch, SR is the reconstructed image, and label is the label of the real image.

[0036] Optionally, the reconstruction criteria are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM);

[0037] The peak signal-to-noise ratio is:

[0038]

[0039] Among them, PNSR is the peak signal-to-noise ratio, and MAX l is the maximum value of the image color, MSE is the mean square error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i,j) is the pixel value of the original image at position (i,j), K(i,j) is the pixel value of the reconstructed or predicted image at position (i,j), i is the row index with a value range of 0 ≤ i < m, and j is the column index with a value range of 0 ≤ j < n;

[0040] The structural similarity is as follows:

[0041] SSIM(X,Y) = l(X,Y)·c(X,Y)·s(X,Y)

[0042]

[0043] Among them, SSIM is the structural similarity, l(X,Y) is the luminance contrast term, μ X and μ Y respectively represent the means of images X and Y, c(X,Y) is the contrast term, σ X and σ Y respectively represent the variances of images X and Y, s(X,Y) is the structural similarity term, C1, C2, and C3 are constants respectively, σ XY represents the covariance of images X and Y, X(i,j) is the pixel value at the i-th row and j-th column in image X, Y(i,j) is the pixel value at the i-th row and j-th column in image Y, H is the height (number of rows) of the image, and W is the width (number of columns) of the image.

[0044] Specifically, according to another aspect of the present application, a hyperspectral super-resolution reconstruction system based on Transformer and RNN is provided, which includes: an encoding module, a spectral-spatial residual module (SSRM, Spectral-Spatial Residual Module), and a decoding module;

[0045] The encoding module includes a 3D convolution and a weight-normalized hidden function. Among them, the 3D convolution kernel can perform convolution operations in three dimensions, namely the spatial dimensions (height, width) and the spectrum;

[0046] The spectral-spatial residual module includes a double 3D convolution module and a spatial and channel concatenated hybrid module. Among them, the double 3D convolution module contains two basic 3D convolutions, and the basic 3D convolution contains a 3D convolution, a normalization processing hidden function, and a ReLu activation function; the spatial and channel concatenated hybrid module is in the form of a series connection of a spatial hybrid module and a channel hybrid module;

[0047] The decoding module is used to decode the reconstructed image using inverse 3D convolution.

[0048] Compared with the prior art, the present application provides a hyperspectral super-resolution reconstruction method based on Transformer and RNN, which reduces the deviation of data in certain dimensions by implementing mean normalization; in the encoding stage, 3D convolution, normalization function and activation function are used to capture deep feature information, and through four SSRM modules that integrate spatial and channel fusion, the linear attention mechanism and CUDA parallel computing are combined, which significantly reduces the computational complexity of the model, enabling the model to efficiently process high-resolution data. In addition, through special CUDA optimization, the algorithm's computing speed and resource utilization efficiency have been improved, enabling the algorithm to quickly process global dependencies in high-resolution images and long sequence data, and ultimately achieve fast and accurate reconstruction of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0050] Figure 1 The figure illustrates a flowchart of a hyperspectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of the present application.

[0051] Figure 2 The figure illustrates the SSRM structure diagram of the high-spectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of the present application.

[0052] Figure 3 The diagram illustrates a spatial and channel series hybrid module structure diagram of a high-spectral super-resolution reconstruction method based on Transformer and RNN according to an embodiment of the present application.

[0053] Figure 4 The figure illustrates the overall architecture of a model of a hyperspectral super-resolution reconstruction system based on Transformer and RNN according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] Below, the embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0055] Embodiment 1:

[0056] like Figure 1 As shown, the hyperspectral super-resolution reconstruction method based on Transformer and RNN according to the embodiment of the present application includes: S1, collecting different types of images as initial data sets, wherein each type contains several images, and using MATLAB software to fuse several images of each type into hyperspectral images; S2, performing mean normalization processing on the hyperspectral image to obtain a normalized feature tensor; S3, adding a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B, 1, C, H, W]; S4, performing 3D convolution and weight normalization on the normalized feature tensor after the dimension increase through the encoding module to obtain a coded feature tensor; S5, performing deep feature extraction on the coded feature tensor through a spectral-spatial residual module (SSRM); S6, repeating step S5 multiple times to obtain a reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module as the input of the next SSRM module; S7, decoding the reconstructed image using inverse 3D convolution.

[0057] In an embodiment of the present application, the step S1 collects different types of images as an initial data set, wherein each type contains several images, and uses MATLAB software to fuse several images of each type into a hyperspectral image. Specifically, there are 31 images of each type in the initial data set, in PNG format, representing 31 bands of the hyperspectral spectrum, and these 31 images are fused into a hyperspectral image in .mat format using MATLAB software. At this point, the hyperspectral image contains two key values, namely low spatial resolution (LR) and high-resolution multispectral (HR), wherein the size of LR is half that of HR. Hyperspectral super-resolution technology aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images, which can improve the detail clarity of the image while maintaining the integrity of the spectral information.

[0058] In an embodiment of the present application, the step S2 is to perform mean normalization processing on the hyperspectral image to obtain a normalized feature tensor. It should be understood that the hyperspectral image is large in size, contains more objects, and the objects are unevenly distributed. Therefore, in order to prevent the data of certain dimensions from having a large deviation, thereby affecting the training of the network, the preprocessed data is mean normalized so that the distribution of the input data is closer to zero mean, thereby improving the super-resolution reconstruction effect. Specifically, the hyperspectral image is mean normalized to obtain a normalized feature tensor, wherein the data shape of the normalized feature tensor is (batch size, channel, height, weight) (12, 31, 32, 32).

[0059] In an embodiment of the present application, step S3 adds a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B, 1, C, H, W]. It should be understood that when performing batch processing, adding a dimension can maintain the dimensional consistency of the data. Even when processing a single sample, this dimensional consistency helps to simplify the code implementation and avoid conditional judgment and adjustment of the data dimension in different situations. Therefore, a dimension is further added to the first dimension of the normalized feature tensor, and the data shape at this time becomes [B, 1, C, H, W], that is, (batch size, 1, channel, height, weight) (12, 1, 31, 32, 32).

[0060] In an embodiment of the present application, the step S4, through the coding module, performs 3D convolution and weight normalization on the normalized feature tensor after increasing the dimension to obtain the coded feature tensor. It should be understood that 3D convolution is a powerful feature extraction tool that can process the spatial information and spectral information of the image at the same time, and weight normalization is a method of reparameterizing the weights of the neural network, which adjusts the weights of the convolution layer so that the activation outputs of different channels have similar scales. This helps to accelerate the convergence of the network and improve the stability of the training. Combining the two helps to extract and fuse the spatial-spectral features of hyperspectral images, improve the training efficiency and stability of the model, and enhance the generalization ability of the model. Therefore, the normalized feature tensor after increasing the dimension is further subjected to 3D convolution and weight normalization operations by the coding module.

[0061] Specifically, the normalized feature tensor after the dimension increase is fed into the 3D convolution, and the 3D convolution kernel performs convolution operations on the normalized feature tensor after the dimension increase in the spatial dimension (height, width) and the spectral dimension, thereby learning the complex cross-dimensional correlation, compressing redundant information, reducing the number of network parameters, and improving computational efficiency. Among them, the shape of the data after convolution becomes (batch size, 1+channel, channel, height, weight) (12, 32, 31, 32, 32). Then, the features extracted by the 3D convolution are weighted normalized to obtain the encoded feature tensor.

[0062] In the embodiment of the present application, the step S5 performs deep feature extraction on the encoded feature tensor through a spectral-spatial residual module (SSRM). Specifically, Figure 2 As shown, the step S5 includes: S51, sending the encoded feature tensor to three dual 3D convolution modules in sequence, adding the data processed by the dual 3D convolution module to the result obtained by the previous dual 3D convolution as the input of the next dual 3D convolution module, repeating twice to form three residual shapes, wherein the outputs of the three dual 3D convolution modules are the first residual feature tensor, the second residual feature tensor and the third residual feature tensor respectively; S52, converting the third residual feature tensor into a 4D tensor, performing deep feature extraction on the converted 4D tensor through three space and channel series mixing modules to obtain a spatial channel mixed feature tensor, and restoring it to a 5D shape; S53, multiplying the first residual feature tensor and the second residual feature tensor by the super learning parameters respectively, and superimposing them with the spatial channel mixed feature tensor after restoring the 5D shape; S54, after extracting features from the superimposed data through the basic 3D convolution, applying the ReLU activation function to introduce nonlinearity, and inputting the processed data into a dual 3D convolution for feature refinement.

[0063] It should be understood that through the processing of the dual 3D convolution module, features of different scales can be captured, which is crucial for understanding the local and global structures in the image. Each dual 3D convolution module can be regarded as extracting features at different scales, which helps the model learn richer spatial and spectral information. In addition, by using the idea of ​​residual network (ResNet), by adding the data processed by the dual 3D convolution to the result obtained by the previous dual 3D convolution, the depth of the network can be increased without causing the gradient vanishing problem, while enhancing the feature transfer, making deep network training possible. Therefore, in order to better improve the super-resolution reconstruction quality of hyperspectral images, multi-level and multi-scale feature extraction and residual connection are performed on the encoded feature tensor. In this way, this method can better retain the detail information of the image while reducing noise and artifacts, thereby improving the accuracy and interpretability of the reconstructed image.

[0064] Specifically, the S51 includes: performing convolution encoding on the encoded feature tensor through a first dual 3D convolution module to obtain a first 3D convolution feature tensor, adding the first 3D convolution feature tensor to the encoded feature tensor to obtain a first residual feature tensor; performing convolution encoding on the first residual feature tensor through a second dual 3D convolution module to obtain a second 3D convolution feature tensor, adding the second 3D convolution feature tensor to the first residual feature tensor to obtain a second residual feature tensor; performing convolution encoding on the second residual feature tensor through a third dual 3D convolution module to obtain a third 3D convolution feature tensor, and adding the third 3D convolution feature tensor to the second residual feature tensor to obtain a third residual feature tensor.

[0065] Next, it should be understood that the spatial and channel series hybrid module is able to perform feature fusion in both spatial and channel dimensions. This hybrid can enhance the model's understanding of the spatial structure and the relationship between channels, thereby extracting richer feature representations. Therefore, the spatial and channel series hybrid module is used to perform deeper feature extraction on the third residual feature tensor. Further, considering that in deep learning, especially when processing images or hyperspectral data, 4D tensors (usually batch size, number of channels, depth, height, width) are a common data format, which allows the model to operate simultaneously in spatial dimensions (depth, height, width) and channel dimensions. Based on this, in order to better adapt to the processing requirements of the spatial and channel series hybrid module, the third residual feature tensor is first converted to a 4D tensor before feature extraction. In addition, in deep learning models, it is very important to maintain the dimensional consistency of the data, which ensures that the data can flow and process correctly between different parts of the network. Therefore, it is also necessary to restore the data after feature extraction to its original 5D shape.

[0066] Specifically, in S52, the data shape of the 4D tensor is [(B*C), C+1, H, W], that is, (batchsize*channel, channel+1, height, weight) (12*31, 32, 32, 32). At this time, the number of channels is channel+1. By increasing the number of channels, the network has more capacity to represent more detailed feature levels. Furthermore, considering that the subsequent modules use GPU computing, modern acceleration hardware usually optimizes the processing of spatial dimensional data during continuous memory access. Based on this, the data shape at this time is changed to [(B*C), H, W, C+1], that is, (batch size*channel, height, weight, channel+1) (12*31, 32, 32, 32).

[0067] In particular, if Figure 3 As shown in the figure, the implementation process of the spatial mixing module is as follows: first, the input data is shifted (similar to the relative position encoding in Transformer) and fed into three parallel linear layers, similar to the self-attention module of Transform, to obtain three key values ​​(i.e., matrices R, K, and V). K and V calculate the global attention result through a linear complexity bidirectional attention mechanism, and the calculated global attention result is multiplied by σ(R) that controls the output probability to adjust the intensity of attention. The sigmoid function compresses the value of R between 0 and 1 as a control factor for the output probability. The adjusted attention result is linearly projected for further feature processing. After the output linear projection, layer normalization is used to stabilize the features and reduce internal covariate shift. Then, the normalized features are multiplied with the original input features using element-by-element multiplication (⊙) to enhance important features and suppress unimportant features. Finally, the features processed by element-by-element multiplication are input into a linear layer to obtain the final output of the spatial mixing module. The specific formula is as follows:

[0068] R = OmniShift(x)W R , K = OmniShift(x)W K , V = OmniShift(x)W V

[0069]

[0070] sR=σ(R)

[0071] X'=sR⊙X

[0072] Among them, x is the input to the spatial mixing module, R is the query matrix (Query), which is used to calculate the attention weight with the key K, and W Ris the weight matrix of the query, K is the key matrix, representing the query features of the input features, used as the key features in the self-attention mechanism, and W K is the weight matrix for the key, which is used to generate the key matrix from the input features. V is the value matrix, which represents the value feature in the self-attention mechanism and contains the basic information for the output. W V is the weight matrix for values, used to generate the value matrix from the input, X is the global attention result, RUN_CUDA is the execution flag of the CUDA operation, used to control whether to run the code on the GPU, and the RUN_CUDA function is usually used for some complex operations in neural networks. By running custom operations on the GPU, it can speed up the calculation. B is the batch size, T is the time step, used for the time dimension in sequence data or time series analysis, C is the number of channels, the number of channels of the input feature, indicating the depth of the feature, spatialdecayj is the spatial decay factor, which may control the decay in the spatial dimension, used to weaken the feature response far from the center, spatialfirstj is the spatial initial parameter, which sets the feature distribution of the initial space, sR is the data of the query matrix R after the sigmoid activation function, σ represents the sigmoid function, X' is the output of the spatial mixing module, and ⊙ represents element-by-element multiplication.

[0073] Among them, OmniShift is data displacement, which shifts the input data x in spatial position to generate different feature arrangements. The structure consists of three 2D convolutions, and the convolution kernel windows are 1*1, 3*3 and 5*5 respectively. The data is fed into the three convolutions respectively, and the output results are multiplied by a hyperparameter with the input data respectively. After the input data is shifted, it enters the bidirectional attention mechanism together with the key (K) and value (V) generated by different features, which helps the model use a variety of different spatial feature arrangements in calculations and avoids the model relying only on features at fixed positions. This feature diversity helps the model capture complex spatial structures and contextual information.

[0074] In particular, if Figure 3 As shown in the figure, the implementation process of the channel mixing module is as follows: the output data of the spatial mixing module is used as the input data, the input data is shifted, the shifted data is processed by the activation function ReLu, and then the square root of the activated data is taken to obtain K1, and K1 is sent to the linear layer to obtain the matrix KV. The specific formula is as follows:

[0075]

[0076] KV=W KV K1+b KV

[0077] R1=OmniShift(x1)W R1

[0078] Output=σ(R1)⊙KV

[0079] Among them, W KV is the weight matrix of the linear layer, b KV is the bias term of the linear layer.

[0080] Further, it should be understood that the super-learning parameters enable the model to automatically optimize the feature combination according to the training data, improving the performance and generalization ability of the model. Based on this, the first residual feature tensor and the second residual feature tensor are multiplied by the super-learning parameters to obtain weighted feature tensors (these tensors reflect the importance of each feature and the contribution to the final task), so that the model can dynamically adjust the importance of different feature tensors, so that more important features contribute more to the final result. Further superimposing these weighted feature tensors with the spatial channel mixed feature tensors after restoring the 5D shape can not only integrate feature information from different sources to obtain a more comprehensive image representation. It can also promote the flow of gradients, improve training stability, and reduce the gradient disappearance problem in deep networks, similar to the skip connection or residual connection in the residual network (ResNet). Therefore, in order to enhance the model's learning and representation capabilities for features and improve the performance and efficiency of super-resolution reconstruction of hyperspectral images, the first residual feature tensor and the second residual feature tensor are multiplied by the super-learning parameters and then superimposed with the spatial channel mixed feature tensor after restoring the 5D shape.

[0081] Specifically, the specific implementation process of S53 is:

[0082] 1. Multiply by super-learning parameters: Multiply the first residual feature tensor and the second residual feature tensor by their respective super-learning parameters to obtain the first weighted feature tensor and the second weighted feature tensor. The super-learning parameters are learned during the model training process and are used to adjust the contribution or weight of the feature tensor in the final superposition.

[0083] 2. Superimpose feature tensors: Superimpose the first weighted feature tensor, the second weighted feature tensor, and the spatial channel mixed feature tensor after restoring the 5D shape. This step can be achieved by element-by-element addition (Add operation).

[0084] Through this process, the model is able to integrate feature information from different stages and adjust the weight of each feature through super-learning parameters in the hope of obtaining better feature representation, thereby improving the performance of hyperspectral image super-resolution reconstruction.

[0085] Then, considering that different feature tensors may contain redundant information or conflict in certain areas, direct superposition can easily lead to information confusion and may lack deep features extracted from the data. In addition, deep learning models require nonlinearity to learn and simulate complex functions. If the superimposed data is not processed by a nonlinear activation function, the model may only be able to capture linear relationships, which limits the model's expressive power and ability to handle complex tasks. Therefore, in order to capture deeper features and enable the model to better learn the complex patterns and relationships in the data, the superimposed data is subjected to feature extraction through the basic 3D convolution, and the ReLU activation function is applied to introduce nonlinearity, and the processed data is input into a double 3D convolution for feature refinement. This helps to improve the model's feature extraction ability, nonlinear mapping ability, and feature fusion effect, thereby improving the performance of hyperspectral image super-resolution reconstruction.

[0086] In an embodiment of the present application, step S6 repeats step S5 multiple times to obtain a reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module as the input of the next SSRM module. It should be understood that the feature tensor information is sent to the four SSRM modules in sequence, and each feature tensor processed by the SSRM module is fused with the initial feature tensor, which can better provide supplementary clues for learning. SSMR extracts high-frequency features, and the original features provide low-frequency features, which enhances the model's understanding of the image content, thereby achieving high-quality hyperspectral image reconstruction. Therefore, in order to better complete the image reconstruction work, the feature tensor is processed continuously through four SSRM modules, and after each processing, the obtained output feature tensor is merged with the initial input feature tensor as the input of the next SSRM module.

[0087] Specifically, the specific implementation process of S6 is:

[0088] 1. Initialize input: Pass the initial superposition data as input to the first SSRM module.

[0089] 2. First SSRM processing: The data is processed by the first SSRM module, which extracts and fuses spatial-spectral features through its internal 3D convolutional layers, activation functions, and other feature extraction mechanisms.

[0090] 3. Feature fusion: The output of the first SSRM module is fused with the output of the previous module (in the first iteration, this will be the initial input data). This fusion can be achieved through simple addition, more complex fusion mechanisms, or attention mechanisms, with the goal of integrating feature information at different levels.

[0091] 4. Subsequent SSRM iterations: The fused feature data is passed as input to the next SSRM module. This process is repeated four times, with each iteration further refining and improving the feature representation, gradually improving the quality of the reconstructed image.

[0092] 5. Iterative feature refinement: Each iteration allows the model to more deeply explore and utilize the spectral-spatial information in the data, thereby improving the accuracy and quality of the reconstructed image.

[0093] 6. Final reconstruction: After four iterations, the final output will be a reconstructed hyperspectral image with improved spectral and spatial resolution.

[0094] In the embodiment of the present application, the step S7 uses inverse 3D convolution to decode the reconstructed image. It should be understood that inverse 3D convolution is an inverse operation, which can quickly reconstruct the image and restore the detailed information of the image. Since step S2 performs mean normalization, inverse 3D convolution is used for decoding, and the mean subtracted in step S2 is added back, thereby restoring the data to the initial state to ensure the accuracy of the reconstructed image.

[0095] In particular, the overall loss of the Transformer and RNN-based hyperspectral super-resolution reconstruction method consists of two parts: the mean absolute error (L1) and the frequency domain high frequency loss (L_HFL), as well as a hyperparameter λ for balancing the two parts of the loss. The specific formula is expressed as:

[0096] L_total=L1+λL_HFL

[0097] L1 = nn.L1Loss(SR, label)

[0098] Among them, L_total is the overall loss; L1 is the mean absolute error, which is a measure of the pixel-level difference between the reconstructed image (SR) and the real image (label); L_HFL is the frequency domain high-frequency loss, which focuses on the high-frequency components of the image in the frequency domain. These components usually contain the edge and detail information of the image; λ is a hyperparameter used to balance the weight between L1 loss and L_HFL loss; nn.L1Loss is the loss function used to calculate the mean absolute error in PyTorch, SR is the reconstructed image, and label is the label of the real image.

[0099] It is worth mentioning that the criteria for measuring reconstruction are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0100] Among them, the Peak Signal-to-Noise Ratio (PSNR) is an index for measuring image quality, which measures the reconstruction quality of an image based on the Mean Squared Error (MSE). The higher the PSNR value, the better the image quality. The specific formula is expressed as:

[0101]

[0102] where PNSR is the Peak Signal-to-Noise Ratio, MAX l is the maximum value of the image color, MSE is the mean squared error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i, j) is the pixel value of the original image at position (i, j), K(i, j) is the pixel value of the reconstructed or predicted image at position (i, j), i is the row index with a value range of 0 ≤ i < m, and j is the column index with a value range of 0 ≤ j < n;

[0103] Among them, the Structural Similarity (SSIM) is used to evaluate the perceptual similarity of images, especially focusing on the similarity of brightness, contrast, and structural information of the images. The closer the SSIM value is to 1, the more similar the two images are in structure and the better the image quality. The specific formula is expressed as:

[0104] SSIM(X, Y) = l(X, Y)·c(X, Y)·s(X, Y)

[0105]

[0106] where l(X, Y) is the luminance contrast term, μ X 、μ Y represent the means of images X and Y respectively, c(X, Y) is the contrast term, σ X 、σ Y represent the variances of images X and Y respectively, s(X, Y) is the structural similarity term, C1, C2, and C3 are constants respectively, σ XY represents the covariance of images X and Y, X(i, j) is the pixel value at the i-th row and j-th column in image X, Y(i, j) is the pixel value at the i-th row and j-th column in image Y, H is the height (number of rows) of the image, and W is the width (number of columns) of the image.

[0107] In summary, the Transformer and RNN-based hyperspectral super-resolution reconstruction method according to the embodiment of the present application is explained, which first preprocesses and mean-normalizes the initial data set to obtain a normalized feature tensor, and adds a dimension to the first dimension of the normalized feature tensor, and then performs convolution operations and weight normalization of the spatial and spectral dimensions on the processed data through the encoding module to obtain a coded feature tensor, and then extracts deeper features of the coded feature tensor through four spectral-spatial residual modules to obtain a reconstructed image, and finally uses inverse 3D convolution to decode the reconstructed image. The present application achieves efficient and accurate reconstruction of hyperspectral images by connecting spatial mixing and channel mixing in series and adopting an improved linear attention mechanism to replace the traditional transform.

[0108] Embodiment 2:

[0109] Figure 4 The figure shows the overall architecture of the model of the high-spectral super-resolution reconstruction system based on Transformer and RNN according to an embodiment of the present application. Figure 4 As shown, the high-spectral super-resolution reconstruction system based on Transformer and RNN according to the embodiment of the present application includes: an encoding module, four spectral-spatial residual modules (SSRM, Spectral-Spatial Residual Module) and a decoding module;

[0110] The encoding module includes a 3D convolution and a weight normalized hidden function, wherein the 3D convolution kernel can perform convolution operations in three dimensions, namely, spatial dimensions (height, width) and spectrum;

[0111] The spectral-spatial residual module includes four dual 3D convolution modules, one basic 3D convolution, one ReLu activation function, and three spatial and channel series mixing modules, wherein the dual 3D convolution module includes two basic 3D convolutions, and the basic 3D convolution contains a 3D convolution, a normalization hidden function, and a ReLu activation function;

[0112] Wherein, the space and channel series mixing module is in the form of a space mixing module and a channel mixing module connected in series;

[0113] The decoding module is used to decode the reconstructed image using inverse 3D convolution.

[0114] Here, those skilled in the art will appreciate that the specific functions and operations of each step in the above-mentioned Transformer and RNN-based hyperspectral super-resolution reconstruction system have been referenced above. Figure 1It has been introduced in detail in the description of the Transformer and RNN based hyperspectral super-resolution reconstruction method, and therefore, its repeated description will be omitted.

[0115] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. A hyperspectral super-resolution reconstruction method based on Transformer and RNN, characterized in that: include: S1. Collect different types of images as the initial data set, where each type contains several images, and use MATLAB software to fuse several images of each type into a hyperspectral image; S2, performing mean normalization processing on the hyperspectral image to obtain a normalized feature tensor; S3. Add a dimension to the first dimension of the normalized feature tensor so that the data shape is adjusted to [B, 1, C, H, W]; S4, performing 3D convolution and weight normalization on the normalized feature tensor after the dimension increase through the encoding module to obtain an encoded feature tensor; S5, performing deep feature extraction on the encoded feature tensor through the spectral-spatial residual module (SSRM); S6, repeating step S5 multiple times to obtain a reconstructed image, wherein the data processed by the SSRM module is fused with the output of the previous SSRM module as the input of the next SSRM module; S7, decoding the reconstructed image using inverse 3D convolution; Wherein, the S5 includes: S51, sending the encoded feature tensor to three double 3D convolution modules in sequence, adding the data processed by the double 3D convolution module to the result obtained by the previous double 3D convolution as the input of the next double 3D convolution module, repeating twice to form three residual shapes, wherein the outputs of the three double 3D convolution modules are the first residual feature tensor, the second residual feature tensor and the third residual feature tensor respectively; S52, converting the third residual feature tensor into a 4D tensor, performing deep feature extraction on the converted 4D tensor through three spatial and channel series hybrid modules to obtain a spatial channel hybrid feature tensor, and restoring it to a 5D shape; S53, multiplying the first residual feature tensor and the second residual feature tensor by the super learning parameter respectively, and superimposing them with the spatial channel mixed feature tensor after the 5D shape is restored; S54, after extracting features from the superimposed data through basic 3D convolution, a ReLU activation function is applied to introduce nonlinearity, and the processed data is input into a double 3D convolution for feature refinement.

2. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 1 is characterized in that: In the S1, there are 31 images of each type in the initial data set, in PNG format, representing 31 bands of the hyperspectral spectrum, and MATLAB software is used to fuse the 31 images into a hyperspectral image in .mat format. The hyperspectral image contains two key values, namely low spatial resolution (LR) and high resolution multispectral (HR), where the size of the LR image is half of that of the HR image.

3. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 2 is characterized in that: The S51 includes: Convolutionally encoding the encoded feature tensor through a first dual 3D convolution module to obtain a first 3D convolutional feature tensor, and adding the first 3D convolutional feature tensor to the encoded feature tensor to obtain a first residual feature tensor; Convolutionally encoding the first residual feature tensor through a second dual 3D convolution module to obtain a second 3D convolution feature tensor, and adding the second 3D convolution feature tensor to the first residual feature tensor to obtain a second residual feature tensor; The second residual feature tensor is convolutionally encoded through a third dual 3D convolution module to obtain a third 3D convolution feature tensor, and the third 3D convolution feature tensor is added to the second residual feature tensor to obtain a third residual feature tensor.

4. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 3 is characterized in that: In S52, the data shape of the 4D tensor is [(B*C), C+1, H, W], and it is changed to [(B*C), H, W, C+1] to reduce memory jumps.

5. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 4 is characterized in that: In the S53, the spatial mixing module includes: R=OmniShift(x)W R ,K=OmniShift(x)W K ,V=OmniShift(x)W V sR=σ(R) X, =sR⊙X Where x is the input to the spatial blending module, OmniShift is the data shift, R is the query matrix, and W R is the query weight matrix, K is the key matrix, W K is the key weight matrix, V is the value matrix, W V is the weight matrix of the value, X is the global attention result, RUN_CUDA is the execution flag of the CUDA operation, B is the batch size, T is the time step, C is the number of channels, spatialdecayj is the spatial decay factor, spatialfirstj is the spatial initial parameter, sR is the data of the query matrix R after the sigmoid activation function, σ represents the sigmoid function, X' is the output of the spatial mixing module, and ⊙ represents element-by-element multiplication.

6. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 5, characterized in that: In the S53, the channel mixing module includes: KV=W KV ·K1+b KV R1=OmniShift(x1)W R1 Output=σ(R1)⊙KV Among them, is the weight matrix of the linear layer, b KV is the bias term of the linear layer.

7. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 6, characterized in that: The overall loss of hyperspectral super-resolution reconstruction is: L_total=L1+λL_HFL L1 = nn.L1Loss(SR, label) Among them, L_total is the overall loss, L1 is the mean absolute error, λ is a hyperparameter, L_HFL is the high-frequency loss in the frequency domain, nn.L1Loss is the loss function used to calculate the mean absolute error in PyTorch, SR is the reconstructed image, and label is the label of the real image.

8. The hyperspectral super-resolution reconstruction method based on Transformer and RNN according to claim 7, characterized in that: The reconstruction criteria are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The peak signal-to-noise ratio is: where PNSR is the peak signal-to-noise ratio, MAX l is the maximum value of the image color, MSE is the mean square error between the original image and the processed image, m is the vertical resolution of the image, n is the horizontal resolution of the image, I(i, j) is the pixel value of the original image at position (i, j), K(i, j) is the pixel value of the reconstructed or predicted image at position (i, j), i is the row index with a value range of 0 ≤ i < m, and j is the column index with a value range of 0 ≤ j < n; The structural similarities are: SSIM(X,Y)=l(X,Y)·c(X,Y)·s(X,Y) Among them, SSIM is the structural similarity, l(X,Y) is the brightness contrast term, μ X , μ Y Represent the mean of images X and Y respectively, c(X,Y) is the contrast term, σ X , σ Y denote the variance of images X and Y respectively, s(X,Y) is the structural similarity term, C1, C2, C3 are constants, σ XY Represents the covariance of images X and Y, where X(i,j) is the pixel value at the i-th row and j-th column in image X, Y(i,j) is the pixel value at the i-th row and j-th column in image Y, H is the height of the image (number of rows), and W is the width of the image (number of columns).

9. A high-spectral super-resolution reconstruction system based on Transformer and RNN, characterized in that: include: Encoding module, Spectral-Spatial Residual Module (SSRM) and decoding module; The encoding module includes 3D convolution and weight normalization hidden functions, wherein the 3D convolution kernel can perform convolution operations in three dimensions, namely spatial dimensions (height, width) and spectrum; The spectral-spatial residual module includes a dual 3D convolution module and a spatial and channel series mixing module, wherein the dual 3D convolution module includes two basic 3D convolutions, and the basic 3D convolution contains a 3D convolution, a normalized hidden function and a ReLu activation function; the spatial and channel series mixing module is a form of a spatial mixing module and a channel mixing module connected in series; The decoding module is used to decode the reconstructed image using inverse 3D convolution.

Images