Hyperspectral image super-resolution method and system based on channel MLP extraction of spectral features
Patent Information
- Application Number
- CN202310658797.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2026-09-08
- Estimated Expiration
- 2043-06-05
AI Technical Summary
[0005]本发明要解决的技术问题：针对现有技术的上述问题，提供一种基于空谱特征的高光谱图像超分方法及系统，本发明旨在实现一种效果更好、计算资源消耗更小的高光谱图像超分辨率技术，以解决高光谱图像空间分辨率低的问题
[0041] Compared with existing technologies, the present invention has the following advantages: Unlike previous methods that use convolution or Transformer, the present invention can complete the hyperspectral super-resolution process using only a channel MLP module. By using the channel MLP module and band grouping, sliding, and shuffling operations, local and global spectral information can be fully captured. Combining the channel MLP module with band grouping, sliding, and shuffling operations can fully extract spectral information to improve hyperspectral image reconstruction. Furthermore, by using band grouping and channel grouping methods to reduce computational consumption, the computational load of the network can be reduced, resulting in better reconstruction effect and lower computational consumption.
Smart Images
Figure CN116664406B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hyperspectral image processing technology, specifically relating to a hyperspectral image super-resolution method and system based on channel MLP for extracting spatial spectral features. Background Technology
[0002] Hyperspectral images (HSIs) are cubic images containing hundreds of spectral bands. They possess not only spatial information but also rich spectral information. HSIs are widely used in urban planning, agricultural applications, and military applications. However, due to limitations in sensor hardware, the number of photoelectrons reaching each spectral band is limited, resulting in low spatial resolution of hyperspectral images. This leads to difficulties in clearly identifying the texture and structure of ground features, and low distinction between different types of ground features, severely impacting the practical applications of hyperspectral images. Therefore, improving the spatial resolution of hyperspectral images is crucial.
[0003] Super-resolution (SMR) aims to reconstruct high-resolution images from low spatial resolution images. Hyperspectral image SMR algorithms can be divided into two types: fusion-based hyperspectral SMR methods and single-image-based hyperspectral SMR methods. Fusion-based hyperspectral SMR methods are techniques that obtain high spatial resolution hyperspectral images by fusing high spatial resolution auxiliary images and low spatial resolution hyperspectral images. Due to the high complexity and cost of imaging systems, it is difficult to obtain accurately registered high spatial resolution auxiliary images; therefore, fusion-based hyperspectral SMR methods have low practicality.
[0004] Hyperspectral super-resolution based on single images is more practical because it does not require auxiliary images. These methods can be categorized into: image interpolation-based, sparse representation, low-rank tensor-based, and deep learning-based methods. Algorithms based on image interpolation, sparse representation, and low-rank tensors struggle to fully extract the intrinsic features of hyperspectral images. Currently, deep learning-based hyperspectral single-image super-resolution algorithms utilize the powerful feature extraction capabilities of convolutional neural networks (CNNs) or Transformers to reconstruct hyperspectral images. However, existing methods cannot fully extract the spectral information of hyperspectral images, and the reconstruction results need improvement. Multilayer perceptrons (MLPs) have been successfully applied to natural image processing such as RGB images. Unlike RGB images, hyperspectral images possess rich spectral information and are characterized by large data volumes. CNN and Transformer-based methods struggle to fully capture spectral features, leading to spectral distortion and poor performance; moreover, previous methods are computationally expensive when processing hyperspectral images. Therefore, how to fully capture spectral detail and reduce computational cost remains an important challenge in the field of hyperspectral super-resolution. Summary of the Invention
[0005] The technical problem to be solved by this invention is to provide a hyperspectral image super-resolution method and system based on spatial spectral features, which addresses the above-mentioned problems in the prior art. This invention aims to achieve a hyperspectral image super-resolution technology with better performance and lower computational resource consumption, so as to solve the problem of low spatial resolution of hyperspectral images.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features includes generating a high-resolution hyperspectral image from an input low-resolution hyperspectral image using a super-resolution network model. The super-resolution network model includes:
[0008] The channel MLP module is used to extract shallow features F0 from the input low-resolution hyperspectral image;
[0009] The spatial spectral feature extraction unit, composed of n cascaded spatial spectral feature extraction modules, performs complex spatial spectral feature extraction on the shallow feature F0, and sums the outputs of all spatial spectral feature extraction modules through residual connection to obtain the spatial spectral feature F. m The spatial spectral feature extraction module consists of a channel MLP module combined with band grouping, band sliding, band shuffling, and channel grouping operations, used for spectral spatial feature extraction and reducing computational consumption.
[0010] The channel MLP module is used to process the input spatial spectral features F m Extract feature F;
[0011] The upsampling module is used to upsample the feature F to obtain the upsampled feature F. up ;
[0012] The reconstruction module is used to interpolate and upscale the input low-resolution hyperspectral image, and then combine the interpolated and upscaled image with the upsampled feature F. up The summation yields a high-resolution hyperspectral image.
[0013] Optionally, the spatial spectral feature extraction module includes:
[0014] Normalization layer LN is used to normalize the input features F i-1 Normalization is performed to obtain normalized features;
[0015] The local and global spectral information extraction module extracts channel features from normalized features using a channel MLP module, then splits the output into two paths along the channel dimension. One output uses only channel MLP to extract channel features, while the other output uses a channel MLP module combined with band grouping and band sliding operations to capture and fuse two different local spectral information. The fused local spectral feature map is then processed using a channel MLP module combined with band shuffling and band grouping operations to obtain global spectral features. Finally, the two outputs are fused using a channel MLP module to obtain the output of the local and global spectral modules.
[0016] Normalization layer LN is used to normalize the input feature map F i-1 The feature map obtained by adding the feature map output by the local and global spectral information modules is normalized;
[0017] The spatial group feature extraction module is used to extract spatial features from the normalized local and global spectral information output by combining CycleMLP with channel grouping operations.
[0018] The feedforward network module is used to process the input features F i-1 The feature map H is obtained by adding the output of the spatial group feature module. n After feature extraction, the resulting feature map is then compared with H. n The output feature F is obtained by adding the residuals. i .
[0019] Optionally, the local and global spectral information extraction module includes a three-channel MLP module and a group-sliding-shuffled MLP extraction module, with the feature F input to the local and global spectral information extraction module. i0 After feature extraction in the first channel MLP module, the output is divided into two equal outputs along the channel dimension. One output is the feature F. i1 As input to the second channel MLP module for extracting channel dimension information, the feature F of the other output is used. i2 The outputs of the second-channel MLP module and the group-sliding-shuffled MLP extraction module are combined and used as the input feature F of the third-channel MLP module for local and global spectral information extraction. i3 The grouping-sliding-shuffling MLP extraction module is used for feature F i2 Two different local spectral features are extracted using the channel MLP module, which combines band grouping and band sliding extraction. The two local features are then fused and the channel MLP module is used again to extract global spectral features by combining band shuffling and band grouping extraction. The third channel MLP module also includes a dropout function at the end to alleviate the overfitting problem of the super-resolution network model.
[0020] Optionally, the group-slide-shuffle MLP extraction module includes: input F i2 First, after band grouping and band sliding plus band grouping operations, two parallel outputs are obtained; the grouped spectral feature aggregation module LSIB is used to process feature F. i2 One output after band grouping is used to extract a local spectral feature using the channel MLP module; the sliding grouped spectral feature aggregation module SSIB is used to aggregate feature F. i2 The output of the band sliding and band grouping operations is used to extract a second type of local spectral feature using the channel MLP module. A merging module combines the two types of local spectral features output from the grouped spectral feature aggregation module LSIB and the sliding grouped spectral feature aggregation module SSIB. The channel MLP module fuses the merged local spectral features. The global spectral feature extraction module GSIB extracts global spectral features from the fused local spectral features by sequentially shuffling and grouping them by band, and outputs the extracted global spectral features as the grouping-sliding-shuffling MLP extraction module. The band grouping operation groups the feature map along the band dimension, the band sliding operation slides the feature map along the band dimension, and the band shuffling operation shuffles the feature map along the band dimension.
[0021] Optionally, the spatial group feature extraction module includes:
[0022] The deformation module is used to deform the feature map of size N×C×B×H×W formed by the normalized local and global spectral features to obtain a feature map of size (N×B)×C×H×W.
[0023] The channel MLP module is used to extract channel features from a feature map of size (N×B)×C×H×W.
[0024] The shuffling module is used to shuffle the channel order of a feature map of size (N×B)×C×H×W along the channel dimension, and group the shuffled feature map along the channel dimension to generate multiple groups of sub-feature maps.
[0025] CycleMLP unit is used for spatial feature extraction from each group of sub-feature maps;
[0026] The merging module is used to merge multiple sets of spatial features output by CycleMLP units;
[0027] The channel MLP module is used for the fusion of multiple sets of spatial features after merging.
[0028] The deformation module is used to deform the spatial group features of size (N×B)×C×H×W to obtain a feature map of size N×C×B×H×W. At the end, a dropout function is also included to alleviate the overfitting problem of the super-resolution network model.
[0029] Optionally, the CycleMLP unit includes multiple parallel CycleMLP networks with identical structures. The number of CycleMLP networks is the same as the number of sub-feature map groups. Each CycleMLP network is used to extract spatial features for a corresponding group of sub-feature maps.
[0030] Optionally, the step of generating a high-resolution hyperspectral image from an input low-resolution hyperspectral image using a super-resolution network model includes:
[0031] S1) Obtain real high-resolution hyperspectral images and use the real high-resolution hyperspectral images to create a training set that pairs low-resolution and high-resolution hyperspectral images.
[0032] S2) Construct a super-resolution network model;
[0033] S3) Input the low-resolution hyperspectral images from the paired training set into the super-resolution network model, obtain the super-resolution hyperspectral images based on the super-resolution network model, calculate the loss of the super-resolution hyperspectral images and their corresponding real high-resolution hyperspectral images using a preset loss function, iteratively update the network weight parameters of the super-resolution network model according to the loss until convergence, and save the network weight parameters of the final super-resolution network model to obtain the trained super-resolution network model.
[0034] S4) Obtain the low-resolution hyperspectral image to be processed;
[0035] S5) Input the low-resolution hyperspectral image to be processed into the trained super-resolution network model to obtain the corresponding super-resolution hyperspectral image.
[0036] Optionally, the loss function used in step S3) has the following functional expression:
[0037]
[0038] In the above formula, L1 represents the loss function, B is the number of bands, H is the height of the image, W is the width of the image, and I... GT For true high-resolution hyperspectral images, I SR For the super-resolution hyperspectral image obtained from the super-resolution network model, I GT (b,i,j) is I GT The pixel value at coordinates (i,j) in the b-th band, I SR(b,i,j) is I SR The pixel value at coordinates (i,j) in the b-th band.
[0039] Furthermore, the present invention also provides a hyperspectral image super-resolution system based on channel MLP for extracting spatial spectral features, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features.
[0040] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to execute the hyperspectral image super-resolution method for extracting spatial spectral features based on channel MLP.
[0041] Compared with existing technologies, the present invention has the following advantages: Unlike previous methods that use convolution or Transformer, the present invention can complete the hyperspectral super-resolution process using only a channel MLP module. By using the channel MLP module and band grouping, sliding, and shuffling operations, local and global spectral information can be fully captured. Combining the channel MLP module with band grouping, sliding, and shuffling operations can fully extract spectral information to improve hyperspectral image reconstruction. Furthermore, by using band grouping and channel grouping methods to reduce computational consumption, the computational load of the network can be reduced, resulting in better reconstruction effect and lower computational consumption. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the network structure of the super-resolution network model in an embodiment of the present invention.
[0043] Figure 2 This is a schematic diagram of the network structure of the spatial spectrum feature extraction module in an embodiment of the present invention.
[0044] Figure 3 This is a schematic diagram of the network structure of the local and global spectral information extraction module in an embodiment of the present invention.
[0045] Figure 4 This is a schematic diagram of the network structure of the group-sliding-shuffled MLP feature extraction module in an embodiment of the present invention.
[0046] Figure 5 This is a schematic diagram of the network structure of the spatial group feature extraction module in an embodiment of the present invention.
[0047] Figure 6 This is a schematic diagram comparing the conventional channel MLP module and the CycleMLP in an embodiment of the present invention.
[0048] Figure 7 This is a schematic diagram of the network structure of the upsampling module in an embodiment of the present invention.
[0049] Figure 8 This is a schematic diagram illustrating the training and testing process of the super-resolution network model in an embodiment of the present invention. Detailed Implementation
[0050] This embodiment of the hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features includes generating a high-resolution hyperspectral image from an input low-resolution hyperspectral image using a super-resolution network model.
[0051] like Figure 1 As shown, the super-resolution network model includes:
[0052] The channel MLP (Multilayer Perceptron) module is used to extract shallow features F0 from the input low-resolution hyperspectral image;
[0053] The spatial spectral feature extraction unit, composed of n cascaded spatial spectral feature extraction modules, performs complex spatial spectral feature extraction on the shallow feature F0, and sums the outputs of all spatial spectral feature extraction modules through residual connection to obtain the spatial spectral feature F. m The spatial spectral feature extraction module consists of a channel MLP module combined with band grouping, band sliding, band shuffling, and channel grouping operations, used for spectral spatial feature extraction and reducing computational consumption.
[0054] The channel MLP module is used to process the input spatial spectral features F m Extract feature F;
[0055] The upsampling module is used to upsample the feature F to obtain the upsampled feature F. up ;
[0056] The reconstruction module is used to interpolate and upscale the input low-resolution hyperspectral image, and then combine the interpolated and upscaled image with the upsampled feature F. up The summation yields a high-resolution hyperspectral image.
[0057] It should be noted that the number n of spatial spectral feature extraction modules in the spatial spectral feature extraction unit can be chosen according to actual needs. For example, as an optional implementation, n = 4 in this embodiment. The residual connection of the n spatial spectral feature extraction modules adopts a known connection structure, specifically, the n spatial spectral feature extraction modules are cascaded sequentially, and the outputs of the n spatial spectral feature extraction modules are respectively connected to the output of the last spatial spectral feature extraction module through skip connections, such as... Figure 1 As shown.
[0058] It should be noted that the channel MLP (Multilayer Perceptron) module is a fully connected network module along the channel dimension, and the weight initialization of all channel MLP modules adopts the Kamin initialization method.
[0059] like Figure 2 As shown, the spatial spectral feature extraction module includes:
[0060] A normalization layer (Layer Norm (LN)) is used to normalize the input features F. i-1 Normalization is performed to obtain normalized features;
[0061] The local and global spectral information extraction module extracts channel features from normalized features using a channel MLP module, then splits the output into two paths along the channel dimension. One output uses only channel MLP to extract channel features, while the other output uses a channel MLP module combined with band grouping and band sliding operations to capture and fuse two different local spectral information. The fused local spectral feature map is then processed using a channel MLP module combined with band shuffling and band grouping operations to obtain global spectral features. Finally, the two outputs are fused using a channel MLP module to obtain the output of the local and global spectral modules.
[0062] Normalization layer LN is used to normalize the input feature map F i-1 The feature map obtained by adding the feature map output by the local and global spectral information modules is normalized;
[0063] The spatial group feature extraction module is used to extract spatial features from normalized local and global spectral information by combining CycleMLP with channel grouping.
[0064] The feedforward network module is used to process the input features F i-1 The feature map H is obtained by adding the output of the spatial group feature module. n After feature extraction, the extracted feature map is then compared with H. n The output feature F is obtained by adding the residuals. i In this embodiment, the feedforward network module consists of a cascaded MLP module, a nonlinear activation function, a dropout function, and a channel MLP module, used to extract channel information and enhance the ability to represent channel information.
[0065] like Figure 3 As shown, the local and global spectral information extraction module includes a three-channel MLP module and a group-sliding-shuffled MLP extraction module (GSSMB module for short). The feature F input to the local and global spectral information extraction module is... i0 After feature extraction in the first channel MLP module, the output is split into two paths: one output shows the feature F. i1 As input to the second channel MLP module for extracting channel dimension information, the feature F of the other output is used. i2The outputs of the second-channel MLP module and the group-sliding-shuffled MLP extraction module are combined and used as the feature F of the third-channel MLP module as the input for extracting local and global spectral information. i3 The grouping-sliding-shuffling MLP extraction module is used for feature F i2 This embodiment employs a channel MLP module that combines band grouping and band sliding operations to extract two types of local spectral features in parallel and then fuses them. The fused local spectral features are then further processed using the channel MLP module with band shuffling and band grouping operations to extract global spectral features. In this embodiment, the third channel MLP module also includes a dropout function to mitigate overfitting. The dropout function in this embodiment is a well-known function for preventing overfitting; it randomly discards some network neurons during training, thus preventing each neuron from becoming abnormally sensitive to activation of specific neurons. This allows the network to learn more generalized features, thereby alleviating the overfitting problem.
[0066] like Figure 4 As shown, the group-slide-shuffle MLP extraction module includes: input F i2 First, after band grouping and band sliding plus band grouping operations, two parallel outputs are obtained; the grouped spectral feature aggregation module LSIB is used to process the feature map F i2 One output after band grouping is used to extract a local spectral feature using the channel MLP module; the sliding grouped spectral feature aggregation module SSIB is used to aggregate feature F. i2 The output after sequential band sliding and band grouping operations is used to extract a second type of local spectral feature using the channel MLP module; the merging module is used to merge the two types of local spectral features output by the grouped spectral feature aggregation module LSIB and the sliding grouped spectral feature aggregation module SSIB; the channel MLP module is used to fuse the merged local spectral features; the global spectral feature extraction module GSIB is used to sequentially perform band shuffling and band grouping operations on the fused local spectral features and then use the channel MLP module to extract global spectral features, and the extracted global spectral features are used as the feature output of the grouping-sliding-shuffling MLP extraction module; the band grouping operation is to group the feature map along the band dimension, the band sliding operation is to slide the feature map along the band dimension, and the band shuffling operation is to shuffle the feature map along the band dimension. Figure 4In this context, the input feature map has a size of H×W×B×c, where H (height) represents the input height, W (width) represents the input width, B (band) represents the number of input bands, and c (channel) represents the input channel dimension. D represents that the input feature map is divided into D sub-feature maps along the spectral dimension; T represents the number of bands contained in each sub-feature map; CMLP represents the channel MLP (Multilayer Perceptron) module. Band grouping refers to grouping the feature map along the band dimension B, such as... Figure 4 Input F in i2 There are 6 bands in total, numbered "1, 2, 3, 4, 5, 6". They are grouped into pairs, resulting in "(1, 2), (3, 4), (5, 6)". Band sliding refers to sliding the feature map along the band dimension B in order of the bands. The original positions will be moved and changed, for example, from the number "1, 2, 3, 4, 5, 6" to "2, 3, 4, 5, 6, 1". Band shuffling refers to rearranging the band order of the feature map along the band dimension B. The original positions of some bands will be changed, for example, from the number "1, 2, 3, 4, 5, 6" to "1, 3, 5, 2, 4, 6".
[0067] like Figure 5 As shown, the spatial group feature extraction module includes:
[0068] The deformation module is used to deform the feature map of size N×C×B×H×W formed by the normalized local and global spectral features to obtain a feature map of size (N×B)×C×H×W.
[0069] The channel MLP module is used to extract channel features from a feature map of size (N×B)×C×H×W.
[0070] The shuffling module is used to shuffle the channel order of a feature map of size (N×B)×C×H×W along the channel dimension, and group the shuffled feature map along the channel dimension to generate multiple groups of sub-feature maps.
[0071] The CycleMLP unit is used to extract spatial features from each group of sub-feature maps;
[0072] The merging module is used to merge multiple sets of spatial features output by CycleMLP units;
[0073] The channel MLP module is used for the fusion of multiple sets of spatial features after merging.
[0074] The deformation module is used to deform the feature map of size (N×B)×C×H×W to obtain an N×C×B×H×W feature map. Furthermore, in this embodiment, the spatial group feature extraction module also includes a dropout function at the end to alleviate the overfitting problem of the spatial feature group information extraction module. The dropout function in this spatial group feature extraction module is consistent with the dropout function in the aforementioned local and global spectral information extraction modules.
[0075] In this embodiment, the CycleMLP unit comprises multiple parallel CycleMLP networks with identical structures. The number of CycleMLP networks is the same as the number of sub-feature map groups. Each CycleMLP network is used to extract spatial features for a corresponding group of sub-feature maps. The reference for CycleMLP is Chen, S, et al. "CycleMLP: A MLP-like Architecture for Dense Prediction". ICLR 2021, arXiv:2107.10224. The CycleMLP network is a well-known network structure, consisting of three parallel network branches: the first branch consists of a single-channel MLP module; the second branch consists of a single-channel MLP module and a translation / sliding operation along the feature map's height dimension; the third branch consists of a single-channel MLP module and a translation / sliding operation along the feature map's width dimension. After obtaining the outputs of the three branches, they are added and fused, and a single-channel MLP module is used to further extract features from the fused features. Figure 6 As shown in (a), a conventional channel MLP (Multilayer Perceptron) module extracts features along the channel C dimension for a fully connected layer; a conventional channel MLP module aggregates channel information along the channel dimension; from a spatial perspective, it can only aggregate channel information for a single pixel spatial location. Figure 6 As shown in (b), the CycleMLP network aggregates information along the channel dimension, but it samples information from different channels along the channel dimension through a stair-step sampling strategy; from a spatial perspective, it can aggregate information from different spatial locations, thereby capturing a larger spatial receptive field and aggregating more spatial details.
[0076] like Figure 7 As shown, the upsampling module includes a 3-channel MLP module and a 1-pixel rearrangement operation module. Figure 7The input feature map is represented by a circled S and is used to upsample the input feature map in a band-by-band manner. In this example, all bands of the input feature map are first split into single-band feature maps along the band dimension, and the number of single-band feature maps is the same as the number of bands. Each single-band feature map is passed through the first channel MLP module to increase the number of channels, then through the pixel rearrangement operation module to perform pixel rearrangement operation to enlarge the image, and then through the second channel MLP to reduce the number of channels, resulting in a single-band super-resolution image after the single-band feature is magnified. Finally, all single-band super-resolution images are merged, and the third channel MLP is used to fuse spectral information along the band dimension to avoid spectral distortion. Figure 7 The example demonstrates the upsampling process for a single-band feature map, but it is applicable to all single-band feature maps. In this example, to reduce the number of parameters, the weight parameters of the three-channel MLP modules of the upsampling module are shared by all single-band feature maps.
[0077] like Figure 8 As shown, generating a super-resolution hyperspectral image from an input low-resolution hyperspectral image using a super-resolution network model includes:
[0078] S1) Obtain real high-resolution hyperspectral images and use the real high-resolution hyperspectral images to create a training set that pairs low-resolution and high-resolution hyperspectral images.
[0079] S2) Construct a super-resolution network model;
[0080] S3) Input the low-resolution hyperspectral images from the paired training set into the super-resolution network model. Based on the super-resolution network model, obtain the super-resolution hyperspectral images and their corresponding real high-resolution hyperspectral images. Use a preset loss function to calculate the loss, and iteratively update the network weight parameters of the super-resolution network model according to the loss until convergence. Save the final network weight parameters of the super-resolution network model to obtain the trained super-resolution network model.
[0081] S4) Obtain the low-resolution hyperspectral image to be processed;
[0082] S5) Input the low-resolution hyperspectral image to be processed into the trained super-resolution network model to obtain the corresponding super-resolution hyperspectral image.
[0083] In this embodiment, the function expression of the loss function used in step S3) is:
[0084]
[0085] In the above formula, L1 represents the loss function, B is the number of bands, H is the height of the image, W is the width of the image, and I... GT For true high-resolution hyperspectral images, I SRFor the super-resolution hyperspectral image obtained from the super-resolution network model, I GT (b,i,j) is I GT The pixel value at coordinates (i,j) in the b-th band, I SR (b,i,j) is I SR The pixel value at coordinates (i,j) in the b-th band.
[0086] To verify the method of this embodiment, the effectiveness and practicality of the method are tested using two publicly available datasets, Houston and Chikusei. The Houston dataset has 48 bands, and the Chikusei dataset has 128 bands. Super-resolution experiments are conducted on both the Houston and Chikusei datasets. The high-resolution images in the training set required for the experiments are first cropped from the original datasets, while low-resolution images are obtained by degrading the high-resolution images using Gaussian downsampling. The method of this embodiment is compared with existing methods such as Bicubic, VDSR, EDSR, 3DFCNN, GRDN, HLNnet, and Interactformer. For the Bicubic method, see Miller, FP, et al., “Bicubic Interpolation”; Alphascript Publishing: Riga, Latvia, 2010; for VDSR, see Kim, J, et al., “Accurate Image Super-Resolution Using Very Deep Convolutional Networks”, Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016; for EDSR, see Lim, B, et al., “Enhanced Deep Residual Networks for Single Image Super-Resolution”, Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2017; for 3DFCNN, see Mei, S, et al., “Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network”. Sens.2017,9,1139; For GRDN, please refer to the literature Zhao,M,et.al. "Hyperspectral Image Super-Resolution under the Guidance of DeepGradient Information".Remote Sens.2021, 13, 2382; HLNnet can be found in Yang, J, et al., “Hybrid Local and Nonlocal 3-DAttentive CNN for Hyperspectral Image Super-Resolution”. IEEE Geosci. Remote Sens. Lett. 2021, 18, 1274–1278; Interactformer can be found in Liu, Y, et al., “Interactformer: Interactive Transformer and CNN for Hyperspectral Image Super-Resolution”. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–15. Through practical testing, the method of this embodiment achieves better reconstruction results than other methods on the Houston dataset, particularly in the reconstruction of architectural details; other methods exhibit blurring or boundary artifacts. The super-resolution effect of the method of this embodiment is also better than other methods on the Chikusei dataset, and the reconstruction result image of the method of this embodiment can reconstruct more details. The super-resolution results reconstructed by all methods show that the method in this embodiment performs best. The superiority of this method is also evident through subjective evaluation of image quality.
[0087] Furthermore, this embodiment also verifies the effectiveness of the invention through objective evaluation metrics; the objective evaluation metrics used are MPSNR (Mean Peak Signal-to-Noise Ratio), MSSIM (Mean Structural Similarity), and SAM (Spectral Angle Mapping). MPSNR is used to evaluate the pixel difference between the super-resolution image and the reference image; MSSIM is used to evaluate the structural similarity between the super-resolution image and the reference image; and SAM is used to evaluate the fidelity of the spectral information of the super-resolution image. Higher MPSNR and MSSIM values indicate better image reconstruction, while lower SAM values indicate better preservation of the image's spectral information. The objective evaluation metrics for different methods are shown in Tables 1 and 2.
[0088] Table 1 Evaluation metrics for the 16 test images in the Houston dataset
[0089]
[0090] Table 2 Evaluation metrics for 14 test images from the Chikusei dataset
[0091]
[0092] In Tables 1 and 2, ×2, ×3, and ×4 represent 2x, 3x, and 4x super-resolution experiments, respectively. From the values of the three objective evaluation metrics in Tables 1 and 2, it can be seen that the method in this embodiment outperforms other convolutional or Transformer-based methods.
[0093] Furthermore, the method in this embodiment can effectively reduce computational consumption. This embodiment also uses the objective metric of floating-point operations (FLOPs) to evaluate the computational consumption of the network; a higher FLOPs indicates greater computational consumption. The objective metrics of computational consumption for different methods are shown in Table 3. FLOPs were calculated using a 144×144 Chikusei image to be processed during network testing.
[0094] Table 3 Comparison of consumption calculation methods
[0095] FLOPs(G) 220 208 640 556 2960 2530 124 MPSNR (dB) 28.41971 27.94085 28.23195 28.70123 29.13901 29.15101 29.51723
[0096] As can be seen from Table 3, the method in this embodiment has the least amount of floating-point operations, proving that the method of the present invention has the least computational consumption and the best performance.
[0097] In summary, this embodiment provides a hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features. The method includes constructing a paired training set of low-resolution and high-resolution hyperspectral images; building a hyperspectral image super-resolution model, inputting the training set into the network, iteratively updating the network weights until convergence, and obtaining the final network model weights; obtaining an unprocessed low-resolution hyperspectral image; inputting the unprocessed low-resolution hyperspectral image into the pre-trained hyperspectral super-resolution model, and outputting a high-resolution hyperspectral image, thus completing the super-resolution process. Unlike previous methods using convolution or Transformer, this embodiment only uses a channel MLP module to complete the hyperspectral super-resolution process. Utilizing the channel MLP module and band grouping, sliding, and shuffling operations can fully capture local and global spectral information. Combining the channel MLP module with band grouping and channel grouping operations reduces the computational load of the network, resulting in better reconstruction performance, lower computational resource consumption, and higher reliability.
[0098] Furthermore, this embodiment also provides a hyperspectral image super-resolution system based on channel MLP for extracting spatial spectral features, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features.
[0099] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to execute the hyperspectral image super-resolution method for extracting spatial spectral features based on channel MLP.
[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0101] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A hyperspectral image super-resolution method based on channel MLP for extracting spatial-spectral features, characterized in that, This includes using a super-resolution network model to generate a high-resolution hyperspectral image from an input low-resolution hyperspectral image, wherein the super-resolution network model includes: The channel MLP module is used to extract shallow features F0 from the input low-resolution hyperspectral image; The spatial spectral feature extraction unit, composed of n cascaded spatial spectral feature extraction modules, performs complex spatial spectral feature extraction on the shallow feature F0, and sums the outputs of all spatial spectral feature extraction modules through residual connection to obtain the spatial spectral feature F. m The spatial spectral feature extraction module consists of a channel MLP module combined with band grouping, band sliding, band shuffling, and channel grouping operations, used for spectral spatial feature extraction and reducing computational consumption. The channel MLP module is used to process the input spatial spectral features F m Extract feature F; The upsampling module is used to upsample the feature F to obtain the upsampled feature F. up ; The reconstruction module is used to interpolate and upscale the input low-resolution hyperspectral image, and then combine the interpolated and upscaled image with the upsampled features F. up Adding them together yields a high-resolution hyperspectral image; The spatial spectral feature extraction module includes: Normalization layer LN is used to normalize the input features F i-1 Normalization is performed to obtain normalized features; The local and global spectral information extraction module extracts channel features from normalized features using a channel MLP module, then splits the output into two paths along the channel dimension. One output uses only channel MLP to extract channel features, while the other output uses a channel MLP module combined with band grouping and band sliding operations to capture and fuse two different local spectral information. The fused local spectral features are then processed using a channel MLP module combined with band shuffling and band grouping operations to obtain global spectral features. Finally, the two outputs are fused using a channel MLP module to obtain the output of the local and global spectral modules. Normalization layer LN is used to normalize the input feature map F i-1 The feature map obtained by adding the feature maps output by the local and global spectral information modules is normalized; The spatial group feature extraction module is used to extract spatial features from the normalized local and global spectral information feature maps using CycleMLP combined with channel grouping operations. The feedforward network module is used to process the input features F i-1 The feature map H is obtained by adding the output of the spatial group feature module. n After feature extraction, the resulting feature map is then compared with H. n The output feature F is obtained by adding the residuals. i .
2. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 1, characterized in that, The local and global spectral information extraction module includes a three-channel MLP module and a group-sliding-shuffle MLP extraction module. The input features F of the local and global spectral information extraction module are... i0 After feature extraction in the first channel MLP module, the output is split into two paths: one output shows the feature F. i1 As input to the second channel MLP module for extracting channel information, the feature F of the other output is used. i2 The outputs of the second-channel MLP module and the group-sliding-shuffled MLP extraction module are combined and used as the input feature F of the third-channel MLP module to extract local and global spectral information. i3 The grouping-sliding-shuffling MLP extraction module is used for feature F i2 The channel MLP module combines two different local spectral features—band grouping and band sliding extraction—and then fuses them. The fused feature map is then used to extract global spectral features using the channel MLP module combined with band grouping and band shuffling operations. The third channel MLP module also includes a dropout function at the end to alleviate the overfitting problem of the super-resolution network model.
3. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 2, characterized in that, The grouping-sliding-shuffled MLP extraction module includes: input F i2 First, after band grouping and band sliding plus band grouping operations, two parallel outputs are obtained; the grouped spectral feature aggregation module LSIB is used to process feature F. i2 After band grouping, one output channel uses the channel MLP module to extract local spectral features; the sliding group spectral feature aggregation module SSIB is used to aggregate features F. i2 The system sequentially performs band sliding and band grouping, and then uses the channel MLP module to extract local spectral features. A merging module combines the local spectral features output from the grouped spectral feature aggregation module LSIB and the sliding grouped spectral feature aggregation module SSIB. The channel MLP module fuses the merged local spectral features. The global spectral feature extraction module GSIB sequentially performs band shuffling and band grouping on the fused local spectral features, and then uses the channel MLP module to extract global spectral features, using the extracted global spectral features as the output of the grouping-sliding-shuffling MLP extraction module. The band grouping operation groups the feature map along the band dimension, the band sliding operation slides the feature map along the band dimension, and the band shuffling operation shuffles the feature map along the band dimension.
4. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 2, characterized in that, The spatial group feature extraction module includes: The deformation module is used to deform the feature map of size N×C×B×H×W formed by the normalized local and global spectral features to obtain a feature map of size (N×B)×C×H×W. The channel MLP module is used to extract channel features from a feature map of size (N×B)×C×H×W. The shuffling module is used to shuffle the channel order of a feature map of size (N×B)×C×H×W along the channel dimension, and group the shuffled feature map along the channel dimension C to generate multiple groups of sub-feature maps. CycleMLP unit is used for spatial feature extraction from each group of sub-feature maps; The merging module is used to merge multiple sets of spatial features output by CycleMLP units; The channel MLP module is used to fuse multiple sets of merged spatial features. The deformation module is used to deform the feature map of size (N×B)×C×H×W to obtain the feature map of size N×C×B×H×W. It also includes a dropout function at the end to alleviate the overfitting problem of the super-resolution network model.
5. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 4, characterized in that, The CycleMLP unit comprises multiple parallel CycleMLP networks with identical structures. The number of CycleMLP networks is the same as the number of sub-feature map groups. Each CycleMLP network is used to extract spatial features for a corresponding group of sub-feature maps.
6. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 1, characterized in that, The process of generating a high-resolution hyperspectral image from an input low-resolution hyperspectral image using a super-resolution network model includes: S1) Obtain real high-resolution hyperspectral images and use the real high-resolution hyperspectral images to create a training set that pairs low-resolution and high-resolution hyperspectral images; S2) Construct a super-resolution network model; S3) Input the low-resolution hyperspectral images from the paired training set into the super-resolution network model, obtain the super-resolution hyperspectral images based on the super-resolution network model, calculate the loss between the super-resolution hyperspectral images and their corresponding real high-resolution hyperspectral images using a preset loss function, iteratively update the network weight parameters of the super-resolution network model based on the loss until convergence, and save the final network weight parameters of the super-resolution network model to obtain the trained super-resolution network model. S4) Obtain the low-resolution hyperspectral image to be processed; S5) Input the low-resolution hyperspectral image to be processed into the trained super-resolution network model to obtain the corresponding super-resolution hyperspectral image.
7. The hyperspectral image super-resolution method based on channel MLP for extracting spatial spectral features according to claim 6, characterized in that, The loss function used in step S3) has the following expression: In the above formula, Represents the loss function. B For the number of bands, H For the height of the image, W The width of the image, For true high-resolution hyperspectral images, The super-resolution hyperspectral image is obtained from the super-resolution network model. for The Middle b Coordinates in each band ( i , j The pixel value at () for The Middle b Coordinates in each band ( i , j The pixel value at ().
8. A hyperspectral image super-resolution system for extracting spatial-spectral features based on channel MLP, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the hyperspectral image super-resolution method for extracting spatial spectral features based on channel MLP as described in any one of claims 1 to 7.
9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the hyperspectral image super-resolution method for extracting spatial spectral features based on channel MLP as described in any one of claims 1 to 7.