Hyperspectral image super-resolution reconstruction method based on multi-scale space-spectrum feature learning

Abstract

A hyperspectral image super-resolution reconstruction method based on multi-scale space-spectrum feature learning, a high-resolution hyperspectral image-low-resolution hyperspectral image image pair training set is established, a multi-scale space-spectrum feature learning network is constructed, the initial shallow feature map F0 of the low-resolution hyperspectral image is extracted by the shallow feature extraction module, the multi-scale shallow space-spectrum feature map F is obtained based on F0 s ; the DPMSSFN adopts the cascaded DPMSSFB to extract residual features in different receiving domains, and the space-spectrum feature maps F1,..., F d ,..., F D of different levels are obtained, the feature fusion module fuses them to obtain the deep space-spectrum feature map F r ; the image reconstruction module obtains the reconstructed super-resolution image based on F0 and F r ; the multi-scale space-spectrum feature learning network is trained, and the obtained network model can be used for reconstructing the hyperspectral image to be repaired; the present application considers the shallow feature information, the deep space-spectrum feature information and the spectral correlation of the image, and the result is clearer.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral remote sensing image technology, specifically hyperspectral image super-resolution reconstruction technology. More particularly, it relates to a hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning, and especially to a method utilizing a hybrid spectral attention mechanism-guided multi-scale feature learning approach to fully leverage the characteristics of shallow and deep multi-scale spatial-spectral information. This method has achieved good performance validation on publicly available datasets. Background Technology

[0002] Due to the rapid development of imaging spectroscopy and the widespread application of computers, hyperspectral imaging technology has developed rapidly, becoming one of the important methods for land cover identification and detection in the field of remote sensing. Hyperspectral imaging systems can acquire images of the same scene across hundreds of consecutive bands. The acquired hyperspectral image is a three-dimensional data cube with two-dimensional spatial information and one-dimensional spectral information. Because of its fine and continuous spectral information, it has excellent resolution and is commonly used in remote sensing, vegetation detection, disaster management, and other fields. However, limited by sensor technology, increasing the spectral resolution of an image requires increasing the sensor's instantaneous field of view to obtain sufficient photons to meet the acceptable signal-to-noise ratio. Increasing the instantaneous field of view also increases the area acquired within the same sensor, meaning a single pixel in a hyperspectral image represents a wider area of ​​the Earth's surface. Therefore, the design of hyperspectral image acquisition equipment requires a trade-off between spatial and spectral information, often sacrificing some spatial content to ensure the number of spectral channels. Low spatial resolution, however, cannot provide detailed texture features, limiting the further applications of hyperspectral images, such as advanced tasks like small target detection and change detection. The spatial resolution of images is typically improved by enhancing hardware and utilizing algorithms. However, improving hardware is expensive and places high demands on existing engineering techniques. Therefore, using algorithms to enhance image spatial resolution has become the mainstream technology in related fields. The main purpose of hyperspectral image super-resolution technology is to recover high-resolution images from low spatial resolution hyperspectral images while ensuring that the spectrum remains undistorted. In recent years, the rapid development of deep learning, especially convolutional neural networks (CNNs), has shown great potential in image processing due to their powerful representation capabilities, providing new ideas for image super-resolution technology. Existing network model designs mainly focus on how to mine and utilize the spatial-spectral information of hyperspectral images, neglecting the importance of shallow feature information and the correlation between spectra. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning, so as to solve the problems of insufficient utilization of spatial-spectral information mining in existing algorithms and neglect of the importance of shallow information and the correlation between spectra.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A hyperspectral image super-resolution reconstruction method based on multi-scale space-spectral feature learning includes the following steps:

[0006] Step 1: Obtain the image training set, which is a pair of high-resolution hyperspectral images (HR) and low-resolution hyperspectral images (LR);

[0007] Step 2: Construct a multi-scale spatial-spectral feature learning network; the multi-scale spatial-spectral feature learning network structure consists of five parts: a shallow feature extraction module, a multi-scale shallow spatial-spectral feature extraction module MSIM, a multi-scale deep spatial-spectral feature extraction sub-network DPMSSFN, a feature fusion module, and an image reconstruction module;

[0008] The shallow feature extraction module extracts the initial shallow feature map F0 from the input low-resolution hyperspectral image LR; the MSIM consists of a multi-scale shallow spatial-spectral information module MSSSIM and an attention mechanism SE-Net, and obtains the multi-scale shallow spatial-spectral feature map F based on F0. s The DPMSSFN consists of D cascaded dual-path multi-scale deep spatial-spectral feature extraction modules (DPMSSFB). The cascaded DPMSSFBs are used to extract residual features from different receiving domains, resulting in spatial-spectral feature maps F1,…,F1 at different levels. d ,…,F D The feature fusion module will integrate F1, ..., F d ,…,F D The fusion yields the deep spatial-spectral feature map F r The image reconstruction module is based on F0 and F... r Obtain the reconstructed super-resolution image;

[0009] Step 3: Input the training set obtained in Step 1 into the multi-scale spatial-spectral feature learning network for iterative supervised training, and save the trained network model;

[0010] Step 4: Input the hyperspectral image to be repaired into the network model trained in Step 3 to obtain the reconstructed hyperspectral super-resolution image.

[0011] In one embodiment, step 1 involves bicubic downsampling of the hyperspectral image to obtain a corresponding low-resolution image, normalizing each obtained low-resolution hyperspectral image to obtain a preliminary HR-LR image pair, and performing data augmentation on the obtained HR-LR image pair.

[0012] In one embodiment, the shallow feature extraction module uses a 3×3×3 3D convolutional layer to extract the initial shallow feature map F0 of the input low-resolution hyperspectral image LR.

[0013] In one embodiment, the MSSSIM first uses a 3×3×3 3D convolutional layer to extract shallow spatial-spectral features of F0 to obtain a shallow spatial-spectral feature map F0′, and then performs a reshape operation on F0′ to obtain the feature map F. 01 F was extracted using three 2D convolutional layers with different kernel sizes. 01 Multi-scale spatial features are used to obtain feature map F. 13 ,F 25 ,F 37 Finally, F 13 ,F 25 ,F 37 By combining these methods and utilizing a channel attention mechanism to adjust channel weights, a multi-scale shallow spatial-spectral feature map F is obtained. s .

[0014] In one embodiment, the shallow spatial-spectral feature map F0′ is 5-dimensional, which is transformed into 4-dimensionality through a reshape operation. 13 ,F 25 ,F 37 The combination involves stacking the spectral data and then using a 1×1×1 3D convolutional layer to reduce the spectral dimension, resulting in the feature map F. ss The channel attention mechanism applies to the feature map F. ss Channel weights are adjusted to obtain multi-scale shallow spatial-spectral feature maps F. s .

[0015] In one embodiment, each DPMSFB learns spatial-spectral and spatial feature information via a dual-path learning approach. The spatial-spectral feature learning branch consists of M multi-scale spatial-spectral information modules (MSSSIs), each MSSSI employing a hybrid spectral attention mechanism built with a 3D-Res2Net network. 3D-Res2Net replaces the 2D convolutions in the original Res2Net with 3D convolutions, enabling the acquisition of multi-scale spatial-spectral feature information. The spatial feature learning branch consists of N 2D convolutions. The DPMSFB first extracts the feature map F using a 3×3×3 convolutional layer. m The spatial-spectral features are used to obtain the feature map F. d,1 The F m By F s Adding F0 yields the result; secondly, the spatial-spectral feature information learning branch and the spatial feature information learning branch respectively apply F... d,1 Extracting feature information yields the spatial-spectral feature map F.d,ms and spatial feature map F d,si , will F d,ms and F d,si A concat operation is performed along the spectral dimension, followed by dimensionality reduction using a 1×1×1 3D convolutional layer to obtain the feature map F. d This convolution operation simultaneously fuses spatial-spectral features and spatial information, making the two types of features complementary.

[0016] In one embodiment, the 3D-Res2Net is first composed of a 1×1×1 3D convolutional layer for F... d,1 Feature information is extracted to obtain feature map F d,1r Secondly, the feature map F d,1r The spectral channels are equally divided into X1, X2, X3, and X4. X1 is processed without any operation to obtain feature map Y1. X2 is processed through a 3×3×3 3D convolutional layer to obtain feature map Y2. Y2 and X3 are added together and then processed through a 3×3×3 3D convolutional layer to obtain feature map Y3. Y3 and X4 are added together and then processed through a 3×3×3 3D convolutional layer to obtain feature map Y4. Feature maps Y1, Y2, Y3, and Y4 are added together along the spectral dimension to obtain feature map Y. The input feature map F is then processed. d,1 Adding it to Y yields the feature map F. d,Y And use a 1×1×1 3D convolutional layer to extract feature map F. d,Y Feature information is used to obtain feature map F d,1re .

[0017] In one embodiment, the hybrid spectral attention mechanism extracts feature map F through a series of operations including global average pooling (GAP), a 1×1×1 3D convolutional layer, and channel scaling. d,1re First-order statistical features yield feature maps Feature map F is extracted through a series of operations including second-order covariance pooling (SOCP), a single 1×1×1 3D convolutional layer, and channel scaling. d,1re The second-order spectral correlation features yielded the feature map. Will and A concat operation is performed along the spectral dimension, followed by dimensionality reduction using a 1×1×1 3D convolutional layer to obtain the feature map F. d,1c Then, the Sigmoid activation function is used to apply F. d,1c Activation is performed to obtain the feature map. Finally, the input feature map F is processed through local residual learning. d,1re and Adding them together yields the feature map F. d,ms .

[0018] In one embodiment, the feature fusion module first combines the feature maps F1,…,F1 of the cascaded DPMSFB outputs in the spectral dimension. d ,…,F D The connections are made, and then a 1×1×1 3D convolution is used to reduce the dimensionality and obtain the feature map F. c Using 3×3×3 convolutional layers to apply F c Feature extraction is performed to obtain feature map F mc Introducing a channel attention mechanism to the feature map F mc By adjusting the channel weights, the utilization rate of feature information is optimized and improved, maximizing the use of feature information to obtain the fused feature map F. r .

[0019] In one embodiment, the image reconstruction module divides image reconstruction into two parts. The first part is global residual learning, which uses long skip connections to connect the initial shallow feature map F0 and the deep spatial-spectral feature map F. r The feature map F is obtained by adding the features together. f , for F f Upsampling is performed, followed by deconvolution to enlarge the image and obtain a preliminary reconstructed super-resolution image. Secondly, since the input low-resolution image and the preliminary reconstructed super-resolution image differ significantly, Nearest interpolation is performed on the input low-resolution image to obtain the feature map F. n , will F n The final reconstruction result is obtained by adding the image to the initially reconstructed super-resolution image.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] The network model designed for hyperspectral image super-resolution in this invention comprises five parts: a shallow feature extraction module, a multi-scale shallow spatial-spectral feature extraction module, a multi-scale deep spatial-spectral feature extraction sub-network, a feature fusion module, and a reconstruction module. The shallow feature extraction module expands the dimensions of the input low-resolution image and extracts shallow feature information. The multi-scale shallow spatial-spectral feature extraction module mines multi-scale shallow information of the hyperspectral image. The multi-scale deep spatial-spectral feature extraction sub-network can effectively mine spatial-spectral information and spectral adjacency information. The feature fusion module can distinguish information from features at different levels and better fuse feature information. The super-resolution model proposed in this invention fully considers the shallow feature information, deep spatial-spectral information, and spectral correlation of the image, making it more efficient and producing clearer super-resolution results. Attached Figure Description

[0022] Figure 1 This is a flowchart proposed in this invention.

[0023] Figure 2This presents the overall framework of a hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning proposed in this invention.

[0024] Figure 3 yes Figure 2 A schematic diagram of the multi-scale shallow spatial-spectral feature extraction module MSIM is shown in the embodiment.

[0025] Figure 4 yes Figure 2 A schematic diagram of the dual-path multi-scale deep spatial-spectral feature extraction module DPMSFB is shown in the embodiment.

[0026] Figure 5 yes Figure 4 A schematic diagram of the Multi-Scale Spatial-Spectral Information Module (MSSSI) in the embodiment.

[0027] Figure 6 yes Figure 2 A schematic diagram of the feature fusion module in the embodiment. Detailed Implementation

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0029] As mentioned earlier, existing hyperspectral image reconstruction techniques do not consider the importance of shallow feature information and the correlation between spectra. To address this, the hyperspectral image super-resolution reconstruction method of this invention designs a multi-scale shallow feature extraction module to fully utilize shallow feature information; the designed multi-scale deep feature extraction module improves upon the existing network framework Res2Net by incorporating a hybrid spectral attention mechanism, which fully utilizes spatial-spectral feature information while also paying attention to the correlation between spectra.

[0030] like Figure 1 As shown, this invention includes the following steps: constructing an HR-LR pair dataset, constructing a multi-scale spatial-spectral feature learning network, training the model, inputting the hyperspectral image to be reconstructed into the trained model, and predicting the super-resolution hyperspectral image. The specific process is as follows:

[0031] The first step is the generation of a training set of high-resolution and low-resolution image pairs. This invention uses high-resolution hyperspectral images (HR) and low-resolution hyperspectral images (LR) as the training set.

[0032] Based on existing real hyperspectral datasets, bicubic downsampling is performed on hyperspectral images to obtain corresponding low-resolution images. Each low-resolution hyperspectral image is then normalized to obtain preliminary HR-LR image pairs. Data augmentation is then performed on the obtained HR-LR image pairs, including scaling, rotation, and mirroring operations, to enhance the dataset.

[0033] In one embodiment, the Chikusei hyperspectral image dataset is first bicubic downsampled to obtain a low-resolution hyperspectral image dataset. Then, the obtained HR-LR image pairs are augmented with data including scaling of 1x, 0.75x, and 0.5x, rotation of 90°, 180°, and 270°, as well as horizontal and vertical mirroring operations.

[0034] The second step is to construct a multi-scale spatial-spectral feature learning network.

[0035] like Figure 2 As shown, the network structure mainly consists of five parts: a shallow feature extraction module, a multi-scale shallow spatial-spectral feature extraction module (MSIM), a multi-scale deep spatial-spectral feature extraction sub-network (DPMSSFN), a feature fusion module, and an image reconstruction module. It can also consist of only the above five parts.

[0036] The specific steps are as follows:

[0037] (1) The shallow feature extraction module extracts the initial shallow feature map F0 from the input low-resolution hyperspectral image LR, and inputs F0 into the multi-scale shallow spatial-spectral feature extraction module MSIM to obtain the multi-scale shallow spatial-spectral feature map F. s Furthermore, multi-scale shallow features F can be obtained. m The formula is as follows:

[0038] F0 = f Conv3D,3 (unsqueeze(I LR )) (1)

[0039] F s =H MSIM (F0) (2)

[0040] F m =H MSIM (F0)+F0 (3)

[0041] Where f Conv3D,3 (·) indicates a 3D convolutional layer with a kernel of 3. `unsqueeze(·)` transforms a 4D feature map (B×H×W×C) into a 5D feature map (B×N×H×W×C), where B represents the batch size, H and W represent the height and width of the image, C is the number of bands, and H... MSIM (·) indicates the MSIM module.

[0042] (2) F mThe input is a multi-scale deep spatial-spectral feature extraction subnetwork DPMSSFN. DPMSSFN consists of D cascaded dual-path multi-scale deep spatial-spectral feature extraction modules DPMSSFB. The cascaded DPMSSFB is used to extract residual features from different receptive domains, resulting in spatial-spectral feature maps F1,…,F1 at different levels. d ,…,F D In this embodiment, D = 4; the output F of the d-th DPMSFB block d It can be represented as:

[0043] F d =H DPMSSFB,d (H DPMSSFB,d-1 (…H DPMSSFB,1 (F m (4)

[0044] Where H DPMSSFB,d (·) represents the d-th DPMSFB module;

[0045] (3) Input the four feature maps F1, F2, F3, and F4 obtained in (2) into the feature fusion module for fusion to obtain the deep spatial-spectral feature map F. r F0 and F are connected by a long jump connection. r The connection yields the feature map F f This process can be represented as:

[0046] F r =H fusion ([F1,F2,F3,F4]) (5)

[0047] F f =F r +F0 (6)

[0048] Where H fusion (·) indicates the feature fusion module;

[0049] (4) Perform super-resolution reconstruction on the feature map obtained after fusion in (3):

[0050] I SR =squeeze(f Conv3D (up(F f )))+F n (7)

[0051] Where squeeze(·) represents converting a 5-dimensional (B×N×H×W×C) feature map into a 4-dimensional (B×H×W×C) feature map, and up(·) represents an upsampling operation; F n It is the feature map obtained by performing the Nearest operation on the input low-resolution image.

[0052] (5) The multi-scale shallow spatial-spectral feature extraction module MSIM in (1) is explained. MSIM consists of the multi-scale shallow spatial-spectral information module MSSSIM and the channel attention mechanism; such as Figure 3 As shown, specifically:

[0053] ① The input feature map F0 is first processed by MSSSIM using a 3×3×3 3D convolutional layer to extract shallow spatial-spectral information, resulting in a 5-dimensional shallow spatial-spectral feature map F0′. Then, F0′ is transformed into 4-dimensional features through a reshape operation and input into the next layer of the network to obtain the feature map F. 01 ;

[0054] F 01 =reshape(f Conv3D,3 (F0)) (8)

[0055] ② Secondly, F is extracted using three parallel 2D convolutional layers with kernels of different sizes. 01 Multi-scale spatial features are used to obtain feature map F. 13 ,F 25 ,F 37 The obtained F 1,3 ,F 2,5 ,F 3,7 The spectral dimensions are superimposed, and a 1×1×1 3D convolutional layer is used to reduce the spectral dimension to obtain the feature map F. ss Finally, channel attention mechanism is used to process feature map F. ss By adjusting the weights of the spectral channels, a multi-scale shallow spatial-spectral feature map F is obtained. s .

[0056] F s =f se (f Conv2D,k＝1 (concat(F 1,3 ,F 2,5 ,F 3,7 ))) (9)

[0057] Among them, f se (·) represents the SE-Net operation, f Conv2D,k＝1 (·) represents a 2D convolutional layer with a kernel of 1, F 1,3 ,F 2,5 ,F 3,7 The feature map is obtained from three parallel 2D convolutional layers.

[0058] (6) In this embodiment, the convolutional kernels of the three 2D convolutional layers of different sizes are set to 3, 5, and 7, respectively, and the channel attention mechanism is set to SE-Net. The multi-scale deep spatial-spectral feature extraction sub-network DPMSSFN in (2) is described below. In this embodiment, DPMSSFN consists of four DPMSSFB blocks, each composed of a 3D convolutional network, a 2D convolutional network, and an MSSSI network. Each DPMSSFB learns spatial-spectral feature information and spatial feature information in a dual-path learning manner; such as... Figure 4 As shown, specifically:

[0059] ①DPMSSFB first uses a 3×3×3 3D convolutional layer to extract the feature map F. m The spatial-spectral features are used to obtain the feature map F. d,1 , will F d,1 A dual-path learning framework for input to the network;

[0060] F d,1 =f Conv3D,1 (F m (10)

[0061] ② The spatial-spectral feature information learning branch consists of M multi-scale spatial-spectral information modules (MSSSI), which are used for F d,1 Extracting feature information yields the spatial-spectral feature map F. d,ms :

[0062] F d,ms =H msssi (H msssi (F d,1 (11)

[0063] Among them, H msssi (·) indicates the MSSSI module; the spatial feature information learning branch consists of N 2D convolutions, for example, the convolution size is 3×3, for F d,1 Extracting feature information yields the spatial feature map F. d,si :

[0064] F d,si =reshape(f Conv2D,3 (f Conv2D,3 (reshape(F d,1 (12)

[0065] Among them, f Conv2D,3 (·) indicates a 2D convolutional layer with a kernel size of 3.

[0066] In this embodiment, M = 2, N = 2;

[0067] ③The obtained F d,ms and F d,siA concat operation is performed along the spectral dimension, followed by dimensionality reduction using a 1×1×1 3D convolutional layer to obtain the feature map F. d (That is, the initial features are added using residual learning), and this convolution operation simultaneously fuses spatial-spectral feature information and spatial information, making the two types of feature information complementary.

[0068] F d =(f Conv3D,1 (concat([F d,ms ,F d,si ]))+F d,1 )+F d-1 (13)

[0069] (7) MSSSI of the present invention Figure 5 As shown, a hybrid spectral attention mechanism is constructed using a 3D-Res2Net network. The 2D convolutional layers in the original Res2Net are replaced with 3D convolutional layers, resulting in 3D-Res2Net, which serves as the basic framework for multi-scale spectral information extraction, enabling the acquisition of multi-scale spatial-spectral feature information. Specifically:

[0070] ① First, 3D-Res2Net is used to extract the features of the input;

[0071] F d,1r =H 3D-Res2Net (F d,1 (14)

[0072] Among them, H 3D-Res2Net (·) indicates the 3D-Res2Net module;

[0073] ②The obtained feature map F d,1r Perform mixed-spectral attention mechanism operations;

[0074] F d,ms =H HSAM (F d,1r (15)

[0075] Among them, H HSAM (·) indicates the Hybrid Spectral Attention Mechanism (HSAM) module;

[0076] (8) In this invention, 3D-Res2Net is as follows: Figure 5 As shown, specifically:

[0077] ① Use a single 3×3×3 3D convolutional layer to process the feature map F d,1 Feature extraction is performed to obtain feature map F d,1r ;

[0078] F d,1r =fConv3D,1 (F d,1 (16)

[0079] ②The feature map F obtained in ① d,1r The spectral channels are divided into four subsets with the same number of spectral channels, defined as X1, X2, X3, X4. The corresponding feature map Y can be obtained by equation (17).

[0080]

[0081] That is, X1 obtains feature map Y1 without any operation; X2 is processed through a 3×3×3 3D convolutional layer to obtain feature map Y2; Y2 and X3 are added together and processed through a 3×3×3 3D convolutional layer to obtain feature map Y3; Y3 and X4 are added together and processed through a 3×3×3 3D convolutional layer to obtain feature map Y4. Here, σ represents the ReLU activation function.

[0082] ③ Add Y1, Y2, Y3, and Y4 together along the spectral dimension to obtain the feature map Y; then input the feature map F... d,1 Adding it to Y yields the feature map F. d,Y Connect Y and F via residual link d,Y Addition (i.e., extracting feature map F using a 1×1×1 3D convolutional layer) d,Y From the feature information, we obtain the feature map F. d,1re ;

[0083] F d,1re =f Conv3D,1 (concat[Y i ])+F d,1 (18)

[0084] (9) The hybrid spectral attention mechanism of the present invention, such as Figure 5 As shown, the algorithm consists of two branches, exploring first-order statistical features and second-order spectral correlation features respectively. First-order statistical features are explored using global average pooling (GAP), 3D convolution kernels, and channel scaling operations. Second-order spectral correlation features are explored using covariance pooling (SOCP), 3D convolution kernels, and channel scaling operations. Specifically:

[0085] ① For feature map F respectively d,1re Global average pooling (GAP) and second-order covariance pooling (SOCP) are performed; then, 1×1×1 3D convolutional layers are used to extract features from the pooled feature maps.

[0086] F d,1g =f Conv3D,1 (GAP(F d,1re (19)

[0087] Fd1,s =f Conv3D,1 (SOCP(F d,1re (20)

[0088] Where GAP(·) represents global average pooling operation, and SOCP(·) represents second-order covariance pooling operation;

[0089] ②The feature map F obtained in ① d,1g and F d1,s The channel weights are obtained by utilizing the channel scaling operation in the channel attention mechanism, and F is obtained respectively. d,1re First-order statistical characteristic map and F d,1re Second-order spectral correlation feature map

[0090]

[0091]

[0092] ③The feature map obtained in ② and A concat operation is performed along the channel dimension, followed by dimensionality reduction using a 1×1×1 3D convolutional layer to obtain the feature map F. d,1c ;

[0093]

[0094] ④ The feature map F obtained in ③ d,1c Feature maps are obtained by activation using the sigmoid activation function. Then input feature map F d,1re With the activated feature map Performing residual connections yields the following output feature map:

[0095]

[0096]

[0097] Where δ represents the sigmoid activation function;

[0098] (10) The feature fusion module in (3) is described, which fuses the feature maps F1, F2, F3, and F4 from DPMSFB to obtain the deep spatial-spectral feature map F. r Its network structure consists of 3D convolutions and 2D convolutions, such as... Figure 6 As shown, specifically:

[0099] ① Concatenate the feature maps F1, F2, F3, and F4 at different levels along the spectral dimension, and then use a 1×1×1 3D convolutional layer to reduce the dimensionality and obtain the feature map F. c :

[0100] F c =f Conv3D,1 (concat[F1,F2,F3,F4]) (26)

[0101] ② Use a 3×3×3 convolutional layer to apply F c Feature extraction is performed to obtain feature map F mc :

[0102] F mc =f Conv3D,3 (F c (27)

[0103] ③ Introduce a channel attention mechanism for feature map F mc Channel scaling is used to obtain the weights of different channels, optimizing and improving the utilization rate of feature information and maximizing its use. Finally, residual connections are used to connect the feature map F obtained in step ①. c Connect the features to obtain the fused feature map F. r :

[0104]

[0105] (11) The image reconstruction module in (3) is described. The image reconstruction of the image reconstruction module of the present invention is divided into two parts:

[0106] ① First, global residual learning is used, employing long skip connections to connect the initial shallow feature map F0 and the deep spatial-spectral feature map F. r The feature map F is obtained by adding the features together. f For feature map F f Upsampling is performed, followed by deconvolution to enlarge the image, and then a squeeze operation is performed to obtain the preliminary reconstructed super-resolution image I. CSR ;

[0107] F f =F0+F r (29)

[0108] I CSR =squeeze(f Conv3D (up(F f )) (30)

[0109] ② Because the difference between the input low-resolution image and the initially reconstructed super-resolution image is too large, the input low-resolution hyperspectral image is subjected to Nearest interpolation to obtain the feature map F.n , will F n The final reconstruction result is obtained by adding the initially reconstructed super-resolution image to the super-resolution image; it can be expressed as:

[0110] I SR =I CSR +F n (31)

[0111] The third step is to train a multi-scale spatial-spectral feature learning network, with the following specific steps:

[0112] (1) Input the high-resolution-low-resolution image dataset after data augmentation in the first step into the multi-scale spatial-spectral feature learning super-resolution network model;

[0113] (2) The final output value of the network is compared with the high-resolution hyperspectral image, i.e., Ground-Truth, and the error between the two is calculated. In a specific implementation of the present invention, the loss function is as follows:

[0114] L total =L1+αL SSTV (32)

[0115] The first loss function is used to reduce pixel loss, and the second spectral constraint function is used to suppress spectral distortion. α is a factor used to balance the contributions of the two losses; in one embodiment, it is set to a constant, α = 1 × 10⁻⁶. -3 ;

[0116] The first loss function is L1 loss, which can be expressed as:

[0117]

[0118] and It is the ground-truth and reconstructed hyperspectral image, and N is the number of training batches;

[0119] The second loss function can be expressed as:

[0120]

[0121] in and These are the reconstructed images Horizontal, vertical, and spectral gradients.

[0122] (3) Set the number of filters, batch size, epoch, and learning rate. Use the Adaptive Momentestimation ADAM optimizer and gradient descent algorithm to update the parameters of each layer and continuously optimize the entire network.

[0123] In one embodiment, the Chikusei dataset is used, with the number of filters set to 32, batch size to 4, epochs to 200, and initial learning rate to 1×10⁻⁶. -4 Set the ADAM optimizer β1 = 0.9 and β2 = 0.999;

[0124] The fourth step involves inputting the hyperspectral image to be reconstructed into a trained multi-scale spatial-spectral feature learning network model to predict the super-resolution hyperspectral image.

[0125] The quantitative evaluation results of PSNR, SSIM, SAM, RMSE, ERGAS, and UIQI among different algorithms are shown in the table below:

[0126] Table 1. Quantitative evaluation results of the Chikusei dataset at 4x super-resolution.

[0127]

[0128] For the quantitative evaluation metrics, higher values ​​for PSNR, SSIM, and UIQI are better, while lower values ​​for SAM, RMSE, and ERGAS are better. In Table 1, bold text indicates the best results, and italics indicate the second-best results. As shown in Table 1, the algorithm of this invention achieved the best results across all six evaluation metrics on the Chikusei dataset. This demonstrates that the algorithm of this invention achieves good spectral fidelity while improving the spatial resolution of hyperspectral images.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still make modifications or equivalent substitutions to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention are within the protection scope of the claims of the present invention pending approval.

Claims

1. A method for super-resolution reconstruction of hyperspectral images based on multi-scale spatial-spectral feature learning, characterized in that, Includes the following steps: Step 1: Obtain the image training set, which is a pair of high-resolution hyperspectral images (HR) and low-resolution hyperspectral images (LR); Step 2: Construct a multi-scale spatial-spectral feature learning network; the multi-scale spatial-spectral feature learning network structure consists of five parts: a shallow feature extraction module, a multi-scale shallow spatial-spectral feature extraction module MSIM, a multi-scale deep spatial-spectral feature extraction sub-network DPMSSFN, a feature fusion module, and an image reconstruction module; The shallow feature extraction module extracts the initial shallow feature map of the input low-resolution hyperspectral image LR. The MSIM consists of a multi-scale shallow spatial-spectral information module (MSSSIM) and a channel attention mechanism (SE-Net), based on... Obtain multi-scale shallow space-spectral feature maps The DPMSSFN consists of D cascaded dual-path multi-scale deep spatial-spectral feature extraction modules (DPMSSFB). The cascaded DPMSSFBs are used to extract residual features from different receiving domains, resulting in spatial-spectral feature maps at different levels. The feature fusion module will Fusion yields deep spatial-spectral feature maps The image reconstruction module is based on and Obtain the reconstructed super-resolution image; Each DPMSFB learns spatial-spectral and spatial feature information using a dual-path learning approach. The spatial-spectral feature information learning branch consists of M multi-scale spatial-spectral information modules (MSSSI). MSSSI is composed of a hybrid spectral attention mechanism built with a 3D-Res2Net network. 3D-Res2Net replaces the 2D convolutions in the original Res2Net with 3D convolutions. Its network structure can acquire multi-scale spatial-spectral feature information. space The feature information learning branch consists of N 2D convolutions; the DPMSFB first utilizes a Feature maps extracted from convolutional layers The spatial-spectral features are used to obtain the feature map. The Depend on and The sum is obtained; secondly, the spatial-spectral feature information learning branch and the spatial feature information learning branch respectively... Extracting feature information yields a spatial-spectral feature map. and spatial feature map ,Will and Perform a concat join operation along the spectral dimension, followed by using a Dimensionality reduction is performed on the 3D convolutional layer to obtain the feature map. This convolution operation simultaneously fuses spatial-spectral features and spatial information, making the two types of features complementary. The 3D-Res2Net is first composed of a 3D convolutional layers Feature information is extracted to obtain feature maps Secondly, the feature map The spectral channels are equally divided into , The feature map is obtained without any operation. ; After a The 3D convolutional layer obtains the feature map ;Will and After addition, after a The 3D convolutional layer obtains the feature map ;Will and After addition, after a The 3D convolutional layer obtains the feature map ; feature map The feature map is obtained by adding the spectral dimensions. , input feature map and Adding them together yields the feature map. and using a 3D convolutional layer feature map extraction Feature information is used to obtain feature maps ; The hybrid spectral attention mechanism consists of a global average pooling (GAP) and a single... 3D convolutional layers and channel scaling operations are used to extract feature maps. First-order statistical features yield feature maps , consisting of second-order covariance pooling SOCP, 1 3D convolutional layers and channel scaling operations are used to extract feature maps. The second-order spectral correlation features yielded the feature map. ,Will and Perform a concat join operation in the spectral dimension, using one Dimensionality reduction of 3D convolutional layers yields feature maps. Then, the Sigmoid activation function is used to... Activation is performed to obtain the feature map. Finally, the input feature map is processed through local residual learning. and Add them together to obtain the feature map. ; Step 3: Input the training set obtained in Step 1 into the multi-scale spatial-spectral feature learning network for iterative supervised training, and save the trained network model; Step 4: Input the hyperspectral image to be repaired into the network model trained in Step 3 to obtain the reconstructed hyperspectral super-resolution image.

2. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 1, characterized in that, In step 1, the hyperspectral image is downsampled twice to obtain a corresponding low-resolution image. Each low-resolution hyperspectral image is then normalized to obtain a preliminary HR-LR image pair, and the obtained HR-LR image pair is then augmented with data.

3. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 1, characterized in that, The shallow feature extraction module utilizes one The 3D convolutional layer extracts the initial shallow feature map of the input low-resolution hyperspectral image LR. .

4. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 1, characterized in that, The MSSSIM first utilizes one 3D convolutional layer extraction Shallow space-spectral features are used to obtain shallow space-spectral feature maps. Then The feature map is obtained by performing a reshape operation. Extracting data using three 2D convolutional layers with different kernel sizes. Multi-scale spatial features are used to obtain feature maps. Finally By combining these methods and utilizing a channel attention mechanism to adjust channel weights, multi-scale shallow spatial-spectral feature maps are obtained. .

5. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 4, characterized in that, The shallow space-spectral feature map Originally 5-dimensional, it is transformed into 4-dimensional using a reshape operation. Combining involves superimposing them along the spectral dimension and utilizing a... The 3D convolutional layer performs dimensionality reduction on the spectral dimension to obtain the feature map. The SE-Net for feature maps Channel weights are adjusted to obtain multi-scale shallow spatial-spectral feature maps. .

6. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 1, characterized in that, The feature fusion module first integrates the feature maps output by the cascaded DPMSFB in the spectral dimension. Make a connection, and then use one 3D convolution is used to reduce the dimensionality of the feature map. ,use convolutional layers Feature extraction is performed to obtain feature maps. Introducing a channel attention mechanism for feature maps By adjusting the channel weights, the utilization rate of feature information is optimized and improved, maximizing the use of feature information to obtain the fused feature map. .

7. The hyperspectral image super-resolution reconstruction method based on multi-scale spatial-spectral feature learning according to claim 1, characterized in that, The image reconstruction module consists of two parts. The first part is global residual learning, which uses long skip connections to process the initial shallow feature map. and deep space-spectral feature map Adding them together yields the feature map. ,right Upsampling is performed, followed by deconvolution to enlarge the image and obtain a preliminary reconstructed super-resolution image. Secondly, since the input low-resolution image and the preliminary reconstructed super-resolution image differ significantly, Nearest interpolation is performed on the input low-resolution image to obtain a feature map. ,Will The final reconstruction result is obtained by adding the image to the initially reconstructed super-resolution image.

Images