Hyperspectral image reconstruction method based on dual-path fusion

Abstract

The application discloses a hyperspectral image reconstruction method based on double-path fusion. First, the data set is preprocessed to construct a training set and a test set. Then, a reconstruction model with a double-path structure is constructed, including shallow feature extraction, double-path deep feature extraction, adaptive feature fusion and reconstruction. The network parameters are optimized by minimizing the loss, and the optimal reconstruction model weight is saved. Finally, the RGB image is input into the optimal reconstruction model, and the generated hyperspectral image is output. The method according to the application is based on the spatial local similarity and spectral self-similarity of the hyperspectral image, and the double-path structure is designed in the feature extraction to realize targeted extraction, so that the reconstructed image is clearer and the texture is more obvious. In addition, the adaptive fusion module can measure the importance of the spectral features and the spatial features, select the dominant features most conducive to reconstruction for each pixel point, and therefore realize more accurate spectral super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a hyperspectral image reconstruction method based on dual-path fusion. Background Technology

[0002] Hyperspectral images (HSIs) are created by measuring the continuous spectrum of each pixel in a scene using imaging devices. Compared to RGB images, hyperspectral images contain hundreds of continuous narrow spectral bands, storing a wealth of information and helping to distinguish different types of substances. Therefore, they are widely used in environmental monitoring, medical disease detection, food analysis, and other fields. However, due to the limitations of imaging sensors, a trade-off must be made between spatial resolution, spectral resolution, and signal-to-noise ratio. Actual HSI captures often have poor spatial resolution, which severely restricts their practical applications. RGB images are readily available, so reconstructing hyperspectral images from RGB images has become a low-cost alternative method and has received increasing attention in recent years. This process is called spectral super-resolution (SSR).

[0003] Early research focused on designing sparse restoration models, utilizing the statistical distribution of spatial or spectral information as priors to build a complete dictionary, treating the reconstruction process as a linear transformation from RGB to HSI. However, these methods heavily rely on the prior performance of the hyperspectral dataset, and simple linear mapping is insufficient to simulate the correlation between the two datasets. Therefore, these models have poor generalization ability and only achieve satisfactory results in specific scenarios. In recent years, deep learning-based spectral super-resolution frameworks have achieved considerable success in learning the mapping between RGB and HSI by designing complex network structures and extracting rich contextual information. However, these models still have shortcomings: hyperspectral images contain rich spatial and spectral features, and most reconstruction methods do not distinguish between the features of the two types of information, requiring specialized network structures. Furthermore, inappropriate fusion methods lead to disordered spectral bands, and directly fusing spatial and spectral information using cascading and convolutional operations generates redundant information, affecting reconstruction. Therefore, these limitations restrict the expressive power of the models to some extent. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention aims to provide a hyperspectral image reconstruction method based on dual-path fusion.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems includes the following steps:

[0006] Step S1, construct training and test sets: First, crop the RGB images and corresponding hyperspectral images in the training set to 128x128 size, then normalize each cropped block, and finally perform data augmentation to construct the training set. The data in the test set is used directly as the test set without cropping or augmentation.

[0007] Step S2, constructing the reconstruction model: This includes four parts: shallow feature extraction, dual-path deep feature extraction, adaptive feature fusion, and reconstruction. Shallow feature extraction uses a 3x3 convolutional layer to extract shallow features, projecting low-dimensional RGB onto a high-dimensional space. Dual-path deep feature extraction includes multiple spatial residual extraction modules and a spectral dense extraction module to extract deep spatial and spectral features. The adaptive feature fusion module achieves adaptive fusion based on the input feature weights. The reconstruction module uses a 3x3 convolutional layer and a global residual connection to aggregate shallow and deep features, reducing dimensionality and reconstructing the hyperspectral image.

[0008] Step S3, train and rebuild the model: set the model hyperparameters, select the Adam optimizer, use a multinomial decay strategy for the learning rate, optimize the network parameters by minimizing the loss, and select and save the optimal network weights according to the evaluation metrics.

[0009] Step S4, Reconstruct the hyperspectral image: Input the RGB image into the optimal reconstruction model, output and save the generated hyperspectral image.

[0010] Further, in step S1, data augmentation refers to flipping or rotating the cropped image block up / down / left / right by 90° / 180° / 270°; the normalization process refers to normalizing the values ​​of the image pixels to between 0 and 1.

[0011] Further, in step S2, the spatial residual extraction module cascades multiple spatial convolutional groups and a residual connection; the spatial convolutional group consists of a 3×3 convolutional layer, a ReLU activation function, a 3x3 convolutional layer, and a residual connection; the spectral dense extraction module cascades multiple spectral convolutional groups and dense skip connections; the spectral convolutional group consists of a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a 1×1 residual convolutional layer; the dense skip connection refers to using a 1x1 convolution to extract low-level spectral features in different spectral convolutional groups and input them into higher levels for spectral information interaction.

[0012] Furthermore, in step S2, the adaptive feature fusion module measures the importance of spatial and spectral information and selects dominant features for each pixel, specifically including:

[0013] The spatial features obtained by the spatial feature extraction module and the spectral features obtained by the spectral feature extraction module are concatenated to obtain a candidate feature. This candidate feature is then passed through a 1x1 convolutional layer to reduce its dimensionality, resulting in a candidate probability set.

[0014] The candidate probability set is processed by the gumble softmax function to obtain a probability distribution representing the importance of features. The probability distribution is then transformed into candidate labels by the argmax function and one-hot encoding.

[0015] The candidate label and candidate feature are multiplied at corresponding element positions to select spatial or spectral features for each pixel to obtain the modulation feature;

[0016] The candidate features are processed through two 3x3 convolutional layers, two LeakyReLU activation functions, and two dilated convolutional layers to extract high-frequency information from the image, thus obtaining high-frequency features.

[0017] The modulation features are added to the high-frequency features to obtain the fused features.

[0018] Furthermore, in step S3, the loss function uses the mean relative absolute error (MRAE).

[0019]

[0020] in, and This represents the value of the i-th pixel in the reconstructed image and the real image, where N is the total number of pixels, and ε = 1 × 102 -3 It is a constant to prevent zero values ​​in the image from causing overflow.

[0021] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0022] (1) The hyperspectral image reconstruction network based on dual-path fusion designed in this invention solves the problem of spatial spectral inhomogeneity in the feature extraction and fusion stages, and achieves more accurate spectral super-resolution reconstruction.

[0023] (2) The dual-path depth feature extraction module designed in this invention has different extraction modules based on spatial local similarity and spectral self-similarity, and extracts rich contextual information to assist subsequent reconstruction.

[0024] (3) The adaptive feature fusion module designed in this invention can measure the importance of spectral features and spatial features, discard redundant spatial and spectral features, select the dominant feature most beneficial to reconstruction for each pixel, and retain more high-frequency information of the image. Attached Figure Description

[0025] Figure 1This is a flowchart of the algorithm for the hyperspectral image reconstruction method based on dual-path fusion described in an embodiment of the present invention.

[0026] Figure 2 This is a network structure diagram of the hyperspectral image reconstruction method based on dual-path fusion described in an embodiment of the present invention.

[0027] Figure 3 This is a structural diagram of the spatial residual extraction module described in an embodiment of the present invention.

[0028] Figure 4 This is a structural diagram of the spectral density extraction module according to an embodiment of the present invention.

[0029] Figure 5 This is a structural diagram of the adaptive feature fusion module described in an embodiment of the present invention.

[0030] Figure 6 The following is a diagram illustrating the RGB image reconstruction hyperspectral image effect of the hyperspectral image reconstruction method based on dual-path fusion, as provided in this embodiment of the invention. Detailed Implementation

[0031] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example

[0033] like Figure 1 As shown, the hyperspectral image reconstruction method based on dual-path fusion includes the following steps:

[0034] Step S1, construct the training set and test set:

[0035] First, the RGB images and corresponding hyperspectral images in the training set are cropped to 128x128 pixels. Then, each cropped block is normalized. Finally, data augmentation is performed to construct the training set. The data in the test set is used directly without cropping or augmentation.

[0036] In this embodiment, the data augmentation refers to flipping or rotating the cropped image block up / down / left / right by 90° / 180° / 270°; the normalization process refers to normalizing the values ​​of the image pixels to between 0 and 1.

[0037] Step S2, construct the reconstruction model:

[0038] like Figure 2 As shown, the reconstruction model consists of four parts: shallow feature extraction, dual-path deep feature extraction, adaptive feature fusion, and reconstruction. This is implemented through the following steps:

[0039] (S21) Shallow Feature Extraction

[0040] More specifically, shallow feature extraction uses a 3x3 convolutional layer to extract the input RGB image I. ReB shallow features f shallow Projecting low-dimensional RGB onto a high-dimensional space can be expressed by the following formula:

[0041] F shallow =f conv3 (I RGB )

[0042] Where f Conv3 (·) indicates a 3x3 convolutional layer.

[0043] (S22) Dual-path depth features

[0044] More specifically, the dual-path deep feature extraction process includes extracting shallow features through an input spatial feature extraction module and a spectral feature extraction module to achieve differentiated extraction;

[0045] like Figure 3 As shown, the spatial feature extraction module cascades multiple basic spatial units and a residual connection to extract contextual spatial information. Each basic spatial unit includes a 3×3 convolutional layer, a ReLU activation function, a 3x3 convolutional layer, and a residual connection. The r-th spatial depth feature... It can be described as:

[0046]

[0047] Among them, f relu (·) represents the ReLU activation function. This represents the depth space features proposed in the previous basic spatial unit.

[0048] like Figure 4 As shown, the spectral feature extraction module cascades multiple basic spectral units and dense skip connections. Each basic spectral unit consists of a 3×3 convolutional layer, a ReLU activation function, another 3×3 convolutional layer, and a 1×1 residual convolutional layer. The 1×1 convolution captures spectral properties only from the spectral dimension of the hyperspectral image, without introducing spatial information from nearby pixels. Residual connections are used to merge the 1×1 convolutions within each unit to enhance the strong correlation between adjacent spectral bands. The r-th spectral depth feature... It can be described as:

[0049]

[0050] Among them, f Conv1 (·) represents a 1x1 convolutional layer. This represents the deep spectral features extracted from the previous basic spectral unit. Dense skip connections refer to the use of a 1x1 convolution to extract low-level spectral features from different basic spectral units and feed them into higher-level units for spectral information exchange. The r-th spectral depth feature exchanged. It can be described as:

[0051]

[0052] Among them, f Concat (·) indicates concatenating two feature vectors along the channel dimension.

[0053] (S23) Adaptive Feature Fusion Module

[0054] More specifically, the adaptive feature fusion module measures the importance of spatial and spectral information and selects the dominant feature for each pixel. This is achieved through the following steps:

[0055] (S231) The spatial features F obtained by the spatial feature extraction module are processed. spa The spectral features F obtained by the spectral feature extraction module spe By concatenating the components, a candidate feature F is obtained. a The feature dimension is reduced by a 1x1 convolutional layer to obtain the candidate probability set P, which can be expressed by the following formula:

[0056] F a =f Concat [F spa F spe ]

[0057] P = f Conv1 (F a )

[0058] Here, the channel dimension of P describes the possibilities of each piece of information.

[0059] (S232) The candidate probability set is processed by the gumble softmax function to obtain the weight distribution W representing the importance of the features. The specific formula can be expressed as:

[0060] W = f Softmax ((logP+G) / τ)

[0061] Among them, f Softmax (·) denotes the softmax function, G=-log(-log(ξ)) represents noise following a Gumbel distribution, ξ is a random number sampled from a uniform distribution, and τ is a controllable parameter that controls the randomness of the sampling.

[0062] (S233) The probability distribution is transformed into a candidate label set using the argmax function and one-hot encoding. The specific formula can be expressed as follows:

[0063] F p，i，j =onehot(argmax(W))

[0064] Among them, F p，i，j Let represent the set of candidate labels, where the value at pixel (i, j) is either 0 or 1, and p represents the type of information. If the spatial context features dominate at point (i, j), then the candidate label is . The gradient of the parameters in W is directly updated. Therefore, the weight distribution can be adaptively learned.

[0065] (S234) Multiply the candidate label set with the candidate features at corresponding element positions, and select spatial or spectral features for each pixel to obtain the modulation feature F. modulate The specific formula can be expressed as:

[0066]

[0067] (S234) The candidate features are processed through two 3x3 convolutional layers, two Leaky ReLU activation functions, and two dilated convolutional layers to extract high-frequency information from the image, resulting in high-frequency feature F. high The specific formula can be expressed as:

[0068] F high =f LReLU (f Dconv …(f LReLU (f conv3 (F a ))))

[0069] Among them, f LReLU (·) denotes the LeakyReLU activation function, used to avoid gradient vanishing, f Dconv (·) represents a dilated convolution with a dilation ratio of 2.

[0070] (S235) The modulation feature and the high-frequency feature are added together to obtain the fused feature, and the specific formula can be expressed as follows:

[0071] F fusion =F high +F modulate

[0072] Among them, F fusion It is then sent to the next spatial feature extraction module and spectral feature extraction module for further extraction.

[0073] (S22) Reconstruction

[0074] More specifically, the reconstruction process uses a 3x3 convolutional layer and global residual connections to aggregate shallow and deep features, reducing dimensionality and reconstructing the hyperspectral image. HSI .

[0075] Step S3, train the reconstruction model:

[0076] Set the model hyperparameters, select the Adam optimizer, optimize the network parameters by minimizing the loss, set the initial learning rate to 0.0001, and adopt a decay strategy based on a polynomial function with a power of 1.5. Set the batch size to 16, stop training the network after 100 epochs, and select and save the optimal network weights according to the evaluation metrics.

[0077] Specifically, the loss function uses the mean relative absolute error (MRAE).

[0078]

[0079] in, and This represents the value of the i-th pixel in the reconstructed image and the real image, where N is the total number of pixels, and ε = 1 × 102 -3 It is a constant to prevent zero values ​​in the image from causing overflow.

[0080] Step S4, reconstruct the hyperspectral image:

[0081] Input an RGB image into the optimal reconstruction model, and output and save the generated hyperspectral image.

[0082] The invention will be further explained below with reference to experiments:

[0083] Simulation experiments were conducted using the PyTorch deep learning framework on NVIDIA TESLA P40 GPUs and the Ubuntu operating system. The proposed method was compared with HSCNN, Multi, HRNet, AWAN, and MST methods on the NTIRE 2022, CAVE, and Harvard datasets. Four widely used evaluation metrics were employed to compare the performance of the proposed method with other competing methods: root mean square error (RMSE), spectral angle mapper (SAM), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).

[0084] Table 1 presents the quantitative results of different methods on the NTIRE 2022, CAVE, and Harvard datasets. Bold data represents the best results, and underlined data represents the second-best results. The experimental results show that the present invention achieves the best reconstruction results on all three datasets, with all evaluation metrics reaching the highest levels. Specifically, on the NTIRE 2022 dataset, the average PSNR and SSIM are 0.41 dB and 0.0028 higher than the second-best method MST, respectively. On the Harvard dataset, the present invention's method reduces SAM by 0.22 and improves the average PSNR by 1.33 dB. These data fully demonstrate the superiority and effectiveness of the present invention, which is due to its ability to adaptively explore spectral and spatial information on different datasets.

[0085] Table 1. Performance comparison of different spectral super-resolution models on NTIRE 2022, CAVE, and Harvard datasets.

[0086]

[0087] Figure 5 This is a heatmap of errors in images reconstructed using different methods according to embodiments of the present invention. Six bands were selected from 31 bands to visualize the reconstruction errors of different methods. The first column represents the true bands of each image, and the other columns show the error mapping between the reconstructed images and the true images using different methods. Figure 5 As can be seen, the HSCNN+ and HRNet methods exhibit significant errors in areas with background information. While the MST method mitigates this to some extent, it still suffers from substantial errors in brighter regions. In contrast, the method presented in this invention demonstrates high reconstruction accuracy, exhibiting almost no errors even in brighter areas. These visualization results further provide additional evidence for the effectiveness of our method.

[0088] In summary, the hyperspectral image reconstruction method based on dual-path fusion provided in this embodiment, by cascading multiple spatial and spectral feature extraction modules according to the characteristics of hyperspectral images, can extract rich feature information, resulting in clearer reconstructed images with more pronounced textures. Furthermore, the adaptive fusion module can further select dominant features for each pixel based on feature importance, discarding redundant information and adapting to different datasets while retaining more high-frequency information. Therefore, this embodiment achieves minimal reconstruction error and can generate high-fidelity and high-quality images.

[0089] The embodiments listed above are detailed descriptions of the specific steps and effects of the present invention. These embodiments are merely preferred implementations of the present invention and should not be construed as limiting the scope of the invention. Within the scope of the claims, those skilled in the art can make various changes or modifications. Any modifications, equivalent substitutions, or improvements made to the present invention within the spirit and principles of the invention should be included within the protection scope of the present invention.

Claims

1. A hyperspectral image reconstruction method based on dual-path fusion, characterized in that, Specifically, the following steps are included: Step S1, construct training and test sets: First, crop the RGB images and corresponding hyperspectral images in the training set to 128x128 size, then normalize each cropped block, and finally perform data augmentation to construct the training set. The data in the test set is used directly as the test set without cropping or augmentation. Step S2, constructing the reconstruction model: The reconstruction model includes four parts: shallow feature extraction, dual-path deep feature extraction, adaptive feature fusion, and reconstruction; The shallow feature extraction uses a 3x3 convolutional layer to extract shallow features, projecting low-dimensional RGB onto a high-dimensional space; The dual-path deep feature extraction includes multiple spatial residual extraction modules and a spectral density extraction module to extract deep spatial and spectral features; The adaptive feature fusion includes: concatenating the spatial features obtained by the spatial residual extraction module with the spectral features obtained by the spectral dense extraction module to obtain candidate features, and passing them through a 1×1 convolutional layer to reduce the feature dimension to obtain a candidate probability set; passing the candidate probability set through the Gumbel-Softmax function to obtain a weight distribution representing the importance of features, and transforming the weight distribution into a candidate label set through the argmax function and one-hot encoding; multiplying the candidate label set with the candidate features at corresponding element positions to select spatial features or spectral features for each pixel to obtain modulation features; extracting high-frequency information from the candidate features to obtain high-frequency features; adding the modulation features to the high-frequency features to obtain fused features, and sending the fused features to the next spatial residual extraction module and spectral dense extraction module for further feature extraction; The reconstruction module uses a 3x3 convolutional layer and global residual connections to aggregate shallow and deep features, reducing dimensionality to reconstruct the hyperspectral image; Step S3, train and rebuild the model: set the model hyperparameters, select the Adam optimizer, use a multinomial decay strategy for the learning rate, optimize the network parameters by minimizing the loss, and select and save the optimal network weights according to the evaluation metrics. Step S4, Reconstruct the hyperspectral image: Input the RGB image into the optimal reconstruction model, output and save the generated hyperspectral image.

2. The hyperspectral image reconstruction method based on dual-path fusion according to claim 1, characterized in that, The spatial residual extraction module cascades multiple spatial convolutional groups and a residual connection; the spatial convolutional group includes a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a residual connection.

3. The hyperspectral image reconstruction method based on dual-path fusion according to claim 1, characterized in that, The dense spectral extraction module cascades multiple spectral convolutional groups and dense skip connections; the spectral convolutional group includes a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a 1×1 residual convolutional layer; the dense skip connection uses a 1×1 convolution in different spectral convolutional groups to extract low-level spectral features and passes the low-level spectral features into higher levels for spectral information interaction.

4. The hyperspectral image reconstruction method based on dual-path fusion according to claim 1, characterized in that, Extracting high-frequency information from the candidate features includes passing the candidate features through two 3×3 convolutional layers, two LeakyReLU activation functions, and two dilated convolutional layers with a dilation ratio of 2 to obtain the high-frequency features.

5. The hyperspectral image reconstruction method based on dual-path fusion according to claim 1, characterized in that, In step S3, the initial learning rate is set to 0.0001, a decay strategy using a polynomial function with a power of 1.5 is adopted, the batch size is set to 16, and training of the network is stopped after 100 epochs; the loss function uses the mean relative absolute error (MRAE), expressed as: ,in, and Represents the first in the reconstructed image and the real image The value of each pixel. The total number of pixels. It is a constant to prevent zero values ​​in the image from causing overflow.

Images