A hyperspectral image reconstruction method and system based on RGB image

Abstract

The present invention discloses a hyperspectral image reconstruction method and system based on RGB images, the method comprising: performing channel expansion on the RGB image through convolution processing; spectrally mapping and reconstructing the feature map or intermediate feature map after channel expansion through three cascaded RSRMs, each RSRM downsampling the feature map to obtain feature information of different scales; extracting and fusing features using a 2D convolution module and V3DCM; modeling and capturing the fused features in spatial, channel, and spectral dimensions through spatial and channel attention and spectral self-attention mechanisms; fusing and filtering features of different scales using GMFF, and finally generating a hyperspectral reconstruction result through jump connections and convolution operations. The technical solution of the present invention can effectively avoid information loss while improving the hyperspectral reconstruction accuracy, achieve a balance between the hyperspectral reconstruction accuracy and the efficient 3D convolution, and improve the accuracy of the reconstruction process.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a hyperspectral image reconstruction method based on RGB images and a hyperspectral image reconstruction system based on RGB images. Background Art

[0002] Hyperspectral images (HSI) are obtained by decomposing incident light into multiple bands using a hyperspectral imaging system. Compared to conventional RGB images, they capture not only spatial information but also rich spectral features. Many substances with similar appearance or color can be easily distinguished in HSI due to the information in different bands. Therefore, HSI has a wide range of applications in geological exploration, agricultural testing, food safety, and other fields. However, hyperspectral images obtained by hyperspectral imaging systems are not only expensive and time-consuming, but traditional hyperspectral imagers are also difficult to use in dynamic scenes, which greatly limits the scope of hyperspectral applications.

[0003] In recent years, the use of RGB images to predict information from multiple spectral bands and reconstruct complete HSIs has attracted considerable attention and research from both domestic and international scholars, addressing issues such as high cost and acquisition speed. Currently, HSI reconstruction from RGB images has been applied to applications such as corn disease monitoring, vein visualization, skin color detection, and art restoration and appraisal, and its research has profound implications for this task.

[0004] Because HSI has multiple bands, while RGB images only have three, an RGB image can be spectrally downsampled from different HSIs. Therefore, reconstructing HSI from RGB images is a highly ill-posed problem. Early methods used sparse dictionaries to construct a library of mappings between hyperspectral images and their corresponding RGB images, and then reconstructed the hyperspectral image through this mapping. However, this approach can only accurately reconstruct hyperspectral images for specific spectral response functions, and its generalization and robustness need improvement. With the rapid development of deep learning in recent years, the powerful nonlinear mapping capabilities of convolutional neural networks (CNNs) have provided a new approach to addressing this problem. The end-to-end mapping of RGB images and HSIs using CNNs can fully capture the complex mapping relationship between the two, while improving accuracy and generalization. Furthermore, the spectral reconstruction process must fully consider the feature similarity in both spatial and spectral dimensions. Many studies have used a dual spatial-channel attention mechanism to adaptively correct information in these two dimensions. Leveraging the spatial sparseness and spectral self-similarity of hyperspectral images, some studies have developed spectral self-attention computation using the self-attention mechanism in the Transformer, which can overcome the shortcomings of CNNs in capturing incomplete information over long distances. In order to better capture the information between adjacent channels, 3D convolution is also widely used in hyperspectral image related tasks.

[0005] Existing deep learning-based methods for reconstructing HSI from RGB have shown good results. However, this task is a complex nonlinear process, and existing methods cannot fully utilize spatial and spectral dimension information during the reconstruction process, resulting in limited accuracy of the final spectral reconstruction. The prior art discloses the use of a CNN-based dual attention mechanism for RGB hyperspectral reconstruction tasks. However, CNN mainly focuses on feature extraction of local information. Although the feature capture of long-range dependencies in the spatial dimension can be compensated by calculating the similarity of multiple features by decomposing features, for multiple bands of the hyperspectral spectrum, it is necessary not only to grasp the global information in the spatial dimension, but also to calculate the feature similarity of the global information in the spectral dimension. In addition, in order to fully utilize the spectral information in the similarity calculation between adjacent bands of the hyperspectral spectrum, some methods have proposed the use of 3D convolution to capture it. However, existing methods cannot take into account the computational complexity of 3D convolution and the memory consumption or the loss of feature details in the spatial and spectral dimensions due to the reduction of complexity. One approach uses space-spectrally symmetric 3D convolution to overcome this challenge. While this approach can balance the complexity of 3D convolution with the utilization of spectral information to a certain extent, the improved 3D convolution filter may result in a loss of detail in the global spatial spectrum, making reconstruction accuracy unreliable. Furthermore, spectral mapping typically requires the fusion and selection of features at different scales. However, during this selection process, a single CNN convolution cannot fully identify useful information, resulting in limited reconstruction results. Summary of the Invention

[0006] To address the above problems, the present invention provides a hyperspectral image reconstruction method and system based on RGB images. By introducing the spatial-spectral multi-attention mechanism SSMAM (Spatial-Spectral Multi-Attention Mechanim), CNN captures local channel and spatial attention respectively, and uses the attention mechanism in Transformer to capture long-distance detail features in the spectral dimension, comprehensively enhancing the feature extraction capability, and making full use of the global information in the image in the spatial and spectral dimensions, thereby improving the accuracy of hyperspectral reconstruction; through the multi-functional 3D convolution module V3DCM (Versatile 3D Convlution Module) that can reduce the large amount of 3D convolution calculation and high memory usage while retaining feature diversity, in addition to feature calculation in all directions, the ECA layer is also used for feature screening, and jump connection is performed with shallow features, so that the high spectral reconstruction accuracy and efficient 3D convolution are balanced; in addition, by adopting the gated convolution GMFF (Gated Multi-Feature Fusion) of multi-feature fusion in the spectral reconstruction stage, the gated convolution is used instead of the ordinary convolution for feature filtering for the selection of multi-feature fusion, so that more effective information can be captured for spectral reconstruction during the mapping process, thereby improving the accuracy of reconstruction.

[0007] To achieve the above object, the present invention provides a hyperspectral image reconstruction method based on RGB image, comprising:

[0008] Perform channel expansion on the three-channel RGB image through convolution processing;

[0009] Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales.

[0010] Use the 2D convolution module and the multifunctional 3D convolution module V3DCM to extract spatial and spectral features of features at different scales, and fuse the obtained features;

[0011] Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled, captured and weightedly fused in the spatial, channel and spectral dimensions respectively;

[0012] The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connections and convolution operations to generate hyperspectral reconstruction results.

[0013] In the above technical solution, preferably, the channel expansion of the three-channel RGB image is performed by convolution processing, and the specific process includes:

[0014] A 2D convolution module with a convolution kernel size of 3 is used to expand the three-channel RGB image so that the number of channels becomes 3;

[0015] The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

[0016] In the above technical solution, preferably, the spectrum mapping and reconstruction are performed by three cascaded residual spectrum reconstruction modules RSRM, each RSRM downsamples the feature map or intermediate feature map after the expansion channel to obtain feature information of different scales, and the specific process includes:

[0017] Three residual spectrum reconstruction modules RSRM are cascaded, and each RSRM performs a downsampling process of ×1, ×2 and ×4 on the features or intermediate features of the RGB image after the expansion channel, and obtains features of three different scales of [31, 128, 128], [62, 64, 64] and [124, 32, 32] from the original features.

[0018] In the above technical solution, preferably, the 2D convolution module and the multifunctional 3D convolution module V3DCM are used to extract spatial and spectral features of features of different scales respectively, and the obtained features are fused. The specific process includes:

[0019] For features of different scales, two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used respectively, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution.

[0020] In the above technical solution, preferably, the specific process of the multifunctional 3D convolution module includes:

[0021] Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension;

[0022] A 1×1×1 filter is used to extract shallow features from the input features. Convolution kernels of 3×1×1, 1×3×1, and 1×1×3 sizes are used to capture feature information in the depth, height, and width directions respectively. To avoid missing details of the global features of the spatial spectrum, a 3×3×3 filter is used for a comprehensive feature calculation.

[0023] Perform a jump connection between the extracted features of each dimension and the input of the corresponding dimension;

[0024] The features of four different dimensions are fused, and then a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension;

[0025] The ECA layer is spliced ​​once to screen and extract the effective spatial spectrum information, and then a long jump connection is performed on the shallow features input into the multifunctional 3D convolution.

[0026] In the above technical solution, preferably, the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism are used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions and perform weighted fusion. The specific process includes:

[0027] Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are spliced ​​along the new channel dimension. Then, a convolution layer is used to convolve the pooled tensor, and the Sigmoid function is used to generate attention weights, which are finally multiplied element-wise with the input tensor.

[0028] Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point. Then, a convolutional layer and ReLU activation function are used for feature optimization. The Sigmoid function is then used to generate channel weights, which are then element-wise multiplied with the original input tensor to achieve channel enhancement.

[0029] Attention weights are calculated along the spectral dimension through Transformer-based spectral self-attention;

[0030] The results of the above attention mechanism are feature fused, and a convolution operation with a convolution kernel size of 1 and a skip connection are performed.

[0031] In the above technical solution, preferably, the gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connections and convolution operations to generate hyperspectral reconstruction results. The specific process includes:

[0032] The features of three different scales are first flattened and then spliced;

[0033] Use three gated convolutions to filter the concatenated features, and perform ordinary Concat connections on the filtered features;

[0034] The features are fused through a convolution operation with a convolution kernel size of 3;

[0035] The fused features are connected to the input features of the residual spectrum reconstruction module RSRM once, and then a convolution operation with a convolution kernel of 3 is performed;

[0036] A skip connection is performed on the tensor of the last output RSRM to obtain the hyperspectral reconstruction result.

[0037] In the above technical solution, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0038] Loss calculation is performed on the hyperspectral reconstruction result and the real spectrum, and the model parameters are optimized by back propagation until the model converges and the training is completed.

[0039] In the above technical solution, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0040] The NTIRE2022 dataset and Harvard dataset were used as validation sets to verify the optimal model obtained through training, and the model was evaluated using preset evaluation indicators.

[0041] The present invention further proposes a hyperspectral image reconstruction system based on RGB images, which is applied to the hyperspectral image reconstruction method based on RGB images disclosed in any one of the above technical solutions, comprising:

[0042] RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing;

[0043] The multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales.

[0044] The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features;

[0045] The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through CNN-based spatial and channel attention and Transformer-based spectral self-attention mechanism;

[0046] The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results through skip connections and convolution operations on the fused features.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The spatial-spectral multi-attention mechanism SSMAM is introduced. Based on the spatial attention mechanism and channel attention mechanism built by CNN, a self-attention mechanism calculated along the spectral dimension based on Transformer is added to model the features of the spectral dimension, capture the long-distance dependencies in the spectral dimension, and improve the ability to extract the similarity and detail features between bands in hyperspectral images. This makes up for the fact that only CNN is used to capture local features in the spatial and channel dimensions, which can effectively avoid information loss and improve the reconstruction accuracy of hyperspectral images.

[0049] (2) The multifunctional 3D convolution module V3DCM, by decomposing the ordinary 3D convolution kernel and performing convolution in different dimensions, can effectively retain feature diversity while reducing computational complexity and memory usage. In addition, while using filters of three different dimensions for calculation, in order to retain the full convolution effect of 3D convolution, a branch using the 3D full convolution kernel is retained. The module also uses the ECA layer to screen features, effectively retaining features important for spectral reconstruction and removing redundant features. After convolution, the features are jump-connected with shallow features to enhance information flow, further improve image reconstruction accuracy, and achieve a balance between high spectral reconstruction accuracy and efficient 3D convolution;

[0050] (3) By adopting the gated convolution GMFF of multi-feature fusion in the spectral reconstruction stage, gated convolution is used instead of ordinary convolution for feature filtering in the selection of multi-feature fusion, which can adaptively select the most relevant features according to the importance of the input features, thereby enhancing the information capture ability in the reconstruction process, so that more effective information can be captured for spectral reconstruction during the mapping process. In addition, in the process of multi-feature fusion, gated convolution can be used to filter irrelevant features to avoid redundant information interference, thereby improving the accuracy of the reconstruction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the network structure of a hyperspectral image reconstruction method based on RGB images disclosed in one embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structure of a multifunctional 3D convolution module V3DCM disclosed in one embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the structure of the spatial-spectral multi-attention mechanism SSMAM and the gated selection GMFF for multi-feature fusion disclosed in one embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a process for reconstructing a hyperspectral image based on RGB images disclosed in an embodiment of the present invention;

[0055] Figure 5 and Figure 6 This is a schematic diagram of the spectral reconstruction effect of the present invention and the comparison method on the NITRE2022 dataset disclosed in one embodiment of the present invention;

[0056] Figure 7 and Figure 8 A schematic diagram of the subjective effects of an embodiment of the present invention and a comparison method on the Harvard dataset. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] The present invention is described in further detail below with reference to the accompanying drawings:

[0059] like Figure 1 As shown, a hyperspectral image reconstruction method based on RGB image provided by the present invention includes:

[0060] Perform channel expansion on the three-channel RGB image through convolution processing;

[0061] Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales.

[0062] Use the 2D convolution module and the multifunctional 3D convolution module V3DCM to extract spatial and spectral features of features at different scales, and fuse the obtained features;

[0063] Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled, captured and weightedly fused in the spatial, channel and spectral dimensions respectively;

[0064] The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connections and convolution operations to generate hyperspectral reconstruction results.

[0065] In this embodiment, by introducing the spatial-spectral multi-attention mechanism SSMAM, CNN captures local channel and spatial attention respectively, and uses the attention mechanism in Transformer to capture long-distance detail features in the spectral dimension, which comprehensively enhances the feature extraction capability and fully utilizes the global information in the image in the spatial and spectral dimensions, thereby improving the accuracy of hyperspectral reconstruction; by using a multifunctional 3D convolution module V3DCM that can reduce the large amount of 3D convolution calculation and memory usage while retaining feature diversity, in addition to feature calculations in all directions, the ECA layer is also used for feature screening, and jump connections are made with shallow features, so that the accuracy of hyperspectral reconstruction and efficient 3D convolution are balanced; in addition, by adopting the gated convolution GMFF for multi-feature fusion in the spectral reconstruction stage, gated convolution is used instead of ordinary convolution for feature filtering for the selection of multi-feature fusion, so that more effective information can be captured for spectral reconstruction during the mapping process, thereby improving the accuracy of reconstruction.

[0066] Specifically, to enable the RGB-reconstructed hyperspectral image to fully utilize the effective spatial-spectral information and reduce the overall error of spectral reconstruction, the present invention, based on the characteristics of sparse spectral self-similarity in HSI space, develops an RGB hyperspectral reconstruction method based on spatial-spectral multi-attention mechanism and multifunctional 3D convolution with gated feature selection from the perspectives of long-range and local features. In the spectral mapping stage of RGB image reconstruction of hyperspectral images, shallow features are easily lost due to excessive network depth. Therefore, the method proposed in this invention uses a residual structure as the backbone.

[0067] In the above embodiment, preferably, the three-channel RGB image is expanded by convolution processing, and the specific process includes:

[0068] A 2D convolution module with a convolution kernel size of 3 is used to expand the three-channel RGB image so that the number of channels becomes 3;

[0069] The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

[0070] In the above embodiment, preferably, spectral mapping and reconstruction are performed by three cascaded residual spectrum reconstruction modules RSRM, each RSRM downsamples the feature map or intermediate feature map after the expansion channel to obtain feature information of different scales. The specific process includes:

[0071] Three residual spectrum reconstruction modules (RSRMs) are cascaded, and each RSRM undergoes multi-scale feature extraction, multiple attention mechanism calculation, feature fusion gate selection, and multiple jump connections.

[0072] Specifically, the features or intermediate features of the RGB image after the expansion channel are downsampled by ×1, ×2 and ×4 respectively, and the original features are converted into features of three different scales: [31, 128, 128], [62, 64, 64] and [124, 32, 32] respectively.

[0073] These three different-scale features can effectively fuse shallow and deep feature information in spectral reconstruction. By fusing multi-scale features, spatial and spectral information at different levels can be captured in the subsequent spectral reconstruction stage, thereby better recovering the high-dimensional spectral structure.

[0074] like Figure 2 As shown, in the above embodiment, preferably, a 2D convolution module and a multifunctional 3D convolution module V3DCM are used to extract spatial and spectral features of features of different scales respectively, and the obtained features are fused. The specific process includes:

[0075] For features of different scales, two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used respectively, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution.

[0076] In the above embodiment, the multifunctional 3D convolution module is one of the key points of the present invention. Preferably, the specific process of the multifunctional 3D convolution module includes:

[0077] Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension;

[0078] A 1×1×1 filter is used to extract shallow features from the input features. Then, convolution kernels of 3×1×1, 1×3×1, and 1×1×3 sizes are used to capture feature information in the depth, height, and width directions respectively. To avoid missing details of the global features of the spatial spectrum, a 3×3×3 filter is used for a comprehensive feature calculation.

[0079] After the features of each dimension are extracted, a jump connection is made between the extracted features of each dimension and the input of the corresponding dimension;

[0080] Then, the features of four different dimensions are fused, and a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension.

[0081] In order to retain the effective spatial spectral information extracted by 3D convolution, the ECA layer is spliced ​​again to filter and extract the effective spatial spectral information. In order not to lose the shallow features, a long jump connection is finally performed on the shallow features of the input multifunctional 3D convolution.

[0082] like Figure 3 As shown, in the above embodiment, preferably, through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are respectively subjected to feature weighted fusion in the spatial, channel and spectral dimensions. The specific process includes:

[0083] Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are spliced ​​along the new channel dimension. Then, a convolution layer is used to convolve the pooled tensor, and the Sigmoid function is used to generate attention weights, which are finally multiplied element-wise with the input tensor.

[0084] Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point. Then, a convolutional layer and ReLU activation function are used for feature optimization. The Sigmoid function is then used to generate channel weights, which are then element-wise multiplied with the original input tensor to achieve channel enhancement.

[0085] Through Transformer-based spectral self-attention, unlike ordinary self-attention, which uses a single pixel as a token to calculate attention weights, the present invention calculates attention weights along the spectral dimension in order to better utilize the spectral information of the hyperspectral, especially long-distance information;

[0086] The results of the above-mentioned multiple attention mechanisms are feature fused, and a convolution operation with a convolution kernel size of 1 and a jump connection are performed to avoid the loss of original feature information.

[0087] Among them, the use of the multi-attention mechanism not only takes advantage of CNN's excellent feature extraction ability in local details, but also takes advantage of Transformer's excellent representation ability in long distances. During the spectral reconstruction process, it can accurately restore more detailed and comprehensive spectral information, thereby improving the reconstruction accuracy and robustness.

[0088] In the above embodiment, the features of three different scales are all processed through the above steps, and the feature information obtained includes the results of 2D convolution, 3D convolution and multiple attention mechanisms. Preferably, for these three different scale features, it is also necessary to use gated selection GMFF based on multi-feature fusion of gated convolution to fuse and filter the features of different scales, and generate hyperspectral reconstruction results for the fused features through skip connection and convolution operation. The specific process includes:

[0089] The features of three different scales are first flattened and then spliced;

[0090] In order to fully filter out effective information, the concatenated features are first filtered using the three-level gated convolution GMFF, and the filtered features are then subjected to ordinary Concat connection.

[0091] The features are then fused through a convolution operation with a convolution kernel size of 3;

[0092] The fused features are connected to the input features of the residual spectrum reconstruction module RSRM through a jump connection, and then a convolution operation with a convolution kernel of 3 is performed;

[0093] Then, a jump connection is performed on the tensor after the expansion channel to obtain the hyperspectral reconstruction result.

[0094] like Figure 4 As shown, in the above embodiment, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0095] The loss is calculated for the hyperspectral reconstruction results and the real spectrum, and the model parameters are optimized through back propagation until the model converges and the training is completed.

[0096] In the above embodiment, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0097] The NTIRE2022 dataset and Harvard dataset were used as validation sets to verify the optimal model obtained through training, and the model was evaluated using preset evaluation indicators.

[0098] In this embodiment, since the present invention uses two data sets for effect verification, the training parameters of the verification method for the two data sets are consistent, but the training parameters and image preprocessing stages of each data set are partially different. The parameter settings corresponding to the two data sets are first introduced separately.

[0099] For the NTIRE2022 dataset, RGB and hyperspectral images were paired and segmented with a stride of 8 and a patch size of 128. Each patch was randomly rotated, horizontally flipped, and vertically flipped. All methods were trained with a batch size of 20 and an epoch of 300, with an initial learning rate of 0.0004.

[0100] The Harvard dataset uses a stride of 64 and a patch_size of 128 to split paired RGB images and hyperspectral images. No data augmentation is performed. The batch_size during training is 8, the epoch is 40, and the learning rate is initialized to 0.0001.

[0101] The Adam optimizer is used for both datasets with β1=0.9 and β2=0.99.

[0102] The present invention further provides a hyperspectral image reconstruction system based on RGB images, which is applied to the hyperspectral image reconstruction method based on RGB images disclosed in any one of the above embodiments, comprising:

[0103] RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing;

[0104] The multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales.

[0105] The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features;

[0106] The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through CNN-based spatial and channel attention and Transformer-based spectral self-attention mechanism;

[0107] The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results through skip connections and convolution operations on the fused features.

[0108] According to the hyperspectral image reconstruction system based on RGB images disclosed in the above embodiment, the functions to be implemented by each module thereof correspond to the respective steps in the hyperspectral image reconstruction method based on RGB images disclosed in the above embodiment. During implementation, the operations are performed with reference to the above embodiment and will not be described in detail here.

[0109] According to the hyperspectral image reconstruction method and system based on RGB images disclosed in the above embodiments, the implementation process and effects of the present invention are described through the following examples.

[0110] Example 1:

[0111] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset to experiment on the effectiveness of the spatial-spectral multi-attention mechanism proposed in this invention.

[0112] First, we preprocessed the 900 training pairs in the NTIRE2022 dataset, partitioning the paired RGB and hyperspectral images into 128-patch_sizes with a step size of 8. Each patch is 128 x 128, and the RGB image is linearly scaled to [0, 1]. Each pair of data (containing an RGB and corresponding hyperspectral image patch) is then rotated, flipped vertically, or flipped horizontally to varying degrees.

[0113] We then trained for 300 epochs using a batch size of 20, feeding the network 20 pairs of patches at a time. We then fed the RGB image into the network for training, and calculated the loss between the resulting pseudo-hyperspectral image and the true hyperspectral image. We used MREA for loss calculation and backpropagation updates, and Adam as the optimizer.

[0114] Using this dataset and the same training parameters, experiments were conducted on a baseline (CNN-based with only spatial and channel attention mechanisms) and a network using a spatial-spectral multi-attention mechanism. MRAE, RMSE, and PSNR were used for validation. The following evaluation metrics were calculated between the reconstructed hyperspectral images and the true hyperspectral images using 50 pairs of RGB images from the validation set:

[0115]

[0116] In this quantitative experiment, it can be seen that the network with the proposed spatial-spectral multi-attention mechanism is superior to the network without it in all evaluation indicators, which shows that the proposed spatial-spectral multi-attention mechanism has a good effect.

[0117] Example 2:

[0118] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset, and experiments were conducted simultaneously on the effectiveness of the spatial-spectral multi-attention mechanism and the multi-functional 3D convolution module proposed in this invention.

[0119] First, we preprocessed the 900 training pairs in the NTIRE2022 dataset, partitioning the paired RGB and hyperspectral images into 128-patch_sizes with a step size of 8. Each patch is 128 x 128, and the RGB image is linearly scaled to [0, 1]. Each pair of data (containing an RGB and corresponding hyperspectral image patch) is then rotated, flipped vertically, or flipped horizontally to varying degrees.

[0120] We then trained for 300 epochs using a batch size of 20, feeding the network 20 pairs of patches at a time. We then fed the RGB image into the network for training, and calculated the loss between the resulting pseudo-hyperspectral image and the true hyperspectral image. We used MREA for loss calculation and backpropagation updates, and Adam as the optimizer.

[0121] Based on Example 1, a multifunctional 3D convolution module was added for experimentation, and the three evaluation indicators of MRAE, RMSE, and PSNR were used for verification. The evaluation indicators calculated between the reconstructed hyperspectral image obtained by using 50 pairs of RGB of the validation set and the real hyperspectral image are as follows:

[0122]

[0123] In this quantitative experiment, it can be seen that the multifunctional 3D convolutional module proposed in this invention achieves the best performance across all evaluation metrics. Furthermore, the network with both the spatial-spectral attention mechanism and the multifunctional 3D convolutional module performs better than the network with only the spatial-spectral multi-attention mechanism. This demonstrates the effectiveness of the multifunctional 3D convolutional module and, once again, the effectiveness of the spatial-spectral multi-attention mechanism.

[0124] Example 3:

[0125] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset, and experiments were conducted simultaneously on the effectiveness of the spatial-spectral multi-attention mechanism, multi-functional 3D convolution module, and gating selection of multi-feature fusion proposed in this invention.

[0126] First, we preprocessed the 900 training pairs in the NTIRE2022 dataset, partitioning the paired RGB and hyperspectral images into 128-patch_sizes with a step size of 8. Each patch is 128 x 128, and the RGB image is linearly scaled to [0, 1]. Each pair of data (containing an RGB and corresponding hyperspectral image patch) is then rotated, flipped vertically, or flipped horizontally to varying degrees.

[0127] We then trained for 300 epochs using a batch size of 20, feeding the network 20 pairs of patches at a time. We then fed the RGB image into the network for training, and calculated the loss between the resulting pseudo-hyperspectral image and the true hyperspectral image. We used MREA for loss calculation and backpropagation updates, and Adam as the optimizer.

[0128] Based on Example 2, we added multi-feature fusion gated selection to the experiment and used the same three evaluation indicators, MRAE, RMSE, and PSNR, for verification. The evaluation indicators calculated between the reconstructed hyperspectral image and the real hyperspectral image using 50 pairs of RGB from the validation set are as follows:

[0129]

[0130] This quantitative experiment shows that the gated selection of multi-feature fusion proposed in this invention achieves optimal results across all evaluation metrics. Furthermore, the combined effect of these three key modules is even greater than that of a network that incorporates only the spatial-spectral multi-attention mechanism and the multi-functional 3D convolutional module. In this experiment, it can be seen that these three modules complement each other, and their combined effect is optimal.

[0131] Example 4:

[0132] The method proposed in the present invention is implemented using the Pytorch framework, using the Adam optimizer with β1=0.9 and β2=0.99. The learning rate is initialized to 0.0004. The training process cuts the dataset into 128×128 RGB and hyperspectral sample pairs and feeds them into the network for training. The RGB images are linearly scaled to [0,1]. The training process is performed using a single NVIDIA GeForce RTX4090. The present invention verifies its final effect on two datasets respectively. Table 1 shows the results on the NTIRE2022 dataset, compared with the existing SOTA methods, using MRAE, RMSE, PSNR, SAM, and ERGAS as evaluation indicators, verifying the effectiveness of the method proposed in the present invention. Table 2 shows the results on the Harvard dataset, using MRAE, RMSE, PSNR, and SSIM as evaluation indicators, which also proves the superiority of the present invention.

[0133] The NTIRE2022 dataset is the official dataset used in the NTIRE2022 spectral reconstruction challenge. It contains 1,000 RGB hyperspectral pairs, including 900 pairs for training, 50 pairs for validation, and 50 pairs for testing. Because the actual hyperspectral images in the 50 test sets are confidential, only 900 pairs are used for training and 50 pairs for validation. Each hyperspectral image is a 482 x 512 image covering 31 bands from 400 nm to 700 nm. The corresponding RGB image is obtained by applying a transformation matrix to the HIS and adding shot noise to simulate real camera conditions. The Harvard dataset consists of 50 hyperspectral images covering the spectral range of 420 nm to 720 nm, containing 31 bands, and each image has a spatial resolution of 1392 x 1042. To obtain the corresponding RGB images, the camera spectral function used in the dataset proposed by the NTIRE2020 spectral challenge is used to obtain the corresponding RGB images. The training set and validation set are divided into 46:4 for training and validation. The images in the validation set are Imgh0, Imgh1, Imgh2, and Imgh3, and the other images are used for training.

[0134] As for the evaluation indicators used, MREA is used to calculate the pixel-level differences between the reconstructed hyperspectral image and the true hyperspectral image in all bands. The smaller the MREA value, the better the reconstruction effect. RMSE is obtained by calculating the mean sum of the squares of the pixel-level differences between the reconstructed image and the true image, and then taking the square root. The smaller the RMSE value, the closer the reconstructed image is to the true image. PSNR is an indicator used to measure the degree of image distortion. The larger the value, the smaller the image distortion and the more accurate the spectral recovery. SAM reflects the evaluation of the restoration quality from a spectral perspective. The smaller the value, the more similar the predicted spectrum is to the true spectrum. ERGAS is a comprehensive evaluation indicator obtained by calculating the weighted average of the relative errors between the reconstructed image and the true image at different wavelengths. The smaller the value, the higher the reconstruction quality. SSIM is used to evaluate the similarity of two images. When the SSIM value is close to 1, it means that the two images are very similar.

[0135] Specifically, Table 1 shows the results of quantitative comparison of the method proposed in the present invention with existing SOTA methods on the NTIRE2022 dataset, including a snapshot compression imaging system reconstruction method (HDNet), two natural image restoration models (HINet, MIRNet) and five spectral reconstruction methods (HSCNN+, HRNet, AWAN, DRCR, MST++). All methods were retrained and tested using the same settings. As can be seen from the table, the method proposed in the present invention achieved the best results in the four evaluation indicators of RMSE, PSNR, SAM and ERGAS, and MRAE ranked third among all methods. In general, the method proposed in the present invention has a good performance on this dataset, which is largely due to the fact that the present invention makes full use of the effective information of the spatial and spectral dimensions in the process of spectral reconstruction.

[0136] Table 1 Experimental results with other SOTA methods on the NTIRE2022 dataset

[0137]

[0138] Table 2 shows the quantitative comparison results of the method proposed in the present invention with the existing SOTA methods on the Harvard dataset, including a natural image restoration model (MIRNet) and four image spectral reconstruction methods (HSCNN+, AWAN, MST++, FMPSA). In order to ensure fairness, all comparison methods are retrained and tested using the same design. As can be seen from the table, the method proposed in the present invention is quite competitive in terms of the four evaluation indicators of MRAE, RMSE, PSNR and SSIM. Among them, PSNR is the best among all methods, and the other three evaluation indicators are the second best values. This may be because the sample size of the Harvad dataset is small, which affects the effectiveness of this method to a certain extent, but in general, the performance is still the best compared with other methods. It fully demonstrates that this method has excellent capabilities in spectral reconstruction tasks.

[0139] Table 2 Experimental results on Harvard dataset with other SOTA methods

[0140]

[0141] In order to more intuitively demonstrate the excellent spectral reconstruction capability of the present invention, Figure 5-8 Qualitative experimental results compared with other methods on two datasets are shown respectively. Figure 5 and Figure 6 The spectral reconstruction effect diagram of the present invention and the above-mentioned comparison method on the NITRE2022 dataset is shown. The first row is the error heat map calculated using RMSE between the reconstruction results of each method and the true hyperspectral image. The darker the color, the smaller the error. The error image is a heat map of the RMSE between the real data and the reconstructed HSI. The second row is an enlarged view of the green box part of the RGB image corresponding to the Ground Truth and the reconstruction results of each method. The first picture in the second row is an enlarged view of the true hyperspectral image in this area. Figure 5 It can be seen from the figure that the spectrum reconstruction effect of the present invention not only has the smallest error, but also from the detail image magnified by the green box, it can be seen that compared with other methods, the spectrum restoration details of this method are also better.

[0142] exist Figure 6 As can be clearly seen in the figure, the proposed method significantly outperforms other methods in spectral reconstruction of the 530nm band of the ARAD_1K_0950 image in the test set. The error plots show that the proposed method achieves excellent transitions at the boundaries between bright and dark areas. While other methods exhibit varying degrees of spectral distortion during reconstruction, the proposed method, while also exhibiting some distortion, achieves the best overall results.

[0143] Specifically, Figure 6The first row shows the visual results and reconstructed HSI error images (first row) of ARAD_1K_0950 predicted by HSCNN+, HRNet, HDNet, MIRNet, MST++, and the proposed method for the 530nm band in the NTIRE2022 test set. The error images are heatmaps of the RMSE between the ground truth and the reconstructed HSI. The second row shows enlarged images of the green-boxed portion of the RGB images corresponding to the ground truth and the reconstruction results of each method.

[0144] Figure 7 and Figure 8 The subjective effect diagram of the present invention and the above comparison method on the Harvard dataset is shown. Figure 7 The first row shows the reconstruction error maps of various methods for Imgh2 at a wavelength of 570 nm in the Harvard test set. This error map is a PNSR heatmap between the real data and the reconstructed data, with darker colors representing smaller errors. It can be seen that the error map of our method achieves the most uniform reconstruction results and minimizes the error overall. Figure 7 The first image in the first row is the RGB image corresponding to Imgh2, and the second row shows enlarged images of the green-boxed portion of the RGB image, representing the true hyperspectral image and the reconstruction results of each method. The first image is an enlarged view of the green-boxed portion of the true hyperspectral image. This detailed image shows that the proposed method's rendering is closest to the true hyperspectral image, particularly in areas with "reflective" RGB, where detail can be easily lost due to inadequate utilization of the available information in the spatial spectral dimension during reconstruction. The proposed method takes this into account, achieving superior reconstruction results compared to other methods.

[0145] Figure 8 Figure 2 shows the reconstruction error plots of various methods for Imgh3 at 490nm in the Harvard test set, along with magnified images of the reconstruction details. The proposed method minimizes the overall error and achieves the clearest detail texture in the magnified image, most similar to a true hyperspectral detail magnified image. Compared to the detailed renderings of other reconstruction methods, the proposed method produces clearer reconstructions and greater pixel smoothness, intuitively demonstrating its effectiveness and robustness.

[0146] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A hyperspectral image reconstruction method based on RGB image, characterized in that: include: Perform channel expansion on the three-channel RGB image through convolution processing; Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales. The 2D convolution module and the multifunctional 3D convolution module V3DCM are used to extract spatial and spectral features of features at different scales, and the obtained features are fused. The specific process includes: For features of different scales, two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used respectively, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution; The specific process of the multifunctional 3D convolution module V3DCM includes: Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension; A 1×1×1 filter is used to perform shallow feature extraction on the input features. Convolution kernels of 3×1×1, 1×3×1, and 1×1×3 sizes are used to capture feature information in depth, height, and width respectively. A 3×3×3 filter is used to perform a comprehensive feature calculation. Perform a jump connection between the extracted features of each dimension and the input of the corresponding dimension; The features of four different dimensions are fused, and then a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension; Splicing an ECA layer once to filter and extract effective spatial spectrum information, and then performing a long skip connection on the shallow features input into the multifunctional 3D convolution; Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled, captured, and weightedly fused in the spatial, channel, and spectral dimensions. The specific process includes: Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are spliced ​​along the new channel dimension. Then, a convolution layer is used to convolve the pooled tensor, and the Sigmoid function is used to generate attention weights, which are finally multiplied element-wise with the input tensor. Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point. Then, a convolutional layer and ReLU activation function are used for feature optimization. The Sigmoid function is then used to generate channel weights, which are then element-wise multiplied with the original input tensor to achieve channel enhancement. Attention weights are calculated along the spectral dimension through Transformer-based spectral self-attention; The results of the above attention mechanism are feature fused, and a convolution operation with a convolution kernel size of 1 and a skip connection are performed; The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connections and convolution operations to generate hyperspectral reconstruction results.

2. The hyperspectral image reconstruction method based on RGB image according to claim 1, characterized in that: The convolution process is used to expand the three-channel RGB image. The specific process includes: A 2D convolution module with a convolution kernel size of 3 is used to expand the three-channel RGB image so that the number of channels becomes 3; The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

3. The hyperspectral image reconstruction method based on RGB image according to claim 2, characterized in that: The spectrum mapping and reconstruction are performed by three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map or intermediate feature map after the expansion channel to obtain feature information of different scales. The specific process includes: Three residual spectrum reconstruction modules RSRM are cascaded, and each RSRM performs a downsampling process of ×1, ×2 and ×4 on the features or intermediate features of the RGB image after the expansion channel, and obtains features of three different scales of [31, 128, 128], [62, 64, 64] and [124, 32, 32] from the original features.

4. The hyperspectral image reconstruction method based on RGB image according to claim 1, characterized in that: The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connections and convolution operations to generate hyperspectral reconstruction results. The specific process includes: The features of three different scales are first flattened and then spliced; Use three gated convolutions to filter the concatenated features, and perform ordinary Concat connections on the filtered features; The features are fused through a convolution operation with a convolution kernel size of 3; The fused features are connected to the input features of the residual spectrum reconstruction module RSRM once, and then a convolution operation with a convolution kernel of 3 is performed; A skip connection is performed on the tensor of the last output RSRM to obtain the hyperspectral reconstruction result.

5. The hyperspectral image reconstruction method based on RGB image according to any one of claims 1 to 4, characterized in that: Also includes: Loss calculation is performed on the hyperspectral reconstruction result and the real spectrum, and the model parameters are optimized by back propagation until the model converges and the training is completed.

6. The hyperspectral image reconstruction method based on RGB image according to claim 5, characterized in that: Also includes: The NTIRE2022 dataset and Harvard dataset were used as validation sets to verify the optimal model obtained through training, and the model was evaluated using preset evaluation indicators.

7. A hyperspectral image reconstruction system based on RGB images, characterized in that: The hyperspectral image reconstruction method based on RGB images applied to any one of claims 1 to 6 comprises: RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing; The multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules (RSRMs). Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information at different scales. The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features; The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through CNN-based spatial and channel attention and Transformer-based spectral self-attention mechanism; The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results through skip connections and convolution operations on the fused features.

Images