Hyperspectral image reconstruction method and system based on RGB image

Abstract

The invention discloses a hyperspectral image reconstruction method and system based on an RGB image. The method comprises the following steps: carrying out channel expansion on the RGB image through convolution processing; performing spectral mapping and reconstruction on the channel-expanded feature map or the intermediate feature map through three cascaded RSRMs, and performing down-sampling on the feature map by each RSRM to obtain feature information of different scales; extracting and fusing the features by using a 2D convolution module and a V3DCM; modeling and capturing fusion features in space, channel and spectrum dimensions through space and channel attention and a spectrum self-attention mechanism; and fusing and filtering the features of different scales by using a GMFF, and finally generating a hyperspectral reconstruction result through jump connection and convolution operation. According to the technical scheme, information loss can be effectively avoided, meanwhile, the hyperspectral reconstruction precision is improved, the hyperspectral reconstruction precision and efficient 3D convolution are balanced, and the precision of the reconstruction process is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a hyperspectral image reconstruction method based on RGB images and a hyperspectral image reconstruction system based on RGB images. Background Art

[0002] Hyperspectral Image (HSI) is obtained by decomposing the incident light into multiple bands by a hyperspectral imaging system. Compared with ordinary RGB images, it can not only capture information in the spatial dimension but also obtain rich spectral features. Many substances with similar appearance or color can be easily distinguished in HSI because of the information in different bands. Therefore, HSI has a wide range of applications in geological exploration, agricultural testing, food safety and other fields. However, the hyperspectral images obtained by the hyperspectral imaging system are not only expensive and time-consuming, but also difficult to use in dynamic scenes. This greatly limits the application scope of hyperspectral.

[0003] In recent years, the method of using RGB images to predict information of multiple spectral bands and reconstruct complete HSI can solve the problems of high cost and acquisition speed, which has attracted the attention and research of domestic and foreign scholars. At present, the application of HSI reconstructed from RGB images has been applied to corn disease monitoring, vein visualization, skin color detection, art restoration and identification, etc., which has profound significance for the research of this task.

[0004] Since HSI has multiple bands, while RGB images have only three bands, an RGB image can be obtained by spectrally downsampling different HSIs. Therefore, reconstructing HSI from RGB images is a highly ill-posed problem. Early methods used sparse dictionary methods to construct a mapping library between hyperspectral images and corresponding RGB images, and reconstructed hyperspectral images by mapping. However, this method can only reconstruct accurate hyperspectral images for specific spectral response functions, and its generalization and robustness need to be improved. With the rapid development of deep learning in recent years, the powerful nonlinear mapping ability of convolutional neural networks (CNNs) has provided a new solution to this problem. The end-to-end mapping of RGB images and HSIs by CNN can fully capture the complex mapping relationship between the two, while improving accuracy and generalization. At the same time, in the process of spectral reconstruction, the feature similarity of spatial and spectral dimensions should be fully considered. Many studies use the space-channel dual attention mechanism to perform adaptive correction of information in these two dimensions. Based on the characteristics of sparse spectral self-similarity in hyperspectral space, some studies have studied spectral self-attention calculation by using the self-attention mechanism in Transformer, which can make up for the shortcomings of CNN in capturing incomplete information at long distances. In order to better capture the information between adjacent channels, 3D convolution is also widely used in hyperspectral image related tasks.

[0005] Existing methods based on deep learning to reconstruct HSI from RGB have shown good results. However, this task is a complex nonlinear process. Existing methods cannot fully utilize spatial and spectral dimension information during the reconstruction process, resulting in limited accuracy of the final spectral reconstruction. The prior art discloses a dual attention mechanism based on CNN for RGB hyperspectral reconstruction tasks. However, CNN mainly focuses on feature extraction of local information. Although feature capture in the long-distance dependency of the spatial dimension can be compensated by calculating the similarity of multiple features by decomposing features, for multiple bands of the hyperspectral spectrum, it is necessary not only to grasp the global information in the spatial dimension, but also to calculate the feature similarity of the global information in the spectral dimension. In addition, in the calculation of similarity between adjacent bands of the hyperspectral spectrum, in order to make full use of spectral information, some methods propose to use 3D convolution to capture it. However, existing methods cannot take into account the computational complexity of 3D convolution and the consumption of memory or the loss of feature details in the spatial spectral dimension due to the reduction of complexity. There is a method to use space-spectrum symmetric 3D convolution to overcome this problem. Although it can balance the complexity of 3D convolution and the utilization of spectral information to a certain extent, the improvement of the 3D convolution filter may lead to the loss of space-spectrum global feature details and the inability to guarantee the accuracy of the reconstruction process. In addition, in the process of spectral mapping, it is usually necessary to fuse and select features of different scales. However, in the selection process, a single CNN convolution cannot fully identify useful information, resulting in limited reconstruction effect. Summary of the invention

[0006] In view of the above problems, the present invention provides a hyperspectral image reconstruction method and system based on RGB images. By introducing the spatial-spectral multi-attention mechanism SSMAM (Spatial-Spectral Multi-Attention Mechanim), CNN captures local channel and spatial attention respectively, and uses the attention mechanism in Transformer to capture long-distance detail features in the spectral dimension, comprehensively strengthens the feature extraction capability, and fully utilizes the global information in the image in the spatial and spectral dimensions, thereby improving the accuracy of hyperspectral reconstruction; through the multifunctional 3D convolution module V3DCM (Versatile 3D Convlution Module) that can reduce the large amount of 3D convolution calculation and memory usage and retain feature diversity, in addition to feature calculation in all directions, the ECA layer is also used for feature screening, and jump connection is performed with shallow features, so that the high spectral reconstruction accuracy and efficient 3D convolution are balanced; in addition, by adopting the gated convolution GMFF (Gated Multi-Feature Fusion) of multi-feature fusion in the spectral reconstruction stage, the gated convolution is used instead of the ordinary convolution for feature filtering for the selection of multi-feature fusion, so that more effective information can be captured in the mapping process for spectral reconstruction, thereby improving the accuracy of reconstruction.

[0007] To achieve the above object, the present invention provides a hyperspectral image reconstruction method based on RGB image, comprising:

[0008] The three-channel RGB image is expanded through convolution processing;

[0009] Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales.

[0010] Use 2D convolution module and multifunctional 3D convolution module V3DCM to extract spatial and spectral features of different scales respectively, and fuse the obtained features;

[0011] Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled and captured in the spatial, channel and spectral dimensions, and then weighted and fused;

[0012] The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are used to generate hyperspectral reconstruction results through skip connections and convolution operations.

[0013] In the above technical solution, preferably, the channel expansion of the three-channel RGB image is performed by convolution processing, and the specific process includes:

[0014] A 2D convolution module with a convolution kernel size of 3 is used to expand the channels of the three-channel RGB image so that the number of channels becomes 3;

[0015] The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

[0016] In the above technical solution, preferably, the spectrum mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM, each RSRM downsamples the feature map or intermediate feature map after the expansion channel to obtain feature information of different scales, and the specific process includes:

[0017] Three residual spectrum reconstruction modules RSRM are cascaded, and each RSRM performs a downsampling process of ×1, ×2 and ×4 on the features or intermediate features of the RGB image after the expansion channel, and obtains features of three different scales of [31, 128, 128], [62, 64, 64] and [124, 32, 32] from the original features.

[0018] In the above technical solution, preferably, the 2D convolution module and the multifunctional 3D convolution module V3DCM are used to extract the features of different scales in spatial and spectral dimensions respectively, and the obtained features are fused. The specific process includes:

[0019] Two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used for features of different scales, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution.

[0020] In the above technical solution, preferably, the specific process of the multifunctional 3D convolution module includes:

[0021] Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension;

[0022] A 1×1×1 filter is used to extract shallow features from the input features. Convolution kernels of 3×1×1, 1×3×1, and 1×1×3 are used to capture feature information from the depth, height, and width directions respectively. To avoid missing details of the global features of the spatial spectrum, a 3×3×3 filter is used to perform a comprehensive feature calculation.

[0023] Make a jump connection between the extracted features of each dimension and the input of the corresponding dimension;

[0024] The features of four different dimensions are fused, and then a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension;

[0025] The ECA layer is spliced ​​once to screen and extract the effective spatial spectrum information, and then a long jump connection is performed on the shallow features input into the multifunctional 3D convolution.

[0026] In the above technical solution, preferably, the fusion features of 2D convolution and V3DCM are modeled and captured in the spatial, channel and spectral dimensions respectively through the spatial and channel attention based on CNN and the spectral self-attention mechanism based on Transformer, and the specific process includes:

[0027] Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are concatenated along the new channel dimension. Then, the pooled tensor is convolved using a convolutional layer, and the attention weight is generated using the Sigmoid function, which is then multiplied element-by-element with the input tensor.

[0028] Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point, and then the convolution layer and ReLU activation function are used to optimize the features. The Sigmod function is then used to generate channel weights, which are multiplied element-by-element with the original input tensor to achieve channel enhancement.

[0029] Attention weights are calculated along the spectral dimension through Transformer-based spectral self-attention;

[0030] The results of the above attention mechanism are feature fused, and a convolution operation with a convolution kernel size of 1 and a skip connection are performed.

[0031] In the above technical solution, preferably, the gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connection and convolution operations to generate hyperspectral reconstruction results, and the specific process includes:

[0032] The features of three different scales are flattened first and then concatenated;

[0033] Use three gated convolutions to filter the concatenated features, and perform ordinary Concat connections on the filtered features;

[0034] The features are fused through a convolution operation with a convolution kernel size of 3;

[0035] The fused features are jump-connected with the input features of the residual spectrum reconstruction module RSRM, and then a convolution operation with a convolution kernel of 3 is performed;

[0036] A jump connection is performed on the tensor of the last output RSRM to obtain the hyperspectral reconstruction result.

[0037] In the above technical solution, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0038] The loss calculation is performed on the hyperspectral reconstruction result and the real spectrum, and the model parameters are optimized by back propagation until the model converges and the training is ended.

[0039] In the above technical solution, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0040] The NTIRE2022 dataset and Harvard dataset were used as validation sets to verify the optimal model obtained through training, and the model was evaluated using preset evaluation indicators.

[0041] The present invention further proposes a hyperspectral image reconstruction system based on RGB images, which is applied to the hyperspectral image reconstruction method based on RGB images disclosed in any one of the above technical solutions, comprising:

[0042] RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing;

[0043] A multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales.

[0044] The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features;

[0045] The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through the spatial and channel attention based on CNN and the spectral self-attention mechanism based on Transformer;

[0046] The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results for the fused features through skip connections and convolution operations.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The spatial-spectral multi-attention mechanism SSMAM is introduced. Based on the spatial attention mechanism and channel attention mechanism built by CNN, a self-attention mechanism based on Transformer along the spectral dimension is added to model the features of the spectral dimension, capture the long-distance dependencies in the spectral dimension, and improve the ability to extract similarities and detail features between bands in hyperspectral images. This makes up for the fact that only CNN is used to capture local features in the spatial and channel dimensions, which can effectively avoid information loss and improve the reconstruction accuracy of hyperspectral images.

[0049] (2) The multifunctional 3D convolution module V3DCM can effectively retain feature diversity while reducing computational complexity and memory usage by decomposing ordinary 3D convolution kernels and performing convolutions in different dimensions. In addition, while using filters of three different dimensions for calculations, in order to retain the effect of the full convolution of the 3D convolution, a branch using the 3D full convolution kernel is retained. The module also uses the ECA layer to screen features, effectively retaining features that are important for spectral reconstruction and removing redundant features. After convolution, the features are jump-connected to shallow features to enhance information flow, further improve image reconstruction accuracy, and achieve a balance between high spectral reconstruction accuracy and efficient 3D convolution;

[0050] (3) By adopting the gated convolution GMFF for multi-feature fusion in the spectral reconstruction stage, gated convolution is used instead of ordinary convolution for feature filtering in the selection of multi-feature fusion. The most relevant features can be adaptively selected according to the importance of the input features, thereby enhancing the information capture capability in the reconstruction process, so that more effective information can be captured for spectral reconstruction during the mapping process. In addition, irrelevant features can be filtered out with gated convolution in the multi-feature fusion process to avoid redundant information interference, thereby improving the accuracy of the reconstruction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the network structure of a hyperspectral image reconstruction method based on RGB images disclosed in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structure of a multifunctional 3D convolution module V3DCM disclosed in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the structure of a spatial-spectral multi-attention mechanism SSMAM and a gated selection GMFF for multi-feature fusion disclosed in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a process of a hyperspectral image reconstruction method based on RGB images disclosed in an embodiment of the present invention;

[0055] Figure 5 and Figure 6 A schematic diagram of the spectral reconstruction effect of the present invention and the comparison method on the NITRE2022 dataset disclosed in an embodiment of the present invention;

[0056] Figure 7 and Figure 8 A schematic diagram of the subjective effect of a method disclosed in an embodiment of the present invention on a Harvard data set compared with a comparison method. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0059] like Figure 1 As shown, a hyperspectral image reconstruction method based on RGB image provided by the present invention includes:

[0060] The three-channel RGB image is expanded through convolution processing;

[0061] Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales.

[0062] Use 2D convolution module and multifunctional 3D convolution module V3DCM to extract spatial and spectral features of different scales respectively, and fuse the obtained features;

[0063] Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled and captured in the spatial, channel and spectral dimensions, and then weighted and fused;

[0064] The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are used to generate hyperspectral reconstruction results through skip connections and convolution operations.

[0065] In this embodiment, by introducing the spatial-spectral multi-attention mechanism SSMAM, CNN captures local channel and spatial attention respectively, and uses the attention mechanism in Transformer to capture long-distance detail features in the spectral dimension, which comprehensively enhances the feature extraction capability and fully utilizes the global information in the image in the spatial and spectral dimensions, thereby improving the accuracy of hyperspectral reconstruction; by using a multifunctional 3D convolution module V3DCM that can reduce the large amount of 3D convolution calculations and memory usage while retaining feature diversity, in addition to feature calculations in all directions, an ECA layer is used for feature screening, and jump connections are made with shallow features, so that the accuracy of hyperspectral reconstruction and efficient 3D convolution are balanced; in addition, by using a gated convolution GMFF for multi-feature fusion in the spectral reconstruction stage, gated convolution is used instead of ordinary convolution for feature filtering for the selection of multi-feature fusion, so that more effective information can be captured for spectral reconstruction during the mapping process, thereby improving the accuracy of reconstruction.

[0066] Specifically, in order to make the hyperspectral image reconstructed by RGB fully utilize the effective information of the spatial spectrum to reduce the overall error of the spectral reconstruction, the present invention, based on the characteristics of sparse spectral self-similarity in HSI space, based on the attention mechanism, 3D convolution and gating mechanism, invented a RGB hyperspectral reconstruction method based on the spatial spectrum multi-attention mechanism and multifunctional 3D convolution gating feature selection from the perspectives of long-range and local features. In the reconstruction of hyperspectral images from RGB images, the spectral mapping stage is prone to the loss of shallow features due to the excessive depth of the network, so the method proposed in the present invention uses the residual structure as the backbone.

[0067] In the above implementation, preferably, the three-channel RGB image is expanded through convolution processing, and the specific process includes:

[0068] A 2D convolution module with a convolution kernel size of 3 is used to expand the three-channel RGB image so that the number of channels becomes 3;

[0069] The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

[0070] In the above implementation, preferably, spectrum mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM, each RSRM downsamples the feature map or intermediate feature map after the expansion channel to obtain feature information of different scales, and the specific process includes:

[0071] The three residual spectrum reconstruction modules RSRM are cascaded, and each RSRM is constructed by multi-scale feature extraction, multiple attention mechanism calculation, feature fusion gate selection and multiple jump connections.

[0072] Specifically, the features or intermediate features of the RGB image after the expansion channel are downsampled by ×1, ×2, and ×4 times, respectively, and the original features are converted into features of three different scales: [31, 128, 128], [62, 64, 64], and [124, 32, 32], respectively.

[0073] These three different scale features can effectively fuse shallow feature information and deep feature information in spectral reconstruction. Through the fusion of multi-scale features, different levels of spatial and spectral information can be captured in the subsequent spectral reconstruction stage, thereby better restoring the high-dimensional spectral structure.

[0074] like Figure 2 As shown, in the above implementation, preferably, a 2D convolution module and a multifunctional 3D convolution module V3DCM are used to extract features of different scales in spatial and spectral dimensions respectively, and the obtained features are fused. The specific process includes:

[0075] Two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used for features of different scales, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution.

[0076] In the above implementation, the multifunctional 3D convolution module is one of the key points of the present invention. Preferably, the specific process of the multifunctional 3D convolution module includes:

[0077] Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension;

[0078] A 1×1×1 filter is used to extract shallow features from the input features, and then convolution kernels of 3×1×1, 1×3×1, and 1×1×3 are used to capture feature information from the depth, height, and width directions respectively. In order to avoid the loss of spatial spectrum global feature details, a 3×3×3 filter is used to perform a comprehensive feature calculation;

[0079] After the features of each dimension are extracted, a jump connection is made between the extracted features of each dimension and the input of the corresponding dimension;

[0080] Then the features of four different dimensions are fused, and a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension;

[0081] In order to retain the effective spatial spectral information extracted by 3D convolution, the ECA layer is spliced ​​again to filter and extract the effective spatial spectral information. In order not to lose the shallow features, a long jump connection is finally made to the shallow features of the input multifunctional 3D convolution.

[0082] like Figure 3 As shown, in the above implementation, preferably, through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are respectively weightedly fused in the spatial, channel and spectral dimensions, and the specific process includes:

[0083] Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are concatenated along the new channel dimension. Then, the pooled tensor is convolved using a convolutional layer, and the attention weight is generated using the Sigmoid function, which is then multiplied element-by-element with the input tensor.

[0084] Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point, and then the convolution layer and ReLU activation function are used to optimize the features. The Sigmod function is then used to generate channel weights, which are multiplied element-by-element with the original input tensor to achieve channel enhancement.

[0085] Through Transformer-based spectral self-attention, unlike ordinary self-attention, ordinary self-attention uses a single pixel as a token to calculate attention weights, while in order to better utilize the spectral information of the hyperspectral, especially the long-distance information, the present invention calculates the attention weights along the spectral dimension;

[0086] The results of the above-mentioned multiple attention mechanisms are feature fused, and a convolution operation with a convolution kernel size of 1 and a jump connection are performed to avoid the loss of original feature information.

[0087] Among them, the use of multi-attention mechanism not only utilizes the excellent feature extraction ability of CNN in local details, but also utilizes the excellent representation ability of Transformer in long-distance. In the process of spectral reconstruction, it can accurately restore more detailed and comprehensive spectral information, thereby improving the reconstruction accuracy and robustness.

[0088] In the above implementation, the features of three different scales are processed through the above steps, and the obtained feature information includes the results of 2D convolution, 3D convolution and various attention mechanisms. Preferably, for the features of three different scales, it is also necessary to use the gated selection GMFF based on multi-feature fusion of gated convolution to fuse and filter the features of different scales, and generate the hyperspectral reconstruction results through skip connection and convolution operation for the fused features. The specific process includes:

[0089] The features of three different scales are flattened first and then concatenated;

[0090] In order to fully filter out effective information, we first use three gated convolutions GMFF to filter the concatenated features, and then perform ordinary Concat connection on the filtered features;

[0091] The features are then fused through a convolution operation with a convolution kernel size of 3;

[0092] The fused features are skipped and connected with the input features of the residual spectrum reconstruction module RSRM, and then a convolution operation with a convolution kernel of 3 is performed;

[0093] Then, a jump connection is performed on the tensor after the expansion channel to obtain the hyperspectral reconstruction result.

[0094] like Figure 4 As shown, in the above embodiment, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0095] The loss is calculated for the hyperspectral reconstruction results and the real spectrum, and the model parameters are optimized through back propagation until the model converges and the training is ended.

[0096] In the above implementation, preferably, the hyperspectral image reconstruction method based on RGB image further includes:

[0097] The NTIRE2022 dataset and Harvard dataset are used as validation sets to verify the optimal model obtained through training, and the model is evaluated using preset evaluation indicators.

[0098] In this embodiment, since the present invention uses two data sets for effect verification, the training parameters of the verification method for the two data sets are consistent, but the training parameters and image preprocessing stages of each data set are partially different. The parameter settings corresponding to the two data sets are first introduced separately.

[0099] For the NTIRE2022 dataset, RGB images and hyperspectral images are split into pairs with stride 8 and patch_size 128. In addition, each patch is randomly rotated, horizontally flipped, and vertically flipped. During training, all methods use batch_size 20, epoch 300, and the learning rate is initialized to 0.0004.

[0100] The Harvard dataset uses a stride of 64 and a patch_size of 128 to split paired RGB images and hyperspectral images. No data augmentation is performed. The batch_size during training is 8, the epoch is 40, and the learning rate is initialized to 0.0001.

[0101] The Adam optimizer is used for both datasets, with β1=0.9 and β2=0.99.

[0102] The present invention further proposes a hyperspectral image reconstruction system based on RGB images, which is applied to the hyperspectral image reconstruction method based on RGB images disclosed in any one of the above embodiments, comprising:

[0103] RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing;

[0104] A multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales.

[0105] The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features;

[0106] The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through the spatial and channel attention based on CNN and the spectral self-attention mechanism based on Transformer;

[0107] The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results for the fused features through skip connections and convolution operations.

[0108] According to the hyperspectral image reconstruction system based on RGB images disclosed in the above-mentioned embodiments, the functions to be implemented by each module thereof correspond to the respective steps in the hyperspectral image reconstruction method based on RGB images disclosed in the above-mentioned embodiments. During the implementation process, the operations are performed with reference to the above-mentioned embodiments, which will not be described in detail herein.

[0109] According to the hyperspectral image reconstruction method and system based on RGB images disclosed in the above embodiments, the implementation process and effects of the present invention are described through the following examples.

[0110] Embodiment 1:

[0111] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset to experiment with the effectiveness of the spatial-spectral multi-attention mechanism proposed in this invention.

[0112] First, the 900 pairs of training sets in the NTIRE2022 dataset are preprocessed, and the paired RGB and hyperspectral images are divided with a step size of 8, and the size of each patch_size is 128. After segmentation, the size of each patch is 128x 128, and the RGB image is linearly scaled to [0,1]. Then, each pair of data (containing a patch of RGB and the corresponding hyperspectral image) is rotated, flipped vertically, or flipped horizontally to varying degrees.

[0113] Then, we train for 300 epochs with a batch size of 20, that is, we send 20 pairs of patches to the network each time. We send the RGB image to the network for training, and calculate the loss of the obtained pseudo-hyperspectral image and the real hyperspectral image. Here, we use MREA for loss calculation and back-propagation update, and Adam is used as the optimizer.

[0114] The same dataset and training parameters were used to conduct experiments on the baseline (only spatial and channel attention mechanisms based on CNN) and the network using the spatial-spectral multi-attention mechanism, and the three evaluation indicators MRAE, RMSE and PSNR were used for verification. The various evaluation indicators calculated between the reconstructed hyperspectral image and the real hyperspectral image obtained by using 50 pairs of RGB of the validation set are as follows:

[0115]

[0116] In this quantitative experiment, it can be seen that the network with the spatial-spectral multi-attention mechanism proposed in the present invention is superior to the network without it in all evaluation indicators, which shows that the spatial-spectral multi-attention mechanism proposed in the present invention has a good effect.

[0117] Embodiment 2:

[0118] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset, and experiments were conducted simultaneously on the effectiveness of the spatial-spectral multi-attention mechanism and the multi-functional 3D convolution module proposed in this invention.

[0119] First, the 900 pairs of training sets in the NTIRE2022 dataset are preprocessed, and the paired RGB and hyperspectral images are divided with a step size of 8, and the size of each patch_size is 128. After segmentation, the size of each patch is 128x 128, and the RGB image is linearly scaled to [0,1]. Then, each pair of data (containing a patch of RGB and the corresponding hyperspectral image) is rotated, flipped vertically, or flipped horizontally to varying degrees.

[0120] Then, we train for 300 epochs with a batch size of 20, that is, we send 20 pairs of patches to the network each time. We send the RGB image to the network for training, and calculate the loss of the obtained pseudo-hyperspectral image and the real hyperspectral image. Here, we use MREA for loss calculation and back-propagation update, and Adam is used as the optimizer.

[0121] On the basis of Example 1, a multifunctional 3D convolution module was added for experimentation, and the three evaluation indicators of MRAE, RMSE and PSNR were used for verification. The various evaluation indicators calculated between the reconstructed hyperspectral image and the real hyperspectral image obtained by using 50 pairs of RGB of the verification set are as follows:

[0122]

[0123] In this quantitative experiment, it can be seen that the addition of the multifunctional 3D convolution module proposed in the present invention is optimal in all evaluation indicators, and the effect of adding both the spatial-spectral attention mechanism and the multifunctional 3D convolution module is better than that of the network with only the spatial-spectral multi-attention mechanism. This proves the effectiveness of the multifunctional 3D convolution module and once again proves the effectiveness of the spatial-spectral multi-attention mechanism.

[0124] Embodiment three:

[0125] In this experiment, the NTIRE2022 dataset was selected as the experimental dataset, and experiments were conducted simultaneously on the effectiveness of the spatial-spectral multi-attention mechanism, the multi-functional 3D convolution module, and the gating selection of multi-feature fusion proposed in the present invention.

[0126] First, the 900 pairs of training sets in the NTIRE2022 dataset are preprocessed, and the paired RGB and hyperspectral images are divided with a step size of 8, and the size of each patch_size is 128. After segmentation, the size of each patch is 128x 128, and the RGB image is linearly scaled to [0,1]. Then, each pair of data (containing a patch of RGB and the corresponding hyperspectral image) is rotated, flipped vertically, or flipped horizontally to varying degrees.

[0127] Then, we train for 300 epochs with a batch size of 20, that is, we send 20 pairs of patches to the network each time. We send the RGB image to the network for training, and calculate the loss of the obtained pseudo-hyperspectral image and the real hyperspectral image. Here, we use MREA for loss calculation and back-propagation update, and Adam is used as the optimizer.

[0128] Based on Example 2, we added the gated selection of multi-feature fusion to conduct experiments, and used the three evaluation indicators of MRAE, RMSE and PSNR for verification. The various evaluation indicators calculated between the reconstructed hyperspectral image and the real hyperspectral image obtained by using 50 pairs of RGB of the verification set are as follows:

[0129]

[0130] In this quantitative experiment, it can be seen that the gated selection of multi-feature fusion proposed by the present invention is optimal in all evaluation indicators, and the effect of adding the three key modules at the same time is better than that of adding only the spatial-spectral multi-attention mechanism and the multi-functional 3D convolution module. In this experiment, it can be seen that the three modules complement each other, and the effect of adding the three modules at the same time is the best.

[0131] Embodiment 4:

[0132] The method proposed in the present invention is implemented by using the Pytorch framework, using the Adam optimizer, β1=0.9, β2=0.99. The learning rate is initialized to 0.0004. The training process cuts the data set into 128×128 RGB and hyperspectral sample pairs and feeds them into the network for training. The RGB images are linearly scaled to [0,1]. The training process uses a single NVIDIA GeForce RTX4090 for training. The present invention verifies its final effect on two data sets respectively. Table 1 shows the results of comparing with the existing SOTA methods on the NTIRE2022 data set, using MRAE, RMSE, PSNR, SAM, and ERGAS as evaluation indicators, which verifies the effectiveness of the method proposed in the present invention. Table 2 shows the results on the Harvard data set, using MRAE, RMSE, PSNR, and SSIM as evaluation indicators, which also proves the superiority of the present invention.

[0133] The NTIRE2022 dataset is the official dataset used in the NTIRE2022 spectral reconstruction challenge. It contains 1,000 RGB hyperspectral pairs, including 900 pairs of training sets, 50 pairs of validation sets, and 50 pairs of test sets. Since the actual hyperspectral images of the 50 test sets are confidential, only 900 pairs are used for training and 50 pairs for validation. Each hyperspectral image is a 482x 512 image with 31 bands from 400nm to 700nm. The corresponding RGB image is obtained by using a conversion matrix on the HIS and adding shot noise to simulate the real camera situation. The Harvard dataset consists of 50 hyperspectral images with a spectral coverage range of 420nm-720nm, including 31 bands, and the spatial resolution of each image is 1392x 1042. In order to obtain the corresponding RGB image, the camera spectral function used in the dataset proposed by the NTIRE2020 spectral challenge is used to obtain the corresponding RGB image. The training set and validation set were divided into 46:4 for training and validation. The images in the validation set were Imgh0, Imgh1, Imgh2, and Imgh3, and the other images were used for training.

[0134] For the evaluation indicators used, MREA is used to calculate the pixel-level difference between the reconstructed hyperspectral image and the real hyperspectral image in all bands. The smaller the MREA value, the better the reconstruction effect. RMSE is obtained by calculating the mean of the squares of the pixel-level differences between the reconstructed image and the real image, and then taking the square root. The smaller the RMSE value, the closer the reconstructed image is to the real image. PSNR is an indicator used to measure the degree of image distortion. The larger the value, the smaller the image distortion and the more accurate the spectral recovery. SAM reflects the evaluation of the restoration quality from a spectral perspective. The smaller the value, the more similar the predicted spectrum is to the real spectrum. ERGAS is a comprehensive evaluation indicator obtained by calculating the weighted average of the relative errors between the reconstructed image and the real image at different wavelengths. The smaller the value, the higher the reconstruction quality. SSIM is used to evaluate the similarity of two images. When the SSIM value is close to 1, it indicates that the two images are very similar.

[0135] Specifically, Table 1 shows the results of quantitative comparison of the method proposed in the present invention with existing SOTA methods on the NTIRE2022 dataset, including a snapshot compression imaging system reconstruction method (HDNet), two natural image restoration models (HINet, MIRNet) and five spectral reconstruction methods (HSCNN+, HRNet, AWAN, DRCR, MST++). All methods were retrained and tested using the same settings. As can be seen from the table, the method proposed in the present invention achieved the best results in the four evaluation indicators of RMSE, PSNR, SAM and ERGAS, and MRAE ranked third among all methods. In general, the method proposed in the present invention has a good performance on this dataset, which is largely due to the fact that the present invention makes full use of the effective information of the spatial and spectral dimensions in the process of spectral reconstruction.

[0136] Table 1 Experimental results with other SOTA methods on the NTIRE2022 dataset

[0137]

[0138] Table 2 shows the quantitative comparison results of the method proposed in the present invention with the existing SOTA methods on the Harvard dataset, including a natural image restoration model (MIRNet) and four image spectral reconstruction methods (HSCNN+, AWAN, MST++, FMPSA). In order to ensure fairness, all comparison methods are retrained and tested using the same design. As can be seen from the table, the method proposed in the present invention is quite competitive in terms of the four evaluation indicators of MRAE, RMSE, PSNR and SSIM. Among them, PSNR is the best among all methods, and the other three evaluation indicators are the second best values. This may be because the sample size of the Harvad dataset is small, which affects the effectiveness of this method to a certain extent, but in general, the performance is still the best compared with other methods. It fully demonstrates that this method has excellent capabilities in spectral reconstruction tasks.

[0139] Table 2 Experimental results with other SOTA methods on Harvard dataset

[0140]

[0141] In order to more intuitively demonstrate the excellent spectral reconstruction capability of the present invention, Figure 5-8 Qualitative experimental results compared with other methods on two datasets are shown respectively. Figure 5 and Figure 6 The spectral reconstruction effect diagram of the present invention and the above-mentioned comparison method on the NITRE2022 dataset is shown, where the first row is a heat map of the error calculated using RMSE between the reconstruction results of each method and the true hyperspectral image. The darker the color, the smaller the error. The error image is a heat map of the RMSE between the real data and the reconstructed HSI. The second row is an enlarged view of the green box part of the RGB image corresponding to the Ground Truth and the reconstruction results of each method. The first picture in the second row is an enlarged view of the true hyperspectral image in this area. Figure 5 It can be seen from the figure that the spectral reconstruction effect of the present invention not only has the smallest error, but also from the detail image magnified by the green box, it can be seen that compared with other methods, the spectral restoration details of this method are also better.

[0142] exist Figure 6 It can be clearly seen that the spectral reconstruction effect of the present invention in the test set ARAD_1K_0950 with a band number of 530nm is significantly better than that of other methods. From the error graph, the present method can make a good transition at the boundary between light and dark. Other methods have different degrees of spectral distortion during the reconstruction process. Although the present invention also has a certain degree of distortion, the overall effect is the best.

[0143] Specifically, Figure 6The visual results and reconstructed HSI error images (first row) of ARAD_1K_0950 in the NTIRE2022 test set predicted by HSCNN+, HRNet, HDNet, MIRNet, MST++ and the method of the present invention at a band number of 530nm. The error image is a heat map of the RMSE between the real data and the reconstructed HSI. The second row is an enlarged view of the green box part of the RGB image corresponding to the Ground Truth and the reconstruction results of each method.

[0144] Figure 7 and Figure 8 The subjective effect diagram of the present invention and the above comparison method on the Harvard dataset is shown. Figure 7 The first row shows the reconstruction effect error map of each method for Imgh2 in the Harvard test set with a band number of 570nm. The error map is a PNSR heat map between the real data and the reconstructed data. The darker the color, the smaller the error. It can be seen that the error map of the present invention has the most uniform reconstruction effect and the smallest error overall. Figure 7 The first picture in the first row is the RGB picture corresponding to Imgh2, and the second line is the enlarged picture of the real hyperspectral picture and the reconstructed effect pictures of each method in the green box marked in the RGB picture, among which the first picture is the enlarged picture of the real hyperspectral picture in the green box part. It can be seen from the detailed picture that the effect picture of the present invention is closest to the real hyperspectral picture, especially in RGB, some "reflective" parts are easy to cause detail loss because the effective information of the empty spectrum dimension cannot be fully utilized in the reconstruction. The present invention fully considers this point, and the reconstruction effect is the best compared with other methods.

[0145] Figure 8 The figure shows the reconstruction error diagram of each method at 490nm band number of Imgh3 in the Harvard test set and the detailed magnified diagram of the reconstruction effect. It can be seen that the overall error of the present invention is the smallest, the detailed texture in the magnified diagram is the clearest, and it is closest to the real hyperspectral detailed magnified diagram. Compared with the detailed effect diagram of other reconstruction methods, the reconstruction effect of the present invention is clearer and the smoothness between pixels is higher, which intuitively demonstrates the effectiveness and robustness of the present invention.

[0146] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A hyperspectral image reconstruction method based on RGB image, characterized in that: include: The three-channel RGB image is expanded through convolution processing; Spectral mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales. Use 2D convolution module and multifunctional 3D convolution module V3DCM to extract spatial and spectral features of different scales respectively, and fuse the obtained features; Through the CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism, the fusion features of 2D convolution and V3DCM are modeled and captured in the spatial, channel and spectral dimensions, and then weighted and fused; The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are used to generate hyperspectral reconstruction results through skip connections and convolution operations.

2. The hyperspectral image reconstruction method based on RGB image according to claim 1, characterized in that: The three-channel RGB image is expanded through convolution processing, and the specific process includes: A 2D convolution module with a convolution kernel size of 3 is used to expand the channels of the three-channel RGB image so that the number of channels becomes 3; The size of the feature map is adjusted from [3, 128, 128] to [31, 128, 128].

3. The hyperspectral image reconstruction method based on RGB image according to claim 2, characterized in that: The spectrum mapping and reconstruction are performed through three cascaded residual spectrum reconstruction modules RSRM, each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales. The specific process includes: Three residual spectrum reconstruction modules RSRM are cascaded, and each RSRM performs a downsampling process of ×1, ×2 and ×4 on the features or intermediate features of the RGB image after the expansion channel, and obtains features of three different scales of [31, 128, 128], [62, 64, 64] and [124, 32, 32] from the original features.

4. The hyperspectral image reconstruction method based on RGB image according to claim 3 is characterized in that: The 2D convolution module and the multifunctional 3D convolution module V3DCM are used to extract the features of different scales in spatial and spectral dimensions respectively, and the obtained features are fused. The specific process includes: Two 2D convolutions and one PReLU activation and two multifunctional 3D convolutions and one PReLU activation are used for features of different scales, and the features obtained by 2D convolution are fused with the features obtained by multifunctional 3D convolution.

5. The hyperspectral image reconstruction method based on RGB image according to claim 4, characterized in that: The specific process of the multifunctional 3D convolution module V3DCM includes: Use the unsqueeze operation to expand the input feature by one dimension to align the feature dimension with the 3D convolution feature dimension; Use a 1×1×1 filter to perform shallow feature extraction on the input features, use convolution kernels of 3×1×1, 1×3×1, and 1×1×3 sizes to capture feature information in three directions: depth, height, and width, respectively, and use a 3×3×3 filter to perform a comprehensive feature calculation; Make a jump connection between the extracted features of each dimension and the input of the corresponding dimension; The features of four different dimensions are fused, and then a 1×3×3 convolution is used to perform a fine feature extraction in the spatial dimension; The ECA layer is spliced ​​once to screen and extract the effective spatial spectrum information, and then a long jump connection is performed on the shallow features input into the multifunctional 3D convolution.

6. The hyperspectral image reconstruction method based on RGB image according to claim 5, characterized in that: The CNN-based spatial and channel attention and the Transformer-based spectral self-attention mechanism are used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions and perform weighted fusion. The specific process includes: Through CNN-based spatial attention, the maximum and average values ​​of the tensor in the channel dimension are calculated, and the results are concatenated along the new channel dimension. Then, the pooled tensor is convolved using a convolutional layer, and the attention weight is generated using the Sigmoid function, which is then multiplied element-by-element with the input tensor. Through CNN-based channel attention, the input tensor is first globally averaged pooled to map the channel features to a single point, and then the convolution layer and ReLU activation function are used to optimize the features. The Sigmod function is then used to generate channel weights, which are multiplied element-by-element with the original input tensor to achieve channel enhancement. Attention weights are calculated along the spectral dimension through Transformer-based spectral self-attention; The results of the above attention mechanism are feature fused, and a convolution operation with a convolution kernel size of 1 and a skip connection are performed.

7. The hyperspectral image reconstruction method based on RGB image according to claim 6, characterized in that: The gated selection GMFF based on gated convolution for multi-feature fusion is used to fuse and filter features of different scales, and the fused features are subjected to skip connection and convolution operations to generate hyperspectral reconstruction results. The specific process includes: The features of three different scales are flattened first and then concatenated; Use three gated convolutions to filter the concatenated features, and perform ordinary Concat connections on the filtered features; The features are fused through a convolution operation with a convolution kernel size of 3; The fused features are jump-connected with the input features of the residual spectrum reconstruction module RSRM, and then a convolution operation with a convolution kernel of 3 is performed; A jump connection is performed on the tensor of the last output RSRM to obtain the hyperspectral reconstruction result.

8. The hyperspectral image reconstruction method based on RGB image according to any one of claims 1 to 7, characterized in that: Also includes: The loss calculation is performed on the hyperspectral reconstruction result and the real spectrum, and the model parameters are optimized by back propagation until the model converges and the training is ended.

9. The hyperspectral image reconstruction method based on RGB image according to claim 8, characterized in that: Also includes: The NTIRE2022 dataset and Harvard dataset were used as validation sets to verify the optimal model obtained through training, and the model was evaluated using preset evaluation indicators.

10. A hyperspectral image reconstruction system based on RGB image, characterized in that: A hyperspectral image reconstruction method based on RGB images as described in any one of claims 1 to 9, comprising: RGB image preprocessing module, used to perform channel expansion on three-channel RGB images through convolution processing; A multi-scale feature extraction module is used to perform spectral mapping and reconstruction through three cascaded residual spectrum reconstruction modules RSRM. Each RSRM downsamples the feature map after the expansion channel or the intermediate feature map to obtain feature information of different scales. The multifunctional 3D convolution module is used to extract features of different scales in spatial and spectral dimensions using the 2D convolution module and the multifunctional 3D convolution module V3DCM, and fuse the obtained features; The spatial-spectral multi-attention calculation module is used to model and capture the local and long-range dependencies of the fused features of 2D convolution and V3DCM in the spatial, channel and spectral dimensions, and perform weighted fusion through the spatial and channel attention based on CNN and the spectral self-attention mechanism based on Transformer; The hyperspectral image generation module is used to fuse and filter features of different scales using the gated selection GMFF based on multi-feature fusion of gated convolution, and generate hyperspectral reconstruction results for the fused features through skip connections and convolution operations.

Images