Hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero sample learning
Through the semi-supervised zero-sample learning method, combined with deep learning and convolutional operations, the trade-off problem between hyperspectral images is solved, and efficient image super-resolution reconstruction is achieved, which improves PSNR performance and is suitable for high-level high-spectrum images. Super-resolution reconstruction of spectral images.
Patent Information
- Application Number
- CN202411062287.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-22
AI Technical Summary
Hyperspectral images are difficult to trade off between spectral resolution and spatial resolution, resulting in poor spatial resolution. Existing supervised learning methods require a large number of true-value images to be difficult to obtain, while unsupervised learning lacks constraints lead to distortion.
Using a hyperspectral-multi-spectral fusion super-resolution reconstruction method based on semi-supervised zero-sample learning, the super-resolution phase of degradation information learning, spectral upsampling network pre-training and unsupervised image super-resolution phase, combined with deep learning models and convolutional operations, the Spectral response function and point diffusion function modeling to realize super-resolution reconstruction of images.
Effectively solves the problems of inaccurate image estimation and difficult to obtain training sets, and realizes the "training as test" function. PSNR has significantly improved on multiple data sets, indicating that the method has good performance in various bands, which solves the hyperspectral imaging system. limitation.
Smart Images

Figure CN119006283A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hyperspectral image super-resolution, and in particular to a hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-sample learning. Background Art
[0002] Hyperspectral images can obtain rich spectral information from visible light to infrared bands, and have broad application prospects in remote sensing, medical treatment, food safety, military and other fields. However, due to the limitations of hyperspectral imaging systems, hyperspectral images must balance spectral resolution and spatial resolution, resulting in poor spatial resolution, which makes it difficult to meet the needs of subsequent refined production. Although the spatial resolution can be enhanced by improving the imaging system, this places higher demands on hardware design, data storage and transmission. Compared with this approach, it is undoubtedly more economical to obtain products with high spectral and spatial resolution by fusing hyperspectral images with high spectral resolution and multispectral images with high spatial resolution. This method of fusing the two data to alleviate the mutual constraints of spectral / spatial resolution is called fusion-based hyperspectral image super-resolution reconstruction.
[0003] Deep learning has further broken through the bottleneck of traditional methods due to its powerful modeling capabilities. At present, the mainstream method relies on a large number of simulated hyperspectral-multispectral-true value image pairs for supervised training to obtain satisfactory results. However, true value images are difficult to obtain in reality; another unsupervised learning method does not require a large amount of data for training, but there is distortion due to the lack of reference images and accurate degradation information for sufficient constraints. Semi-supervised zero-shot learning can combine the advantages of supervised and unsupervised methods. Therefore, the research on hyperspectral-multispectral fusion super-resolution reconstruction methods based on semi-supervised zero-shot learning has important research significance. Summary of the invention
[0004] The purpose of the present invention is to provide a hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-sample learning to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solution: a hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-sample learning, comprising the following steps:
[0006] Step 1: In the degradation information learning stage, deep degradation information is modeled based on the deep learning model. The hyperspectral image Y is converted into a low spatial and low spectral resolution image through the spectral response function, and the multispectral image Z is converted into a low spatial and low spectral resolution image through the point spread function. The SRF and PSF are modeled using the convolution operation with updateable parameters, and the formula is:
[0007] K1=Fy (Y;θ (SRF) ) K2=F z (Z;θ (PSF) );
[0008] The loss function is constructed as:
[0009] L1=||K1-K2||1;
[0010] Among them, SRF is the spectral response function, PSF is the point spread function, k1 is the low spatial and low spectral resolution image, and k2 is the low spatial and low spectral resolution image. The degradation model is obtained to guide the model reconstruction and obtain the spectral upsampling network;
[0011] Step 2: Pre-training of the spectral upsampling network: supervised pre-training modeling of degradation information and noisy hyperspectral images is performed based on the spectral upsampling network to ensure that the spectral upsampling network can effectively reconstruct the spectrum and obtain a spectral upsampling pre-training model;
[0012] Step 3: Based on step 2, the original hyperspectral image is passed through a spatial upsampling network that focuses on spectral information to obtain rich spectral information, which is used as the initialization input to the pre-trained spectral upsampling network for learnable iterative updates to obtain the final super-resolution image model.
[0013] Step 4: In the unsupervised image super-resolution stage, the degradation model, the spectral upsampling pre-trained model, the blurred super-resolution image and the original multispectral image are combined and iteratively updated in an unsupervised learning manner to obtain the final fused super-resolution image.
[0014] Step five, based on step four, in the image super-resolution stage, the degradation information of the blurred super-resolution image and the final fused super-resolution image is used to constrain the final super-resolution image model.
[0015] Furthermore, in step 2, the high-resolution and low-resolution images first pass through a convolution layer respectively, and then enter the cross-residual spectral reconstruction module to realize the interaction of high-resolution and low-resolution image information. Then, they pass through the subsequent dense residual network respectively, and finally obtain the final super-resolution image through channel fusion and convolution. The number of module stacking is set to 3 through the comparison of the peak signal-to-noise ratio, spectral angle mapping, relative global dimensionality error and parameter quantity.
[0016] Furthermore, in step 4, the original hyperspectral image first passes through a multi-stream super-resolution framework;
[0017] The first pipeline is a generator network with an attention mechanism and learnable transposed convolutions;
[0018] The second process is a multi-scale 3D convolution module used to learn rich spectral priors for hyperspectral images, and finally the multi-stream data elements are added together to obtain the blurred super-resolution image.
[0019] Furthermore, in the first process and the second process, learnable transposed convolution and bilinear interpolation are used to upsample to the required resolution respectively, and finally the output super-resolution image is added.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] (1) This scheme combined with the semi-supervised zero-shot strategy can effectively solve the problems of inaccurate reconstructed image estimation and difficulty in obtaining a large number of training sets in mainstream hyperspectral super-resolution reconstruction technology.
[0022] (2) The proposed method can achieve "training is testing" by relying only on the observed hyper-multispectral images. Experimental results show that compared with the best results among the nine methods, the proposed method has a PSNR improvement of 2.3891dB on the Houston dataset, 2.0789dB on the Chikusei dataset, 2.0106dB on the TianGong-1 dataset, and 3.7302dB on the CAVE dataset.
[0023] (3) From the PSNR curve, it can be seen that the method of this scheme has achieved good overall performance in each band, which is consistent with the quantitative results, indicating that it is feasible for super-resolution reconstruction of hyperspectral images based on fusion.
[0024] (4) This scheme can optimize the network using only the observed hyperspectral-multispectral image pairs, achieving “training is testing” without the need for additional training data sets. This solves the limitations of the hyperspectral imaging system to a certain extent and breaks through the technical difficulty of balancing the spectral resolution and spatial resolution of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a framework diagram of the hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-shot learning of the present invention;
[0026] Figure 2 It is a structural diagram of the spectral upsampling network (SpeUNet) of the present invention;
[0027] Figure 3 This is a structural diagram of the spatial upsampling network (SpaUNet) of the present invention;
[0028] Figure 4 A visual comparison diagram of all methods in the Houston dataset of the present invention;
[0029] Figure 5 A visual comparison diagram of all methods in the TianGong-1 dataset of the present invention;
[0030] Figure 6 A schematic diagram of the visual comparison of all methods in the CAVE dataset of the present invention;
[0031] Figure 7 A visual comparison diagram of all methods in the Chikusei dataset of the present invention;
[0032] Figure 8 is a schematic diagram of PSNR values along spectral bands of the first data set of the present invention;
[0033] Fig. 9 is a schematic diagram of PSNR values along spectral bands of the second data set of the present invention;
[0034] Fig.10 is a schematic diagram of PSNR values along spectral bands of the third data set of the present invention;
[0035] Fig.11 FIG. 4 is a schematic diagram of PSNR values along spectral bands of the fourth data set of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] Example:
[0038] See also Figure 1-8 ,The present invention provides a technical solution: a hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-sample learning;
[0039] 1. Method flow
[0040] Figure 1 The framework diagram of the hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-shot learning in this embodiment includes:
[0041] A. Degraded Information Learning Stage (DIL Stage)
[0042] In this stage, deep learning models are used to model deep degradation information. Specifically, the hyperspectral image Y is converted into a low spatial and low spectral resolution image (k1) through the spectral response function (SRF), and the multispectral image Z is converted into a low spatial and low spectral resolution image (k2) through the point spread function (PSF), where SRF and PSF are modeled using convolution operations with updateable parameters. The formula is:
[0043] K1=F y (Y;θ (SRF) ) K2=F z (Z;θ (PSF) );
[0044] The loss function is constructed as:
[0045] L1=||K1-K2||1.
[0046] B. Pre-training of spectral upsampling network (SRPT Stage)
[0047] In this stage, the designed spectral upsampling network is used to perform supervised pre-training on the degraded information and noisy hyperspectral images. The framework is as follows: Figure 2 As shown in the figure, considering the importance of spatial information for image restoration, the high-resolution and low-resolution images first pass through a convolution layer respectively, and then enter the cross residual spectrum reconstruction module to realize the interaction of high-resolution and low-resolution image information, and then each passes through the subsequent dense residual network, and finally obtains the final super-resolution image through channel fusion and a series of convolutions. The number of module stacking is set to 3 through the comparison of the peak signal-to-noise ratio, spectral angle mapping, relative global dimensionality error and parameter quantity. The formula is:
[0048]
[0049] The loss function is constructed as:
[0050]
[0051] C. Unsupervised Image Super-Resolution Stage (ISR Stage)
[0052] In this stage, the degradation model, the spectral upsampling pre-trained model, the blurred super-resolution image and the original multispectral image are iteratively updated in an unsupervised learning manner to obtain the final fused super-resolution result. Specifically, the original hyperspectral image is firstly Figure 3In the multi-stream super-resolution framework shown in the figure, the first process is a generator network containing an attention mechanism and a learnable transposed convolution, which can obtain the optimal upsampling method through network learning; the second process is a multi-scale 3D convolution module used to learn the rich spectral priors of hyperspectral images, and finally the multi-stream data elements are added to obtain a blurred super-resolution image. Then the original hyperspectral and blurred super-resolution images are input into the pre-trained spectral upsampling network for learnable iterative updates to obtain the final super-resolution image (super-resolution image model). The degradation model is used to obtain accurate degradation information estimates for the blurred super-resolution image and the final fused super-resolution image to constrain the entire model (super-resolution image model). The formula and loss function are expressed as:
[0053]
[0054] D. Experimental Details
[0055] In order to ensure stable optimization of the super-resolution image model, the Adam optimizer adopts a linear decay strategy. The learning rate of DILStage and SRPT Stage is initially fixed at 0.001, and linearly decays to zero after 3000 rounds of training. The total number of training rounds is 6000 and 8000, respectively. The learning rates of SpaUNet and SpeUNet of ISR Stage are initially fixed at 0.003 and 0.001, respectively. They linearly decay to zero after 5000 rounds of training. The total number of training rounds is 14000. All super-resolution image models are run on Intel(R)Xeon(R)Gold 6348A800-80GB GPU under the PyTorch framework.
[0056] 2. Results Analysis and Feasibility of the Method
[0057] The performance of the method proposed in this embodiment is compared with the fusion-based super-resolution model. Six indicators are selected to quantitatively evaluate the fusion performance in terms of spatial, spectral and global quality. They include: correlation coefficient (CC), root mean square error (RMSE), peak signal-to-noise ratio (PSNR), universal image quality index (UIQI), spectral angle (SAM) and ERGAS. Among them, for PSNR, UIQI and CC, the higher the value, the better the fusion performance, while the ideal value of RMSE, SAM and ERGAS is zero. ;
[0058] Five traditional methods, CNMF, GLPHS, GSA, HySure, and SFIMH, and four advanced deep learning methods, CUCaNet, HyCoNet, MIAE, and SURE, were selected for performance comparison. SAM heat map, MRAE heat map, and RGB RMSE heat map were used for visual evaluation to show their differences;
[0059] Four datasets, TianGong-1, Houston, CAVE, and Chikusei, are selected to evaluate the generalization of the method proposed in this embodiment. Due to the unavailability of high-resolution hyperspectral (HrHSI) in real scenes, the datasets used are set as shown in Table 1, where low-resolution hyperspectral (LrHSI) is generated by downsampling the reference image through Gaussian blur kernel and downsampling operator, and high-resolution multispectral (HrMSI) is generated using the multispectral SRF of WorldView 2. The quantitative performance of each dataset is shown in Table 2.
[0060] Table 1. Simulation specifications for the four datasets
[0061]
[0062] Table 2. Quantitative performance of each dataset, the best results are in bold
[0063]
[0064]
[0065] Figure 4 Visual comparison of all methods on the Houston dataset. First row: Pseudo-color images of SR output. Second row: RGB RMSE heatmap. Third row: MRAE heatmap. Fourth row: SAM error heatmap.
[0066] Figure 5 Visual comparison of all methods on the TianGong-1 dataset. First row: Pseudo-color images of SR output. Second row: RGB RMSE heatmap. Third row: MRAE heatmap. Fourth row: SAM error heatmap.
[0067] Figure 6 Visual comparison of all methods in CAVE dataset. First row: Pseudo-color images of SR output. Second row: RGB RMSE heatmap. Third row: MRAE heatmap. Fourth row: SAM error heatmap.
[0068] Figure 7 Visual comparison of all methods on the Chikusei dataset. First row: Pseudo-color images of SR output. Second row: RGB RMSE heatmap. Third row: MRAE heatmap. Fourth row: SAM error heatmap.
[0069] This embodiment, combined with the semi-supervised zero-sample strategy, can effectively solve the problems of inaccurate estimation of reconstructed images and difficulty in obtaining a large number of training sets in mainstream hyperspectral super-resolution reconstruction technology. The method proposed in this embodiment can achieve "training is testing" by relying only on observed high-multispectral images. The experimental results show that the method proposed in this embodiment has a PSNR improvement of 2.3891dB on the Houston dataset, 2.0789dB on the Chikusei dataset, 2.0106dB on the TianGong-1 dataset, and 3.7302dB on the CAVE dataset compared to the best result among the nine methods. In addition, it can be seen from the PSNR curve that the method of this embodiment has achieved good overall performance in each band, which is consistent with the quantitative results, indicating that it is feasible for super-resolution reconstruction of hyperspectral images based on fusion.
[0070] Figure 1 The Chinese and English interpretations are as follows:
[0071] DIL Stage: Degraded Information Learning Stage;
[0072] HrMSI: High-resolution hyperspectral;
[0073] LrHSI: low-resolution high-spectral;
[0074] SRPT Stage: Pre-training of spectral upsampling network;
[0075] SpeUNet: Spectral Upsampling Network;
[0076] ISR Stage: unsupervised image super-resolution stage;
[0077] SpaUNet: Spatial Upsampling Network;
[0078] PSF: point spread function;
[0079] SRF: spectral response function;
[0080] Data flow direction: data flow direction;
[0081] Loss: loss;
[0082] Frozen: frozen;
[0083] Trainable: trainable;
[0084] n noise: noisy hyperspectral image;
[0085] Figure 2 The Chinese and English interpretations are as follows:
[0086] Cross-Residual Spectral Reconstruction Module: Cross-Residual Spectral Reconstruction Module;
[0087] Figure 4 , Figure 5 , Figure 6 , Figure 7 The Chinese and English interpretations are as follows:
[0088] REF: reference;
[0089] CNMF: A coupled non-negative matrix factorization method;
[0090] GLPHS: A method based on the generalized Laplace pyramid;
[0091] GSA: A Schmidt orthogonalization-based adaptation method;
[0092] HySure: Super-resolution fusion method;
[0093] SFIMHS: Smoothing filter based intensity modulation method;
[0094] CUCaNet: A coupled unmixing network with cross-attention;
[0095] HyCoNet: A coupled convolutional neural network with adaptive response function learning.
[0096] MIAE: A model-based unsupervised hyperspectral image super-resolution autoencoder;
[0097] SURE: A fusion framework using unsupervised convolutional neural networks and Stein unbiased risk estimation;
[0098] Ours: The model proposed in the current study;
[0099] Output: RGB image of fusion result;
[0100] RGB RMSE: root mean square error of RGB image;
[0101] MRAE: A mean relative absolute error metric;
[0102] SAM: A metric for evaluating spectral similarity of hyperspectral images;
[0103] Figure 8 , Figure 8 , Fig.10 , Fig.11 The Chinese and English interpretations are as follows:
[0104] PSNR: Peak Signal-to-Noise Ratio
[0105] Houston: Dataset;
[0106] Chikusei: Dataset;
[0107] Band Number: the band in the image;
[0108] TianGong-1: dataset;
[0109] CAVE: dataset.
[0110] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A hyperspectral-multispectral fusion super-resolution reconstruction method based on semi-supervised zero-shot learning, characterized in that: The following steps are involved: Step 1: In the degradation information learning stage, deep degradation information is modeled based on the deep learning model. The hyperspectral image Y is converted into a low spatial and low spectral resolution image through the spectral response function, and the multispectral image Z is converted into a low spatial and low spectral resolution image through the point spread function. The SRF and PSF are modeled using the convolution operation with updateable parameters, and the formula is: K1=F y (Y;θ (SRF) ) K2=F z (Z;θ (PSF) ) The loss function is constructed as: L1=||K1-K2||1; Among them, SRF is the spectral response function, PSF is the point spread function, k1 is the low spatial and low spectral resolution image, and k2 is the low spatial and low spectral resolution image. The degradation model is obtained to guide the model reconstruction and obtain the spectral upsampling network; Step 2: pre-training of the spectral upsampling network, based on the spectral upsampling network, supervised pre-training modeling is performed on the degradation information and noisy hyperspectral image to obtain a spectral upsampling pre-training model; Step 3: Based on step 2, the original hyperspectral image is passed through a spatial upsampling network that focuses on spectral information to obtain rich spectral information, which is used as the initialization input to the pre-trained spectral upsampling network for learning and iterative update to obtain the final super-resolution image model; Step 4: In the unsupervised image super-resolution stage, the degradation model, the spectral upsampling pre-training model, the blurred super-resolution image and the original multispectral image are combined and iteratively updated in an unsupervised learning manner to obtain the final fused super-resolution image. Step five, based on step four, in the image super-resolution stage, the degradation information of the blurred super-resolution image and the final fused super-resolution image is used to constrain the final super-resolution image model.
2. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 1, characterized in that: In step 2, the high-resolution and low-resolution images first pass through a convolution layer respectively, and then enter the cross-residual spectral reconstruction module to realize the interaction of high-resolution and low-resolution image information. Then they pass through the subsequent dense residual network respectively, and finally obtain the final super-resolution image through channel fusion and convolution. The number of module stacking is set to 3 through the comparison of the peak signal-to-noise ratio, spectral angle mapping, relative global dimensionality error and parameter quantity.
3. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 1, characterized in that: In step 4, the original hyperspectral image first passes through the multi-stream super-resolution framework; The first pipeline is a generator network with an attention mechanism and learnable transposed convolutions; The second process is a multi-scale 3D convolution module used to learn rich spectral priors for hyperspectral images, and finally the multi-stream data elements are added together to obtain the blurred super-resolution image.
4. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 3, characterized in that: In the first process and the second process, learnable transposed convolution and bilinear interpolation are used to upsample to the required resolution respectively, and finally added together to obtain the output super-resolution image.
5. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 1, characterized in that: The final super-resolution image model is based on the Adam optimizer and adopts a linear decay strategy.
6. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 5, characterized in that: In the degradation information learning stage, the learning rate was initially fixed at 0.001 and linearly decayed to zero after 3000 training rounds, and the total number of training rounds was 6000.
7. The method for hyperspectral-multispectral fusion super-resolution reconstruction based on semi-supervised zero-shot learning according to claim 5, characterized in that: The pre-training learning rate of the spectral upsampling network was initially fixed to 0.001 and linearly decayed to zero after 3000 training rounds, with a total of 8000 training rounds.
Citation Information
Cited By
Semi-supervised laser microscopic imaging enhancement method
CN120088135A
Hyperspectral image super-resolution method and device based on physical diffusion model
CN120634865A
Spectrum super-resolution method and system based on FP cavity and deep learning
CN121430814A
Hyperspectral fusion imaging method and system of double-branch diffusion model framework
CN121707841A
Mars mineral mapping identification method and mapping identification device
CN122385521A
