Hsi super-resolution reconstruction method based on transfer learning and spectral recovery and related device

Abstract

The present application relates to the field of hyperspectral image, and discloses a HSI super-resolution reconstruction method based on transfer learning and spectral recovery and related equipment, which is used for realizing the prediction of HSI from low-resolution hyperspectral image through a machine learning model, and the method comprises the following steps: acquiring a hyperspectral image set training set and a color image training set; performing spectral down-sampling on the hyperspectral image set training set to obtain a multispectral image set; training a preset transfer learning model according to the color image training set to obtain a spatial super-resolution model, and optimizing the spatial super-resolution model to obtain a target spatial super-resolution model; training a preset deep learning model according to a low-resolution multispectral image set and a low-resolution hyperspectral image set to obtain a target spectral super-resolution model; acquiring a low-resolution hyperspectral image to be reconstructed, and performing transfer learning and spectral recovery processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain a corresponding high-resolution hyperspectral image.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral imaging technology based on deep learning, and in particular to a method, apparatus, device, and storage medium for HSI super-resolution reconstruction based on transfer learning and spectral restoration. Background Technology

[0002] Hyperspectral images (HSI) can simultaneously capture rich spectral and spatial information, providing a more comprehensive reflection of image characteristics. Furthermore, high-resolution spectral information can effectively distinguish objects in the same color gamut or with similar textures. Therefore, HSI is widely used in remote sensing and visual tasks, such as environmental protection, vegetation analysis, and target tracking. High-resolution images, whether in the spatial or spectral domain, can enhance the effectiveness of real-world applications. Therefore, acquiring high-resolution HSI images is crucial.

[0003] Image super-resolution reconstruction is a challenging computer vision problem that can effectively increase image resolution and improve the visual effect of HSI (High-Resolution Image Reconstruction). For HSI, super-resolution research mainly focuses on improving image resolution in the spatial and spectral domains. The main objective of this invention is to improve the spatial resolution of HSI; the super-resolution research discussed here refers to spatial domain super-resolution. HSI super-resolution reconstruction can be divided into fusion-based methods and single-image-based methods. Single-image-based super-resolution reconstruction methods mainly utilize deep learning, employing techniques such as SCT-SDCNN, 3D-FCN, and SSIN. While these methods achieve good super-resolution reconstruction results, the deep network models used have extremely high temporal and spatial complexity and require external training sets for multi-network parameter tuning. However, the high cost and complex techniques make obtaining HSI training sets difficult.

[0004] To address the difficulty in obtaining training sets for HSI super-resolution reconstruction, this invention proposes to employ transfer learning to study the HSI super-resolution reconstruction problem. Compared to HSI, obtaining training sets for color or multispectral images is much easier. This invention utilizes color image training sets to optimize the super-resolution reconstruction model based on transfer learning. However, due to the significant domain separation between HSI and color images, direct transfer may lead to severe spectral distortion in the super-resolution reconstructed HSI.

[0005] To address the issue that direct transfer may result in spectral distortion of the obtained HSI, this invention proposes a super-resolution HSI reconstruction method based on transfer learning and spectral restoration. By utilizing the spectral response function to perform spectral dimensionality reduction on the hyperspectral image, the domain gap between the hyperspectral image and the color image can be effectively reduced. Furthermore, this invention utilizes a trained spectral restoration network to reconstruct a high-resolution HSI based on transfer learning. Summary of the Invention

[0006] The main objective of this invention is to address the problem that direct transfer of existing spatial super-resolution reconstruction models trained on color image training sets may result in spectral distortion of the obtained HSI. By utilizing the spectral response function to perform spectral dimensionality reduction on hyperspectral images, and further reconstructing high-resolution HSI using a trained spectral restoration model based on transfer learning, this invention achieves the goal of HSI super-resolution reconstruction by predicting HSI from low-resolution hyperspectral images through a machine learning model.

[0007] To achieve the above objectives, the first aspect of the present invention provides an HSI super-resolution reconstruction method based on transfer learning and spectral restoration, comprising:

[0008] Obtain the training set of hyperspectral images T = {(X i Y i ), i = 1, 2, ..., Z}, and the color image training set C = {(L i H i ), i = 1, 2, ..., N}, where X = {X i Let i = 1, 2, ..., Z be a set of low-resolution hyperspectral images, and Y = {Y} i Let i = 1, 2, ..., Z be a set of high-resolution hyperspectral images, and L = {L i Let i = 1, 2, ..., N} be a set of low-resolution color images, and H = {H i , i = 1, 2, ..., N} is a set of high-resolution color images;

[0009] Based on the hyperspectral image training set, a corresponding multispectral image set T is generated using a preset spectral response function. C ={(X C i Y C i ), i = 1, 2, ..., Z}, where X C ={X C i Let i = 1, 2, ..., Z be a set of low-resolution multispectral images, and Y be a set of low-resolution multispectral images. C ={Y C i , i = 1, 2, ..., Z} is a high-resolution multispectral image set;

[0010] The preset transfer learning model is trained based on the color image training set to obtain a trained spatial super-resolution model.

[0011] The spatial super-resolution model is validated and optimized based on the multispectral image set to obtain the optimal target spatial super-resolution model.

[0012] Based on the aforementioned low-resolution multispectral image set and low-resolution hyperspectral image set, a synthetic dataset (X) is obtained. C Based on the synthetic dataset, a preset deep learning model is trained to obtain a target spectral super-resolution model;

[0013] A low-resolution hyperspectral image to be reconstructed is acquired, and transfer learning and spectral restoration processing are performed based on the target spatial super-resolution model and the target spectral super-resolution model, respectively, to obtain the corresponding high-resolution hyperspectral image.

[0014] Optionally, in another implementation of the first aspect of the present invention, the step of acquiring the low-resolution hyperspectral image to be reconstructed, and performing transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image includes:

[0015] Obtain the low-resolution hyperspectral image to be reconstructed, and generate the corresponding low-resolution multispectral image using the preset spectral response function;

[0016] Based on the target space super-resolution model, the low-resolution multispectral image is processed by transfer learning to obtain the corresponding high-resolution multispectral image as the predicted high-resolution image.

[0017] Based on the target spectral super-resolution model, the predicted high-resolution image is subjected to spectral restoration processing to obtain the corresponding high-resolution hyperspectral image.

[0018] Optionally, in another implementation of the first aspect of the present invention, training a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model includes:

[0019] Based on the color image training set, it is divided into a first training subset, a first verification subset, and a first test subset according to a preset ratio. The preset transfer learning model is trained in an end-to-end supervised manner. The preset transfer learning model is a SAN network model.

[0020] According to the first preset objective function, the preset transfer learning is used to learn the mapping relationship between each low-resolution color image in the low-resolution color image set and each corresponding high-resolution color image in the high-resolution color image set, until the loss of the first preset objective function reaches a minimum and convergence is achieved, thus obtaining a trained spatial super-resolution model.

[0021] Optionally, in another implementation of the first aspect of the present invention, the first preset objective function is:

[0022]

[0023] Where θ represents the spatial super-resolution model parameters, and N represents the number of training samples.

[0024] Optionally, in another implementation of the first aspect of the present invention, the step of verifying and optimizing the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model specifically includes:

[0025] The multispectral image set is used as the optimization training set, and divided into a second training subset, a second verification subset, and a second test subset according to the preset ratio. The spatial super-resolution model is trained in an end-to-end supervised manner.

[0026] Based on the first preset objective function, the spatial super-resolution model is used to learn the mapping relationship between low-resolution multispectral images and high-resolution multispectral images in the multispectral image set until the loss of the first preset objective function is minimized and convergence is achieved, thus obtaining the optimal target spatial super-resolution model.

[0027] Optionally, in another implementation of the first aspect of the present invention, the step of training a preset deep learning model based on the synthetic dataset to obtain a target spectral super-resolution model specifically includes:

[0028] Based on the synthetic dataset as the training set, it is divided into a third training subset, a third validation subset, and a third test subset according to the preset ratio. The preset deep learning model is trained in an end-to-end supervised manner. The preset deep learning model is an AWAN network model.

[0029] According to the second preset objective function, the preset deep learning model is used to learn the mapping relationship between the low-resolution multispectral images and the low-resolution hyperspectral image set in the synthetic dataset, until the loss of the second preset objective function reaches the minimum and convergence is achieved, thus obtaining the trained target spectral super-resolution model.

[0030] Optionally, in another implementation of the first aspect of the present invention, the second preset objective function includes a loss function and spectral information divergence, wherein the loss function is used to reduce pixel loss and the spectral information divergence is used to suppress spectral distortion;

[0031] The loss function is:

[0032]

[0033] The formula for calculating the spectral information divergence is:

[0034]

[0035]

[0036] Where Φ is the parameter of the spectral super-resolution model, ε is a constant, and Z is the number of training samples.

[0037] A second aspect of the present invention also provides an HSI super-resolution reconstruction device based on transfer learning and spectral restoration, the device comprising:

[0038] The training set acquisition module is used to acquire the hyperspectral image training set T = {(X i Y i ), i = 1, 2, ..., Z}, and the color image training set C = {(L i H i ), i = 1, 2, ..., N}, where X = {X i Let i = 1, 2, ..., Z be a set of low-resolution hyperspectral images, and Y = {Y} i Let i = 1, 2, ..., Z be a set of high-resolution hyperspectral images, and L = {L i Let i = 1, 2, ..., N} be a set of low-resolution color images, and H = {H i , i = 1, 2, ..., N} is a set of high-resolution color images;

[0039] The spectral downsampling module is used to generate a corresponding multispectral image set T based on the hyperspectral image set training set and using a preset spectral response function. C ={(X C i Y C i ), i = 1, 2, ..., Z}, where X C ={X C i Let i = 1, 2, ..., Z be a set of low-resolution multispectral images, and Y be a set of low-resolution multispectral images. C ={Y C i , i = 1, 2, ..., Z} is a high-resolution multispectral image set;

[0040] The spatial super-resolution model training module is used to train a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model.

[0041] The spatial super-resolution model tuning module is used to verify and tune the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model.

[0042] The spectral super-resolution model training module is used to obtain a synthetic dataset (X) based on the low-resolution multispectral image set and the low-resolution hyperspectral image set. CBased on the synthetic dataset, a preset deep learning model is trained to obtain a target spectral super-resolution model;

[0043] The super-resolution reconstruction module is used to acquire the low-resolution hyperspectral image to be reconstructed, and to perform transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image.

[0044] A third aspect of the present invention provides an HSI super-resolution reconstruction device based on transfer learning and spectral restoration, the device comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a circuit; the at least one processor calling the instructions in the memory to cause the HSI super-resolution reconstruction device based on transfer learning and spectral restoration to perform the HSI super-resolution reconstruction method based on transfer learning and spectral restoration as described in any of the preceding claims.

[0045] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the HSI super-resolution reconstruction method based on transfer learning and spectral restoration as described in any one of the preceding claims.

[0046] The technical solution provided by this invention involves acquiring a hyperspectral image training set and a color image training set; performing spectral downsampling on the hyperspectral image training set to obtain a multispectral image set; training a preset transfer learning model on the color image training set to obtain a spatial super-resolution model, and then optimizing the spatial super-resolution model to obtain a target spatial super-resolution model; training a preset deep learning model on a low-resolution multispectral image set and a low-resolution hyperspectral image set to obtain a target spectral super-resolution model; acquiring the low-resolution hyperspectral image to be reconstructed, and performing transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image. This embodiment of the invention utilizes the spectral response function to perform spectral dimensionality reduction on the hyperspectral image, which can effectively reduce the domain gap between the hyperspectral image and the color image. Furthermore, based on transfer learning, a trained spectral restoration network is used to reconstruct a high-resolution HSI, thereby achieving the purpose of HSI super-resolution reconstruction. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of an embodiment of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration in this invention.

[0049] Figure 2 This is a schematic diagram of an embodiment of the HSI super-resolution reconstruction device based on transfer learning and spectral restoration in this invention.

[0050] Figure 3 This is a schematic diagram of an embodiment of the HSI super-resolution reconstruction device based on transfer learning and spectral restoration in this invention. Detailed Implementation

[0051] This invention provides a method, apparatus, device, and storage medium for super-resolution HSI reconstruction based on transfer learning and spectral restoration, which is used to predict HSI from low-resolution hyperspectral images through machine learning models.

[0052] To enable those skilled in the art to better understand the present invention, the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0053] To address the difficulty in obtaining training sets for high-resolution hyperspectral image (HSI) reconstruction in existing technologies, this invention proposes to employ transfer learning to study HSI super-resolution reconstruction. Compared to HSI, training sets for color or multispectral images are easier to obtain. This invention utilizes color image training sets to fine-tune the super-resolution reconstruction model based on transfer learning. However, due to the significant domain separation between HSI and color images, direct transfer may lead to severe spectral distortion in the reconstructed HSI. To solve the problem of spectral distortion in the obtained HSI due to direct transfer, this invention proposes an HSI super-resolution reconstruction method based on transfer learning and spectral restoration. By using the spectral response function to perform spectral dimensionality reduction on the hyperspectral image, the domain separation between the hyperspectral and color images can be effectively reduced. Furthermore, this invention utilizes a trained spectral restoration network based on transfer learning to reconstruct a high-resolution HSI, thus achieving the goal of HSI super-resolution reconstruction. These points will be explained in detail below.

[0054] See Figure 1 One embodiment of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention includes:

[0055] Step 101: Obtain the hyperspectral image training set T = {(X i Y i ), i = 1, 2, ..., Z}, and the color image training set C = {(L i H i), i = 1, 2, ..., N}, where X = {X i Let i = 1, 2, ..., Z be a set of low-resolution hyperspectral images, and Y = {Y} i Let i = 1, 2, ..., Z be a set of high-resolution hyperspectral images, and L = {L i Let i = 1, 2, ..., N} be a set of low-resolution color images, and H = {H i , i = 1, 2, ..., N} is a set of high-resolution color images;

[0056] Step 102: Based on the hyperspectral image training set, generate the corresponding multispectral image set T using a preset spectral response function. C ={(X C i Y C i ), i = 1, 2, ..., Z}, where X C ={X C i Let i = 1, 2, ..., Z be a set of low-resolution multispectral images, and Y be a set of low-resolution multispectral images. C ={Y C i , i = 1, 2, ..., Z} is a high-resolution multispectral image set;

[0057] Step 103: Train the preset transfer learning model based on the color image training set to obtain the trained spatial super-resolution model;

[0058] Step 104: Validate and optimize the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model.

[0059] Step 105: Based on the low-resolution multispectral image set and the low-resolution hyperspectral image set, obtain the synthetic dataset (X). C Based on the synthetic dataset, a preset deep learning model is trained to obtain a target spectral super-resolution model;

[0060] Step 106: Obtain the low-resolution hyperspectral image to be reconstructed, and perform transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image.

[0061] As is well known, hyperspectral images (HSI) contain rich spectral information. For example, typical terrestrial hyperspectral images usually have more than 30 bands; typical airborne or spaceborne remote sensing hyperspectral images usually have 200 or even more bands. Super-resolution network models achieve good super-resolution results through rich feature extraction capabilities. For example, the number of intermediate convolutional features in a color image is about 20 times that of the channels. If the same settings are used to process hyperspectral images, the time and space complexity of the network model is enormous, especially when processing spaceborne or airborne remote sensing images. At the same time, acquiring hyperspectral data is relatively difficult, and training super-resolution networks requires a sufficient training set; otherwise, overfitting is likely. High costs and complex techniques make hyperspectral data relatively scarce. Transfer learning addresses the data scarcity problem by transferring knowledge across domains. In the area of ​​hyperspectral image super-resolution, some scholars have attempted to use transfer learning theory to solve the problem of scarce hyperspectral images, thereby utilizing rich color image training data. However, these methods usually require post-processing to address the severe spectral distortion present in super-resolution images, mainly because there are significant differences between the source and target domains, and direct transfer leads to significant errors. To utilize the source domain (color image or multispectral image) and achieve super-resolution reconstruction of hyperspectral images by directly transferring pre-trained CNNs of color images, the spatial super-resolution part of this paper does not directly transfer the color image super-resolution model to HSI, but instead transfers it to intermediate images with spectral downsampling for spatial super-resolution reconstruction.

[0062] Accordingly, in steps 101 and 102, the server obtains the hyperspectral image training set T = {(X i Y i ), i = 1, 2, ..., Z}, and the color image training set C = {(L i H i ), i = 1, 2, ..., N}, where X = {X i Let i = 1, 2, ..., Z be a set of low-resolution hyperspectral images, and Y = {Y} i Let i = 1, 2, ..., Z be a set of high-resolution hyperspectral images, and L = {L i Let i = 1, 2, ..., N} be a set of low-resolution color images, and H = {H i Let i = 1, 2, ..., N be a set of high-resolution color images. Only a small amount of data needs to be acquired for this high-resolution color image set, primarily for model tuning. This invention uses a color image training set to train and obtain a spatial super-resolution model, and then uses a small set of hyperspectral images as a training set to fine-tune the obtained spatial super-resolution model, thereby making the obtained super-resolution model more accurate.

[0063] Furthermore, in order to effectively reduce the domain gap between hyperspectral images and color images, a preset spectral response function is used to perform spectral downsampling on the hyperspectral image training set to generate a corresponding multispectral image set T. C ={(X C i Y C i ), i = 1, 2, ..., Z}, where X C ={X C i Let i = 1, 2, ..., Z be a set of low-resolution multispectral images, and Y be a set of low-resolution multispectral images. C ={Y C i , i = 1, 2, ..., Z} is a high-resolution multispectral image set, wherein each high-resolution color image in the high-resolution color image set is a one-to-one correspondence with each low-resolution color image in the low-resolution color image set.

[0064] The method embodiments of this invention are used to separately complete spatial super-resolution reconstruction and spectral reconstruction functions. If no usable hyperspectral training data is available, the two parts are trained separately. This invention uses a small number of available hyperspectral image training sets, and then performs joint training on the two parts during the knowledge transfer stage. Since the spectral reconstruction used in this invention is based on the spatial super-resolution model, the quality of the final predicted hyperspectral image is greatly affected by intermediate images. Compared with training the super-resolution model separately, joint training of spectral reconstruction can provide feedback to the super-resolution stage, making the knowledge transfer more inclined towards multispectral images, thereby promoting better hyperspectral image reconstruction.

[0065] Furthermore, all network models of the present invention are implemented based on the PyTorch framework. In step 103, during the pre-training stage of the spatial super-resolution network, in another implementation of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention, the step of training a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model includes:

[0066] Based on the color image training set, it is divided into a first training subset, a first verification subset, and a first test subset according to a preset ratio. The preset transfer learning model is trained in an end-to-end supervised manner. The preset transfer learning model is a SAN network model.

[0067] According to the first preset objective function, the preset transfer learning is used to learn the mapping relationship between each low-resolution color image in the low-resolution color image set and each corresponding high-resolution color image in the high-resolution color image set, until the loss of the first preset objective function reaches a minimum and convergence is achieved, thus obtaining a trained spatial super-resolution model.

[0068] In practice, the DIV2K dataset is used for training. This dataset contains 1000 images from different scenes, divided into a first training set, a first validation set, and a first test set in a ratio of {800, 100, 100}. The super-resolution model is implemented using a SAN network, and the model pre-training uses the ADAM optimizer with parameters β1 = 0.9, β2 = 0.99, and an initial learning rate of 1 × 10⁻⁶. -4 .

[0069] Optionally, the first preset objective function is:

[0070]

[0071] Where θ represents the spatial super-resolution model parameters, and N represents the number of training samples.

[0072] To better transfer knowledge from color images to the multispectral image domain, we fine-tune the trained model on a given low / high resolution multispectral image set. The fine-tuning process is the same as the pre-training process described above. The final trained target space super-resolution model is then used to predict high resolution images.

[0073] Further, in step 104, in another implementation of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention, the step of verifying and optimizing the spatial super-resolution model according to the multispectral image set to obtain the optimal target spatial super-resolution model specifically includes:

[0074] The multispectral image set is used as the optimization training set, and divided into a second training subset, a second verification subset, and a second test subset according to the preset ratio. The spatial super-resolution model is trained in an end-to-end supervised manner.

[0075] Based on the first preset objective function, the spatial super-resolution model is used to learn the mapping relationship between low-resolution multispectral images and high-resolution multispectral images in the multispectral image set until the loss of the first preset objective function is minimized and convergence is achieved, thus obtaining the optimal target spatial super-resolution model.

[0076] Specifically, it can be done using the formula This indicates that the optimized target space super-resolution model is used to predict high-resolution images, where F SR This represents the optimized target space super-resolution model. X represents the predicted high-resolution image. C Low-resolution multispectral images representing spectral degradation.

[0077] The above-described optimization and training process uses the same parameters as the pre-training stage of the spatial super-resolution network, and will not be elaborated upon here.

[0078] Further, in step 105, in another implementation of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention, the step of training a preset deep learning model based on the synthetic dataset to obtain the target spectral super-resolution model specifically includes:

[0079] Based on the synthetic dataset as the training set, it is divided into a third training subset, a third validation subset, and a third test subset according to the preset ratio. The preset deep learning model is trained in an end-to-end supervised manner. The preset deep learning model is an AWAN network model.

[0080] According to the second preset objective function, the preset deep learning model is used to learn the mapping relationship between the low-resolution multispectral images and the low-resolution hyperspectral image set in the synthetic dataset, until the loss of the second preset objective function reaches the minimum and convergence is achieved, thus obtaining the trained target spectral super-resolution model.

[0081] In practice, given a low-resolution hyperspectral image set X, the corresponding low-resolution multispectral image set X can be generated using the aforementioned preset spectral response function. C Spectral super-resolution models with synthetic datasets (X C The network parameters were optimized using low-resolution hyperspectral images (X) as the training set. The super-resolution spectral model was trained using the CAVE dataset, which contains 32 hyperspectral images with a spatial resolution of 512×512. Each image has 31 bands, covering the spectral range of 400 nm to 700 nm. These images were divided into a second training set, a second validation set, and a second test set according to the aforementioned proportions. The super-resolution model was implemented using the AWAN network, and the ADAM optimizer was used for pre-training with parameters β1 = 0.9 and β2 = 0.99. The initial learning rate was set to 1×10⁻⁶. -4 The number of network training iterations was 100.

[0082] Optionally, in another implementation of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention, the second preset objective function includes a loss function and spectral information divergence, wherein the loss function is used to reduce pixel loss and the spectral information divergence is used to suppress spectral distortion;

[0083] The loss function is:

[0084]

[0085] The formula for calculating the spectral information divergence is:

[0086]

[0087]

[0088] Where Φ is the parameter of the spectral super-resolution model, ε is a constant, and Z is the number of training samples.

[0089] In a specific implementation of this invention, the network model is trained using the L(Φ) loss function and spectral information divergence (SID) as objective functions. The L(Φ) loss is used to reduce pixel loss between the super-resolution image and the reference image, while the spectral information divergence (SID) loss is used to suppress spectral distortion between the super-resolution image and the reference image.

[0090] Optionally, in another implementation of the HSI super-resolution reconstruction method based on transfer learning and spectral restoration of the present invention, the step of acquiring the low-resolution hyperspectral image to be reconstructed, and performing transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image includes:

[0091] Obtain the low-resolution hyperspectral image to be reconstructed, and generate the corresponding low-resolution multispectral image using the preset spectral response function;

[0092] Based on the target space super-resolution model, the low-resolution multispectral image is processed by transfer learning to obtain the corresponding high-resolution multispectral image as the predicted high-resolution image.

[0093] Based on the target spectral super-resolution model, the predicted high-resolution image is subjected to spectral restoration processing to obtain the corresponding high-resolution hyperspectral image.

[0094] The objective of this invention is to predict a high-resolution HSI from a low-resolution HSI. Specifically, the method uses a high-resolution image predicted by a spatial super-resolution model as an intermediate image, and then utilizes a spectral super-resolution model to recover the high-resolution HSI. Specifically, the low-resolution hyperspectral image to be reconstructed is spectrally downsampled using a preset spectral response function to obtain a low-resolution multispectral image. The preset spectral response function can be any response function used in existing technologies for spectral downsampling to achieve spectral dimensionality reduction, and is not limited here. A target spatial super-resolution model more suitable for the multispectral image domain is obtained through optimization. The low-resolution multispectral image is then transferred to obtain the predicted high-resolution image. Finally, the trained target spectral super-resolution model is used to generate the high-resolution HSI from the predicted high-resolution image.

[0095] To verify the effectiveness of the algorithm proposed in this patent, we compared it with several other baseline methods, including: Bicubic: Bicubic interpolation is applied to low-resolution images; SRCNN: The input and output settings of the network are modified to meet the requirements of hyperspectral images, while other parameters remain unchanged, and the network is trained using a simulated dataset; TL-CNMF: A pre-trained network model is used to super-resolution low-resolution images in a band-by-band manner, and then optimized using CNMF; SSIN: Paired low-resolution / high-resolution image sets are first constructed using the Bicubic method, and then the network is trained according to the settings in the original paper.

[0096] Table 1. Accuracy comparison of super-resolution reconstruction of Balloons (from CAVE(×3)) using different methods

[0097]

[0098] Table 2. Accuracy comparison of super-resolution reconstruction of Balloons (from CAVE(×4)) using different methods

[0099]

[0100] Tables 1 and 2 show the super-resolution reconstruction results of the comparison method on the "Balloons" test image, while Tables 3 and 4 show the average super-resolution reconstruction results of the comparison method on all test images, with magnification factors of ×3 and ×4, respectively.

[0101] Table 3. Average super-resolution reconstruction results of different methods on all test images of CAVE (×3).

[0102]

[0103] Table 4. Average super-resolution reconstruction results of different methods on all test images of CAVE (×4).

[0104]

[0105] According to Tables 3 and 4, the Bicubic interpolation method shows the worst reconstruction performance. While the SRCNN method achieves better spatial domain reconstruction results compared to the Bicubic method, it suffers from severe spectral distortion. This is mainly because SRCNN was originally designed for color images, with only 64 convolutional features extracted, which is insufficient to fully represent the spatial-spectral characteristics of hyperspectral images, thus resulting in significant spectral distortion. The TL-CNMF method predicts high-resolution images band-by-band, maintaining spectral consistency by utilizing CNMF, thus improving super-resolution reconstruction performance. The SSIN method utilizes images with good super-resolution results; the original paper sets the base-features of grouped convolutions to 16, indicating the extraction of 16 convolutional features from each band image, demonstrating that strong feature extraction capabilities are beneficial for improving super-resolution reconstruction performance, but leading to higher spatial complexity. The proposed method uses spectrally downsampled images as intermediate images, effectively leveraging super-resolution knowledge in the color image domain to improve super-resolution reconstruction performance and reduce the computational complexity of the super-resolution reconstruction network. Experimental results demonstrate that the proposed method achieves good super-resolution reconstruction performance.

[0106] In summary, the HSI super-resolution reconstruction method based on transfer learning and spectral restoration provided by this invention involves: acquiring a hyperspectral image training set and a color image training set; performing spectral downsampling on the hyperspectral image training set to obtain a multispectral image set; training a preset transfer learning model on the color image training set to obtain a spatial super-resolution model, and then optimizing the spatial super-resolution model to obtain a target spatial super-resolution model; training a preset deep learning model on a low-resolution multispectral image set and a low-resolution hyperspectral image set to obtain a target spectral super-resolution model; acquiring the low-resolution hyperspectral image to be reconstructed, and performing transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image. In this embodiment, the low-resolution HSI is downsampled into a multispectral image using the spectral response function during the transfer learning stage, and this image is used as an intermediate image for transferring the trained super-resolution model to multispectral image super-resolution reconstruction; in the spectral restoration stage, the trained spectral restoration model is used to reconstruct the multispectral image into a high-resolution HSI. This involves using the spectral response function to perform spectral dimensionality reduction on hyperspectral images, which can effectively reduce the domain gap between hyperspectral and color images. Furthermore, based on transfer learning, a trained spectral restoration network is used to reconstruct high-resolution HSI, thereby achieving the goal of HSI super-resolution reconstruction.

[0107] The HSI super-resolution reconstruction method based on transfer learning and spectral restoration in the embodiments of the present invention has been described above. The HSI super-resolution reconstruction apparatus based on transfer learning and spectral restoration in the embodiments of the present invention is described below. Please refer to [link to relevant documentation]. Figure 2One embodiment of the HSI super-resolution reconstruction device based on transfer learning and spectral restoration in this invention includes:

[0108] Training set acquisition module 11 is used to acquire the hyperspectral image training set T = {(X i Y i ), i = 1, 2, ..., Z}, and the color image training set C = {(L i H i ), i = 1, 2, ..., N}, where X = {X i Let i = 1, 2, ..., Z be a set of low-resolution hyperspectral images, and Y = {Y} i Let i = 1, 2, ..., Z be a set of high-resolution hyperspectral images, and L = {L i Let i = 1, 2, ..., N} be a set of low-resolution color images, and H = {H i , i = 1, 2, ..., N} is a set of high-resolution color images;

[0109] Spectral downsampling module 12 is used to generate a corresponding multispectral image set T based on the hyperspectral image set training set and using a preset spectral response function. C ={(X C i Y C i ), i = 1, 2, ..., Z}, where X C ={X C i Let i = 1, 2, ..., Z be a set of low-resolution multispectral images, and Y be a set of low-resolution multispectral images. C ={Y C i , i = 1, 2, ..., Z} is a high-resolution multispectral image set;

[0110] The spatial super-resolution model training module 13 is used to train a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model.

[0111] The spatial super-resolution model tuning module 14 is used to verify and tune the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model.

[0112] The spectral super-resolution model training module 15 is used to obtain a synthetic dataset (X) based on the low-resolution multispectral image set and the low-resolution hyperspectral image set. C Based on the synthetic dataset, a preset deep learning model is trained to obtain a target spectral super-resolution model;

[0113] The super-resolution reconstruction module 16 is used to acquire the low-resolution hyperspectral image to be reconstructed, and to perform transfer learning and spectral restoration processing based on the target spatial super-resolution model and the target spectral super-resolution model respectively to obtain the corresponding high-resolution hyperspectral image.

[0114] Optionally, in another implementation of the device, the super-resolution reconstruction module includes:

[0115] A spectral downsampling unit is used to acquire a low-resolution hyperspectral image to be reconstructed and to generate a corresponding low-resolution multispectral image using the preset spectral response function.

[0116] The transfer learning processing unit is used to perform transfer learning processing on the low-resolution multispectral image based on the target space super-resolution model to obtain the corresponding high-resolution multispectral image as the predicted high-resolution image.

[0117] The spectral restoration processing unit is used to perform spectral restoration processing on the predicted high-resolution image based on the target spectral super-resolution model to obtain the corresponding high-resolution hyperspectral image.

[0118] Optionally, in another implementation of the device, the spatial super-resolution model training module includes:

[0119] The first training set processing unit is used to divide the color image training set into a first training subset, a first verification subset, and a first test subset according to a preset ratio, and to train the preset transfer learning model in an end-to-end supervised manner. The preset transfer learning model is a SAN network model.

[0120] The spatial super-resolution model acquisition unit is used to learn the mapping relationship between each low-resolution color image in the low-resolution color image set and each corresponding high-resolution color image in the high-resolution color image set according to the first preset objective function and the preset transfer learning, until the loss of the first preset objective function reaches the minimum and converges, thereby obtaining the trained spatial super-resolution model.

[0121] Optionally, in another implementation of the device, the first preset objective function is:

[0122]

[0123] Where θ represents the spatial super-resolution model parameters, and N represents the number of training samples.

[0124] Optionally, in another implementation of the device, the spatial super-resolution model tuning module includes:

[0125] The second training set processing unit is used to use the multispectral image set as an optimization training set, divide it into a second training subset, a second verification subset and a second test subset according to the preset ratio, and train the spatial super-resolution model in an end-to-end supervised manner.

[0126] The target space super-resolution model acquisition unit is used to learn the mapping relationship between low-resolution multispectral images and high-resolution multispectral images in the multispectral image set according to the first preset objective function and the spatial super-resolution model, until the loss of the first preset objective function reaches the minimum and converges, thereby obtaining the optimal target space super-resolution model.

[0127] Optionally, in another implementation of the device, the spectral super-resolution model training module specifically includes:

[0128] The third training set processing unit is used to divide the synthetic dataset into a third training subset, a third validation subset, and a third test subset according to the preset ratio, and to train the preset deep learning model in an end-to-end supervised manner. The preset deep learning model is an AWAN network model.

[0129] The target spectral super-resolution model acquisition unit is used to learn the mapping relationship between the low-resolution multispectral images and the low-resolution hyperspectral image set in the synthetic dataset using the preset deep learning model according to the second preset objective function, until the loss of the second preset objective function reaches the minimum and converges, thus obtaining the trained target spectral super-resolution model.

[0130] Optionally, in another implementation of the device, the second preset objective function includes a loss function and spectral information divergence, wherein the loss function is used to reduce pixel loss and the spectral information divergence is used to suppress spectral distortion;

[0131] The loss function is:

[0132]

[0133] The formula for calculating the spectral information divergence is:

[0134]

[0135]

[0136] Where Φ is the parameter of the spectral super-resolution model, ε is a constant, and Z is the number of training samples.

[0137] It should be noted that the apparatus in the embodiments of the present invention can be used to implement all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above examples, which will not be repeated here.

[0138] above Figure 2 The HSI super-resolution reconstruction device based on transfer learning and spectral restoration in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The HSI super-resolution reconstruction device based on transfer learning and spectral restoration in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0139] Figure 3 This is a schematic diagram of the structure of an HSI super-resolution reconstruction device based on transfer learning and spectral restoration provided in an embodiment of the present invention. The HSI super-resolution reconstruction device 300 based on transfer learning and spectral restoration can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 301 (e.g., one or more processors) and a memory 309, and one or more storage media 308 (e.g., one or more mass storage devices) for storing application programs 307 or data 306. The memory 309 and storage media 308 can be temporary or persistent storage. The program stored in the storage media 308 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations stored in Boolean variables for graph computation. Furthermore, the processor 301 may be configured to communicate with the storage media 308 and execute the series of instruction operations in the storage media 308 on the HSI super-resolution reconstruction device 300 based on transfer learning and spectral restoration.

[0140] The HSI super-resolution reconstruction device 300 based on transfer learning and spectral restoration may also include one or more power supplies 302, one or more wired or wireless network interfaces 303, one or more input / output interfaces 304, and / or one or more operating systems 305, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The HSI super-resolution reconstruction device structure based on transfer learning and spectral restoration shown in the figure does not constitute a limitation on HSI super-resolution reconstruction devices based on transfer learning and spectral restoration. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0142] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0143] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium, which can be non-volatile or volatile. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A super-resolution HSI reconstruction method based on transfer learning and spectral restoration, characterized in that, The method includes: Obtain the training set of hyperspectral images and color image training set ,in, For low-resolution hyperspectral image sets, It is a high-resolution hyperspectral image set. For low-resolution color image sets, A set of high-resolution color images; Based on the hyperspectral image training set, a corresponding multispectral image set is generated using a preset spectral response function. ,in For low-resolution multispectral image sets, It is a high-resolution multispectral image set; The preset transfer learning model is trained based on the color image training set to obtain a trained spatial super-resolution model. The spatial super-resolution model is validated and optimized based on the multispectral image set to obtain the optimal target spatial super-resolution model. Based on the low-resolution multispectral image set and the low-resolution hyperspectral image set, a synthetic dataset is obtained. The target spectral super-resolution model is obtained by training a preset deep learning model based on the synthetic dataset. Obtain the low-resolution hyperspectral image to be reconstructed, and generate the corresponding low-resolution multispectral image using the preset spectral response function; Based on the target space super-resolution model, the low-resolution multispectral image is processed by transfer learning to obtain the corresponding high-resolution multispectral image as the predicted high-resolution image. Based on the target spectral super-resolution model, the predicted high-resolution image is subjected to spectral restoration processing to obtain the corresponding high-resolution hyperspectral image.

2. The HSI super-resolution reconstruction method based on transfer learning and spectral restoration according to claim 1, characterized in that, The step of training a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model includes: Based on the color image training set, it is divided into a first training subset, a first verification subset, and a first test subset according to a preset ratio. The preset transfer learning model is trained in an end-to-end supervised manner. The preset transfer learning model is a SAN network model. According to the first preset objective function, the preset transfer learning is used to learn the mapping relationship between each low-resolution color image in the low-resolution color image set and each corresponding high-resolution color image in the high-resolution color image set, until the loss of the first preset objective function reaches a minimum and convergence is achieved, thus obtaining a trained spatial super-resolution model.

3. The HSI super-resolution reconstruction method based on transfer learning and spectral restoration according to claim 2, characterized in that, The first preset objective function is: ； Where θ represents the spatial super-resolution model parameters, and N represents the number of the first training samples.

4. The HSI super-resolution reconstruction method based on transfer learning and spectral restoration according to claim 2, characterized in that, The step of verifying and optimizing the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model specifically includes: The multispectral image set is used as the optimization training set, and divided into a second training subset, a second verification subset, and a second test subset according to the preset ratio. The spatial super-resolution model is trained in an end-to-end supervised manner. Based on the first preset objective function, the spatial super-resolution model is used to learn the mapping relationship between low-resolution multispectral images and high-resolution multispectral images in the multispectral image set until the loss of the first preset objective function is minimized and convergence is achieved, thus obtaining the optimal target spatial super-resolution model.

5. The HSI super-resolution reconstruction method based on transfer learning and spectral restoration according to claim 4, characterized in that, The step of training a preset deep learning model based on the synthetic dataset to obtain the target spectral super-resolution model specifically includes: Based on the synthetic dataset as the training set, it is divided into a third training subset, a third validation subset, and a third test subset according to the preset ratio. The preset deep learning model is trained in an end-to-end supervised manner. The preset deep learning model is an AWAN network model. According to the second preset objective function, the preset deep learning model is used to learn the mapping relationship between the low-resolution multispectral images and the low-resolution hyperspectral image set in the synthetic dataset, until the loss of the second preset objective function reaches the minimum and convergence is achieved, thus obtaining the trained target spectral super-resolution model.

6. The HSI super-resolution reconstruction method based on transfer learning and spectral restoration according to claim 5, characterized in that, The second preset objective function includes a loss function and spectral information divergence. The loss function is used to reduce pixel loss, and the spectral information divergence is used to suppress spectral distortion. The loss function is: ， The formula for calculating the spectral information divergence is: ； ， ， Where Φ is the parameter of the spectral super-resolution model, ε is a constant, and Z is the number of the second training samples.

7. A super-resolution HSI reconstruction device based on transfer learning and spectral restoration, characterized in that, The device includes: The training set acquisition module is used to acquire the training set of hyperspectral images. and color image training set ,in, For low-resolution hyperspectral image sets, It is a high-resolution hyperspectral image set. For low-resolution color image sets, A set of high-resolution color images; The spectral downsampling module is used to generate a corresponding multispectral image set based on the hyperspectral image set training set and using a preset spectral response function. ,in For low-resolution multispectral image sets, It is a high-resolution multispectral image set; The spatial super-resolution model training module is used to train a preset transfer learning model based on the color image training set to obtain a trained spatial super-resolution model. The spatial super-resolution model tuning module is used to verify and tune the spatial super-resolution model based on the multispectral image set to obtain the optimal target spatial super-resolution model. The spectral super-resolution model training module is used to obtain a synthetic dataset based on the low-resolution multispectral image set and the low-resolution hyperspectral image set. The target spectral super-resolution model is obtained by training a preset deep learning model based on the synthetic dataset. A low-resolution multispectral image generation module is used to acquire a low-resolution hyperspectral image to be reconstructed and generate a corresponding low-resolution multispectral image using the preset spectral response function. A high-resolution multispectral image generation module is used to perform transfer learning processing on the low-resolution multispectral image based on the target space super-resolution model to obtain the corresponding high-resolution multispectral image as the predicted high-resolution image. The high-resolution hyperspectral image acquisition module is used to perform spectral restoration processing on the predicted high-resolution image based on the target spectral super-resolution model to obtain the corresponding high-resolution hyperspectral image.

8. An HSI super-resolution reconstruction device based on transfer learning and spectral restoration, characterized in that, The HSI super-resolution reconstruction device based on transfer learning and spectral restoration includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the HSI super-resolution reconstruction device based on transfer learning and spectral restoration to perform the HSI super-resolution reconstruction method based on transfer learning and spectral restoration as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the HSI super-resolution reconstruction method based on transfer learning and spectral restoration as described in any one of claims 1-6.

Images