Spectrum super-resolution reconstruction method and system based on deep synchronous spectrum unmixing network

Abstract

The invention discloses a spectral super-resolution reconstruction method and system based on a deep synchronous spectral unmixing network. According to the deep synchronous spectrum unmixing network constructed by the invention, non-pairwise hyperspectral surface feature end member information is mined by coupling the relation of a hyperspectral image and a multispectral image in surface feature end members; the specific structure comprises five modules: a hyperspectral encoder module, a spectrum down-sampling module, a multispectral discriminator module, a multispectral encoder module and a hyperspectral decoder module. According to the method, the correlation between the end members of the hyperspectral image and the multispectral image is coupled by utilizing the ground feature end member homogeneity between the non-paired hyperspectral image and the multispectral image to be hyper-divided, so that collaborative spectral unmixing is carried out, and the network is promoted to learn correct ground feature end member information.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and hyperspectral image reconstruction, and specifically relates to a spectral super-resolution reconstruction method and system based on a deep synchronous spectral unmixing network. Background Art

[0002] Hyperspectral remote sensing technology can not only capture the two-dimensional spatial information of ground objects in the observation scene, but also record the spectral signals of the observation scene in continuous spectral bands, which helps to accurately identify different ground objects in the observation scene and has wide applications in disaster prevention and mitigation, smart cities and other fields. However, due to the limitations of existing remote sensing imaging hardware, the imaging speed of hyperspectral remote sensing technology is slow, making it difficult to quickly and dynamically monitor the observation scene, limiting the application scenarios of hyperspectral remote sensing technology.

[0003] Spectral super-resolution methods aim to improve the spectral resolution of multispectral images. They are an effective method for obtaining hyperspectral remote sensing images with fast speed and low cost. In recent years, a large number of spectral super-resolution methods based on deep learning have been proposed to learn the spectral mapping between multispectral images and hyperspectral images. For example, Fu et al. proposed a spectral-space convolutional neural network in the document "Fu Y, Zhang T, Zheng Y, Zhang D, Huang H. Joint camera spectral response selection and hyperspectral image recovery [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (1): 256-272.", which includes a spectral network for learning spectral nonlinear mapping and a spatial network for exploring spatial similarity. Mei et al. proposed a lightweight multi-resolution feature fusion network in the paper "Mei S, Zhang G, Wang N, Wu B, Ma M, Zhang Y, Feng Y. Lightweight multiresolution feature fusion network for spectral super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5501414." to fully explore the complementary information at different resolution scales.

[0004] These methods usually focus on learning the hyperspectral data distribution information from multispectral-hyperspectral image sample pairs. However, it is very difficult to collect a large number of paired samples under different seasons, different atmospheric conditions, etc. In practice, it is easy to obtain a large number of unpaired hyperspectral remote sensing images, but existing methods find it difficult to effectively mine and utilize the rich end-member information of ground objects contained in these unpaired hyperspectral remote sensing images to improve the quality of reconstruction. Summary of the invention

[0005] The present invention discloses a novel spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network, which mainly overcomes the problem that the existing spectral super-resolution reconstruction method relies on spectral super-resolution reconstruction under paired training samples and is difficult to effectively utilize the end-member information of ground objects contained in unpaired hyperspectral images.

[0006] The deep synchronous spectral unmixing network disclosed in the present invention mines the ground object endmember information of non-paired hyperspectral images by coupling the connection between hyperspectral images and multispectral images in ground object endmembers. The specific structure of the deep synchronous spectral unmixing network includes five modules: a hyperspectral encoder module, a spectral downsampling module, a multispectral discriminator module, a multispectral encoder module and a hyperspectral decoder module. The specific scheme is as follows:

[0007] 1. Abundance information extraction of unpaired hyperspectral images. First, a hyperspectral encoder module is constructed to extract the abundance information of the objects contained in the hyperspectral image. The module consists of multiple 1×1 convolutions and multi-head self-attention components stacked in series. Based on the linear spectral mixture model, the last layer of the hyperspectral encoder module is set to a truncation function to force the abundance values ​​of each object to be in the interval [0,1] to meet the non-negativity of the abundance. Furthermore, the L1 norm is used to construct a regular term with the sum of abundances being 1. Then, the abundance sparsity regularization is constructed using the Kullback-Leibler divergence formula.

[0008] 2. Spectral response function estimation. Construct a spectral downsampling module to estimate the spectral response function from a hyperspectral image to a multispectral image, which can be used to downsample the hyperspectral image in the spectral dimension to obtain a reconstructed multispectral image. According to the principles of hyperspectral and multispectral imaging, the spectral downsampling module can be constructed using a 1×1 convolutional layer. Further construct a multispectral discrimination module, based on an adversarial learning strategy to force the reconstructed multispectral image obtained by the spectral downsampling module and the real multispectral image to follow the same distribution, so as to improve the accuracy of the spectral response function estimation.

[0009] 3. Abundance information extraction from multispectral images. A multispectral encoder module is constructed to extract abundance information from the reconstructed multispectral images. The structure of the multispectral encoder module is also composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series. Like the hyperspectral encoder module, this module also contains abundance non-negativity, sum-to-1, and sparsity regularization terms. Next, since the unpaired hyperspectral image and the reconstructed multispectral image contain the same distribution of objects, an abundance consistency regularization term is constructed to improve the accuracy of the abundance information of the multispectral image.

[0010] 4. Hyperspectral image reconstruction. Construct a hyperspectral decoder module to reconstruct the corresponding hyperspectral image from the abundance matrix. According to the linear spectral unmixing model, this module is designed as a 1×1 convolution layer without an activation function, which is multiplied with the abundance matrix to obtain the reconstructed hyperspectral image; according to the nonlinear spectral unmixing model, the model can also stack multiple 1×1 convolution layers for nonlinear learning. Then, the abundance matrices of the hyperspectral image and the multispectral image are respectively input into the hyperspectral decoder module to reconstruct the corresponding hyperspectral image, and the consistency regularization of the reconstruction result is constructed to promote the convergence of the network parameters.

[0011] 5. Optimization of network weight parameters. Use the gradient descent method to optimize the weight parameters of the deep synchronous spectral unmixing network until convergence.

[0012] 6. Test the spectral super-resolution reconstruction performance of the deep synchronous spectral unmixing network. After the training is completed, the multispectral image to be spectrally super-resolved is input into the multispectral encoder module to extract its ground feature abundance information, and then the ground feature abundance information is input into the hyperspectral decoder to reconstruct the required hyperspectral image and compare it with the real hyperspectral image to evaluate the quality of the reconstructed hyperspectral image.

[0013] The technical solution of the present invention is a spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network, comprising the following steps:

[0014] Step 1, construct a hyperspectral encoder module to extract the abundance information of the objects contained in the unpaired hyperspectral image;

[0015] Step 2, construct a spectral downsampling module to estimate the spectral response function from the hyperspectral image to the multispectral image, downsample the hyperspectral image in the spectral dimension, and obtain a reconstructed multispectral image;

[0016] Construct a multispectral discrimination module, and use adversarial learning strategy to identify whether the reconstructed multispectral image obtained by the spectrum downsampling module follows the same distribution as the multispectral image to be spectrally super-resolved.

[0017] Step 3, construct a multispectral encoder module to extract abundance information from the reconstructed multispectral image;

[0018] Step 4, construct a hyperspectral decoder module to reconstruct the corresponding hyperspectral image from the abundance information, that is, input the abundance information of the unpaired hyperspectral image in step 1 into the hyperspectral decoder module to reconstruct the corresponding hyperspectral image, and input the abundance information of the multispectral image in step 3 into the decoder module to reconstruct the corresponding hyperspectral image;

[0019] Step 5, combining the above five modules to build a deep synchronous spectral unmixing network, and using the gradient descent method to optimize the weight parameters of the deep synchronous spectral unmixing network until convergence;

[0020] Step 6: After the training is completed, the multispectral image to be spectrally super-resolved is input into the multispectral encoder module to extract its ground feature abundance information, and then the ground feature abundance information is input into the hyperspectral decoder to reconstruct the required hyperspectral image.

[0021] Furthermore, the hyperspectral encoder module is composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series, where the 1×1 convolution is used to extract the abundance information of the hyperspectral image in the spectral dimension, and the multi-head self-attention component is used to introduce global spatial information.

[0022] Furthermore, according to the linear spectral mixture model, in order to ensure that the abundance matrix of the hyperspectral image satisfies the non-negativity, the sum is 1, and the sparsity, the last layer of the hyperspectral encoder module is set to a truncation function to force the abundance values ​​of various objects to be in the interval [0,1] to meet the abundance non-negativity; further, a loss function L with a sum of 1 regularization is constructed based on the L1 norm. Ahsi_sum2one , which makes the abundance matrix A hsi The sum of abundances is 1; then the sparse regularized loss function L is constructed using the Kullback-Leibler divergence formula KL(·||·) Ahsi_sparse , which makes the abundance matrix sparse, where ε is a parameter representing sparsity;

[0023]

[0024] in, and ρ are two independent variables, W un ×H un is the hyperspectral image Y un Space size, B h is the hyperspectral image Y un The number of bands, k represents Y un The number of end members of the middle ground object.

[0025] Furthermore, in step 2, the spectral degradation is modeled as spectral downsampling from the hyperspectral image to the multispectral image, which is expressed by the following formula:

[0026]

[0027] in, is the reconstructed multispectral image, Y un (i,j,b) represents the hyperspectral image Y un The reflectivity of the b-th band in the i-th row, j-th column, is the spectral response function, B m is the number of bands of the multispectral image, B h is the number of bands of the hyperspectral image, and formula (4) is regarded as a 1×1 convolution operation, where C is the weight parameter; therefore, the spectral downsampling module is constructed using a 1×1 convolution layer, where the number of convolution kernels in the convolution layer is equal to the number of spectral bands of the multispectral image.

[0028] Furthermore, the multispectral discrimination module is composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series; the multispectral image X to be spectrally super-resolved is used as the supervision signal of the multispectral discrimination module for adversarial training, and the loss function L of the multispectral discrimination module is adv for:

[0029]

[0030] Among them, L adv is the adversarial loss function, D(·) represents the multi-spectral discrimination module, It represents the expectation of the multispectral discrimination module on the judgment result after inputting the real multispectral image X. Represents the input reconstructed multispectral image Finally, the multi-spectral discrimination module expects its judgment results.

[0031] Furthermore, the multispectral encoder module is composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series. In order to ensure that the abundance of the multispectral image satisfies the non-negative, sum-to-1, and sparsity, first, the last layer of the multispectral encoder module is set to a truncation function to force the abundance values ​​of each object to be in the interval [0,1]. Then, the abundance matrix A is msi Apply L with sum equal to 1 Amsi_sum2one Regularization term and sparsity L Amsi_sparse Regularization term;

[0032]

[0033] because Yes un Generated by the spectral downsampling module, and Y un Contains the same spatial distribution of features, so and Y unThere should be the same abundance information, so the abundance consistency regularization term formula (8) is constructed to optimize the parameters of the multispectral encoding module;

[0034]

[0035] Furthermore, in step 4, according to the linear spectral unmixing model, the hyperspectral decoder module is designed as a 1×1 convolutional layer without an activation function, which is multiplied with the abundance matrix to obtain the reconstructed hyperspectral image; according to the nonlinear spectral unmixing model, the hyperspectral decoder module can also stack multiple 1×1 convolutional layers for nonlinear learning to reconstruct the hyperspectral image.

[0036] Furthermore, the hyperspectral image Y un The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image Multispectral image The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image and It is consistent, so there is a consistent reconstruction loss L rec :

[0037]

[0038] Furthermore, the loss function of the deep synchronous spectral unmixing network is as follows:

[0039] Loss = L rec +α(L Ahsi_sum2one +L Ahsi_sparse +L Amsi_sum2one +L Amsi_sparse )+βL A_same +ηL Adv (10)

[0040] Among them, α, β, and η are hyperparameters used to control the balance between various regularization terms.

[0041] The present invention also provides a spectral super-resolution reconstruction system based on a deep synchronous spectral unmixing network, comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as described in the above technical solution.

[0042] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0043] 1. The deep synchronous spectral unmixing network disclosed in the present invention can make full use of unpaired hyperspectral images for training and use their endmember information to improve the quality of reconstructed images.

[0044] 2. This method utilizes the homogeneity of ground object endmembers between unpaired hyperspectral images and multispectral images to be super-resolved, and couples the association between the endmembers of hyperspectral images and multispectral images to perform collaborative spectral unmixing, prompting the network to learn the correct ground object endmember information.

[0045] 3. Experiments were conducted on the public Paiva University hyperspectral remote sensing dataset and demonstrated the spectral super-resolution reconstruction performance of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention in the training stage.

[0047] Figure 2 It is the overall flow chart of the present invention in the testing and reasoning stages. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0049] Reference Figure 1 The embodiment of the present invention provides a spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network, and the specific implementation steps are as follows:

[0050] Step 1: Extract abundance information from unpaired hyperspectral images.

[0051] (1a) Order represents an unpaired hyperspectral remote sensing image, where W un ×H un is the image space size, B h is the number of bands of the hyperspectral image. First, a hyperspectral encoder module is constructed to extract the hyperspectral image Y un Abundance information of included features Where k represents Y un The hyperspectral encoder module is composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series, where the 1×1 convolution is used to extract the abundance information of the hyperspectral image in the spectral dimension, and the multi-head self-attention component is used to introduce global spatial information to improve the accuracy of abundance information extraction.

[0052] (1b) According to the linear spectral mixture model, in order to ensure that the abundance matrix of the hyperspectral image satisfies non-negativity, sums to 1, and sparsity, the last layer of the hyperspectral encoder module is set to a truncation function to force the abundance values ​​of each object to be in the interval [0,1] to meet the abundance non-negativity. Furthermore, a loss function L is constructed based on the L1 norm with a sum-to-1 regularization. Ahsi_sum2one , which makes the abundance matrix A hsiThe sum of abundances is 1. Then, the sparse regularized loss function L is constructed using the Kullback-Leibler divergence formula KL(·||·) Ahsi_sparse , which makes the abundance matrix sparse, where ε is a parameter representing sparsity and is set to a small constant.

[0053]

[0054] in, and ρ are two independent variables.

[0055] Step 2: Spectral response function estimation.

[0056] (2a) Construct a spectral downsampling module to estimate the spectral response function between the hyperspectral image and the multispectral image, which can be used to downsample the hyperspectral image Y in the spectral dimension. un Down-sample to obtain the reconstructed multispectral image Among them B m is the number of bands of the multispectral image. The spectral response function between the hyperspectral sensor and the multispectral sensor will be affected by factors such as atmosphere and instrument noise, and will differ from the actual spectral response function. Therefore, this module is constructed to accurately estimate the spectral response function. According to the principles of hyperspectral and multispectral imaging, spectral degradation can be modeled as spectral downsampling from hyperspectral images to multispectral images, which can be expressed by the following formula:

[0057]

[0058] Among them, Y un (i,j,b) represents the reflectivity of the bth band in the i-th row, j-th column, is the spectral response function. Formula (4) can be regarded as a 1×1 convolution operation, where C is the weight parameter. Therefore, the spectral downsampling module can be constructed with a 1×1 convolution layer, where the number of convolution kernels in this layer is equal to the number of spectral bands of the multispectral image.

[0059] (2b) Construct a multispectral discrimination module to identify the reconstructed multispectral image obtained by the spectral downsampling module based on the adversarial learning strategy. Whether to use the multispectral image to be super-resolved The multispectral discrimination module is also composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series. The multispectral image X to be spectrally super-resolved can be used as the supervision signal of the multispectral discrimination module for adversarial training. The loss function L of the multispectral discrimination module adv for:

[0060]

[0061] Among them, L adv is the adversarial loss function, D(·) represents the multi-spectral discrimination module, It represents the expectation of the multispectral discrimination module on the judgment result after inputting the real multispectral image X. Represents the input reconstructed multispectral image Finally, the multi-spectral discrimination module expects its judgment results.

[0062] Step 3: Extract the abundance information of multispectral images.

[0063] (3a) This module is used to reconstruct multispectral images Extract the abundance information of its features The structure of the multispectral encoder module consists of multiple 1×1 convolutions and multi-head self-attention components stacked in series. In order to ensure that the abundance of the multispectral image satisfies the non-negative, sum-to-1, and sparsity, first, the last layer of the multispectral encoder module is set to a truncation function to force the abundance values ​​of each object to be in the interval [0,1]. Then, the abundance matrix A is msi Apply L that sums to 1 Amsi_sum2one Regularization term and sparsity L Amsi_sparse Regularization term.

[0064]

[0065] (3b) Abundance consistency rule. Yes un Generated by the spectral downsampling module, and Y un Contains the same spatial distribution of features, so and Y un There should be the same abundance information, then the abundance consistency regularization term formula (8) can be constructed to optimize the parameters of the multispectral encoding module.

[0066]

[0067] Step 4: Hyperspectral image reconstruction.

[0068] (4a) A hyperspectral decoder module is constructed to reconstruct the corresponding hyperspectral image from the abundance matrix. According to the linear spectral unmixing model, this module is designed as a 1×1 convolutional layer without an activation function, which is multiplied with the abundance matrix to obtain the reconstructed hyperspectral image; according to the nonlinear spectral unmixing model, this model can also stack multiple 1×1 convolutional layers for nonlinear learning to reconstruct the hyperspectral image.

[0069] (4b) Reconstruction result consistency regularization. un The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image Multispectral image The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image and It should be consistent, so there is a consistent reconstruction loss L rec :

[0070]

[0071] Step 5: Optimization of network weight parameters.

[0072] Combining the above five modules, the overall loss function of the deep synchronous spectral unmixing network is formula (10), where α, β, and η are hyperparameters used to control the balance between various regularization terms. Finally, the gradient descent method is used to optimize the weight parameters of the deep synchronous spectral unmixing network until the overall loss function converges.

[0073] Loss = L rec +α(L Ahsi_sum2one +L Ahsi_sparse +L Amsi_sum2one +L Amsi_sparse )+βL A_same +ηL Adv (10)

[0074] Step 6: Test the spectral super-resolution reconstruction performance of the deep synchronous spectral unmixing network.

[0075] When the training is completed, the multispectral image to be super-resolved is input into the multispectral encoder module to extract its ground feature abundance information. The ground feature abundance information is then input into the hyperspectral decoder to reconstruct the required hyperspectral image and compare it with the real hyperspectral image to evaluate the quality of the reconstructed hyperspectral image.

[0076] On the other hand, an embodiment of the present invention also provides a spectral super-resolution reconstruction system based on a deep synchronous spectral unmixing network, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as described in the above technical solution.

[0077] The effect of the present invention can be further illustrated by the following experiments.

[0078] Experimental conditions:

[0079] The relevant experiments were performed with a CPU of 13th Gen. The test was carried out on a workstation with an i9-13900K CPU, 64G memory, RTX4090GPU, and Ubuntu 22.04.2LTS operating system.

[0080] Experimental content:

[0081] Experiments were conducted on the public Paiva University hyperspectral remote sensing dataset to test the spectral super-resolution reconstruction performance of the method of the present invention.

[0082] J-SLoL (Gao L, Hong D, Yao J, et al. Spectral superresolution of multispectral imagery with joint parse and low-rank learning [J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3): 2269-2280.), SSCNN (FuY, Zhang T, Zheng Y, et al. Joint camera spectral response selection and hyperspectral image) were used. recovery[J].IEEE Transactions on Pattern Analysis andMachine Intelligence,2022,44(1):256-272.),SSJSRN(Mei S,Jiang R,Li X,etal.Spatial and spectral joint super-resolution using convolutional neuralnetwork[J].IEEE Transactions on Geoscience and Remote Sensing,2020,58(7):4590-4603.), SSRAN(Zheng X, Chen W, Lu X.Spectral super-resolution of multispectral images using spatial-spectral residual attention network[J].IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-14.) is used as a comparison method to verify the effectiveness of the method of the present invention.

[0083] The present invention uses Peak Signal-to-Noise Ratio (PSNR), Spectral Angle Mapping (SAM), and Root Mean Square Error (RMSE) to measure the performance of each method. The performance comparison results of the method of the present invention and the comparative method are shown in Table 1. It can be seen from Table 1 that the present invention has achieved the best performance in all indicators.

[0084] Table 1 Quantitative performance of different spectral super-resolution reconstruction methods on the Pavia University dataset

[0085]

[0086] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network, characterized in that: The steps include: Step 1, construct a hyperspectral encoder module to extract the abundance information of the objects contained in the unpaired hyperspectral image; Step 2, construct a spectral downsampling module to estimate the spectral response function from the hyperspectral image to the multispectral image, downsample the hyperspectral image in the spectral dimension, and obtain a reconstructed multispectral image; Construct a multispectral discrimination module, and use adversarial learning strategy to identify whether the reconstructed multispectral image obtained by the spectrum downsampling module follows the same distribution as the multispectral image to be spectrally super-resolved. Step 3, construct a multispectral encoder module to extract abundance information from the reconstructed multispectral image; Step 4, construct a hyperspectral decoder module to reconstruct the corresponding hyperspectral image from the abundance information, that is, input the abundance information of the unpaired hyperspectral image in step 1 into the hyperspectral decoder module to reconstruct the corresponding hyperspectral image, and input the abundance information of the multispectral image in step 3 into the decoder module to reconstruct the corresponding hyperspectral image; Step 5, combining the above five modules to build a deep synchronous spectral unmixing network, and using the gradient descent method to optimize the weight parameters of the deep synchronous spectral unmixing network until convergence; Step 6: After the training is completed, the multispectral image to be spectrally super-resolved is input into the multispectral encoder module to extract its ground feature abundance information, and then the ground feature abundance information is input into the hyperspectral decoder to reconstruct the required hyperspectral image.

2. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network according to claim 1, characterized in that: The hyperspectral encoder module consists of multiple 1×1 convolutions and multi-head self-attention components stacked in series, where the 1×1 convolution is used to extract the abundance information of the hyperspectral image in the spectral dimension, and the multi-head self-attention component is used to introduce global spatial information.

3. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as claimed in claim 1, characterized in that: According to the linear spectral mixture model, in order to ensure that the abundance matrix of the hyperspectral image satisfies the non-negativity, the sum is 1, and the sparsity, the last layer of the hyperspectral encoder module is set to a truncation function to force the abundance values ​​of various objects to be in the interval [0,1] to meet the abundance non-negativity; further, a loss function L1 with a sum of 1 regularization is constructed based on the L1 norm. Ahsi_sum2one , which makes the abundance matrix A hsi The sum of abundances is 1; then the sparse regularized loss function L is constructed using the Kullback-Leibler divergence formula KL(·||·) Ahsi_sparse , which makes the abundance matrix sparse, where ε is a parameter representing sparsity; in, and ρ are two independent variables, W un ×H un is the hyperspectral image Y un Space size, B h is the hyperspectral image Y un The number of bands, k represents Y un The number of end members of the middle ground object.

4. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network according to claim 1, characterized in that: In step 2, the spectral degradation is modeled as spectral downsampling from the hyperspectral image to the multispectral image, which is expressed by the following formula: in, is the reconstructed multispectral image, Y un (i,j,b) represents the hyperspectral image Y un The reflectivity of the bth band in the i-th row, j-th column, is the spectral response function, B m is the number of bands of the multispectral image, B h is the number of bands of the hyperspectral image, and formula (4) is regarded as a 1×1 convolution operation, where C is the weight parameter; therefore, the spectral downsampling module is constructed using a 1×1 convolution layer, where the number of convolution kernels in the convolution layer is equal to the number of spectral bands of the multispectral image.

5. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as claimed in claim 3, characterized in that: The multispectral discrimination module is composed of multiple 1×1 convolutions and multi-head self-attention components stacked in series. The multispectral image X to be spectrally super-resolved is used as the supervision signal of the multispectral discrimination module for adversarial training. The loss function L of the multispectral discrimination module is adv for: Among them, L adv is the adversarial loss function, D(·) represents the multi-spectral discrimination module, It represents the expectation of the multispectral discrimination module on the judgment result after inputting the real multispectral image X. Represents the input reconstructed multispectral image Finally, the multi-spectral discrimination module expects its judgment results.

6. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as claimed in claim 5, characterized in that: The multispectral encoder module consists of multiple 1×1 convolutions and multi-head self-attention components stacked in series. In order to ensure that the abundance of the multispectral image satisfies the non-negative, sum-to-1, and sparsity, first, the last layer of the multispectral encoder module is set to a truncation function to force the abundance values ​​of each object to be in the interval [0,1]. Then, the abundance matrix A is msi Apply L that sums to 1 Amsi_sum2one Regularization term and sparsity L Amsi_sparse Regularization term; because Yes un Generated by the spectral downsampling module, and Y un Contains the same spatial distribution of features, so and Y un There should be the same abundance information, so the abundance consistency regularization term formula (8) is constructed to optimize the parameters of the multispectral encoding module; 7. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network according to claim 1, characterized in that: In step 4, according to the linear spectral unmixing model, the hyperspectral decoder module is designed as a 1×1 convolutional layer without an activation function, which is multiplied with the abundance matrix to obtain the reconstructed hyperspectral image; According to the nonlinear spectral unmixing model, the hyperspectral decoder module can also stack multiple 1×1 convolutional layers for nonlinear learning to reconstruct the hyperspectral image.

8. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network according to claim 6, characterized in that: The hyperspectral image Y un The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image Multispectral image The abundance matrix is ​​input into the hyperspectral decoder module to reconstruct the hyperspectral image and It is consistent, so there is a consistent reconstruction loss L rec :

9. The spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network according to claim 8, characterized in that: The loss function of the deep synchronous spectral unmixing network is as follows: Loss=L rec +α(L Ahsi_sum2one +L Ahsi_sparse +L Amsi_sum2one +L Amsi_sparse )+βL A_same +ηL Adv (10) Among them, α, β, and η are hyperparameters used to control the balance between various regularization terms.

10. A spectral super-resolution reconstruction system based on a deep synchronous spectral unmixing network, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a spectral super-resolution reconstruction method based on a deep synchronous spectral unmixing network as described in any one of claims 1 to 8.

Images