A hyperspectral spatial spectrum quality enhancement method based on multi-level detail injection

Abstract

The present invention belongs to the technical field of image processing, and discloses a method for enhancing the quality of hyperspectral spatial spectrum based on multi-level detail injection, generating training images; constructing a dual-branch fusion network for training. Constructing a dual-branch fusion network to extract and fuse spatial details, and conducting training, extracting multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images; injecting details to reconstruct high-resolution hyperspectral images. In the detail injection part, the details extracted by the predefined gain coefficients will be injected into the upsampled hyperspectral image step by step. The present invention fuses the spatial features of low-resolution hyperspectral images and high-resolution panchromatic images through a dual-branch fusion network, and hierarchically processes low-resolution hyperspectral images and high-resolution panchromatic images in two bidirectional branches; for each layer, the spatial features captured from the high-resolution panchromatic image and the low-resolution hyperspectral image are combined and injected into the upsampled hyperspectral image to obtain a high-resolution hyperspectral image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for enhancing the quality of hyperspectral spatial spectrum. The obtained high-quality hyperspectral fusion image with improved spatial resolution and rich spatial spectrum information can be used for accurate classification of remote sensing images and target detection. Background Art

[0002] At present, hyperspectral imaging can obtain dozens or even hundreds of densely sampled spectral bands about the same scene from visible light to near-infrared light, which makes it possible to distinguish various materials with similar spectral characteristics on the surface. However, due to physical limitations, the spatial resolution of hyperspectral images is relatively low. Hyperspectral images with low spatial performance cannot meet the challenges posed by application tasks. Hyperspectral sharpening combines the spectral content of hyperspectral images and the spatial details of panchromatic images to obtain sharpened hyperspectral images with high spatial resolution and rich spectral information. It overcomes the disadvantage that a single sensor cannot meet both spectral and spatial requirements.

[0003] At present, image sharpening methods are mainly divided into the following four categories: classic component substitution (CS) based on simple models, multiresolution analysis (MRA) sharpening methods, and the recently proposed matrix decomposition and Bayesian sharpening methods based on complex models.

[0004] CS and MRA sharpening methods are simple and fast to implement and are very popular in large-scale real-time application systems. They can be represented by a unified model, that is, the entire sharpening process can be divided into two independent detail extraction processes and detail injection processes. The main difference between these two types of sharpening algorithms lies in the extraction of details. The CS sharpening method uses a panchromatic image to replace the intensity component of the hyperspectral image, while the MRA method directly extracts the high-frequency component of the panchromatic image and injects it into the low-resolution hyperspectral image to achieve sharpening. The widely used CS sharpening methods are Gram–Schmidt (GS), adaptive GS (GSA), intensity hue saturation (IHS) transform, principal component analysis (PCA), and guided filter PCA (GFPCA). Due to the mismatch of spectral range, the spectral quality of the sharpened image provided by the CS method is not ideal. Spectral range mismatch is an inherent problem of hyperspectral image sharpening, which affects the performance of hyperspectral image sharpening methods. The CS sharpening method that uses a panchromatic image instead of the intensity component of a hyperspectral image is greatly affected by the spectral range mismatch problem and produces significant spectral distortion. At present, the research on CS sharpening methods mainly focuses on the estimation of context-adaptive injection coefficients. The MRA sharpening algorithm has excellent spectral performance and is still under research. This type of method focuses on accurately estimating the missing spatial details of hyperspectral images.

[0005] Recently proposed Bayesian and matrix decomposition sharpening methods rely on regularization solutions to an ill-posed problem. Although Bayesian decomposition and matrix decomposition sharpening methods perform better in terms of spectral fidelity and spatial resolution than traditional sharpening methods, they are time-consuming. In addition, it is difficult to find a suitable statistical model to jointly characterize hyperspectral images, panchromatic images, and sharpened images, which greatly limits their application in sharpening. In order to reduce the computational complexity, Wei Q et al. proposed a fusion algorithm for fast solving optimization equations using the Sylvester equation in the literature "Wei Q, Dobigeon N, Tourneret J Y. Fast Fusion of Multi-Band Images Based on Solving a Sylvester Equation. IEEE Signal Processing Letters, 2016, 23(11): 1632-1636." The algorithm constructs the optimization equation into the Sylvester equation and designs a fast solution process. Although this type of fusion algorithm based on spectral separation and factorization can reduce the complexity of solving the sparse dictionary and Sylvester equation, it still has the disadvantage of relying on the selection of complex prior information of hyperspectral images.

[0006] Coupled nonnegative matrix factorization (CNMF) is a popular unmixing-based sharpening method, which was originally proposed to achieve the fusion of panchromatic and multispectral images. However, how to trade off between time complexity and sharpening performance remains a challenge for Bayesian and matrix factorization sharpening methods.

[0007] In recent years, deep neural networks, especially convolutional neural networks (CNNs), have been widely used in remote sensing image processing. C. Dong et al. first developed a super-resolution CNN (SRCNN) model in the journal "C. Dong, C. C. Loy, K. He, and X. Tang. Image Super-resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295–307, Feb. 2015." This is a milestone in the field of CNN in image super-resolution. So far, various CNN-based image super-resolution architectures have been designed. Sharpening can be regarded as a special case of super-resolution, which uses panchromatic and low-resolution hyperspectral images to reconstruct high-resolution hyperspectral images. In recent years, CNN-based hyperspectral image sharpening methods have also emerged. He et al. proposed a spectral prediction hyperspectral sharpening framework (HyperPNN) based on CNN in the document "L.He, JWZhu, J.Li, A.Plaza, J.Chanussot, and B.Li.HyperPNN:Hyperspectral Pansharpening Via Spectrally PredictiveConvolutional Neural Networks.IEEE J.Sel.Topics Appl.Earth Observ.RemoteSens.,vol.12,no.8,pp.3092–3100,Jan.2019." The framework introduces a spectral prediction structure to alleviate spectral distortion and spatial blur. Li et al. proposed a deep Laplacian hyperspectral sharpening method (DDLPS) in the document "K.Li, W.Xie, Q.Du, and Y.Li.DDLPS:Detail-based Deep Laplacian Pansharpeningfor Hyperspectral Imagery.IEEE Trans.Geosci.Remote Sens.,vol.57,no.10,pp.8011-8025,Oct.2019.", focusing on spatial and spectral fidelity.W. Dong et al. proposed a Laplacian pyramid dense network (LPPNet) for hyperspectral image sharpening in the journal "W. Dong, T. Zhang, J. Qu, S. Xiao, J. Liang, and Y. Li. Laplacian Pyramid Dense Network for Hyperspectral Pansharpening. IEEE Trans. Geosci. Remote Sens., April 2021, doi: 10.1109 / TGRS.2021.3076768." The Laplacian pyramid decomposition technology was applied to CNN, and the sharpening problem was simplified to several pyramid-level learning problems, which improved the efficiency of the network. However, most existing CNN-based sharpening methods train a specific CNN model for each type of dataset with the same number of spectral bands.

[0008] Through the above analysis, the problems and defects of the existing technology are as follows: the fused image obtained by the existing hyperspectral and panchromatic image fusion algorithm has too little improvement on spatial information and too much loss on spectral information, and cannot fully estimate the spatial details missing from the low-resolution hyperspectral image, and cannot achieve satisfactory results.

[0009] The significance of solving the above problems and defects is:

[0010] The present invention improves the spatial resolution of a hyperspectral image on the basis of retaining spectral information. The present invention adaptively estimates the missing details of the hyperspectral image through CNN, and mainly solves the defect that the existing sharpening algorithm cannot fully estimate the missing spatial details of a low-resolution hyperspectral image.

[0011] Most traditional CNN-based sharpening methods were originally designed for super-resolution and work like black boxes. The proposed hyperspectral spatial quality enhancement method based on multi-level detail injection has a clear physical explanation for hyperspectral image sharpening. Detail extraction is achieved through a two-branch fusion network, in which the first principal component (PC 1 ) is upsampled in one branch, while the full-color image is downsampled in another separate branch.

[0012] The existing DNN-based pan-sharpening algorithm extracts details from the panchromatic image as an additional frequency band of the low-resolution hyperspectral image, which obviously cannot extract the deep features of the panchromatic image. To solve this problem, the proposed hyperspectral spatial quality enhancement method based on multi-level detail injection designs a special branch with a residual network in the dual-branch fusion network to process the panchromatic image. In order to better maintain the spectral diversity, the extracted details are proportionally injected into the upsampled hyperspectral image to keep the spectral ratio of the original hyperspectral image in the fused hyperspectral image unchanged. Summary of the invention

[0013] In view of the problems existing in the prior art, the present invention provides a method for enhancing the quality of a hyperspectral spatial spectrum.

[0014] The present invention is implemented as follows: a method for enhancing the quality of a hyperspectral spatial spectrum, the method comprising: generating a training image, mainly used for training a model, so that the model obtains better prior information; constructing a dual-branch fusion network for training; constructing a dual-branch fusion network to extract and fuse spatial details, training, extracting multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images, this part is mainly used to extract spatial spectrum detail information that needs to be fused; injecting details, reconstructing a high-resolution hyperspectral image; in the detail injection part, the details extracted by a predefined gain coefficient will be injected step by step into the upsampled hyperspectral image, this part mainly fuses the extracted spatial spectrum details losslessly into the required high-resolution hyperspectral image.

[0015] Furthermore, generating training images includes: acquiring a public low-resolution hyperspectral image dataset and a high-resolution panchromatic image dataset.

[0016] Furthermore, we obtain the public low-resolution hyperspectral image dataset H∈R m×n×λ , and a high-resolution full-color image dataset P∈R M×N , where m×n represent the resolution of the hyperspectral image, λ is the number of spectral bands, and M×N is the resolution of the panchromatic image.

[0017] Furthermore, the construction of the dual-branch fusion network and the training thereof include: in the hyperspectral image sharpening algorithm based on PCA, the extracted PC 1 Contains the main spatial information of the HS image; the basic principle of the PCA-based sharpening method is to replace the PCA with a higher resolution image. 1 , and a sharpened image is obtained by inverse PCA spectral transformation; high spatial resolution panchromatic images are often used as an alternative to PCA. 1The ideal detail map of the image is obtained; the spatial component is extracted through a dual-branch fusion network, in which the spatial components of the hyperspectral image and the panchromatic image are processed hierarchically in two independent branches; the dual-branch fusion contains two branches: a panchromatic image detail extraction branch and a detail fusion branch.

[0018] Further including:

[0019] (1) The full-color image detail extraction branch consists of two levels. For the first level of the detail extraction branch, the full-color image is input into the convolutional layer to generate a 64-channel feature map. Then, the residual features are extracted through the residual network, and the output of the residual network is input into the convolutional layer. The output of the convolutional layer is the input of the second layer, which is used to fuse the feature map obtained by the second layer of the hyperspectral image detail extraction branch. The first level of the full-color image detail extraction branch is given by the following formula:

[0020]

[0021] The subscript 1 indicates the first layer, represents the output of the first layer, and f represents the convolutional layer and residual network function. There are two differences between the first and second layers: first, the input of the second level needs to pass through a maximum pooling layer to downsample the feature map to the size of the hyperspectral image; second, the output of the second level is fused with the output of the first level of the hyperspectral image detail extraction branch to reconstruct the spatial component. The second level of the panchromatic image detail extraction branch is given by the following formula:

[0022]

[0023] Among them, the subscript 2 indicates the second level, represents the output of the second level;

[0024] (2) Detail fusion branch, each level contains two convolutional layers and one deconvolutional layer:

[0025] At the first level of the detail fusion branch, PC 1 It is input into a convolutional layer containing 32 filters of size 3×3×1. The second convolutional layer contains 4 filters of size 3×3×32; when the feature map with 32 channels is input into the second convolutional layer, a feature map with 4 channels is obtained; the feature map with 4 channels is input into a deconvolutional layer, and the output of the deconvolutional layer is an interpolated feature map of double size, and the detail fusion branch outputs It is expressed as:

[0026]

[0027] Among them, g represents two layers of convolution and one layer of deconvolution operations;

[0028] Except for the input and output, the second level is the same as the first level. The second level of the detail fusion branch is expressed as:

[0029]

[0030] in is the output of the second layer of the detail fusion branch;

[0031] (3) Detail injection to reconstruct high-resolution hyperspectral images;

[0032] 1) Another key step in the CS and MRA-based sharpening method is detail injection, the injection gain g k The definition of is the core issue of detail injection; the injection gain is defined according to the ratio between spectral bands, and the injection gain G of the first-order k-th band is k,1 It is expressed as:

[0033]

[0034] Here, the subscript 1 indicates the number of stages, and the injection gain 2 according to the 4-fold upsampling is also defined in this way;

[0035] 2) Reconstruct high-resolution hyperspectral images;

[0036] After extracting the spectral details and calculating the injection gain, sharpen the image:

[0037]

[0038] Where H is a low-resolution hyperspectral image, and ↑ is an upsampling symbol;

[0039] (4) Construct the loss function and add the spectral angle mapper (SAM) to constrain the two-branch network. The overall loss function is described as:

[0040]

[0041] in, Indicates L 1 loss, α is set to 10 -3 Parameters, Indicates SAM loss.

[0042] Furthermore, the loss function is constructed: Let H, P, and F represent the low-resolution hyperspectral image, the high-resolution panchromatic image, and the reference hyperspectral image, respectively, and the training set is expressed as Where M represents the sample size, the purpose of training the network is to predict high-resolution hyperspectral images. And expressed as:

[0043]

[0044] Where m = 1, 2, ..., M, represents the mth predicted high-resolution hyperspectral image, f net The proposed method for enhancing the quality of hyperspectral spatial spectrum based on multi-level fusion is represented; the proposed method is optimized using the L1 loss function; the spectral angle mapper SAM constrained dual-branch network is added, and the overall loss function is described as:

[0045]

[0046] in, Indicates L 1 loss, α is set to 10 -3 Parameters, represents the SAM loss, expressed as:

[0047]

[0048] In the formula, and for and F (m) The spectral vector of the j-th pixel.

[0049] The object of the present invention is to provide a hyperspectral spatial spectrum quality enhancement system for implementing the hyperspectral spatial spectrum quality enhancement method, the hyperspectral spatial spectrum quality enhancement system comprising:

[0050] A training image generation module, used for generating training images;

[0051] A dual-branch fusion network construction module is used to construct a dual-branch fusion network for training;

[0052] Multi-level spatial detail extraction module, used to construct a dual-branch fusion network to extract and fuse spatial details, and to perform training to extract multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images;

[0053] Detail injection module for reconstructing high-resolution hyperspectral images.

[0054] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: The present invention includes: generating training images; constructing a dual-branch fusion network to extract and fuse spatial details, and training it. The two bidirectional branches of the detail extraction network extract multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images. The details of the hyperspectral and panchromatic images are extracted in two independent branches. The present invention has a great impact on the PC of hyperspectral images. 1Upsampling is performed and the panchromatic image is correspondingly downsampled to improve the efficiency of the network; detail injection is used to reconstruct a high-resolution hyperspectral image. In the detail injection part, the details extracted by the predefined gain coefficients will be injected step by step into the upsampled hyperspectral image.

[0055] The present invention improves the spatial resolution of a hyperspectral image on the basis of retaining spectral information. The present invention adaptively estimates the missing details of the hyperspectral image through CNN, and mainly solves the defect that the existing sharpening algorithm cannot fully estimate the missing spatial details of a low-resolution hyperspectral image.

[0056] Most traditional CNN-based sharpening methods were originally designed for super-resolution and work like black boxes. The proposed hyperspectral spatial quality enhancement method based on multi-level detail injection has a clear physical explanation for hyperspectral image sharpening. Detail extraction is achieved through a two-branch fusion network, in which the first principal component (PC 1 ) is upsampled in one branch, while the full-color image is downsampled in another separate branch.

[0057] The existing DNN-based pan-sharpening algorithm extracts details from the panchromatic image as an additional frequency band of the low-resolution hyperspectral image, which obviously cannot extract the deep features of the panchromatic image. To solve this problem, the proposed hyperspectral spatial quality enhancement method based on multi-level detail injection designs a special branch with a residual network in the dual-branch fusion network to process the panchromatic image. In order to better maintain the spectral diversity, the extracted details are proportionally injected into the upsampled hyperspectral image to keep the spectral ratio of the original hyperspectral image in the fused hyperspectral image unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a method for enhancing the quality of a hyperspectral spatial spectrum provided by an embodiment of the present invention.

[0059] Figure 2 It is a structural schematic diagram of a hyperspectral spatial spectrum quality enhancement system provided by an embodiment of the present invention;

[0060] Figure 2 In: 1. Training image generation module; 2. Dual-branch fusion network construction module; 3. Multi-level spatial detail extraction module; 4. Detail injection module.

[0061] Figure 3 It is a flow chart for implementing the method for enhancing the quality of high-spectral spatial spectrum provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] In view of the problems existing in the prior art, the present invention provides a method for enhancing the quality of a hyperspectral spatial spectrum, which is described in detail below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the hyperspectral spatial spectrum quality enhancement method provided by the present invention comprises the following steps:

[0065] S101: Generate training images;

[0066] S102: construct a dual-branch fusion network and perform training;

[0067] S103: Construct a dual-branch fusion network to extract and fuse spatial details, perform training, and extract multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images;

[0068] S104: Detail injection,reconstruction of high-resolution hyperspectral images.

[0069] The method for enhancing the quality of the hyperspectral space spectrum provided by the present invention can also be implemented by other steps by ordinary technicians in the industry. Figure 1 The hyperspectral spatial spectrum quality enhancement method provided by the present invention is only a specific embodiment.

[0070] like Figure 2 As shown, the hyperspectral spatial spectrum quality enhancement system provided by the present invention comprises:

[0071] The training image generation module 1 is used to generate training images.

[0072] The dual-branch fusion network construction module 2 is used to construct a dual-branch fusion network for training.

[0073] The multi-level spatial detail extraction module 3 is used to construct a dual-branch fusion network to extract and fuse spatial details, perform training, and extract multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images.

[0074] The detail injection module 4 is used to reconstruct high-resolution hyperspectral images.

[0075] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0076] like Figure 3As shown, the hyperspectral spatial spectrum quality enhancement method provided by the present invention includes three stages: preprocessing the hyperspectral image data set to generate training images; constructing a dual-branch fusion network and training it; injecting details to reconstruct high-resolution hyperspectral images. The specific implementation is as follows:

[0077] Step 1: Generate training images.

[0078] 1.1) Obtain a public low-resolution hyperspectral image dataset H∈R m×n×λ , and a high-resolution full-color image dataset P∈R M×N . Where m×n represents the resolution of the hyperspectral image, λ is the number of spectral bands, and M×N is the resolution of the panchromatic image.

[0079] Step 2: Build a dual-branch fusion network and train it.

[0080] Different from the existing PCA-based hyperspectral image sharpening methods, this method extracts the spatial component through a two-branch fusion network, where the spatial components of the hyperspectral image and the panchromatic image are hierarchically processed in two independent branches. In particular, the two-branch fusion contains two branches: a panchromatic image detail extraction branch and a detail fusion branch.

[0081] 2.1) Full-color image detail extraction branch:

[0082] The main task of the panchromatic detail extraction branch is to extract the spatial features of the panchromatic image layer by layer. Usually, the resolution of the required hyperspectral image with high spatial quality is four times that of the existing hyperspectral image. Therefore, in this case, the panchromatic image detail extraction branch consists of two levels.

[0083] For the first level of the detail extraction branch, the full-color image is input into the convolutional layer to generate a 64-channel feature map. The remaining features are then extracted through the residual network. In order to convert the multi-channel feature map into a single-channel detailed image, the output of the residual network is input into the convolutional layer. The output of this convolutional layer is the input to the second layer, which is also used to fuse the feature map obtained by the second layer of the hyperspectral image detail extraction branch. The first level of the full-color image detail extraction branch can be given by the following formula:

[0084]

[0085] The subscript 1 indicates the first layer, represents the output of the first layer, and f represents the convolutional layer and residual network function. There are two differences between the first and second layers: first, the input of the second level needs to pass through a maximum pooling layer to downsample the feature map to the size of the hyperspectral image; second, the output of the second level is fused with the output of the first level of the hyperspectral image detail extraction branch to reconstruct the spatial component. The second level of the panchromatic image detail extraction branch can be given by the following formula:

[0086]

[0087] Among them, the subscript 2 indicates the second level, Represents the output of the second stage.

[0088] 2.2) Detail fusion branch:

[0089] At the first level of the detail fusion branch, PC 1 It is input into a convolutional layer with 32 filters of size 3×3×1. The second convolutional layer contains 4 filters of size 3×3×32. Then, when the feature map with 32 channels is input into the second convolutional layer, a feature map with 4 channels is obtained. Then, the 4-channel feature map is input into a deconvolutional layer. The output of the deconvolutional layer is an interpolated feature map of double size. Detail fusion branch output It can be expressed as:

[0090]

[0091] Among them, g represents two layers of convolution and one layer of deconvolution operations.

[0092] Except for the input and output, the second level is the same as the first level. The second layer of the detail fusion branch can be expressed as:

[0093]

[0094] in It is the output of the second layer of the detail fusion branch.

[0095] Step 3: Detail injection, reconstructing high-resolution hyperspectral images.

[0096] 3.1) The injection gain G of the first stage k-th band k,1 It can be expressed as:

[0097]

[0098] Where the subscript 1 indicates the number of stages. In this case, it means 2 times upsampling. Similarly, the injection gain 2 based on 4 times upsampling is also defined in this way.

[0099] 3.2) Reconstruct high-resolution hyperspectral images:

[0100] After extracting the spectral details and calculating the injection gain, the sharpened image can be generated according to the sharpening model (1):

[0101]

[0102] Where H is a low-resolution hyperspectral image and ↑ is an upsampling symbol.

[0103] Step 4: Construct the loss function.

[0104] Let H, P, and F represent low-resolution hyperspectral images, high-resolution panchromatic images, and reference hyperspectral images, respectively. The training set is represented as Where M represents the sample size. The purpose of this training network is to predict high-resolution hyperspectral images. And represent it as:

[0105]

[0106] Where m = 1, 2, ..., M, represents the mth predicted high-resolution hyperspectral image, f net The proposed method for enhancing the quality of hyperspectral spatial spectrum based on multi-level fusion is shown in Figure 2. The proposed method is optimized using the L1 loss function. The present invention adds a spectral angle mapper (SAM) to constrain the dual-branch network. The overall loss function is described as:

[0107]

[0108] in, Indicates L 1 loss, α is set to 10 -3 Parameters, represents the SAM loss, which can be expressed as:

[0109]

[0110] In the formula, and for and F (m) The spectral vector of the j-th pixel.

[0111] The technical effects of the present invention are described in detail below in conjunction with experiments.

[0112] 1. Experimental conditions

[0113] The present invention is experimented on the Pavia Center dataset and the Columbia computer vision laboratory (CAVE) dataset.

[0114] The Pavia Center dataset hyperspectral image has a spectral range of 0.4-0.9 μm and provides data from 115 bands. After removing the water vapor absorption and noise bands, this paper uses 102 bands for experiments. Pavia Center has 1096×1096 pixels. This experiment intercepted a part with a size of 960×600×102 for the experiment.

[0115] CAVE dataset hyperspectral images. The CAVE dataset has 31 bands in the spectral range of 0.4 to 0.7 μm, and the image size is 512×512. The available hyperspectral images are used as reference images. Low-resolution hyperspectral images and high-resolution panchromatic images are generated according to the Wald protocol. Specifically, the panchromatic image is generated by averaging the visible bands of the reference image. The simulated hyperspectral image is obtained by applying a 9×9 Gaussian filter and then downsampling with a scale factor of 4. Therefore, the size of the simulated hyperspectral image is 128×128 pixels, while the size of the panchromatic image is 512×512.

[0116] To evaluate the performance of the fusion technique, we use several widely accepted objective metrics, including cross correlation (CC), spectral angle mapper (SAM), root mean squared error (RMSE), and error relative globaleadimensionnelle desynthse (ERGAS), to evaluate the sharpening performance.

[0117] The comparative experimental results of the present invention and other four multi- / hyperspectral image fusion methods on the Pavia Center dataset and CAVE dataset are shown in Tables 1 and 2.

[0118] Table 1 shows the values ​​of objective indicators for the Pavia Center dataset. Compared with other compared methods, the proposed hyperspectral spatial quality enhancement method based on fine-tuning the training model produces higher CC values ​​and lower SAM, RMSE and ERGAS values, indicating that the proposed method is superior to the latter.

[0119] Table 2 shows the values ​​of objective indicators for the CAVE dataset. The proposed hyperspectral spatial-spectral quality enhancement method based on fine-tuning the training model significantly improves the performance of degraded low-resolution hyperspectral images and achieves optimal values ​​in all objective indicators, indicating that the proposed method improves the spatial and spectral quality of the fused image.

[0120] Table 1. Performance indicators of the hyperspectral image sharpening results of the present invention and the four existing methods on the Pavia Center dataset

[0121]

[0122] Table 2. Performance indicators of the hyperspectral image sharpening results of the present invention and the four existing methods on the CAVE dataset

[0123]

[0124] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0125] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for enhancing the quality of a hyperspectral spatial spectrum, characterized in that: The hyperspectral spatial spectrum quality enhancement method comprises: generating a training image; constructing a dual-branch fusion network and performing training; constructing a dual-branch fusion network to extract and fuse spatial details, performing training, and extracting multi-level spatial details from a low-resolution hyperspectral image and a high-resolution panchromatic image; injecting details and reconstructing a high-resolution hyperspectral image; in the detail injection part, the details extracted by a predefined gain coefficient will be injected into the upsampled hyperspectral image step by step; The dual-branch fusion network is constructed and trained, including: in the hyperspectral image sharpening algorithm based on PCA, the extracted PC1 contains the main spatial information of the HS image; the basic principle of the PCA-based sharpening method is to replace PC1 with a higher resolution image, and obtain a sharpened image through inverse PCA spectral transformation; a panchromatic image with high spatial resolution is usually used as an ideal detail image to replace PC1; a spatial component is extracted through a dual-branch fusion network, wherein the spatial components of the hyperspectral image and the panchromatic image are hierarchically processed in two independent branches; the dual-branch fusion includes two branches: a panchromatic image detail extraction branch and a detail fusion branch; Further including: (1) The full-color image detail extraction branch consists of two levels. For the first level of the detail extraction branch, the full-color image is input into the convolutional layer to generate a 64-channel feature map. Then, the residual features are extracted through the residual network, and the output of the residual network is input into the convolutional layer. The output of the convolutional layer is the input of the second layer, which is used to fuse the feature map obtained by the second layer of the hyperspectral image detail extraction branch. The first level of the full-color image detail extraction branch is given by the following formula: The subscript 1 indicates the first layer. represents the output of the first layer, f represents the convolutional layer and residual network function, and there are two differences between the first and second layers: first, the input of the second level needs to pass through a maximum pooling layer to downsample the feature map to the size of the hyperspectral image; second, the output of the second level is fused with the output of the first level of the hyperspectral image detail extraction branch to reconstruct the spatial component. The second level of the panchromatic image detail extraction branch is given by the following formula: Among them, the subscript 2 indicates the second level, represents the output of the second level; (2) Detail fusion branch, each level contains two convolutional layers and one deconvolutional layer: In the first level of the detail fusion branch, PC1 is input into a convolutional layer containing 32 filters of size 3×3×1, and the second convolutional layer contains 4 filters of size 3×3×32; when the feature map with 32 channels is input into the second convolutional layer, a feature map with 4 channels is obtained; the 4-channel feature map is input into a deconvolution layer, and the output of the deconvolution layer is an interpolated feature map of double size. The detail fusion branch outputs It is expressed as: Among them, g represents two layers of convolution and one layer of deconvolution operations; Except for the input and output, the second level is the same as the first level, and the second layer of the detail fusion branch is expressed as: in is the output of the second layer of the detail fusion branch; (3) Detail injection to reconstruct high-resolution hyperspectral images; 1) Another key step in the CS and MRA-based sharpening method is detail injection, the injection gain g k The definition of is the core issue of detail injection; the injection gain is defined according to the ratio between spectral bands, and the injection gain G of the first-order k-th band is k,1 It is expressed as: Here, the subscript 1 indicates the number of stages, and the injection gain 2 according to the 4-fold upsampling is also defined in this way; 2) Reconstruct high-resolution hyperspectral images; After extracting the spectral details and calculating the injection gain, sharpen the image: Where H is a low-resolution hyperspectral image, and ↑ is an upsampling symbol; (4) Construct the loss function and add the spectral angle mapper (SAM) to constrain the two-branch network. The overall loss function is described as: in, represents L1 loss, α is set to 10 -3 Parameters, Indicates SAM loss.

2. The method for enhancing the quality of a hyperspectral spatial spectrum according to claim 1, characterized in that: The generating of training images comprises: acquiring a public low-resolution hyperspectral image dataset and a high-resolution panchromatic image dataset.

3. The method for enhancing the quality of a hyperspectral spatial spectrum according to claim 2, wherein: Obtain the public low-resolution hyperspectral image dataset H∈R m×n×λ , and a high-resolution full-color image dataset P∈R M×N , where m×n represent the resolution of the hyperspectral image, λ is the number of spectral bands, and M×N is the resolution of the panchromatic image.

4. The method for enhancing the quality of a hyperspectral spatial spectrum according to claim 1, wherein: Loss function construction: Let H, P, and F represent low-resolution hyperspectral images, high-resolution panchromatic images, and reference hyperspectral images, respectively. The training set is expressed as Where M represents the sample size, the purpose of training the network is to predict high-resolution hyperspectral images. And expressed as: In the formula, represents the mth predicted high-resolution hyperspectral image, f net The proposed method for enhancing the quality of hyperspectral spatial spectrum based on multi-level fusion is represented; the proposed method is optimized using the L1 loss function; the spectral angle mapper SAM constrained dual-branch network is added, and the overall loss function is described as: in, represents L1 loss, α is set to 10 -3 Parameters, represents the SAM loss, expressed as: In the formula, and for and F (m) The spectral vector of the j-th pixel.

5. A hyperspectral spatial spectrum quality enhancement system for implementing the hyperspectral spatial spectrum quality enhancement method according to any one of claims 1 to 4, characterized in that: The hyperspectral spatial spectrum quality enhancement system comprises: A training image generation module, used for generating training images; A dual-branch fusion network construction module is used to construct a dual-branch fusion network for training; The multi-level spatial detail extraction module is used to construct a dual-branch fusion network to extract and fuse spatial details, perform training, and extract multi-level spatial details from low-resolution hyperspectral images and high-resolution panchromatic images; the detail injection module is used to reconstruct high-resolution hyperspectral images.

Images