Deep spatial-spectral fusion method of hyperspectral images guided by component replacement model

Abstract

A component replacement model-guided deep spatial-spectral fusion method for hyperspectral images comprises the following steps: (1) obtaining publicly available high-resolution hyperspectral data X, preprocessing the high-resolution hyperspectral data X, and preparing a data set; (2) constructing a component replacement algorithm-driven spectral-spatial information fusion convolutional network and training it to enable the model to learn the nonlinear relationship between the fusion result of the hyperspectral image and the panchromatic image and the reference image; (3) inputting the low-spatial-resolution hyperspectral image and the panchromatic image to be fused obtained in the same scene into the trained network model to obtain a fused image with high spectral and high spatial resolution in order to verify the performance and generalization ability of the trained model. The present invention can reduce the spectral information loss of the hyperspectral image during the fusion process and obtain a fused image with both high spatial resolution and high spectral resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model. Background Art

[0002] Existing optical remote sensing imaging systems often struggle to balance spatial resolution, spectral resolution, and signal-to-noise ratio (SNR) in their data, resulting in the inability of the imaging platform to provide images with both high spatial and high spectral resolution. Therefore, integrating spatial and spectral information into a single image can significantly improve image quality. Reducing spectral information loss and making the model learning process interpretable when fusing panchromatic and hyperspectral images are unresolved challenges. Existing methods for spatial-spectral fusion of hyperspectral and panchromatic images can be broadly categorized into model-based and deep learning approaches.

[0003] Model-based methods can be broadly divided into two categories: component replacement methods and multiresolution analysis methods. Component replacement-based fusion methods primarily use a panchromatic image to replace the spatial components of an upsampled hyperspectral image according to certain rules, or they append the spatial information of the panchromatic image to the upsampled hyperspectral image according to established rules. Multiresolution-based methods employ wavelet transforms or other multiresolution analysis tools to decompose the hyperspectral image and its corresponding histogram-matched panchromatic image, respectively, to obtain high-frequency and low-frequency coefficients. These coefficients are then fused separately using specific fusion rules. Finally, an inverse multiresolution analysis transform is performed on the fused coefficients to obtain the fused image. These methods are sensitive to spatial correspondences. Local discrepancies arise when the spatial information of the panchromatic image does not match that of the low-resolution hyperspectral image.

[0004] In recent years, deep learning has also been widely used in remote sensing image fusion processing. Examples of deep learning-based remote sensing image fusion methods include: J. Peng et al. (in "PSMD-Net: A Novel Pan-Sharpening Method Based on a Multiscale Dense Network") proposed a multiscale dense fusion network that fully utilizes the local and global spatial features and spectral information of the original PAN and MS images; and M. Zhou et al. (in "PAN-Guided Band-Aware Multi-Spectral Feature Enhancement for Pan-Sharpening") designed a panchromatic image-guided band-aware multispectral feature enhancement module and a multi-focus feature fusion module to achieve efficient pan-sharpening. These deep learning-based fusion methods primarily rely on the design of network architectures. However, the "black-box" nature of deep learning methods makes it difficult for these methods to leverage prior knowledge of hyperspectral imagery. Furthermore, they lack appropriate mathematical tools to model the learning process of deep learning models, resulting in poor generalization.

[0005] Deep learning methods have powerful feature extraction capabilities, but their internal operating mechanisms are opaque during training and testing, lacking mathematical theoretical proof to explain the generation logic from input to output and the optimization methods of the model. This not only discards the use of prior information on hyperspectral images, but also limits the generalization ability of neural network models. Summary of the Invention

[0006] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a deep spatial-spectral fusion method of hyperspectral images guided by a component replacement model, so as to reduce the spectral information loss of hyperspectral images during the fusion process and obtain a fused image with both high spatial resolution and high spectral resolution; the proposed algorithm embeds the mathematical principles of the component replacement algorithm into the neural network and guides the construction of the model, clarifies the inherent generation mechanism of the network in the process of fusing spatial-spectral information, and realizes the interpretability of the deep learning fusion model.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A component replacement model guided deep spatial spectral fusion method for hyperspectral images includes the following steps:

[0009] (1) Obtain public high-resolution hyperspectral data X, preprocess the high-resolution hyperspectral data X and create a dataset; obtain a panchromatic image training set P and a low-spatial-resolution hyperspectral image training dataset H, and upsample the obtained H to obtain a hyperspectral image of the same size as P

[0010] (2) In order to enable the model to learn the nonlinear relationship between the fusion results of the hyperspectral image and the panchromatic image and the reference image, a spectral-spatial information fusion convolutional network driven by the component replacement algorithm is constructed and trained;

[0011] (3) In order to verify the performance and generalization ability of the trained model, the low spatial resolution hyperspectral image and panchromatic image to be fused obtained in the same scene are input into the trained network model to obtain a fused image with high spectral and high spatial resolution.

[0012] The step (1) is specifically as follows:

[0013] (1.1) Obtain openly downloaded high-resolution hyperspectral data X, and use the Wald protocol to sample the real reference image to generate the initial low spatial resolution hyperspectral image H for model training. t and the initial full-color image P t ;

[0014] (1.2) Let the high-resolution hyperspectral image X be the reference image, and the initial panchromatic image P be t The 160*160*1 panchromatic image training dataset P is obtained by cropping, and the initial low spatial resolution hyperspectral image H is t After cropping, a 40*40*102 hyperspectral image training dataset H is obtained;

[0015] (1.3) Use the bicubic interpolation algorithm to upsample the training image H to obtain a hyperspectral image of the same size as P

[0016] In the step (1.1): hyperspectral image data Where N is the number of bands of the high-resolution hyperspectral image X, m = w × h is the total number of pixels in a single band of the hyperspectral image X, and w and h represent the length and width of each band.

[0017] In the step (1.2): H = XBS; wherein represents the point spread function of the sensor, Indicates four times downsampling; N represents the number of bands of the high-resolution hyperspectral image X, m = w × h represents the number of pixels in each band of the hyperspectral image X, w and h represent the length and width of each band of the hyperspectral image X respectively; d represents the spatial scale ratio between the low-resolution hyperspectral training image H and the high-resolution hyperspectral image X, m / d 2 =w / d×h / d represents the number of single-band pixels in the low-resolution hyperspectral training image H, w / d and h / d are the length and width of each band of the low-resolution hyperspectral image H respectively;

[0018] The full-color image training dataset P is:

[0019] P = RX;

[0020] in, represents the spectral response function of the sensor,

[0021] The specific steps of step (2) are:

[0022] (2.1) Construct a spatial component extraction module and obtain the spectral overlap weight vector W between the measured spectral band and the full-color image based on the channel attention mechanism; then calculate the spectral overlap weight vector W by band. The value of and summed in the band dimension to obtain the hyperspectral image The spatial component I of

[0023] (2.2) Construct a histogram matching module with two layers of convolution, each with a kernel size of 3×3. The full-color image is histogram-matched with the spatial component I using this module to improve the correlation between the full-color image and the replacement component, thereby reducing the spectral distortion introduced by the fusion process.

[0024] (2.3) Construct an adaptive injection gain extraction module; concatenate the full-color image and the interpolated hyperspectral image in the channel dimension, and then perform two convolution calculations; use the channel attention module to calculate the scale factor ρ of the spectral band of the cascaded image; Solve the band ratio information of the hyperspectral image and multiply it by ρ to obtain the injection gain G;

[0025] (2.4) The matched full-color image is subtracted from the spatial component I and then multiplied by the injection gain G to obtain the injection spatial information H adjusted by G res , hyperspectral image With H res The fused image is obtained by adding two convolution calculations

[0026] (2.5) Construct the objective function based on the network input and reference image:

[0027]

[0028] Among them, f Fusion (·) represents the deep spatial-spectral fusion network driven by the component replacement model proposed in this paper, λ represents the number of samples, R represents the reference image, and θ represents the model parameters.

[0029] (2.6) Train the model and optimize the objective function. Input the low-resolution hyperspectral image and the full-color image into the spectral spatial information deep convolutional network, use the absolute error loss function for training, and use the gradient descent method to update the network parameters. When the loss function value converges or the training round is greater than the pre-defined value, the training is stopped to obtain the trained spectral spatial information deep convolutional network. The round setting is based on the estimated training time and is a fixed value. When training the network model, the learning rate is set to 1×10 -5 .

[0030] In the step (2.1):

[0031] The channel attention module CA1 consists of an average pooling layer, a maximum pooling layer, and two convolutional layers. The convolution kernel size of the two convolutional layers is 1×1, and the output of each layer is input into the activation function ReLU. Finally, the output is obtained by the activation function Sigmoid.

[0032] Interpolate the hyperspectral image Input into the channel attention module CA1 to obtain the gain coefficient for measuring the spectral information overlap between the spectral bands of the hyperspectral image and the panchromatic image:

[0033] Then, the intensity component I is represented by a linear combination of the interpolated hyperspectral image bands:

[0034] The step (2.2) is specifically as follows: the histogram matching module consists of two convolution layers, and the size of the convolution kernel is 3×3. After the full-color image is input into the histogram matching module, the intensity component I of the interpolated hyperspectral image is subtracted to obtain the spatial information PI required for fusion:

[0035]

[0036] After being processed by the histogram matching module, the correlation between P and I is extremely high, and the difference between the two is calculated to obtain the spatial information missing from the low-resolution hyperspectral image.

[0037] The step (2.3) is specifically as follows:

[0038] 2.3.1) Obtaining interpolated hyperspectral images The ratio of spectral information between each band is:

[0039]

[0040] 2.3.2) Obtain the constraint parameter ρ for controlling the injection gain: refer to Figure 3Construct a channel attention module CA2, which consists of an average pooling layer, a maximum pooling layer, and two convolutional layers; the size of the convolution kernel of the two convolution layers is 1×1, and each layer uses the linear function ReLU as the activation function; finally, the output is through the activation function SoftMax.

[0041] Concatenate the interpolated hyperspectral image with the panchromatic image in the channel dimension:

[0042]

[0043] in, represents the image of the kth band of the interpolated hyperspectral image, Cat(·) represents splicing in the channel dimension, and N represents The total number of bands;

[0044] Then first H cat Input two convolutional layers, and the convolution kernel size of each convolutional layer is 3×3; the obtained results are input into the channel attention module CA2 to obtain the constraint parameter ρ;

[0045] ρ=CA2(Conv(Conv(H cat )))

[0046] 2.3.3) Calculate the adaptive injection gain G:

[0047] G=ρ·Ratio

[0048] In the step (2.4), res The sum is then fed into two convolution layers with a convolution kernel size of 3×3 to obtain a fused image

[0049]

[0050] Where Conv(·) represents a convolutional layer with a convolution kernel size of 3×3.

[0051] Beneficial effects of the present invention:

[0052] 1. This invention combines the component replacement method with a neural network; it can solve the problem of the uninterpretability of deep neural networks and the need for parameters based on the component replacement method to be manually set based on experience or additional calculations, thereby improving the generalization performance of the network model.

[0053] 2. The present invention utilizes the learned ratio matrix between each band of the hyperspectral image to obtain the injection gain, effectively preserving the spectral information of the original hyperspectral image.

[0054] 3. This invention constructs a histogram matching module and performs histogram matching on the spatial components of the panchromatic and hyperspectral images, improving the correlation of spatial information between the two images and thereby obtaining the optimal spatial information required for fusion. This spatial information, when injected into the original hyperspectral image, can reduce the loss of spectral information.

[0055] 4. The present invention uses the channel attention mechanism to obtain the modulation coefficient matrix induced by the hyperspectral image band and the panchromatic band, which can fully consider the ratio of spatial information carried by the hyperspectral band and the panchromatic band and thus adjust the amount of injected spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is an implementation flow chart of the present invention.

[0057] Figure 2 It is a channel attention module used by the present invention to extract the intensity information of hyperspectral images.

[0058] Figure 3 It is a channel attention module of the present invention that solves the limiting parameters of the control space detail information injection amount.

[0059] Figure 4 The low-resolution hyperspectral image of the Pavia dataset to be fused in this invention and the panchromatic image of the same scene; (a) is a low-spatial-resolution Pavia hyperspectral image with a size of 40×40 and 102 bands; (b) is a panchromatic image of the same scene in (a) with a size of 160×160.

[0060] Figure 5 These are pseudo-color images of the results of fusing low-resolution hyperspectral images and panchromatic images of the Pavia dataset using the present invention and seven existing fusion methods. (a)-(h) are, in order, the fusion results of the PCA method, the GSA method, the CNMF method, the HyperPNN method, the UAL method, the MIDP method, the SFIIN method, and the present invention. DETAILED DESCRIPTION

[0061] The present invention will be described in further detail below with reference to the accompanying drawings.

[0062] Reference Figure 1 The present invention is a method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model, which includes two stages: image preprocessing in the first stage and training and testing a component replacement model guided deep spatial spectral fusion network of hyperspectral images in the second stage. The specific implementation is as follows:

[0063] Step 1: Image preprocessing.

[0064] 1.1) Generate training images:

[0065] 1.1.1) Acquiring hyperspectral image data Where N is the number of bands of the high-resolution hyperspectral image X, m = w × h is the total number of pixels in a single band of the hyperspectral image X, and w and h represent the length and width of each band.

[0066] 1.1.2) Generate low spatial resolution hyperspectral training image H by spatial downsampling:

[0067] H = XBS;

[0068] in represents the point spread function of the sensor, Indicates four times downsampling; N represents the number of bands of the high-resolution hyperspectral image X, m = w × h represents the number of pixels in each band of the hyperspectral image X, w and h represent the length and width of each band of the hyperspectral image X respectively; d represents the spatial scale ratio between the low-resolution hyperspectral training image H and the high-resolution hyperspectral image X, m / d 2 =w / d×h / d represents the number of pixels in a single band of the low-resolution hyperspectral training image H, w / d and h / d are the length and width of each band of the low-resolution hyperspectral image H, respectively.

[0069] 1.1.3) Generate a full-color image P by spectral sampling:

[0070] P = RX;

[0071] in represents the spectral response function of the sensor,

[0072] 1.1.4) Use the bicubic interpolation algorithm to upsample the low-resolution hyperspectral training image H to obtain an interpolated hyperspectral image of the same size as P.

[0073] Step 2: Build a component replacement model-guided deep spatial-spectral fusion network for hyperspectral images and train it.

[0074] 2.1) Extract the intensity component of the interpolated hyperspectral image:

[0075] Reference Figure 2The constructed channel attention module CA1 consists of an average pooling layer, a max pooling layer, and two convolutional layers. The convolution kernel size of both layers is 1×1, and the output of each layer is input into the ReLU activation function. The final output is activated by the Sigmoid function. The average pooling operation can encode the global statistical information of the features, while the max pooling operation can complement it by encoding the most salient features. In this way, max pooling and average pooling learn different feature information to achieve more refined channel attention.

[0076] Interpolate the hyperspectral image Input into the channel attention module CA1 to obtain the gain coefficient for measuring the spectral information overlap between the spectral bands of the hyperspectral image and the panchromatic image:

[0077] Then, the intensity component I is represented by a linear combination of the interpolated hyperspectral image bands:

[0078] 2.2) Constructing the histogram matching module:

[0079] The degree of correlation between the spatial components of the hyperspectral image and the panchromatic image will affect the spectral distortion of the fusion result, so the histogram matching operation is used to reduce the distortion.

[0080] The histogram matching module consists of two convolutional layers, and the size of the convolution kernel is 3×3. After the full-color image is input into the histogram matching module, the intensity component I of the interpolated hyperspectral image is subtracted to obtain the spatial information PI required for fusion:

[0081] After being processed by the histogram matching module, the correlation between P and I is extremely high, and the spatial information missing from the low-resolution hyperspectral image contained in P is obtained by subtraction.

[0082] 2.3) Construct injection gain extraction module:

[0083] 2.3.1) Obtaining interpolated hyperspectral images The ratio of spectral information between each band is:

[0084]

[0085] 2.3.2) The degree of spectral distortion of the fusion result is affected by the injection gain definition scheme. In order to achieve adaptive learning of injection gain to reduce spectral distortion, it is necessary to obtain the constraint parameter ρ that controls the injection gain: Figure 3Construct a channel attention module CA2, which consists of an average pooling layer, a maximum pooling layer, and two convolutional layers; the size of the convolution kernel of the two convolution layers is 1×1, and each layer uses the linear function ReLU as the activation function; finally, the output is through the activation function SoftMax.

[0086] The classic component replacement algorithm uses a simple global method to define the injection gain, while the injection gain obtained by the adaptive scheme proposed in this method can provide better reconstruction performance.

[0087] Concatenate the interpolated hyperspectral image with the panchromatic image in the channel dimension:

[0088]

[0089] in, represents the image of the kth band of the interpolated hyperspectral image, Cat(·) represents splicing in the channel dimension, and N represents The total number of bands.

[0090] Then first H cat Input two convolutional layers, and the convolution kernel size of each convolutional layer is 3×3; the obtained results are input into the channel attention module CA2 to obtain the constraint parameter ρ:

[0091] ρ=CA2(Conv(Conv(H cat )))

[0092] 2.3.3) Calculate the adaptive injection gain G:

[0093] G=ρ·Ratio

[0094] 2.4) In order to adjust the amount of spatial detail information required for fusion through the injection gain, the injected spatial information H is calculated using the obtained adaptive injection gain G and spatial information PI. res :H res =G·(PI)

[0095] Get the injection space information H adjusted by the adaptive injection gain res , which can reduce the degree of spectral distortion during fusion.

[0096] 2.5) With H res The sum is then fed into two convolution layers with a convolution kernel size of 3×3 to obtain a fused image

[0097]

[0098] Where Conv(·) represents a convolutional layer with a convolution kernel size of 3×3.

[0099] 2.6) Construct the objective function based on the network input and reference image:

[0100]

[0101] Among them, f Fusion (·) represents the deep spatial-spectral fusion network driven by the component replacement model proposed in this paper, λ represents the number of samples, R represents the reference image, and θ represents the model parameters.

[0102] 2.7) Train the model and optimize the objective function:

[0103] The low-resolution hyperspectral image and the panchromatic image are input into the spectral spatial information deep convolutional network, and the absolute error loss function is used for training. The gradient descent method is used to update the network parameters. When the loss function value converges or the training round is greater than the pre-defined value, the training is stopped to obtain the trained spectral spatial information deep convolutional network. The round setting is based on the estimated training time and is a fixed value. When training the network model, the learning rate is set to 1×10 -5 .

[0104] 2.8) Test model:

[0105] First, bicubic interpolation is used to upsample the low spatial resolution hyperspectral image H to the same size as the panchromatic image. Then the full-color image P and Input into the trained model, and finally obtain the desired fused image with high spatial and high spectral resolution

[0106]

[0107] in, Indicates that the image with the same size as the full-color image is obtained after interpolation operation; f Fusion (·) indicates the deep spatial-spectral fusion network driven by the component replacement model proposed in the present invention. The following is a detailed description of the technical effects of the present invention in conjunction with simulation experiments:

[0108] 1. Experimental conditions:

[0109] 1. Dataset

[0110] The Pavia dataset is hyperspectral data acquired by the German Airborne Reflectance Optical Spectroscopic Imager (ROSIS), covering a wavelength range of 430 to 860 nm. The dataset contains 115 bands, 13 of which were removed for practical purposes. The ground-truth reference image is 960 × 640 × 10². The ground-truth reference image is tiled into 12 images of 160 × 320 × 10². 139 training samples are selected from these images, and the remaining 3 are used for testing. These 12 images are further segmented and smoothly cut using an eight-step window into 21 overlapping images of 160 × 160 pixels. Finally, 139 samples are selected from these images for training and 63 for testing.

[0111] After the above processing, the dataset needs to use the Wald protocol to produce full-color images and low-resolution hyperspectral images on the segmented data.

[0112] 2. Simulation Experiment Conditions

[0113] The experimental basic environment and configuration of the present invention are shown in Table 1.

[0114] Table 1 Experimental basic environment and configuration

[0115]

[0116] 2. Evaluation criteria and experimental content:

[0117] 1. Evaluation Criteria

[0118] In addition to qualitative observations of subjective results, quantitative evaluations should also be performed to measure the performance of the image fusion model. In this paper, this experiment uses PSNR (peak signal-to-noise ratio), CC (correlation coefficient), SAM (spectral angle mapping), RMSE (root mean square error), and ERGAS (global relative spectral loss) to quantify the quality of the fused image. The larger the PSNR value, the better the quality of the fused image; the larger the CC value, the better the quality of the fused image; the smaller the SAM value, the higher the fidelity of the spectral information of the fused image; the smaller the RMSE value, the better the spectral information of the fused image is preserved; the smaller the ERGAS value, the better the spatial and spectral information of the fused image is preserved;

[0119] 2. Experimental content

[0120] The present invention and seven existing fusion methods are used to fuse low-resolution hyperspectral images with panchromatic images of the same scene. The results are as follows: Figure 5 shown.

[0121] It can be found that model-based methods such as PCA, GSA, and CNMF have serious edge blurring problems (such as Figure 5(a), (b)). HyperPNN, UAL, MIDP, SFIIN and other deep learning-based models perform better than traditional model-based methods. Although the spatial distortion problem in the fusion result is greatly reduced, the color distortion problem still exists, such as Figure 5 (d) and (g) have spectral distortion problems. The fusion result of the present invention has clearer shapes of objects and sharper edges, and is closest to the reference image in terms of spatial details and spectral fidelity.

[0122] The present invention and seven existing fusion methods fuse the Pavia low-resolution hyperspectral image with the panchromatic image of the same scene. The performance indicators of the obtained results are shown in Table 2:

[0123] Table 2. Performance indicators of Pavia hyperspectral image fusion results of the present invention and seven existing methods

[0124]

[0125] It can be seen from Table 2 that the CC value of the algorithm of the present invention is the largest, indicating that the spatial details of the hyperspectral image after fusion of the present invention are the richest; the spectral angle mapping SAM of the present invention is the smallest, indicating that the present invention has the least loss of spectral information and the spectral information of the hyperspectral image after fusion is the best; the root mean square error RMSE and global relative error ERGAS of the present invention are the smallest, indicating that whether evaluated from the global scale of space or spectrum, the fusion result obtained by the present invention has the best spatial spectrum information.

[0126] In summary, this invention is a deep spatial-spectral fusion method for hyperspectral images driven by a component replacement model. By encoding the mathematical model of the component replacement algorithm into a deep neural network, we obtain a deeply interpretable fusion model. This method enhances the spatial detail of hyperspectral images while reducing the loss of spectral information. The resulting fused image with high spatial and spectral resolution is well-suited for application in other visual fields. Furthermore, all parameters in the model can be automatically learned end-to-end.

[0127] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A component replacement model-guided deep spatial-spectral fusion method for hyperspectral images, characterized in that: The following steps are included: (1) Obtain public high-resolution hyperspectral data X, preprocess the high-resolution hyperspectral data X and create a dataset to obtain a full-color image training set P and a low-spatial-resolution hyperspectral image training dataset H, and upsample the obtained H to obtain a hyperspectral image of the same size as P. (2) Construct a spectral-spatial information fusion convolutional network driven by a component replacement algorithm and train it; (3) In order to verify the performance and generalization ability of the trained model, the low spatial resolution hyperspectral image and panchromatic image to be fused obtained in the same scene are input into the trained network model to obtain a fused image with high spectral and high spatial resolution; The specific steps of step (2) are: (2.1) Construct a spatial component extraction module and obtain the spectral overlap weight vector W between the measured spectral band and the full-color image based on the channel attention mechanism; then calculate the spectral overlap weight vector W by band. The value of and summed in the band dimension to obtain the hyperspectral image The spatial component I of (2.2) Construct a histogram matching module with two layers of convolution, each with a kernel size of 3×3. The full-color image is histogram-matched with the spatial component I using this module to improve the correlation between the full-color image and the replacement component, thereby reducing the spectral distortion introduced by the fusion process. (2.3) Construct an adaptive injection gain extraction module; concatenate the full-color image and the interpolated hyperspectral image in the channel dimension, and then perform two convolution calculations; use the channel attention module to calculate the scale factor ρ of the spectral band of the cascaded image; Solve the band ratio information of the hyperspectral image and multiply it by ρ to obtain the injection gain G; (2.4) The matched full-color image is subtracted from the spatial component I and then multiplied by the injection gain G to obtain the injection spatial information H adjusted by G res , hyperspectral image With H res The fused image is obtained by adding two convolution calculations (2.5) Construct the objective function based on the network input and reference image: Among them, f Fusion (·) represents the deep spatial-spectral fusion network driven by the component replacement model proposed in this paper, λ represents the number of samples, R represents the reference image, and θ represents the model parameters; (2.6) Training the model and optimizing the objective function; inputting the low-resolution hyperspectral image and the panchromatic image into the spectral spatial information deep convolutional network, using the absolute error loss function for training, and using the gradient descent method to update the network parameters, stopping the training when the loss function value converges or the number of training rounds is greater than a predefined value, and obtaining the trained spectral spatial information deep convolutional network, where the number of rounds is set according to the estimated training time and is a fixed value; when training the network model, the learning rate is set to 1×10 -5 .

2. The method for deep spatial spectral fusion of hyperspectral images guided by component replacement model according to claim 1, characterized in that: The step (1) is specifically as follows: (1.1) Obtain openly downloaded high-resolution hyperspectral data X, and use the Wald protocol to sample the real reference image to generate the initial low spatial resolution hyperspectral image H for model training. t and the initial full-color image P t ; (1.2) Let the high-resolution hyperspectral image X be the reference image, and the initial panchromatic image P be t The panchromatic image training dataset P is obtained by cropping, and the initial low spatial resolution hyperspectral image H t Perform cropping to obtain the hyperspectral image training dataset H; (1.3) Use the bicubic interpolation algorithm to upsample the training image H to obtain a hyperspectral image of the same size as P 3. The method for deep spatial spectral fusion of hyperspectral images guided by component replacement model according to claim 2, characterized in that: In the step (1.1): hyperspectral image data Where N is the number of bands of the high-resolution hyperspectral image X, m = w × h is the total number of pixels in a single band of the hyperspectral image X, and w and h represent the length and width of each band.

4. The method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model according to claim 2, characterized in that: In the step (1.2), H=XBS; wherein represents the point spread function of the sensor, Indicates four times downsampling; N represents the number of bands of the high-resolution hyperspectral image X, m = w × h represents the number of pixels in each band of the hyperspectral image X, w and h represent the length and width of each band of the hyperspectral image X respectively; d represents the spatial scale ratio between the low-resolution hyperspectral training image H and the high-resolution hyperspectral image X, m / d 2 =w / d×h / d represents the number of pixels in a single band of the low-resolution hyperspectral training image H, w / d and h / d are the length and width of each band of the low-resolution hyperspectral image H, respectively.

5. The method for deep spatial spectral fusion of hyperspectral images guided by component replacement model according to claim 2, characterized in that: The full-color image training dataset P is: P = RX; in, represents the spectral response function of the sensor, 6. The method for deep spatial spectral fusion of hyperspectral images guided by component replacement model according to claim 1, characterized in that: In the step (2.1): The channel attention module CA1 consists of an average pooling layer, a maximum pooling layer, and two convolutional layers. The output of each layer is input into the activation function ReLU. Finally, the output is obtained by the activation function Sigmoid. Interpolate the hyperspectral image Input into the channel attention module CA1 to obtain the gain coefficient for measuring the spectral information overlap between the spectral bands of the hyperspectral image and the panchromatic image: Then, the intensity component I is represented by a linear combination of the interpolated hyperspectral image bands:

7. The method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model according to claim 1, characterized in that: The step (2.2) is specifically as follows: the histogram matching module is composed of two convolutional layers; after the full-color image is input into the histogram matching module, the intensity component I of the interpolated hyperspectral image is subtracted to obtain the spatial information PI required for fusion: After being processed by the histogram matching module, the correlation between P and I is extremely high, and the difference between the two is calculated to obtain the spatial information missing from the low-resolution hyperspectral image.

8. The method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model according to claim 1, characterized in that: The step (2.3) is specifically as follows: 2.3.1) Obtaining interpolated hyperspectral images The ratio of spectral information between each band is: 2.3.2) Obtain the constraint parameter ρ for controlling the injection gain: Construct the channel attention module CA2, which consists of an average pooling layer, a maximum pooling layer, and two convolutional layers. Each of the two convolutional layers uses the linear function ReLU as the activation function. Finally, the output is activated by the SoftMax activation function. Concatenate the interpolated hyperspectral image with the panchromatic image in the channel dimension: in, represents the image of the kth band of the interpolated hyperspectral image, Cat(·) represents splicing in the channel dimension, and N represents The total number of bands; Then first H cat Input two convolutional layers, and the results are input into the channel attention module CA2 to obtain the constraint parameter ρ: ρ=CA2(Conv(Conv(H cat ))) 2.3.3) Calculate the adaptive injection gain G: G=ρ·Ratio.

9. The method for deep spatial spectral fusion of hyperspectral images guided by a component replacement model according to claim 1, characterized in that: In the step (2.4), With H res The sum is then sent to two convolutional layers to obtain a fused image Among them, Conv(·) represents the convolutional layer.

Images