Hyperspectral image panchromatic sharpening method based on convolutional neural network with explicit prior

Abstract

The application discloses a hyperspectral image panchromatic sharpening method based on an explicit prior convolutional neural network, which comprises the following steps: acquiring a hyperspectral image and a corresponding panchromatic image; performing image region division on the preprocessed hyperspectral image and the corresponding panchromatic image, so as to construct a training data set and a test data set; constructing an explicit prior convolutional neural network, wherein the explicit prior convolutional neural network comprises a spectral decision submodule, a spectral generation submodule, an explicit prior fusion submodule and a residual compensation layer; initializing the weights and the bias of the convolution included in the explicit prior convolutional neural network; performing forward inference on the explicit prior convolutional neural network according to the input of the training data set, predicting a sharpened image, calculating a loss function value, and obtaining an optimal explicit prior convolutional neural network. The method has important theoretical and practical application values in improving the panchromatic sharpening quality of the hyperspectral image.

Description

Technical Field

[0001] This invention relates to the field of remote sensing images, and in particular to a method for panchromatic sharpening of hyperspectral images based on explicit prior convolutional neural networks. Background Technology

[0002] Extreme resolution ratio hyperspectral image panchromatic sharpening is a branch of hyperspectral image panchromatic sharpening, addressing the sharpening problem when the ratio of panchromatic space resolution to hyperspectral space resolution is too large (generally greater than 8). This presents several new challenges: firstly, the significant spatial differences between panchromatic and hyperspectral images—with panchromatic images possessing rich spatial information while hyperspectral images contain only limited information—make fusion difficult; secondly, the relatively limited information provided by hyperspectral images means a large amount of information needs to be calculated, resulting in a large solution space and unstable results for each calculation. Faced with these challenges, most existing sharpening methods struggle to achieve satisfactory results.

[0003] Extreme resolution ratio hyperspectral image panchromatic sharpening, as a specialized branch of hyperspectral image panchromatic sharpening, focuses on solving image fusion problems when the spatial resolution of the panchromatic image is much higher than that of the hyperspectral image (typically a ratio greater than 8). In this specific case, several new challenges have been encountered in research and practice.

[0004] First, the vast difference in spatial information between panchromatic images and hyperspectral images, due to the rich spatial details of panchromatic images and the relative scarcity of spatial information in hyperspectral images, makes image fusion complex and difficult. Details in panchromatic images often lack direct counterparts in hyperspectral images, necessitating sharpening algorithms that can intelligently infer and reconstruct the missing spatial information. Second, the limited information provided by hyperspectral images means that the amount of information to be solved during sharpening is enormous, resulting in a broad solution space and increasing the instability of the solution process. Faced with these challenges, most existing sharpening methods often fail to achieve the desired results when dealing with hyperspectral images with extreme resolution ratios, thus hindering high-quality image fusion. Therefore, new theories and technologies are needed to better address image sharpening problems under such extreme conditions, enabling effective fusion of hyperspectral and panchromatic images and improving image usability and application value. Summary of the Invention

[0005] To overcome the problems of information loss and excessively large solution space in the panchromatic sharpening of hyperspectral images with extreme resolution ratios in the existing technology, the purpose of this invention is to provide a panchromatic sharpening method for hyperspectral images based on explicit prior convolutional neural networks.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks includes:

[0008] Acquire hyperspectral images and their corresponding panchromatic images, and preprocess the images;

[0009] The preprocessed hyperspectral image and its corresponding panchromatic image are divided into image regions. Two non-overlapping regions are selected to construct the training dataset and the test dataset, respectively.

[0010] A convolutional neural network based on explicit priors is constructed, which includes a spectral decision submodule, a spectral generation submodule, an explicit prior fusion submodule, and a residual compensation layer.

[0011] Set the hyperparameters of the network and initialize the weights and biases of the convolutions included in the convolutional neural network based on explicit priors;

[0012] During the training phase, the convolutional neural network based on explicit priors performs forward inference based on the input of the training dataset to predict the sharpened image and calculate the loss function value.

[0013] The loss function value is optimized, and the network is backpropagated based on the loss function value to update the weights and biases of all convolutions in the convolutional neural network based on explicit priors. When the loss function value converges to the minimum value, the optimal convolutional neural network based on explicit priors is obtained.

[0014] During the testing phase, data from the test dataset is input into the optimal convolutional neural network based on explicit priors to obtain high spatial resolution hyperspectral images.

[0015] further,

[0016] Spectral Decision Submodule: Input the original hyperspectral image and panchromatic image, output the land cover type probability vector for each spatial pixel location, thereby finding the most suitable land cover type;

[0017] Spectrum generation submodule: Input the original hyperspectral image and panchromatic image, as well as the corresponding spectral curves in the spectral library indicated by the spectral decision submodule, and output the predicted spectral image, which is consistent with the target reference image in spectral information;

[0018] Explicit Prior Fusion Submodule: Input the original hyperspectral image and panchromatic image, as well as the spectral image predicted by the spectral generation submodule, and output the feature map;

[0019] Residual Compensation Layer: The explicit prior fusion submodule does not directly reconstruct the high spatial resolution hyperspectral image of the target, but rather subtracts the upsampled low-resolution hyperspectral image from it. Detailed images.

[0020] Furthermore, the hyperspectral image and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship:

[0021] h2 = h1 × r

[0022] w2 = w1 × r

[0023] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image, h1 and w1 are the height and width of the hyperspectral image, respectively, while h2 and w2 represent the height and width of the panchromatic image, and b represents the number of spectral channels contained in the hyperspectral image.

[0024] Furthermore, the preprocessing includes:

[0025] Construct a low-pass filter based on the sensor modulation transfer function and frequency response for the raw hyperspectral image data. and panchromatic image data Perform smoothing;

[0026] The image resolution is further reduced by employing a downsampling method to simulate the degradation effect during the sensor imaging process, thereby generating degraded low-resolution hyperspectral image data. and panchromatic image

[0027] further,

[0028] Low-resolution hyperspectral image data and panchromatic image data In the process, multiple sub-images of the same size are cropped out at fixed intervals without repetition. and The training dataset is formed by randomly shuffling the sub-images.

[0029] Furthermore, including:

[0030] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps

[0031] Channel attention mask (CAM) layers are used to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, thus providing a basis for subsequent feature enhancement or suppression. It uses the feature map... The input is a one-dimensional attention mask of length 64.

[0032] The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationships between channels and enhancing the network's ability to capture key information; it uses channel attention masks... and feature maps As input, output a feature map after feature adjustment.

[0033] The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels to extract features from the convolutional layers. Input and output feature maps

[0034] The feature compression convolutional layer Conv3 uses 128 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps

[0035] The feature compression convolutional layer Conv4 uses 256 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps

[0036] The feature classification layer (CLA) further compresses the feature map and unfolds it into a one-dimensional vector. This vector is then non-linearly processed using a fully connected layer, and finally normalized using the softmax function to obtain the final probability vector.

[0037] Furthermore, the structure of the spectrum generation submodule is as follows:

[0038] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels and outputs the image G from the spectral library. (i) Input: Feature map

[0039] The convolutional generation layer (CGL) uses 64×h²×w² 3×3 convolutional kernels, which generate feature maps. As input, output is the spatially rearranged convolution kernel K. (i) ;

[0040] The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps

[0041] The feature extraction convolutional layer Conv3 uses 64 3×3 convolutional kernels to extract features from the convolutional layers. Input and output feature maps

[0042] The spectral image generation layer SIG utilizes the convolution kernel K generated by the convolution generation layer CGL. (i) For feature maps Perform pixel-by-pixel convolution operations and output a feature map with 64 channels.

[0043] The residual convolutional layer Res1 consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map

[0044] The Res2 residual convolutional layer consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map

[0045] The spectral prediction convolutional layer PConv uses b 3×3 kernels, which are based on feature maps. As input, output predicted spectral image O (i) .

[0046] Furthermore, the structure of the explicit prior fusion submodule is as follows:

[0047] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. Low-resolution panchromatic image and predicted spectral image O (i) Input and output feature maps

[0048] Channel attention mask (CAM) layers are used to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, expressed as a feature map. As input, output a one-dimensional attention mask of length 64.

[0049] The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationship between channels. and feature maps As input, output a feature map after feature adjustment.

[0050] The residual convolutional layer Res1 consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map

[0051] The Res2 residual convolutional layer consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map

[0052] The spectral reconstruction convolution Conv2 layer uses b 3×3 convolutional kernels, which are based on feature maps. The input is the predicted feature image.

[0053] Furthermore, the loss function value is optimized using an adaptive moment estimation algorithm.

[0054] Furthermore, the loss function adopted is the mean absolute error loss function.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1) One of the core modules of the extreme resolution hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks designed in this invention is the spectral decision submodule. This module analyzes the input low-resolution hyperspectral image and panchromatic image, and makes land cover category decisions pixel by pixel. In this process, the relationship between channels is adjusted through a channel attention mechanism to enhance the utility of high-contribution channels and suppress the expression of useless channels; in addition, feature information is condensed and the accuracy of decision-making is improved by reducing the spatial size and increasing the number of channels.

[0057] 2) One of the core modules of the extreme resolution hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks designed in this invention is the spectral generation submodule. This module is responsible for converting the spectral curves of ground features in the spectral library into corresponding reference spectral images. It utilizes dynamic convolution kernel technology to modulate the hyperspectral and panchromatic images using the spectral curves, enabling a thorough combination of the two. Furthermore, this module employs a residual structure, effectively reducing the learning difficulty and improving the accuracy of the predicted spectral images.

[0058] 3) One of the core modules of the extreme resolution hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks designed in this invention is the spectral generation submodule. This module predicts the difference between the target hyperspectral image and the upsampled low-resolution hyperspectral image by fusing the reference spectral image and the low-resolution hyperspectral image with their corresponding panchromatic images. By introducing attention and residual mechanisms, the spatial-spectral characteristics are effectively adjusted to make them closer to the target value.

[0059] 4) The panchromatic sharpening method for extreme resolution ratio hyperspectral images based on explicit prior convolutional neural networks designed in this invention is mainly aimed at the panchromatic sharpening problem of extreme resolution ratio hyperspectral images. This network decomposes the extreme resolution ratio panchromatic sharpening problem into three sub-problems: land cover category determination, reference spectral image generation, and panchromatic sharpening, and corresponds to three sub-networks. In this way, by introducing additional spectral information, the size of the solution space can be effectively constrained, making the sharpening results more stable. Attached Figure Description

[0060] Figure 1 This is a flowchart of a hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks according to an embodiment of the present invention.

[0061] Figure 2 This is a structural diagram of the spectral decision submodule in an embodiment of the present invention.

[0062] Figure 3 This is a structural diagram of the spectrum generation submodule in an embodiment of the present invention.

[0063] Figure 4 This is a structural diagram of the explicit prior fusion submodule in an embodiment of the present invention.

[0064] Figure 5 This is a structural diagram of a convolutional neural network based on explicit priors, according to an embodiment of the present invention.

[0065] Figure 6(a) is a Huston hyperspectral reference image, Figure 6(b) is an image processed by bicubic interpolation, Figure 6(c) is an image processed by matrix factorization algorithm, Figure 6(d) is an image processed by Bayesian algorithm, and Figure 6(e) is an image processed by the method described in this embodiment. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0067] like Figures 1-5 As shown, a panchromatic sharpening method for hyperspectral images with extreme resolution ratios based on explicit priors using convolutional neural networks includes the following steps:

[0068] Step 1: Read the observed hyperspectral image data and the corresponding panchromatic image data, and perform certain preprocessing on the data pairs;

[0069] Step 2: Filter the regions of the hyperspectral image and the corresponding panchromatic data, and select two non-overlapping regions to construct the training set and the test set, respectively.

[0070] Step 3: Design a convolutional neural network structure based on explicit priors, wherein the convolutional neural network structure based on explicit priors includes:

[0071] Based on the characteristics of hyperspectral and panchromatic image data, a spectral decision submodule is designed. The input is the original hyperspectral and panchromatic images, and the output is the land cover type probability vector for each spatial pixel location, thereby finding the best matching land cover type.

[0072] Based on the characteristics of hyperspectral and panchromatic image data, a spectral generation submodule is designed to take the original hyperspectral and panchromatic images as input, as well as the corresponding spectral curves in the spectral library indicated by the spectral decision submodule, and output the predicted spectral image, which is consistent with the target reference image in spectral information.

[0073] An explicit prior fusion submodule is designed based on the characteristics of hyperspectral and panchromatic image data. The input is the original hyperspectral and panchromatic images, as well as the spectral image predicted by the spectral generation submodule, and the output is a feature map.

[0074] Residual Compensation Layer: The explicit prior fusion submodule does not directly reconstruct the high spatial resolution hyperspectral image of the target, but rather subtracts the upsampled low-resolution hyperspectral image from it. Detailed images. The advantage of this is reduced learning difficulty and improved network efficiency. This process can be described as: in This indicates a pixel-by-pixel addition operation.

[0075] Step 4: Set the hyperparameters of the network, and initialize the weights and biases of the convolutions contained in the convolutional neural network based on explicit priors. The hyperparameters include non-learnable parameters such as learning rate and number of convolutions.

[0076] Step 5: During the training phase, the convolutional neural network based on explicit priors performs forward inference based on the input of the training set to predict the sharpened image. The mean absolute error loss function is used, and the loss function value is calculated with the sharpened image and the corresponding reference image as input.

[0077] Step 6: Optimize the loss value using the adaptive moment estimation algorithm, backpropagate the network based on the loss function value, and update the weights and biases of all convolutions in the network.

[0078] Step 7: Repeat steps 5 and 6 until the loss function value converges; when the loss function value converges to the minimum value, save the corresponding network weights and biases as the optimal network parameters.

[0079] Step 8: During the testing phase, load the convolutional neural network structure based on explicit priors, and simultaneously load the optimal network parameters described in Step 10. Input the low-resolution data from the test set into the convolutional neural network based on explicit priors to predict the final high spatial resolution hyperspectral image.

[0080] Furthermore, the hyperspectral image read in step 1 and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship:

[0081] h2 = h1 × r

[0082] w2 = w1 × r

[0083] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image, and in the case of panchromatic sharpening of hyperspectral images with extreme resolution ratios, this value is generally greater than or equal to 8; h1 and w1 are the height and width of the hyperspectral image, respectively, while h2 and w2 represent the height and width of the panchromatic image. b represents the number of spectral channels contained in the hyperspectral image.

[0084] Furthermore, the preprocessing in step 1 includes the following specific steps: constructing a low-pass filter based on the sensor modulation transfer function and frequency response for the original hyperspectral image data. and panchromatic image data Smoothing is then performed. Subsequently, downsampling techniques are applied to further reduce the image resolution, thereby simulating the degradation effect during sensor imaging. This series of steps produces degraded, low-resolution hyperspectral image data. and panchromatic image This laid the foundation for subsequent research on image restoration and sharpening.

[0085] Furthermore, the detailed process of constructing the suspended chain dataset and the test dataset is as follows:

[0086] Low-resolution hyperspectral image data obtained after low-pass filtering and downsampling. and panchromatic image data In the process, a representative region is selected, and multiple small-sized sub-blocks are cropped out at fixed intervals without repetition. and These sub-blocks are randomly shuffled to form the training set. Similarly, non-overlapping sub-images are cropped from another non-overlapping region to form the test set, thus ensuring the diversity and representativeness of the data for model training and evaluation.

[0087] The structure of the spectral decision submodule is as follows:

[0088] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels to extract features from the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively, and cat(·) represents concatenation by channel. Represents a non-linear ReLU activation function;

[0089] Channel Attention Mask (CAM) is a network component specifically designed to process the information relationships between channels in a feature map. The core function of this layer is to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, thus providing a basis for subsequent feature enhancement or suppression. It uses the feature map... As input, output a one-dimensional attention mask of length 64. The specific calculation process can be represented as follows: Where GP(·) represents global average pooling, FC(·) represents fully connected, and σ(·) is a non-linear sigmoid function responsible for normalizing the values;

[0090] The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationships between channels. This effectively learns and emphasizes feature channels that are more important to the current task, while suppressing less important channels, thereby enhancing the network's ability to capture key information. It utilizes the channel attention mask... and feature maps As input, output a feature map after feature adjustment. Specifically, the calculation process can be represented as follows: Where ⊙ represents the channel-by-channel multiplication operation;

[0091] The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels to extract features from the convolutional layers. Input and output feature maps The operation process can be represented as follows: W2 and B2 represent the weights and biases of the convolution kernel, respectively. Represents a non-linear ReLU activation function;

[0092] The feature compression convolutional layer Conv3 uses 128 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps The operation process can be represented as follows: W3 and B3 represent the weights and biases of the convolution kernel, respectively. Represents a non-linear ReLU activation function;

[0093] The feature compression convolutional layer Conv4 uses 256 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps The operation process can be represented as follows: W4 and B4 represent the weights and biases of the convolution kernel, respectively. Represents a non-linear ReLU activation function;

[0094] The feature classification layer (CLA) further compresses the feature map and unfolds it into a one-dimensional vector. This vector is then non-linearly processed using a fully connected layer, and finally normalized using the softmax function to obtain the final probability vector. The calculation process can be represented as follows: Here, GP(·) represents average pooling, FC(·) represents fully connected, and σ(·) is the non-linear sigmoid function, which is responsible for normalizing the values.

[0095] Furthermore, the structure of the designed spectrum generation submodule is as follows:

[0096] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels and outputs an image G from a spectral library. (i) Input: Feature map The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0097] The convolutional generation layer (CGL) uses 64×h²×w² 3×3 convolutional kernels, which generate feature maps. As input, output is the spatially rearranged convolution kernel K. (i) The operation process can be represented as follows: Among them W k and B k These represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0098] The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels to extract features from the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps The operation process can be represented as follows: W2 and B2 represent the weights and biases of the convolution kernel, respectively, and cat(·) represents concatenation by channel. Represents a non-linear ReLU activation function;

[0099] The feature extraction convolutional layer Conv3 uses 64 3×3 convolutional kernels to extract features from the feature maps. Input and output feature maps The operation process can be represented as follows: W3 and B3 represent the weights and biases of the convolution kernel, respectively. Represents a non-linear ReLU activation function;

[0100] Furthermore, the spectral image generation layer SIG utilizes the convolution kernel K generated by the convolution generation layer CGL. (i) For feature maps Perform pixel-by-pixel convolution operations and output a feature map with 64 channels. This process can be represented as: in This represents a pixel-by-pixel convolution operation. This represents a non-linear ReLU activation function.

[0101] The Res1 residual convolutional layer simplifies the learning task and improves efficiency by introducing a residual structure that learns only the difference between the input and output. This layer consists of two sets of 64 3×3 convolutional kernels, with the input feature map... Output feature map This process can be represented as: W4 and W5, and B4 and B5, represent the weights and biases of the convolutional kernel, respectively. Represents a non-linear ReLU activation function. This indicates a pixel-by-pixel addition operation.

[0102] The Res2 residual convolutional layer simplifies the learning task and improves efficiency by introducing a residual structure that learns only the difference between the input and output. This layer consists of two sets of 64 3×3 convolutional kernels, with the input feature map... Output feature map This process can be represented as: W6 and W7, and B6 and B7, represent the weights and biases of the convolutional kernel, respectively. Represents a non-linear ReLU activation function. This indicates a pixel-by-pixel addition operation;

[0103] The spectral prediction convolutional layer PConv uses b 3×3 convolutional kernels, which are based on feature maps. As input, output predicted spectral image O (i) The operation process can be represented as follows: W8 and B8 represent the weights and biases of the convolution kernel, respectively.

[0104] Furthermore, the structure of the explicit prior fusion submodule is as follows:

[0105] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels to extract features from the upsampled low-resolution hyperspectral image. Low-resolution panchromatic image and predicted spectral image O (i) Input and output feature maps The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively, and cat(·) represents concatenation by channel. Represents a non-linear ReLU activation function;

[0106] Channel Attention Mask (CAM) is a network component specifically designed to process the information relationships between channels in a feature map. The core function of this layer is to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, thus providing a basis for subsequent feature enhancement or suppression. It uses the feature map... As input, output a one-dimensional attention mask of length 64. The specific calculation process can be represented as follows: Where GP(·) represents global average pooling, FC(·) represents fully connected, and σ(·) is a non-linear sigmoid function responsible for normalizing the values;

[0107] The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationships between channels. This effectively learns and emphasizes feature channels that are more important to the current task, while suppressing less important channels, thereby enhancing the network's ability to capture key information. It utilizes the channel attention mask... and feature maps As input, output a feature map after feature adjustment. Specifically, the calculation process can be represented as follows: Where ⊙ represents the channel-by-channel multiplication operation;

[0108] The Res1 residual convolutional layer simplifies the learning task and improves efficiency by introducing a residual structure that learns only the difference between the input and output. This layer consists of two sets of 64 3×3 convolutional kernels, with the input feature map... Output feature map This process can be represented as: Where W2 and W3, and B2 and B3 represent the weights and biases of the convolution kernel, respectively. Represents a non-linear ReLU activation function. This indicates a pixel-by-pixel addition operation;

[0109] The Res2 residual convolutional layer simplifies the learning task and improves efficiency by introducing a residual structure that learns only the difference between the input and output. This layer consists of two sets of 64 3×3 convolutional kernels, with the input feature map... Output feature map This process can be represented as: W4 and W5, and B4 and B5, represent the weights and biases of the convolutional kernel, respectively. Represents a non-linear ReLU activation function. This indicates a pixel-by-pixel addition operation;

[0110] The spectral reconstruction convolution Conv2 layer uses b 3×3 convolutional kernels, which are based on feature maps. Input: Predicted feature image The operation process can be represented as follows: W6 and B6 represent the weights and biases of the convolution kernel, respectively.

[0111] In this embodiment, Mr. Han chose the mean absolute error loss function, whose expression is:

[0112]

[0113] Where Ω represents the input-output mapping relationship of the explicit prior convolutional neural network, θ represents the network parameters, and N b Represents the number of training samples input in each iteration, ||·|| F This represents the Frobenius norm.

[0114] This embodiment uses hyperspectral and panchromatic images of the University of Houston captured by a DiMAC ULTRALIGHT spectral imager to verify the effect of panchromatic sharpening. The hyperspectral image used covers the spectral range of 308-1050nm, with a total of 48 bands and a spatial resolution of 1m. The spatial resolution of the panchromatic image is 5cm, and the ratio of their spatial resolutions is 1:20.

[0115] Figure 6(a) shows the reference image, i.e., the original hyperspectral image, used to evaluate the quality of subsequent processing results. Figure 6(b) shows the hyperspectral image obtained by upsampling using bicubic interpolation. It can be observed that while this method improves the spatial resolution of the image, it leads to severe distortion of spatial spectral information, resulting in blurred and smooth image contours and difficulty in discerning details. Figures 6(c) and 6(d) show the hyperspectral images processed by principal component analysis and adaptive Gram-Schmidt orthogonal transform, respectively. These methods improve the image sharpening effect to some extent, but blurring and low contour recognition still exist, indicating limitations in detail recovery. Finally, Figure 6(e) presents the hyperspectral image processed using the method described in this embodiment. The results show that this method performs excellently in reconstructing hyperspectral images, with ground feature details very close to the reference image, clearly visible, and without significant spectral distortion. This method not only demonstrates good spectral recovery capabilities in flat areas but also achieves significant spatial detail enhancement in complex structural areas such as buildings. These results confirm the superiority and effectiveness of this method in panchromatic sharpening of hyperspectral images.

[0116] This method aims to effectively alleviate the problem of insufficient information and narrow the solution space by introducing additional spectral data as explicit priors, thereby significantly improving the sharpening effect. The core of this method consists of three key sub-modules: a spectral decision sub-module, a spectral generation sub-module, and an explicit prior fusion sub-module. First, the spectral decision sub-module receives a low-resolution hyperspectral image and its corresponding panchromatic image as input. Through pixel-by-pixel analysis, it identifies the types of land features in the image and determines their corresponding locations in the spectral library. This step provides precise guidance for subsequent spectral information recovery. Second, the spectral generation sub-module uses the low-resolution hyperspectral image, the panchromatic image, and selected spectral curves from the spectral library to generate a reference spectral image with consistent spectral characteristics. This reference spectral image provides important spectral information for the sharpening process, helping to reconstruct a high-quality hyperspectral image. Finally, the explicit prior fusion sub-module combines the low-resolution hyperspectral image, the panchromatic image, and the reference spectral image to recover the difference between the target hyperspectral image and the upsampled low-resolution hyperspectral image. Because of the introduction of the reference image as prior information, this sub-module can effectively assist the panchromatic sharpening process, recovering a more accurate target image. This method not only enriches the network's input but also helps constrain the solution space, making the sharpening results more stable and reliable. The method of this invention demonstrates significant advantages in handling panchromatic sharpening of hyperspectral images with extreme resolution ratios, providing a new solution for the field of remote sensing image processing.

[0117] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A hyperspectral image panchromatic sharpening method based on explicit prior convolutional neural networks, characterized in that, include: Acquire hyperspectral images and their corresponding panchromatic images, and preprocess the images; The preprocessed hyperspectral image and its corresponding panchromatic image are divided into image regions. Two non-overlapping regions are selected to construct the training dataset and the test dataset, respectively. A convolutional neural network based on explicit priors is constructed, which includes a spectral decision submodule, a spectral generation submodule, an explicit prior fusion submodule, and a residual compensation layer. Set the hyperparameters of the network and initialize the weights and biases of the convolutions included in the convolutional neural network based on explicit priors; During the training phase, the convolutional neural network based on explicit priors performs forward inference based on the input of the training dataset to predict the sharpened image and calculate the loss function value. The loss function value is optimized, and the network is backpropagated based on the loss function value to update the weights and biases of all convolutions in the convolutional neural network based on explicit priors. When the loss function value converges to the minimum value, the optimal convolutional neural network based on explicit priors is obtained. During the testing phase, data from the test dataset is input into the optimal convolutional neural network based on explicit priors to obtain high spatial resolution hyperspectral images. Spectral Decision Submodule: Input the original hyperspectral image and panchromatic image, output the land cover type probability vector for each spatial pixel location, and determine the land cover type; Spectrum generation submodule: Input the original hyperspectral image and panchromatic image, as well as the corresponding spectral curves in the spectral library indicated by the spectral decision submodule, and output the predicted spectral image, which is consistent with the target reference image in spectral information; Explicit Prior Fusion Submodule: Input the original hyperspectral image and panchromatic image, as well as the spectral image predicted by the spectral generation submodule, and output the feature map; Residual Compensation Layer: The explicit prior fusion submodule does not directly reconstruct the high spatial resolution hyperspectral image of the target, but rather subtracts the upsampled low-resolution hyperspectral image from it. Detailed images.

2. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The hyperspectral image and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship: in Represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image. and These represent the height and width of the hyperspectral image, respectively. and This represents the height and width of the panchromatic image. This represents the number of spectral channels contained in a hyperspectral image.

3. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The preprocessing includes: Construct a low-pass filter based on the sensor modulation transfer function and frequency response for the raw hyperspectral image data. and panchromatic image data Perform smoothing; The image resolution is further reduced by employing a downsampling method to simulate the degradation effect during the sensor imaging process, thereby generating degraded low-resolution hyperspectral image data. and panchromatic image .

4. The hyperspectral image panchromatic sharpening method according to claim 3, characterized in that, Low-resolution hyperspectral image data and panchromatic image data In the process, multiple sub-images of the same size are cropped out at fixed intervals without repetition. and The training dataset is formed by randomly shuffling the sub-image blocks.

5. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The spectral decision submodule includes: The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps ; Channel attention mask (CAM) layers are used to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, thus providing a basis for subsequent feature enhancement or suppression. It uses the feature map... The input is a one-dimensional attention mask of length 64. ; The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationships between channels and enhancing the network's ability to capture key information; it uses channel attention masks... and feature maps As input, output a feature map after feature adjustment. ; The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels to extract features from the convolutional layers. Input and output feature maps ; The feature compression convolutional layer Conv3 uses 128 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps ; The feature compression convolutional layer Conv4 uses 256 3×3 convolutional kernels with a stride of 2, which compress the feature maps. Input and output feature maps ; The feature classification layer (CLA) further compresses the feature map and unfolds it into a one-dimensional vector. This vector is then non-linearly processed using a fully connected layer, and finally normalized using the softmax function to obtain the final probability vector. .

6. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The structure of the spectrum generation submodule is as follows: The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels and outputs the image based on a spectral library. Input: Feature map ; The convolutional generation layer CGL uses 64 Each 3×3 convolutional kernel uses the feature map The input is a spatially rearranged convolutional kernel. ; The feature extraction convolutional layer Conv2 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. and for low-resolution panchromatic images Input and output feature maps ; The feature extraction convolutional layer Conv3 uses 64 3×3 convolutional kernels to extract features from the convolutional layers. Input and output feature maps ; The spectral image generation layer SIG utilizes the convolution kernels generated by the convolution generation layer CGL. For feature maps Perform pixel-by-pixel convolution operations and output a feature map with 64 channels. ; The residual convolutional layer Res1 consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map ; The Res2 residual convolutional layer consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map ; Spectral prediction convolution PConv, this layer uses Each 3×3 convolutional kernel uses the feature map Input: Predicted spectral image .

7. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The structure of the explicit prior fusion submodule is as follows: The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels on the upsampled low-resolution hyperspectral image. Low-resolution panchromatic image and predicted spectral images Input and output feature maps ; Channel attention mask (CAM) layers are used to transform the input multi-channel feature map into a one-dimensional attention mask. This mask characterizes the saliency of each channel, expressed as a feature map. As input, output a one-dimensional attention mask of length 64. ; The Channel Attention Masking (CAMA) layer applies the obtained channel attention mask to the feature map, adjusting the information relationship between channels. and feature maps As input, output a feature map after feature adjustment. ; The residual convolutional layer Res1 consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map ; The Res2 residual convolutional layer consists of two sets of 64 3×3 convolutional kernels, inputting the feature map. Output feature map ; Spectral reconstruction convolution Conv2, this layer uses Each 3×3 convolutional kernel uses the feature map The input is the predicted feature image.

8. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The loss function value is optimized using an adaptive moment estimation algorithm.

9. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The loss function used is the mean absolute error loss function.

Images