An extreme resolution ratio hyperspectral image panchromatic sharpening method

Abstract

The application discloses an extreme resolution ratio hyperspectral image panchromatic sharpening method, which comprises a panchromatic ladder convolution module and a hyperspectral ladder convolution module playing a main role in fusion at each stage. The panchromatic ladder convolution module inputs two panchromatic features of different resolutions, then performs feature extraction and fusion on the two features, and finally improves the resolution of the fused features to be consistent with the higher resolution panchromatic input. The hyperspectral ladder convolution module is based on high-resolution panchromatic features and low-resolution hyperspectral features, which fully fuse the two inputs, and improve the resolution of the fused features to be consistent with the input panchromatic features. The advantage of this is that the resolution improvement is decomposed into multiple sub-problems for solving, effectively reducing the learning difficulty of the sub-modules, and in each stage, the resolution difference between the hyperspectral features and the panchromatic features is small, and more sufficient fusion can be performed.

Description

Technical Field

[0001] This invention relates to the field of remote sensing images, and in particular to a method for panchromatic sharpening of hyperspectral images with extreme resolution ratios. Background Technology

[0002] Remote sensing images have wide applications in various fields such as resource exploration, geological analysis, and military applications. These applications also require remote sensing images to have high spatial and hyperspectral resolution. However, due to limitations in the physical structure of actual sensors, it is often impossible to obtain hyperspectral images with both high spatial and hyperspectral resolution simultaneously. To overcome this difficulty, hyperspectral image panchromatic sharpening technology was proposed. Its principle is to fuse a low spatial resolution hyperspectral image with a corresponding registered high spatial resolution panchromatic image to generate a satisfactory high spatial resolution hyperspectral image. Through years of effort by the panchromatic sharpening community, many types of hyperspectral image panchromatic sharpening methods have been developed. Traditional sharpening methods can be classified into the following categories: component substitution methods, multi-scale analysis methods, and variational optimization methods. Component substitution methods primarily achieve sharpening through domain transformations, with representative methods including principal component analysis (PCA) and Schmidt orthogonal transform. Multi-resolution analysis methods extract and re-inject spatial information from multiple scales, with representative methods including wavelet transform and Laplace pyramid transform. Variational optimization methods utilize variational theory to build a model and then iteratively solve it, with representative methods including nonnegative matrix factorization (NMF) and Bayesian algorithms. In recent years, with the improvement of hardware computing power, convolutional neural networks (CNNs) have seen significant development, and numerous CNN-based methods have emerged in the field of hyperspectral panchromatic sharpening, achieving performance far exceeding traditional methods and becoming a focus of attention.

[0003] Extreme resolution ratio hyperspectral image panchromatic sharpening is a branch of hyperspectral image panchromatic sharpening, addressing the sharpening problem when the ratio of panchromatic space resolution to hyperspectral space resolution is too large (generally greater than 8). This presents several new challenges: firstly, the significant spatial differences between panchromatic and hyperspectral images—with panchromatic images possessing rich spatial information while hyperspectral images contain only limited information—make fusion difficult; secondly, the relatively limited information provided by hyperspectral images means a large amount of information needs to be calculated, resulting in a large solution space and unstable results for each calculation. Faced with these challenges, most existing sharpening methods struggle to achieve satisfactory results. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a method for panchromatic sharpening of hyperspectral images with extreme resolution ratios, and more particularly to a method for panchromatic sharpening of hyperspectral images with extreme resolution ratios based on a multi-step progressive convolutional neural network.

[0005] This invention combines the relationship between hyperspectral and panchromatic images, performing multi-stage, progressive fusion of spatial and spectral features to improve panchromatic sharpening. The panchromatic staircase convolution module and the hyperspectral staircase convolution module play a key role in each stage of fusion. The panchromatic staircase convolution module takes two panchromatic features of different resolutions as input, extracts and fuses them, and finally improves the resolution of the fused features to match the higher-resolution panchromatic input. The hyperspectral staircase convolution module uses high-resolution panchromatic features and low-resolution hyperspectral features, fully fusing the two inputs and improving the resolution of the fused features to match the input panchromatic features. This approach decomposes the resolution improvement into multiple sub-problems, effectively reducing the learning difficulty of the sub-modules. Furthermore, the small resolution difference between hyperspectral and panchromatic features in each stage allows for more thorough fusion. In addition, the introduction of the panchromatic staircase module incorporates panchromatic information at multiple resolutions, further constraining the solution space and stabilizing the final sharpening effect.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A panchromatic sharpening method for hyperspectral images with extreme resolution ratios includes:

[0008] Acquire the original hyperspectral image data and the panchromatic image data registered with the original hyperspectral image data, and construct the training set and the test set;

[0009] Construct a panchromatic staircase convolutional module and a hyperspectral staircase convolutional module to further obtain a multi-staircase progressive convolutional neural network;

[0010] Set the hyperparameters of the multi-stage progressive convolutional neural network and initialize the weights and biases of the convolutions used by the multi-stage progressive convolutional neural network.

[0011] The low-resolution data in the training set is fed into a multi-stage progressive convolutional neural network. The network performs forward inference to obtain the predicted image. The loss function value is calculated by combining the predicted image with the corresponding label image in the dataset.

[0012] Optimize the loss function value, and when the loss function value converges to the minimum value, the obtained network weights and biases are taken as the optimal network parameters;

[0013] The constructed multi-stage progressive convolutional neural network and its optimal parameters are loaded, and the low-resolution data in the test set is input into the multi-stage progressive convolutional neural network to obtain the predicted high-resolution hyperspectral image.

[0014] Furthermore, an adaptive moment estimation algorithm is used to optimize the loss function value.

[0015] Furthermore, the mean absolute error is used as the loss function for the multi-stage progressive convolutional neural network.

[0016] Furthermore, the structure of the full-color step convolution module is as follows:

[0017] The low-resolution feature extraction convolutional layer Conv1_1 consists of 32 3×3 convolutional kernels that extract low-resolution feature maps. Input: Feature map

[0018] The low-resolution deconvolutional layer DConv1_2 consists of 32 3×3 convolutional kernels, which use feature maps... Input: Feature map

[0019] The high-resolution scale convolutional layer Conv2_1 consists of 32 3×3 convolutional kernels that scale with high-resolution feature maps. Input: Feature map

[0020] The high-resolution scale convolutional layer Conv2_2 consists of 32 5×5 convolutional kernels that scale with high-resolution feature maps. Input: Feature map

[0021] The high-resolution scale convolutional layer Conv2_3 consists of 32 7×7 convolutional kernels that scale with high-resolution feature maps. Input: Feature map

[0022] The high-resolution convolutional layer Conv2_4 consists of 64 3×3 convolutional kernels that output feature maps. and Input: Feature map

[0023] The high-resolution convolutional layer Conv2_5 consists of 32 1×1 convolutional kernels that output feature maps. Input: Feature map

[0024] The convolutional layer Conv3_1 is fused, which consists of 32 3×3 convolutional kernels that use feature maps... and Input: Feature map

[0025] The gradient operator Grad is used to obtain texture information from a feature image, which is obtained from a high-resolution feature map. As input, output gradient map D(i) ;

[0026] The convolutional layer Conv3_2 is fused, which consists of 32 3×3 convolutional kernels that use feature maps... and D (i) Input: Feature map

[0027] Furthermore, the hyperspectral step convolution module includes:

[0028] The feature extraction convolutional layer Conv1 consists of 64 3×3 convolutional kernels that extract low-resolution feature maps. Input: Feature map

[0029] The deconvolutional layer DConv2 consists of 64 3×3 convolutional kernels, which use feature maps... Input: Feature map

[0030] The Channel Attention Convergence Layer (CA) transforms the information relationships between channels of the input feature map into a one-dimensional attention vector representing the saliency of each channel. The input is a one-dimensional attention vector of length 64.

[0031] The Channel Adjustment Layer (CAA) applies the attention vectors obtained from the previous layer to the feature map, adjusting the information relationships between channels. Its input is a one-dimensional attention vector. and feature map Output adjusted feature map

[0032] The Conv3 convolutional layer consists of 64 3×3 convolutional kernels, which use feature maps... Input: Feature map

[0033] The Conv4 convolutional layer is fused, consisting of 64 3×3 convolutional kernels that use feature maps... and panchromatic feature map Input: Feature map

[0034] The spatial attention convergence layer (SA) transforms the input feature map into an attention mask representing the salience of each spatial pixel location, using the feature image... For input, output spatial attention mask M (i) ;

[0035] The spatial enhancement layer (SAug) applies the attention mask obtained from the upper layer to the feature map, enhancing the spatial information on the feature map. Spatial attention mask M (i) Input is the enhanced feature map, output is the enhanced feature map.

[0036] The Conv5 convolutional layer is a fusion layer consisting of 64 3×3 convolutional kernels that use feature maps... Input: Feature map

[0037] Furthermore, the structure of the multi-step progressive convolutional neural network includes three hyperspectral step convolutional modules, three panchromatic step convolutional modules, a spectral reconstruction convolutional layer, and a residual compensation layer.

[0038] Furthermore, in the structure of the multi-stage progressive convolutional neural network:

[0039] Three hyperspectral stepped convolutional modules are connected in sequence to sequentially achieve resolution enhancement and sharpening of the hyperspectral image in the first, second and third stages.

[0040] Three panchromatic step convolutional modules are connected in sequence to sequentially extract and fuse information at different resolutions of the panchromatic image in the first, second, and third stages.

[0041] The spectral reconstruction convolutional layer is used to take the output feature map of the last hyperspectral staircase convolutional module as input. Output feature map

[0042] Residual compensation layer, the low-resolution hyperspectral image sampled above this layer. and feature map As input, output a high spatial resolution hyperspectral image.

[0043] Furthermore, the spectral reconstruction convolutional layer comprises 64 1×1 convolutional kernels.

[0044] Furthermore, the construction of the training set and test set specifically involves:

[0045] In low-resolution hyperspectral image data and its registered low-resolution panchromatic image data, a suitable region is selected, and sub-block images are cropped from that region at fixed intervals without repetition. and The training set is formed by randomly shuffling the data; similarly, another non-overlapping region is selected from the low-resolution hyperspectral-panchromatic data pair, and several non-overlapping sub-block images are cropped from the region to form the test set.

[0046] Furthermore, hyperspectral images and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship:

[0047] h2 = h1 × r

[0048] w2 = w1 × r

[0049] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image. In the problem of panchromatic sharpening of hyperspectral images with extreme resolution ratios, h1 and w1 represent the height and width of the hyperspectral image, respectively, h2 and w2 represent the height and width of the panchromatic image, respectively, and b represents the number of spectral channels of the hyperspectral image.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) One of the core modules of the extreme resolution ratio hyperspectral image panchromatic sharpening method based on multi-step progressive convolutional neural network designed in this invention is the panchromatic step convolution module. This module gradually fuses the information of low-resolution panchromatic features and the high-resolution panchromatic feature information of multiple scales extracted by multiple convolutions of different sizes. In addition, the gradient information of the high-resolution panchromatic feature map is extracted by the gradient operator. This module can fully fuse panchromatic feature information of different scales, thereby effectively mining the spatial detail information of the panchromatic image.

[0052] (2) One of the core modules of the extreme resolution ratio hyperspectral image panchromatic sharpening method based on a multi-step progressive convolutional neural network designed in this invention is a hyperspectral stepwise convolutional module. This module gradually fuses hyperspectral features and high-resolution panchromatic features, and uses spatial-spectral attention to adjust the features, which can fully adjust for differences in spatial-spectral features and improve the robustness of the module. In addition, since each hyperspectral stepwise module only needs to improve the spatial resolution by a small ratio, the difference between the corresponding hyperspectral features and panchromatic features is small, so the fusion can be more thorough.

[0053] (3) The panchromatic sharpening method for hyperspectral images with extreme resolution ratios designed in this invention based on a multi-step progressive convolutional neural network is mainly aimed at the panchromatic sharpening problem of hyperspectral images with extreme resolution ratios. This network divides the large ratio spatial resolution enhancement and image fusion into multiple step-by-step convolutional modules. Each module only needs to complete a small ratio of spatial resolution enhancement and image fusion, which effectively improves the overall sharpening effect. In addition, the separate processing of panchromatic features and hyperspectral features enables the network to process spectral and spatial information more effectively. Finally, the artificial introduction of multiple redundant scales of panchromatic features can provide additional information and further constrain the size of the solution space, making the sharpening result more stable. Attached Figure Description

[0054] Figure 1This is a flowchart of an embodiment of the present invention for a panchromatic sharpening method for hyperspectral images with extreme resolution ratios based on a multi-step progressive convolutional neural network;

[0055] Figure 2 This is a structural diagram of the full-color stepped convolution module according to an embodiment of the present invention;

[0056] Figure 3 This is a structural diagram of the hyperspectral stepped convolution module according to an embodiment of the present invention;

[0057] Figure 4 This is a structural diagram of a multi-stage progressive convolutional neural network model, as shown in the embodiment of the present invention.

[0058] Figure 5(a) is an AVIRIS hyperspectral reference image, Figure 5(b) is an image processed by bicubic interpolation, Figure 5(c) is an image processed by matrix factorization algorithm, Figure 5(d) is an image processed by Bayesian algorithm, and Figure 5(e) is an image processed by the method described in this embodiment. Detailed Implementation

[0059] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0060] The hyperspectral image used in this experiment comes from AVIRIS satellite data. This image originally consisted of 224 bands, of which 176 were selected after removing the water vapor absorption bands. The spatial resolution ratio of the panchromatic image to the hyperspectral image is 5:1. The size of the panchromatic image is 395*185, and the size of the hyperspectral image is 79*37.

[0061] like Figures 1-4 As shown, a panchromatic sharpening method for hyperspectral images with extreme resolution ratios based on a multi-staircase progressive convolutional neural network includes the following steps:

[0062] Step 1: Read the raw hyperspectral image data and the panchromatic image data registered with it, and perform certain preprocessing on the data pairs;

[0063] Step 2: Filter the regions of the hyperspectral image data, select two non-overlapping regions, and pair them with the corresponding panchromatic image data regions to construct the training set and test set, respectively.

[0064] Step 3: Design a panchromatic step convolution module based on the characteristics of panchromatic image data;

[0065] Furthermore, based on the characteristics of panchromatic image data, specifically referring to panchromatic images having high spatial resolution but only one band, the focus is on processing spatial information.

[0066] Step 4: Design a hyperspectral stepped convolution module based on the characteristics of hyperspectral image data;

[0067] Furthermore, the hyperspectral image data is characterized by low spatial resolution but many bands, with a focus on processing spectral information.

[0068] Step 5: Design a multi-stage progressive convolutional neural network structure;

[0069] Step 6: Set the hyperparameters of the network, and initialize the weights and biases of the convolutions used by the multi-stage progressive convolutional neural network.

[0070] Step 7: Feed the low-resolution data in the training set into a multi-stage progressive convolutional neural network. The network performs forward inference to obtain the predicted image. The loss function is calculated using the mean absolute error as the loss function, combined with the predicted image and the corresponding label image in the dataset.

[0071] Step 8: Optimize the loss function value using the adaptive moment estimation algorithm. This algorithm updates the weights and biases of all convolutions in the network in reverse order based on the loss function value. Repeat steps 7 and 8 until the loss function value converges to its minimum.

[0072] Step 9: When the loss function value converges to the minimum value, save the network weights and biases at this point as the optimal network parameters;

[0073] Step 10: Load the designed multi-stage progressive convolutional neural network structure, and simultaneously load the optimal network parameters saved in Step 9. Input the low-resolution data in the test set into the multi-stage progressive convolutional neural network to obtain the predicted high-resolution hyperspectral image.

[0074] The hyperspectral image read in step 1 and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship:

[0075] h2 = h1 × r

[0076] w2 = w1 × r

[0077] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image. In the problem of panchromatic sharpening of hyperspectral images with extreme resolution ratios, r is generally greater than or equal to 8. h1 and w1 represent the height and width of the hyperspectral image, respectively, and h2 and w2 represent the height and width of the panchromatic image, respectively. b represents the number of spectral channels of the hyperspectral image.

[0078] The preprocessing in step 1 includes the following specific steps: constructing a suitable low-pass filter based on the sensor's modulation transfer function and frequency response, and then using this filter to process the original hyperspectral image data. and panchromatic image data A smoothing filter is applied, followed by downsampling of the smoothed hyperspectral-panchromatic data pairs to generate degraded low-resolution hyperspectral image data. and low-resolution panchromatic images

[0079] The detailed process of constructing the training and testing datasets in step 2 is as follows: The low-resolution hyperspectral image data obtained in step 1... And the corresponding low-resolution panchromatic image In the process, select a suitable region and crop out small sub-blocks of the image from that region at fixed intervals without repetition. and The training set is formed by randomly shuffling the data; similarly, another non-overlapping region is selected from the low-resolution hyperspectral-panchromatic data pair, and several non-overlapping sub-images are cropped from the region to form the test set.

[0080] The structure of the full-color step convolution module designed in step 3 is as follows:

[0081] The low-resolution feature extraction convolutional layer Conv1_1 uses 32 3×3 convolutional kernels to extract low-resolution feature maps. Input: Feature map The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0082] The low-resolution deconvolutional layer DConv1_2 uses 32 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: in and These represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0083] The high-resolution scale convolutional layer Conv2_1 uses 32 3×3 convolutional kernels to scale high-resolution feature maps. Input: Feature map The operation process can be represented as follows: Where W3 and B3 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0084] The high-resolution scale convolutional layer Conv2_2 uses 32 5×5 convolutional kernels to scale high-resolution feature maps. Input: Feature map The operation process can be represented as follows: Where W4 and B4 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0085] The high-resolution scale convolutional layer Conv2_3 uses 32 7×7 convolutional kernels to scale high-resolution feature maps. Input: Feature map The operation process can be represented as follows: W5 and B5 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0086] The high-resolution convolutional layer Conv2_4 uses 64 3×3 convolutional kernels, which are based on feature maps. and Input: Feature map The operation process can be represented as follows: Where W6 and B6 represent the weights and biases of the convolution kernel, respectively. This indicates that the feature maps are stitched together along the channel dimension. This represents a non-linear ReLU activation function;

[0087] The high-resolution convolutional layer Conv2_5 uses 32 1×1 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: W7 and B7 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0088] The convolutional layer Conv3_1 is fused, and this layer uses 32 3×3 convolutional kernels, which are based on feature maps. and Input: Feature map The operation process can be represented as follows: Where W8B8 represents the weights and biases of the convolution kernel, respectively. This indicates that the feature maps are stitched together along the channel dimension. This represents a non-linear ReLU activation function;

[0089] The gradient operator Grad is used to obtain the texture information, i.e., high-frequency information, of a feature image, which is obtained from a high-resolution feature map. As input, output gradient map D (i) The calculation process can be represented as follows: in, and These are the horizontal gradient feature map and the vertical gradient feature map, respectively, where ||·||2 is the L2 norm for each pixel.

[0090] The convolutional layer Conv3_2 is fused, and this layer uses 32 3×3 convolutional kernels, which are based on feature maps. and D (i) Input: Feature map The operation process can be represented as follows: W9 and B9 represent the weights and biases of the convolution kernel, respectively. This indicates that the feature maps are stitched together along the channel dimension. This represents a non-linear ReLU activation function.

[0091] The structure of the hyperspectral step convolution module designed in step 4 is as follows:

[0092] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels to extract low-resolution feature maps. Input: Feature map The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0093] The deconvolutional layer DConv2 uses 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: in and These represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0094] The Channel Attention Convergence Layer (CA) transforms the information relationships between channels of the input feature map into a one-dimensional attention vector representing the saliency of each channel. The input is a one-dimensional attention vector of length 64. Its operation process can be represented as follows: Where GP(·) represents global average pooling, FC(·) represents fully connected, and σ(·) is the non-linear sigmoid function, which is responsible for normalizing the values;

[0095] The Channel Adjustment Layer (CAA) applies the attention vectors obtained from the previous layer to the feature map, adjusting the information relationships between channels. Its input is a one-dimensional attention vector. and feature map Output adjusted feature map Its operation process can be represented as follows: Where ⊙ represents the channel-by-channel multiplication operation;

[0096] The Conv3 convolutional layer uses 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: Where W3 and B3 represent the weights and biases of the convolution kernel, respectively. This represents a non-linear ReLU activation function;

[0097] The Conv4 convolutional layer is a fusion layer that uses 64 3×3 convolutional kernels to transform feature maps. and panchromatic feature map Input: Feature map The operation process can be represented as follows: Where W4B4 represents the weights and biases of the convolution kernel, respectively. This indicates that the feature maps are stitched together along the channel dimension. This represents a non-linear ReLU activation function;

[0098] The spatial attention convergence layer (SA) transforms the input feature map into an attention mask representing the salience of each spatial pixel location, using the feature image... For input, output spatial attention mask M (i) Its operation process can be represented as: Where σ represents the nonlinear Sigmoid activation function;

[0099] The spatial enhancement layer (SAug) applies the attention mask obtained from the upper layer to the feature map, enhancing the spatial information on the feature map. Spatial attention mask M (i) Input is the enhanced feature map, output is the enhanced feature map. The process can be represented as follows: in This represents a pixel-by-pixel multiplication operation;

[0100] The Conv5 convolutional layer is a fusion layer that uses 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: W5 and B5 represent the weights and biases of the convolution kernel, respectively, and this layer has no activation function.

[0101] The structure of the multi-stage progressive convolutional neural network constructed in step 5 is as follows:

[0102] Hyperspectral Stepped Convolution Module-1. This module takes the original hyperspectral image as input and the output of the panchromatic stepped convolution module-1 as input, and performs the first stage of resolution enhancement and sharpening on the hyperspectral image.

[0103] Hyperspectral Stepped Convolution Module-2. Taking the outputs of Hyperspectral Stepped Convolution Module-1 and Panchromatic Stepped Convolution Module-2 as inputs, this module performs a second stage of resolution enhancement and sharpening of the hyperspectral image.

[0104] Hyperspectral Stepped Convolution Module-3. It takes the outputs of Hyperspectral Stepped Convolution Module-2 and Panchromatic Stepped Convolution Module-2 as inputs and outputs 64 feature maps. Achieve third-stage resolution enhancement and sharpening of hyperspectral images.

[0105] Panchromatic Stepped Convolution Module-1. Takes a first panchromatic image and a second panchromatic image as input, and performs the first-stage extraction and fusion of information from different resolutions of the panchromatic images.

[0106] Panchromatic Stepwise Convolution Module-2. Inputting the output of panchromatic stepwise convolution module-1 and the third panchromatic image, this module performs a second-stage extraction and fusion of information from different resolutions of the panchromatic image.

[0107] Panchromatic Stepwise Convolution Module-3. Taking the output of panchromatic stepwise convolution module-2 and the fourth panchromatic image as input, this module performs the third-stage extraction and fusion of information from different resolutions of the panchromatic image.

[0108] The spectral reconstruction convolutional layer Conv1 uses 64 1×1 convolutional kernels, which output feature maps from the hyperspectral staircase convolutional module-3. Input: Feature map The operation process can be represented as follows: W1 and B1 represent the weights and biases of the convolution kernel, respectively, and this layer has no activation function.

[0109] Residual compensation layer (RC). The main network does not directly reconstruct the high spatial resolution hyperspectral image, but rather its corresponding high-frequency information. Therefore, the low-resolution hyperspectral image sampled above this layer... and feature map The process of taking a high spatial resolution hyperspectral image as input and outputting it can be represented as follows: in This indicates a pixel-by-pixel addition operation.

[0110] Furthermore, the low-resolution hyperspectral image is obtained by directly upsampling the original input of the network.

[0111] The loss function selected in step 7 is the mean absolute error loss function, and its expression is:

[0112]

[0113] Where Ω represents the input-output mapping relationship of a multi-stage progressive convolutional neural network, θ represents the network parameters, and N... b Represents the number of training samples input in each iteration, ||·|| F This represents the Frobenius norm.

[0114] Specifically, this embodiment uses hyperspectral and panchromatic images of the University of Houston captured by a DiMAC ULTRALIGHT spectral imager to verify the effect of panchromatic sharpening. The hyperspectral image used covers the spectral range of 308-1050nm, with a total of 48 bands and a spatial resolution of 1m. The spatial resolution of the panchromatic image is 5cm, and the ratio of their spatial resolutions is 1:20.

[0115] Figure 5(a) is a reference image of the hyperspectral image from the Houston dataset; Figure 5(b) is a hyperspectral image obtained after upsampling using bicubic interpolation; Figure 5(c) is a hyperspectral image obtained after processing using principal component analysis; Figure 5(d) is an image processed using adaptive Gram-Schmidt orthogonal transform; and Figure 5(e) is a hyperspectral image obtained using the method described in this embodiment. As can be seen from the results, the image upsampled using only bicubic interpolation suffers from severe distortion of spatial spectral information, with blurred and smooth contours, making it difficult to discern details. While the image processed by principal component analysis and adaptive Gram-Schmidt orthogonal transform shows some improvement, it still suffers from significant blurring, and the contours remain difficult to identify. The hyperspectral image reconstructed using the method described in this embodiment exhibits the best performance, with ground features closely resembling the reference image, clearly visible, and without significant spectral differences. It not only demonstrates good spectral restoration in flat areas but also shows significant spatial detail enhancement in many areas with buildings.

[0116] In summary, the present invention provides a panchromatic sharpening method for hyperspectral images with extreme resolution ratios based on a multi-step progressive convolutional neural network. By combining the characteristics of hyperspectral and panchromatic images and controlling the spatial resolution enhancement, it can better reconstruct the spatial detail information missing in the hyperspectral image and effectively reduce the spectral distortion of the sharpened image.

[0117] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for panchromatic sharpening of hyperspectral images with extreme resolution ratios, characterized in that, include: Acquire the original hyperspectral image data and the panchromatic image data registered with the original hyperspectral image data, and construct the training set and the test set; Construct a panchromatic staircase convolutional module and a hyperspectral staircase convolutional module to further obtain a multi-staircase progressive convolutional neural network; The structure of the multi-step progressive convolutional neural network includes three hyperspectral step convolutional modules, three panchromatic step convolutional modules, a spectral reconstruction convolutional layer, and a residual compensation layer. In the structure of the multi-stage progressive convolutional neural network: Three hyperspectral step convolutional modules are connected in sequence to sequentially improve and sharpen the resolution of the first, second, and third stages of the hyperspectral image; three panchromatic step convolutional modules are connected in sequence to sequentially extract and fuse the first, second, and third stages of information at different resolutions in the panchromatic image. The spectral reconstruction convolutional layer is used to take the output feature map of the last hyperspectral staircase convolutional module as input. Output feature map ; Residual compensation layer, the low-resolution hyperspectral image sampled above this layer. and feature map As input, output a high spatial resolution hyperspectral image; The panchromatic stepped convolution module gradually fuses information from low-resolution panchromatic features with high-resolution panchromatic feature information from multiple scales extracted through convolutions of multiple different sizes, and then uses a gradient operator to extract gradient information from the high-resolution panchromatic feature map. The hyperspectral stepped convolution module gradually fuses hyperspectral features and high-resolution panchromatic features, and uses spatial-spectral attention to adjust the features, which can fully adjust for differences in spatial-spectral features and improve the robustness of the module. Set the hyperparameters of the multi-stage progressive convolutional neural network and initialize the weights and biases of the convolutions used by the multi-stage progressive convolutional neural network. The low-resolution data in the training set is fed into a multi-stage progressive convolutional neural network. The network performs forward inference to obtain the predicted image. The loss function value is calculated by combining the predicted image with the corresponding label image in the dataset. Optimize the loss function value, and when the loss function value converges to the minimum value, the obtained network weights and biases are taken as the optimal network parameters; The constructed multi-stage progressive convolutional neural network and its optimal parameters are loaded, and the low-resolution data in the test set is input into the multi-stage progressive convolutional neural network to obtain the predicted high-resolution hyperspectral image.

2. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, An adaptive moment estimation algorithm is used to optimize the loss function value.

3. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, The mean absolute error is used as the loss function of the multi-stage progressive convolutional neural network.

4. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, The structure of the full-color step convolution module is as follows: The low-resolution feature extraction convolutional layer Conv1_1 consists of 32 3×3 convolutional kernels that extract low-resolution feature maps. Input: Feature map ; The low-resolution deconvolutional layer DConv1_2 consists of 32 3×3 convolutional kernels, which decompose the feature maps. Input: Feature map ; The high-resolution scale convolutional layer Conv2_1 consists of 32 3×3 convolutional kernels that scale with high-resolution feature maps. Input: Feature map ; The high-resolution scale convolutional layer Conv2_2 consists of 32 5×5 convolutional kernels that scale with high-resolution feature maps. Input: Feature map ; The high-resolution scale convolutional layer Conv2_3 consists of 32 7×7 convolutional kernels that scale with high-resolution feature maps. Input: Feature map ; The high-resolution convolutional layer Conv2_4 consists of 64 3×3 convolutional kernels that output feature maps. , and Input: Feature map ; The high-resolution convolutional layer Conv2_5 consists of 32 1×1 convolutional kernels that output feature maps. Input: Feature map ; The convolutional layer Conv3_1 is fused, which consists of 32 3×3 convolutional kernels that use feature maps... and Input: Feature map ; The gradient operator Grad is used to obtain texture information from a feature image, which is obtained from a high-resolution feature map. Input: Gradient Map ; The convolutional layer Conv3_2 is fused, which consists of 32 3×3 convolutional kernels that use feature maps... and Input: Feature map .

5. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, The hyperspectral stepped convolution module includes: The feature extraction convolutional layer Conv1 consists of 64 3×3 convolutional kernels that extract low-resolution feature maps. Input: Feature map ; The deconvolutional layer DConv2 consists of 64 3×3 convolutional kernels, which use feature maps... Input: Feature map ; The Channel Attention Convergence Layer (CA) transforms the information relationships between channels of the input feature map into a one-dimensional attention vector representing the saliency of each channel. The input is a one-dimensional attention vector of length 64. ; The Channel Adjustment Layer (CAA) applies the attention vectors obtained from the previous layer to the feature map, adjusting the information relationships between channels. Its input is a one-dimensional attention vector. and feature map Output the adjusted feature map ; The Conv3 convolutional layer consists of 64 3×3 convolutional kernels, which use feature maps... Input: Feature map ; The Conv4 convolutional layer is fused, consisting of 64 3×3 convolutional kernels that use feature maps... and panchromatic feature map Input: Feature map ; The spatial attention convergence layer (SA) transforms the input feature map into an attention mask representing the salience of each spatial pixel location, using the feature image... For input, output space attention mask ; The spatial enhancement layer (SAug) applies the attention mask obtained from the upper layer to the feature map, enhancing the spatial information on the feature map. Spatial attention mask Input is the enhanced feature map, output is the enhanced feature map. ; The Conv5 convolutional layer is a fusion layer consisting of 64 3×3 convolutional kernels that use feature maps... Input: Feature map .

6. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, The spectral reconstruction convolutional layer consists of 64 1×1 convolutional kernels.

7. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to any one of claims 1-6, characterized in that, The construction of the training and test sets specifically involves: In low-resolution hyperspectral image data and its registered low-resolution panchromatic image data, a suitable region is selected, and sub-block images are cropped from that region at fixed intervals without repetition. and The training set is formed by randomly shuffling the order of the elements. Similarly, another non-overlapping region was selected from the low-resolution hyperspectral-panchromatic data pair, and several non-overlapping sub-block images were cropped from the region to form a test set.

8. The method for panchromatic sharpening of hyperspectral images with extreme resolution ratios according to claim 1, characterized in that, Hyperspectral images and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship: in This represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image. In the problem of panchromatic sharpening of hyperspectral images with extreme resolution ratios, and These represent the height and width of the hyperspectral image, respectively. and These represent the height and width of the panchromatic image, respectively. This represents the number of spectral channels in a hyperspectral image.

Images