Convolutional neural network based panchromatic sharpening of hyperspectral images with extreme resolution ratio using implicit priors

Abstract

The application discloses a kind of extreme resolution ratio hyperspectral image panchromatic sharpening methods based on implicit priori of convolutional neural network, including reading original hyperspectral image and corresponding panchromatic image, and pretreatment is carried out;Using the hyperspectral image and panchromatic image after pretreatment, training data set and test data set are constructed;The implicit priori convolutional neural network based on is constructed, and the implicit priori convolutional neural network based on includes low-resolution ratio panchromatic sharpening submodule and space spectrum adjustment submodule;Using training data set, the implicit priori convolutional neural network is trained, and the optimal implicit priori convolutional neural network after training is obtained;Using test data set, input optimal implicit priori convolutional neural network and predict high spatial resolution hyperspectral image.The method of the application solves the problem of hyperspectral image panchromatic sharpening under extreme resolution ratio effectively by this step-by-step resolution improvement and space spectrum adjustment strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to remote sensing image processing, in particular to a method for panchromatic sharpening of hyperspectral images with extreme resolution ratio based on implicit prior convolutional neural network. BACKGROUND

[0002] With the continuous progress of remote sensing technology and the increasing maturity of algorithm theory, the requirements for spatial resolution and spectral resolution of remote sensing images are also increasing. However, due to the limitations of the physical structure of the sensor, it is still challenging to directly acquire high spatial resolution hyperspectral images. In order to overcome this difficulty, the conventional strategy is to collect hyperspectral images with lower spatial resolution and high spatial resolution single-band panchromatic images, and to fuse the two through panchromatic sharpening technology to generate hyperspectral images with high spatial resolution. In traditional hyperspectral image panchromatic sharpening methods, there are mainly three types: component substitution method, multi-resolution analysis method and model optimization based method. The component substitution method covers a series of domain transformation based techniques, such as principal component analysis and Schmidt orthogonal transformation, which realize image sharpening through domain transformation operations. The multi-resolution analysis method focuses on using wavelet transform and its derivative techniques, Laplacian pyramid transform, and multi-scale analysis means based on brightness smoothing filter adjustment algorithm. The model optimization based method usually involves sparse representation technology and Bayesian inference algorithm, which improves the sharpening effect by establishing mathematical models and optimizing parameters. These methods have their own advantages, but also have limitations, especially when dealing with hyperspectral images with extreme resolution ratio, traditional panchromatic sharpening technology may not achieve ideal results. Therefore, exploring new sharpening strategies to adapt to higher resolution requirements of remote sensing image processing tasks has become the focus of current research. In recent years, with the rapid development of convolutional neural network technology, its application in panchromatic sharpening field has become increasingly widespread. Numerous studies have shown that hyperspectral image panchromatic sharpening methods based on convolutional neural network significantly outperform traditional methods in sharpening effect. These deep learning based methods can effectively learn complex image features and automatically extract and utilize these features for image fusion and detail enhancement through end-to-end training process.

[0003] Due to the relatively simple and rapid development of panchromatic imaging sensor structure, there is a large spatial resolution difference between the panchromatic image and the hyperspectral image in some emerging unmanned aerial vehicle or aerospace hyperspectral imaging systems. Especially in the case of extreme resolution ratio (usually greater than 8), the spectral information of the hyperspectral image is significantly missing, and due to its low spatial resolution, it is difficult to achieve accurate registration with the high-resolution panchromatic image at each spatial position, thereby hindering the effective fusion of the two images. At present, most of the panchromatic sharpening methods of hyperspectral images are mainly designed for low resolution ratio conditions, and do not fully consider the information loss and feature matching problems under extreme resolution ratio conditions. These methods often cannot achieve ideal results when processing image fusion with high resolution difference. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present application is to provide an implicit prior-based convolutional neural network extreme resolution ratio hyperspectral image panchromatic sharpening method.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] An implicit prior-based convolutional neural network extreme resolution ratio hyperspectral image panchromatic sharpening method, comprising:

[0007] Read the original hyperspectral image and the corresponding panchromatic image, and perform preprocessing;

[0008] Use the preprocessed hyperspectral image and panchromatic image to construct a training data set and a test data set;

[0009] Construct an implicit prior-based convolutional neural network, which includes a low resolution ratio panchromatic sharpening submodule and a spatial-spectral adjustment submodule;

[0010] Use the training data set to train the implicit prior-based convolutional neural network to obtain a trained optimal implicit prior-based convolutional neural network;

[0011] Use the test data set to input the optimal implicit prior-based convolutional neural network to predict a high spatial resolution hyperspectral image.

[0012] Further, the implicit prior-based convolutional neural network includes a feature extraction convolutional layer, three low resolution ratio panchromatic sharpening submodules, and a spatial-spectral adjustment submodule.

[0013] Further,

[0014] The feature extraction convolutional layer Conv1 includes 64 convolutional kernels with a receptive field of 1x1, and inputs the up-sampled hyperspectral image and a low resolution panchromatic image output feature map

[0015] a first low resolution panchromatic sharpening sub-module, inputting the output feature map implementing a first stage of extreme resolution ratio sharpening with implicit prior;

[0016] a second low resolution panchromatic sharpening sub-module, inputting the output of the first stage of extreme resolution ratio sharpening with implicit prior, implementing a second stage of extreme resolution ratio sharpening with implicit prior;

[0017] a third low resolution panchromatic sharpening sub-module, inputting the output of the second stage of extreme resolution ratio sharpening with implicit prior, outputting a reconstructed spectral image O (i) implementing a third stage of extreme resolution ratio sharpening with implicit prior;

[0018] a spectral adjustment sub-module, inputting the output of the third stage of extreme resolution ratio sharpening with implicit prior, outputting a predicted high spatial resolution hyperspectral image H (i) .

[0019] Further,

[0020] the hyperspectral image and the panchromatic image have the following spatial size relationship:

[0021]

[0022] wherein r represents the ratio of the spatial resolution of the observed panchromatic image to the corresponding hyperspectral image, h1 and w1 represent the height and width of the observed hyperspectral image respectively, h2 and w2 represent the height and width of the observed panchromatic image respectively, and c represents the number of spectral channels of the observed hyperspectral image.

[0023] Further, the preprocessing comprises:

[0024] adopting a smoothing filter to process the original hyperspectral image and panchromatic image, specifically:

[0025] the observed hyperspectral image and the panchromatic image applying a low-pass filter based on frequency response adaptation to simulate the frequency response characteristics of the sensor, to obtain the smoothed hyperspectral image and panchromatic image;

[0026] the smoothed hyperspectral image and panchromatic image perform downsampling operation to obtain a low resolution hyperspectral image and a panchromatic image

[0027] Meanwhile, in order to restore the spatial resolution of the image, a polynomial interpolation method is used to up-sample the low-resolution hyperspectral image to obtain an up-sampled hyperspectral image

[0028] Further, a training data set and a test data set are constructed, specifically:

[0029] Regions are selected in the hyperspectral image that has undergone up-sampling processing, and the same number of sub-images of the same size are cropped from the low-resolution panchromatic image in the regions and

[0030] The cropped hyperspectral and panchromatic sub-images are randomly shuffled in order to constitute the training data set;

[0031] Next, sub-images are continuously cropped from the remaining regions of the up-sampled hyperspectral image and the low-resolution panchromatic image , and these sub-images are also used to constitute the test data set.

[0032] Further, the low-resolution-to-pan-sharpening sub-module includes a plurality of convolution layers of different sizes, fusion convolution layers, and residual convolution layers.

[0033] Further, the structure of the low-resolution-to-pan-sharpening sub-module specifically includes:

[0034] a small-scale convolution layer SConv1, which includes 64 convolution kernels with a receptive field of 3x3, input feature maps output feature maps

[0035] a small-scale convolution layer SConv2, which includes 64 convolution kernels with a receptive field of 3x3, input feature maps output feature maps

[0036] a medium-scale convolution layer MConv1, which includes 64 convolution kernels with a receptive field of 5x5, input feature maps output feature maps

[0037] a medium-scale convolution layer MConv2, which includes 64 convolution kernels with a receptive field of 5x5, input feature maps output feature maps

[0038] a large-scale convolution layer LConv1, which includes 64 convolution kernels with a receptive field of 7x7, input feature maps output feature maps

[0039] Large-scale convolutional layer LConv2, which contains 64 convolutional kernels with a receptive field of 7x7, input feature map output feature map

[0040] Fusion convolutional layer FConv, which contains 64 convolutional kernels with a receptive field of 3x3, input feature map and output feature map

[0041] Residual convolutional layer ResConv1, which includes two groups of convolutional kernels, each group including 64 convolutional kernels with a size of 3x3, which are used to extract local features in the input feature map and convert these features into an output feature map, input feature map output feature map

[0042] Residual convolutional layer ResConv2, which includes two groups of convolutional kernels, each group including 64 convolutional kernels with a size of 3x3, which are used to extract local features in the input feature map and convert these features into an output feature map, input feature map output feature map

[0043] Spectral reconstruction convolutional layer HRConv, which includes c convolutional kernels with a receptive field of 3x3, input feature map output reconstructed spectral image O (i) .

[0044] Further, the structure of the spatial-spectral adjustment sub-module is:

[0045] Feature extraction convolutional layer Conv1, which uses 64 convolutional kernels with a size of 3x3, which takes the reconstructed spectral image O (i) as input, output feature map

[0046] Channel attention extraction layer CAE, which converts the input multi-channel feature map into a one-dimensional attention mask vector, which can represent the significance of each channel, which takes the feature map as input, outputs a one-dimensional attention mask with a length of 64

[0047] Channel attention action layer CAA, which takes the channel attention mask and the feature map as input, outputs the feature-adjusted feature map

[0048] a spatial adjustment convolutional layer SConv1, which includes 64 convolutional kernels with a receptive field of 3x3, and an input feature map an output feature map

[0049] a spatial adjustment convolutional layer SConv2, which includes 1 convolutional kernel with a receptive field of 3x3, and an input feature map an output feature map

[0050] a spatial attention extraction layer SAE, which converts an input multi-channel feature map into a spatial attention mask with a spatial size of h2xw2, which can represent the saliency of each pixel position, and takes the feature map as input, and outputs a spatial attention mask

[0051] a spatial feature adjustment layer SFA, which applies the obtained channel attention mask to the feature map, and takes the spatial attention mask and the feature map as input, and outputs a feature-adjusted feature map

[0052] a residual convolutional layer ResConv1, which includes two groups of convolutional kernels, each group including 64 convolutional kernels with a size of 3x3, which are used to extract local features in an input feature map and convert the features into an output feature map, and the input feature map an output feature map

[0053] a residual convolutional layer ResConv2, which includes two groups of convolutional kernels, each group including 64 convolutional kernels with a size of 3x3, which are used to extract local features in an input feature map and convert the features into an output feature map, and the input feature map an output feature map

[0054] a reconstruction convolutional layer RConv, which includes c convolutional kernels with a receptive field of 3x3, and an input feature map an output target high-spatial-resolution hyperspectral image H (i) .

[0055] Further, the loss function used for training the convolutional neural network of the implicit prior is a mean absolute error loss function.

[0056] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0057] (1) The extreme resolution ratio hyperspectral image panchromatic sharpening method based on implicit prior provided by the application designs a low resolution ratio panchromatic sharpening submodule, which utilizes multiple convolution kernels of different sizes to perform multi-scale feature extraction on the input feature map in parallel. Through this multi-path parallel processing, the low resolution ratio panchromatic sharpening submodule can capture and integrate the multi-scale spatial detail information of the feature map. In order to further improve the abstraction and fusion ability of the features, the submodule also adopts a residual structure, so that the network can more effectively learn the residual mapping between the input and the output.

[0058] (2) The extreme resolution ratio hyperspectral image panchromatic sharpening method based on implicit prior provided by the application designs a spatial-spectral adjustment submodule, which focuses on detecting the spatial and spectral differences between the input feature map and the target hyperspectral image. By generating an attention mask, the submodule can identify the spatial-spectral differences of each region in the feature map and accordingly perform differential processing on the input feature map, thereby significantly enhancing the sharpening effect.

[0059] (3) The extreme resolution ratio hyperspectral image panchromatic sharpening method based on implicit prior provided by the application is based on the low resolution ratio panchromatic sharpening submodule and the spatial-spectral adjustment submodule to construct an implicit prior extreme resolution ratio sharpening convolutional neural network. The network decomposes the extreme resolution panchromatic sharpening task into multiple low resolution ratio panchromatic sharpening tasks and spatial-spectral fine-tuning tasks. In each low resolution ratio panchromatic sharpening task, the network only needs to improve a small resolution ratio, which greatly reduces the difficulty of feature matching. At the same time, the parameters of each submodule are pre-set after training with small resolution ratio data, and the mapping relationship contained therein serves as a kind of implicit prior knowledge, which helps to improve the overall sharpening effect. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The flowchart of the extreme resolution ratio hyperspectral image panchromatic sharpening method based on implicit prior of the embodiment of the application;

[0061] Figure 2 The structural diagram of the low resolution ratio panchromatic sharpening submodule of the embodiment of the application;

[0062] Figure 3 The structural diagram of the spatial-spectral adjustment submodule of the embodiment of the application;

[0063] Figure 4 The structural diagram of the implicit prior extreme resolution ratio sharpening convolutional neural network model of the embodiment of the application;

[0064] Fig. 5(a) is a Huston hyperspectral reference image, Fig. 5(b) is an up-sampled spectral image processed using a bicubic interpolation method, Fig. 5(c) is an image processed using a guided filter-based principal component analysis algorithm, Fig. 5(d) is an image processed using a brightness smoothing filter adjustment algorithm, and Fig. 5(e) is an image processed using the method described in the embodiment. DETAILED DESCRIPTION

[0065] The application will be further described in conjunction with the embodiments, but the embodiments of the application are not limited thereto.

[0066] EMBODIMENT

[0067] As shown in the accompanying drawings, a method for panchromatic sharpening of hyperspectral images with extreme resolution ratio based on an implicit prior convolutional neural network includes the following steps: Figures 1-4 P1, reading an original hyperspectral image and a corresponding panchromatic image, and performing specific preprocessing operations thereon;

[0068] P2, using the preprocessed hyperspectral image and panchromatic image to construct a training data set and a test data set;

[0069] P3, designing a low-resolution ratio panchromatic sharpening sub-module and a spectral adjustment sub-module, and constructing an implicit prior convolutional neural network based on the modules;

[0070] P4, setting hyperparameters and initializing parameters of the implicit prior convolutional neural network;

[0071] P5, based on the training data set, selecting a suitable loss function and an optimization algorithm to train the implicit prior convolutional neural network;

[0072] The loss function selected in the step P5 is an average absolute error loss function, and its expression is as follows:

[0073]

[0074] wherein Ω represents an input-output mapping relationship of the implicit prior extreme resolution ratio sharpening convolutional neural network, θ represents parameters of the network, N b represents the number of training samples input in each iteration, and ||·||F F represents a Frobenius norm.

[0075] P6, based on the test data set, using the trained implicit prior convolutional neural network to predict a hyperspectral image with high spatial resolution.

[0076]

[0077] ​Further, the observed hyperspectral image read in step P1 and the corresponding panchromatic image There is the following relationship in spatial size:

[0078]

[0079] where r represents the ratio of spatial resolution of the observed panchromatic image to the corresponding hyperspectral image, hi and wi represent the height and width of the observed hyperspectral image respectively, h2 and w2 represent the height and width of the observed panchromatic image respectively. c represents the number of spectral channels of the observed hyperspectral image.

[0080] Further, the pre-processing includes the following steps:

[0081] A smoothing filter is applied to process the original hyperspectral image and panchromatic image. The observed hyperspectral image and the panchromatic image A low-pass filter based on frequency response adaptation is applied to simulate the frequency response characteristics of the sensor. This step effectively smooths the image data, reducing high-frequency noise while preserving important spectral and spatial information of the image. Subsequently, down-sampling operations are performed on the smoothed hyperspectral image and panchromatic image to simulate the image spatial resolution due to the resolution limitation of the sensor, obtaining a low-resolution hyperspectral image and a panchromatic image with spatial resolution degraded by r times. Meanwhile, in order to restore the spatial resolution of the image, a polynomial interpolation method is used to up-sample the low-resolution hyperspectral image to obtain an up-sampled hyperspectral image

[0082] Further, the detailed process of constructing the training data set and the test data set in P2 is as follows: specific regions are selected in the up-sampled hyperspectral image , and the same number of small size sub-images are cropped from the low-resolution panchromatic image and These sub-images are cropped from the original image without overlap according to a fixed sampling interval to ensure the diversity and representativeness of the training samples. Subsequently, these cropped hyperspectral and panchromatic sub-images are randomly shuffled in order to constitute the training data set. Then, sub-images are continuously cropped from the remaining regions of the up-sampled hyperspectral image and the low-resolution panchromatic image These sub-images are also used to constitute the test data set. In this way, it is ensured that the test data set contains different image regions from the training data set, so as to better evaluate the generalization ability of the model.

[0083] Further, the convolutional neural network based on implicit prior includes a feature extraction convolutional layer, three low-resolution ratio panchromatic sharpening sub-modules, and a spectral adjustment sub-module.

[0084] The three low-resolution ratio panchromatic sharpening sub-modules are respectively a first low-resolution ratio panchromatic sharpening sub-module, a second low-resolution ratio panchromatic sharpening sub-module, and a third low-resolution ratio panchromatic sharpening sub-module.

[0085] Specifically:

[0086] The feature extraction convolutional layer Conv1 includes 64 convolutional kernels with a receptive field of 1*1. The input is the up-sampled hyperspectral image and the low-resolution panchromatic image The output is a feature map The operation process can be represented as where cat(·) represents concatenation in the channel dimension, W1 and B1 represent the weights and bias of the convolutional kernel of the layer, respectively, represents a nonlinear Relu activation function;

[0087] The first low-resolution ratio panchromatic sharpening sub-module inputs the feature map to realize the extreme resolution ratio sharpening of the first-stage implicit prior;

[0088] The second low-resolution ratio panchromatic sharpening sub-module inputs the output of the extreme resolution ratio sharpening of the first-stage implicit prior to realize the extreme resolution ratio sharpening of the second-stage implicit prior;

[0089] The third low-resolution ratio panchromatic sharpening sub-module inputs the output of the extreme resolution ratio sharpening of the second-stage implicit prior and outputs the reconstructed spectral image O (i) to realize the extreme resolution ratio sharpening of the third-stage implicit prior;

[0090] The spectral adjustment sub-module inputs the output of the extreme resolution ratio sharpening of the third-stage implicit prior and outputs the predicted high-spatial-resolution hyperspectral image H (i) .

[0091] Further, the low-resolution ratio panchromatic sharpening sub-module includes:

[0092] The small-scale convolutional layer SConv1 includes 64 convolutional kernels with a receptive field of 3*3. The input is the feature map The output is a feature map The operation process can be described as where W1 and B1 are the weights and bias of the convolutional kernel, respectively, represents a nonlinear Leaky Relu activation function;

[0093] Small-scale convolutional layer SConv2, which contains 64 convolutional kernels with a receptive field of 3x3. The input feature map Output feature map The operation process can be described as where W2 and B2 are the weights and bias of the convolutional kernel, respectively, representing the non-linear Leaky Relu activation function;

[0094] Medium-scale convolutional layer MConv1, which contains 64 convolutional kernels with a receptive field of 5x5. The input feature map Output feature map The operation process can be described as where W3 and B3 are the weights and bias of the convolutional kernel, respectively, representing the non-linear Leaky Relu activation function;

[0095] Medium-scale convolutional layer MConv2, which contains 64 convolutional kernels with a receptive field of 5x5. The input feature map Output feature map The operation process can be described as where W4 and B4 are the weights and bias of the convolutional kernel, respectively, representing the non-linear Leaky Relu activation function;

[0096] Large-scale convolutional layer LConv1, which contains 64 convolutional kernels with a receptive field of 7x7. The input feature map Output feature map The operation process can be described as where W5 and B5 are the weights and bias of the convolutional kernel, respectively, representing the non-linear Leaky Relu activation function;

[0097] Large-scale convolutional layer LConv2, which contains 64 convolutional kernels with a receptive field of 7x7. The input feature map Output feature map The operation process can be described as where W6 and B6 are the weights and bias of the convolutional kernel, respectively, representing the non-linear Leaky Relu activation function;

[0098] Fusion convolutional layer FConv, which contains 64 convolutional kernels with a receptive field of 3x3. The input feature map and Output feature map The operation process can be described as where W7 and B7 are the weights and bias of the convolution kernel respectively, cat(·) represents concatenation along the channel dimension, represents a nonlinear Leaky Relu activation function.

[0099] Residual convolution layer ResConv1, which consists of two groups of convolution kernels, each group containing 64 convolution kernels with a size of 3x3. These convolution kernels are used to extract local features in the input feature map and convert these features into an output feature map. The input feature map The output feature map The operation process can be described as where W8 and W9 represent the weights of the first and second groups of convolution kernels respectively, and B8 and B9 are the corresponding bias terms, Similarly, represents a nonlinear Leaky Relu activation function. The last term The input feature map, which is directly added to the output of the convolution operation through a residual connection, realizes residual learning.

[0100] Residual convolution layer ResConv2, which consists of two groups of convolution kernels, each group containing 64 convolution kernels with a size of 3x3. These convolution kernels are used to extract local features in the input feature map and convert these features into an output feature map. The input feature map The output feature map The operation process can be described as where W 10 and W 11 represent the weights of the first and second groups of convolution kernels respectively, and B 10 and B 11 are the corresponding bias terms, Similarly, represents a nonlinear Leaky Relu activation function. The last term The input feature map, which is directly added to the output of the convolution operation through a residual connection, realizes residual learning.

[0101] Spectral reconstruction convolution layer HRConv, which contains c convolution kernels with a receptive field of 3x3. The input feature map The output reconstructed spectral image O (i) . The operation process can be described as where W 12 and B 12 are the weights and bias of the convolution kernel respectively, represents a nonlinear Leaky Relu activation function.

[0102] Further, the structure of the spectral adjustment sub-module is:

[0103] Feature extraction convolutional layer Conv1, which uses 64 3x3 size convolutional kernels, takes the reconstructed spectral image O (i) as input, and outputs feature maps The operation process can be represented as where W1 and B1 represent the weights and bias of the convolutional kernel respectively, and φ1(x) = max(x, 0) represents a nonlinear Relu activation function.

[0104] Channel attention extraction layer CAE, which converts the input multi-channel feature map into a one-dimensional attention mask vector, which can represent the significance of each channel. It takes the feature map as input, and outputs a one-dimensional attention mask of length 64 The specific operation process can be represented as: where GP(·) represents global average pooling, FC(·) represents full connection, and σ(·) is a nonlinear sigmoid function responsible for normalizing numerical values.

[0105] Channel attention action layer CAA, which applies the obtained channel attention mask to the feature map, adjusts the information relationship between channels, enhances important feature channels, and suppresses secondary channels, thereby enhancing the network's ability to capture key information. It takes the channel attention mask and the feature map as input, and outputs the feature-adjusted feature map The operation process can be represented as: where represents a channel-by-channel multiplication operation.

[0106] Spatial adjustment convolutional layer SConv1, which contains 64 convolutional kernels with a receptive field of 3x3. The input feature map outputs the feature map The operation process can be described as where W2 and B2 are the weights and bias of the convolutional kernel respectively, representing a nonlinear Relu activation function.

[0107] Spatial adjustment convolutional layer SConv2, which contains 1 convolutional kernel with a receptive field of 3x3. The input feature map outputs the feature map The operation process can be described as where W3 and B3 are the weights and bias of the convolutional kernel respectively, representing a nonlinear Relu activation function.

[0108] a spatial attention extraction layer SAE, which converts the input multi-channel feature map into a spatial attention mask with a spatial size of h2 x w2, which can represent the significance of each pixel position. It takes the feature map as input and outputs the spatial attention mask The specific operation process can be represented as: σ(·) is a nonlinear sigmoid function responsible for normalizing the numerical value;

[0109] a spatial feature adjustment layer SFA, which applies the obtained channel attention mask to the feature map to adjust the information relationship between channels, enhance important feature channels, and suppress secondary channels, thereby enhancing the network's ability to capture key information. It takes the spatial attention mask and the feature map as input, and outputs the feature-adjusted feature map The operation process can be represented as: wherein represents a pixel position multiplication operation;

[0110] a residual convolution layer ResConv1, which consists of two groups of convolution kernels, each group containing 64 convolution kernels with a size of 3 x 3. These convolution kernels are used to extract local features from the input feature map and convert these features into the output feature map. The input feature map the output feature map The operation process can be described as wherein W4 and W5 represent the weights of the first and second groups of convolution kernels, respectively, and B4 and B5 are the corresponding bias terms, representing a nonlinear Relu activation function. The last term the input feature map, which is directly added to the output of the convolution operation through a residual connection, realizing residual learning.

[0111] a residual convolution layer ResConv2, which consists of two groups of convolution kernels, each group containing 64 convolution kernels with a size of 3 x 3. These convolution kernels are used to extract local features from the input feature map and convert these features into the output feature map. The input feature map the output feature map The operation process can be described as wherein W6 and W7 represent the weights of the first and second groups of convolution kernels, respectively, and B6 and B7 are the corresponding bias terms, also representing a nonlinear Relu activation function. The last term the input feature map, which is directly added to the output of the convolution operation through a residual connection, realizing residual learning.

[0112] The reconstructed convolutional layer RConv includes c convolutional kernels with a receptive field of 3x3. The input feature map The output target high spatial resolution hyperspectral image H (i) The operation process can be described as where W8 and B8 are the weights and biases of the convolutional kernel, respectively.

[0113] Specifically, the hyperspectral image and panchromatic image of Houston University taken by the DiMAC ULTRALIGHT spectral imager are used to verify the effect of panchromatic sharpening. The hyperspectral image used covers a spectral range of 308-1050 nm, a total of 48 bands, and a spatial resolution of 1 m. The spatial resolution of the panchromatic image is 5 cm, and the ratio of the spatial resolutions of the two is 1:20.

[0114] Fig. 5(a) is a Houston hyperspectral reference image, Fig. 5(b) is an up-sampled hyperspectral image processed using the bicubic interpolation method, Fig. 5(c) is an image processed using the guided filter-based principal component analysis algorithm, Fig. 5(d) is an image processed using the brightness smoothing filter adjustment algorithm, and Fig. 5(e) is an image processed using the method described in the embodiment. As can be clearly observed from the figures, the image processed using the bicubic interpolation method has obviously lost a large amount of spatial details compared with the reference image, resulting in significant spatial distortion. Although this simple interpolation method can improve the spatial resolution of the image, it fails to effectively restore the high-frequency information of the image, especially in complex texture areas. The image processed using the guided filter-based principal component analysis algorithm has made some progress in sharpening, but there are still problems of spatial blurring and spectral distortion in multiple areas. This shows that although the algorithm has improved in global feature extraction, it still has deficiencies in detail restoration. The image processed using the brightness smoothing filter adjustment algorithm exhibits good global sharpening effect and high spectral fidelity. However, in local areas, especially in areas with rich spatial details, the problem of detail loss can still be observed, which limits the application effect of the algorithm in complex scenes. In contrast, the image processed using the method described in the embodiment achieves excellent sharpening effect. In significant areas such as buildings, the method successfully reconstructs fine spatial details, while in non-significant areas such as vegetation, it also maintains excellent spectral fidelity. Overall, the method described in the embodiment has made significant improvements in global sharpening effect, effectively overcoming the deficiencies of other algorithms in detail restoration and spectral preservation, and providing a more effective solution for panchromatic sharpening of hyperspectral images.

[0115] The method comprises two core sub-modules: a low-resolution-to-pan-sharpening sub-module and a spectral adjustment sub-module. The design of the low-resolution-to-pan-sharpening sub-module utilizes multiple convolution kernels of different sizes, which perform multi-scale feature extraction on the input feature map in parallel. Through this multi-path parallel processing, the low-resolution-to-pan-sharpening sub-module can capture and integrate the multi-scale spatial detail information of the feature map. In order to further improve the abstraction and fusion ability of the features, the low-resolution-to-pan-sharpening sub-module also adopts a residual structure, so that the network can more effectively learn the residual mapping between the input and the output.

[0116] The spectral adjustment sub-module focuses on detecting the spatial and spectral differences between the input feature map and the target hyperspectral image. By generating an attention mask, this sub-module can identify the spatial and spectral differences of each region in the feature map and accordingly perform differential processing on the input feature map, thereby significantly enhancing the sharpening effect.

[0117] In combination with the two sub-modules, the present application constructs an implicit prior extreme resolution ratio sharpening convolutional neural network. The network decomposes the extreme resolution ratio panchromatic sharpening task into multiple low-resolution-to-pan-sharpening tasks and spectral fine-tuning tasks. In each low-resolution-to-pan-sharpening task, the network only needs to improve a small resolution ratio, which greatly reduces the difficulty of feature matching. At the same time, the parameters of each sub-module are pre-set after training on small resolution ratio data, and the mapping relationship contained therein serves as an implicit prior knowledge, which helps to improve the overall sharpening effect. The method of the present application effectively solves the problem of panchromatic sharpening of hyperspectral images under extreme resolution ratio through the strategy of step-by-step resolution improvement and spectral adjustment, and provides a new solution for the field of hyperspectral image processing.

[0118] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be included in the protection scope of the present application.

Claims

1. A method for pan-sharpening hyperspectral images with extreme resolution ratios using a convolutional neural network based on implicit priors, characterized in that: include: Read the original hyperspectral image and the corresponding panchromatic image and perform preprocessing; Use the preprocessed hyperspectral images and panchromatic images to construct training and test datasets; Constructing a convolutional neural network based on implicit prior, wherein the convolutional neural network based on implicit prior includes a low-resolution ratio pan-sharpening submodule and a spatial spectrum adjustment submodule; Using a training data set, the implicit prior convolutional neural network is trained to obtain an optimal implicit prior convolutional neural network after training; Using the test dataset, the optimal implicit prior-based convolutional neural network is input to predict high spatial resolution hyperspectral images; The convolutional neural network based on implicit prior includes a feature extraction convolution layer, three low-resolution ratio pan-sharpening submodules and a spatial spectrum adjustment submodule; The convolutional neural network based on implicit prior includes: Feature extraction convolution layer, which contains 64 convolution kernels with a receptive field of 1×1, and inputs the upsampled hyperspectral image and low-resolution panchromatic images Output feature map The first low-resolution full-color sharpening submodule, input feature map Implementing extreme resolution ratio sharpening of the first stage implicit prior; The second low-resolution ratio pan-sharpening submodule inputs the output of the extreme resolution ratio sharpening of the first stage implicit prior to achieve the extreme resolution ratio sharpening of the second stage implicit prior; The third low-resolution ratio panchromatic sharpening submodule inputs the output of the extreme resolution ratio sharpening of the second stage implicit prior and outputs the reconstructed spectral image O (i) , achieving extreme resolution ratio sharpening of the third stage implicit prior; The spatial spectrum adjustment submodule inputs the output of the extreme resolution ratio sharpening of the third stage implicit prior and outputs the predicted high spatial resolution hyperspectral image H (i) .

2. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 1, characterized in that: The hyperspectral image and full-color images The following relationship exists in space: Where r represents the ratio of the spatial resolution of the observed panchromatic image to the corresponding hyperspectral image, h1 and w1 represent the height and width of the observed hyperspectral image, respectively, h2 and w2 represent the height and width of the observed panchromatic image, respectively, and c represents the number of spectral channels of the observed hyperspectral image.

3. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 1, characterized in that: The pretreatment includes: A smoothing filter is used to process the original hyperspectral image and panchromatic image, specifically: Observed hyperspectral images and full-color images A low-pass filter based on frequency response adaptation is applied to simulate the frequency response characteristics of the sensor to obtain smoothed hyperspectral images and panchromatic images; The smoothed hyperspectral image and the panchromatic image are downsampled to obtain a low-resolution hyperspectral image with a spatial resolution degraded by r times. and full-color images At the same time, in order to restore the spatial resolution of the image, the polynomial interpolation method is used to Upsample to obtain an upsampled hyperspectral image 4. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 3, characterized in that: Construct training and test datasets, specifically: After upsampling of the hyperspectral image Select an area and extract the low-resolution full-color image from the area Crop out the same number of sub-images of the same size and The cropped hyperspectral and panchromatic sub-image pairs are randomly shuffled to form a training dataset; Next, we upsample the hyperspectral image and low-resolution panchromatic images Sub-images are further cropped from the remaining areas of the image, and these sub-images are also used to form the test dataset.

5. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 1, characterized in that: The low-resolution pan-sharpening submodule includes a plurality of convolutional layers of different sizes, a fusion convolutional layer and a residual convolutional layer.

6. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 5, characterized in that: The structure of the low-resolution ratio pan-sharpening submodule specifically includes: The small-scale convolution layer SConv1 includes 64 convolution kernels with a receptive field of 3×3 and an input feature map. Output feature map The small-scale convolution layer SConv2 contains 64 convolution kernels with a receptive field of 3×3, and the input feature map Output feature map The medium-scale convolution layer MConv1 contains 64 convolution kernels with a receptive field of 5×5, and the input feature map Output feature map The medium-scale convolution layer MConv2 contains 64 convolution kernels with a receptive field of 5×5, and the input feature map Output feature map Large-scale convolution layer LConv1, which contains 64 convolution kernels with a receptive field of 7×7, and input feature map Output feature map Large-scale convolution layer LConv2, which contains 64 convolution kernels with a receptive field of 7×7, and input feature map Output feature map The fused convolution layer FConv contains 64 convolution kernels with a receptive field of 3×3 and input feature maps as well as Output feature map The residual convolution layer ResConv1 includes two groups of convolution kernels, each group includes 64 convolution kernels of size 3×3, which are used to extract local features in the input feature map and convert these features into output feature maps. Output feature map The residual convolution layer ResConv2 includes two groups of convolution kernels, each group includes 64 convolution kernels of size 3×3, which are used to extract local features in the input feature map and convert these features into output feature maps. Output feature map Spectral reconstruction convolution layer HRConv, the spectral reconstruction convolution layer HRConv includes c convolution kernels with a receptive field of 3×3, and the input feature map Output reconstructed spectral image O (i) .

7. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 6, characterized in that: The structure of the spatial spectrum adjustment submodule is: The feature extraction convolution layer Conv1 uses 64 3×3 convolution kernels to reconstruct the spectral image O (i) is the input and output feature map Channel attention extraction layer CAE, which converts the input multi-channel feature map into a one-dimensional attention mask vector, which can characterize the saliency of each channel. As input, output a one-dimensional attention mask of length 64 Channel attention layer CAA, which uses channel attention mask And feature maps As input, output feature map after feature adjustment The spatial adjustment convolution layer SConv1 includes 64 convolution kernels with a receptive field of 3×3 and an input feature map Output feature map The spatial adjustment convolution layer SConv2 includes a convolution kernel with a receptive field of 3×3 and an input feature map Output feature map The spatial attention extraction layer SAE converts the input multi-channel feature map into a spatial attention mask with a spatial size of h2×w2, which can characterize the significance of each pixel position. is the input and output is the spatial attention mask The spatial feature adjustment layer SFA applies the obtained channel attention mask to the feature map, which uses the spatial attention mask And feature maps As input, output feature map after feature adjustment The residual convolution layer ResConv1 includes two groups of convolution kernels, each group includes 64 convolution kernels of size 3×3, which are used to extract local features in the input feature map and convert these features into output feature maps. Output feature map The residual convolution layer ResConv2 consists of two groups of convolution kernels, each group contains 64 convolution kernels of size 3×3, which are used to extract local features in the input feature map and convert these features into output feature maps. Output feature map Reconstruct the convolution layer RConv, which contains c convolution kernels with a receptive field of 3×3 and input feature maps Output target high spatial resolution hyperspectral image H (i) .

8. The method for pan-sharpening hyperspectral images using extreme resolution ratio of convolutional neural networks according to claim 1, characterized in that: The loss function used to train the implicit prior convolutional neural network is the mean absolute error loss function.

Images