Hyperspectral image panchromatic sharpening method based on gabor filter guided attention network

Abstract

The application discloses a hyperspectral image panchromatic sharpening method based on a Gabor filter guided attention network, and comprises the following steps: acquiring a hyperspectral image and a corresponding panchromatic image, and performing pretreatment; the pretreated images constitute a training set and a test set; a hyperspectral image panchromatic sharpening network based on a Gabor filter guided attention network is constructed, comprising a high-low frequency feature separation module, a Gabor filter guided attention module and a feature fusion module; image data in the training set is taken as input of the hyperspectral image panchromatic sharpening network, a sharpening result is predicted through forward propagation, a difference between the sharpening result and an actual result is calculated, a loss function is calculated, the weight and parameters of the network are constantly updated, and an optimal hyperspectral image panchromatic sharpening network based on a Gabor filter guided attention network is obtained; and image data in the test set is input into the optimal hyperspectral image panchromatic sharpening network, and a predicted sharpening result of the test set is output.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, and in particular to a method for panchromatic sharpening of hyperspectral images based on a Gabor filter-guided attention network. Background Technology

[0002] Hyperspectral images, by continuously acquiring spectral information across dozens to hundreds of bands, can accurately reflect the chemical and physical properties of ground materials, making them invaluable in environmental monitoring, agricultural remote sensing, and geological exploration. However, limited by the physical constraints of imaging sensors (such as the balance between instantaneous field of view and signal-to-noise ratio), hyperspectral images often suffer from insufficient spatial resolution, leading to the loss of detailed information and severely limiting the accuracy of subsequent tasks such as classification and target detection. Meanwhile, panchromatic imaging instruments, thanks to their simple structure, have achieved rapid development in both high spatial resolution and lightweight design. Therefore, by fusing low-spatial-resolution hyperspectral images with high-spatial-resolution panchromatic images, data possessing both hyperspectral and high spatial resolution can be generated.

[0003] Over the past decade, various algorithms have emerged for hyperspectral panchromatic image sharpening, broadly categorized into component substitution-based, multi-scale analysis-based, variational optimization-based, and deep convolutional neural network-based methods. The core idea of ​​component substitution is to perform specific transformations on the hyperspectral image to separate its spatial and spectral information, then replace the original spatial components with the panchromatic image. While this type of method achieves good spatial detail by preserving the complete panchromatic image, it often produces significant spectral distortion. Multi-scale analysis extracts high-frequency information from the panchromatic image and injects it into the hyperspectral image. This method offers good spectral fidelity but inevitably suffers from high-frequency information loss. Variational optimization utilizes variational theory to build a model and then optimizes the solution; representative methods include nonnegative matrix factorization and Bayesian algorithms. In recent years, deep learning technology, with its powerful feature learning capabilities, has provided new approaches to hyperspectral image sharpening. In recent years, with the outstanding performance of deep convolutional networks in image processing, numerous convolutional neural network-based methods have emerged in the field of hyperspectral panchromatic sharpening, achieving performance far exceeding that of traditional methods. However, most current deep learning methods for panchromatic sharpening rarely consider the frequency differences between hyperspectral and panchromatic images, leading to harmful mixing of frequency information and low learning efficiency of the network for spectral and spatial features. Therefore, new theories and methods are needed to better utilize the high and low frequency information of images, thereby achieving better panchromatic sharpening results. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a hyperspectral image panchromatic sharpening method based on Gabor filter-guided attention network.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A hyperspectral image panchromatic sharpening method based on Gabor filter-guided attention network includes:

[0007] Acquire hyperspectral images and their corresponding panchromatic images, and preprocess them. The preprocessed images constitute the training set and the test set.

[0008] A hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network is constructed, including a high- and low-frequency feature separation module, a Gabor filter-guided attention module, and a feature fusion module;

[0009] The image data in the training set is used as the input of the hyperspectral image panchromatic sharpening network. The sharpening result is predicted through forward propagation, the difference between the sharpening result and the true result is calculated, the loss function is calculated, and the weights and parameters of the network are continuously updated to obtain the optimal hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network.

[0010] The image data from the test set is input into the optimal hyperspectral image panchromatic sharpening network, which outputs the predicted sharpening results for the test set.

[0011] Furthermore, the hyperspectral image and its corresponding panchromatic image The size of the space satisfies the following quantitative relationship:

[0012]

[0013] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image, h1 and w1 are the height and width of the hyperspectral image, respectively, while h2 and w2 represent the height and width of the panchromatic image, and c represents the number of spectral channels contained in the hyperspectral image.

[0014] Furthermore, the preprocessing includes:

[0015] A low-pass filter based on the sensor modulation transfer function and frequency response is constructed for the acquired hyperspectral image and the corresponding panchromatic image to perform a smoothing operation on the hyperspectral image and the panchromatic image;

[0016] By employing downsampling methods to reduce image resolution, a low-resolution hyperspectral image with a spatial resolution reduction of r times is obtained. and panchromatic image

[0017] Furthermore, the hyperspectral image panchromatic sharpening network includes a high- and low-frequency feature separation module, a Gabor filter-guided attention module, and a feature fusion module, specifically:

[0018] High- and low-frequency feature separation module: Input the upsampled hyperspectral image and panchromatic image, output the high-frequency and low-frequency feature maps of the upsampled hyperspectral image and panchromatic image, and separate the high- and low-frequency feature maps;

[0019] Gabor filter-guided attention module: Input high-frequency and low-frequency feature maps of the upsampled hyperspectral and panchromatic images, output high-frequency and low-frequency feature maps after feature enhancement;

[0020] Feature fusion module: Input high-frequency and low-frequency feature maps after feature enhancement, output hyperspectral image after full-color sharpening.

[0021] Furthermore, the structure of the high- and low-frequency feature separation module is as follows:

[0022] Feature extraction convolutional layer Conv1: Uses 64 3×3 convolutional kernels, with the input being an upsampled low-resolution hyperspectral image. and the corresponding panchromatic image Output feature map

[0023] Low-pass filter layer LPS: Uses 64×h1×w1 3×3 convolutional kernels, feature map As input, generate different convolution kernels K for each pixel. n For the output, the convolution kernel K n Normalization is performed to make the generated convolution kernel a low-pass filter, and then it is combined with the feature map. Perform pixel-by-pixel convolution to output feature maps. Finally and Subtraction yields the feature map

[0024] Conv2 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output high-frequency feature map H (i) ;

[0025] Feature extraction convolutional layer Conv3: Uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output low-frequency feature map L (i) .

[0026] Furthermore, the Gabor filter-guided attention module specifically comprises:

[0027] Conv4, a convolutional layer for feature extraction, uses 64 3×3 convolutional kernels to extract high-frequency feature maps H. (i) Input: Feature map

[0028] Gabor convolutional layer (GAB): Uses 64 Gabor convolutional kernels of size 3×3 in four different directions, which are based on feature maps. Input: Feature map with enhanced directional features Feature map This refers to the query vector in the attention network;

[0029] Conv5 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract low-frequency feature maps K. (i) Input: Feature map Feature map These are the key vectors and value vectors in the attention network, followed by the feature map. With feature map Multiply to obtain the feature map

[0030] SoftMax weight generation layer: based on feature maps As input, the feature map is processed using the Softmax function. After normalization, the weighted feature map S is output. (i) Then it was combined with Multiply and perform residual join, then combine with L (i) Add them together and output the low-frequency feature map after weight adjustment.

[0031] Conv6 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract low-frequency feature maps. Input: Feature map Then it was combined with Perform residual connections to obtain the high-frequency feature map after high- and low-frequency feature interaction.

[0032] Furthermore, the low-frequency feature map L (i) The low-frequency feature map is obtained by sequentially passing the attention module guided by Gabor filtering. It uses low-frequency features as query vectors and high-frequency features as key vectors and value vectors.

[0033] Furthermore, the structure of the feature fusion module is as follows:

[0034] Conv7 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract high-frequency feature maps. and low-frequency feature maps Input: Feature map

[0035] Residual convolutional layer ResConv1: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map

[0036] Residual convolutional layer ResConv2: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map

[0037] Conv8 feature-adjusted convolutional layer: uses c 3×3 convolutional kernels, which adjust the feature maps... As input, output is a panchromatic sharpened hyperspectral image I (i) .

[0038] Furthermore, the loss function value is optimized using an adaptive moment estimation algorithm.

[0039] Furthermore, the loss function adopted is the mean absolute error loss function.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] (1) The present invention uses a high- and low-frequency feature separation module. This module generates different convolution kernels for each pixel of the hyperspectral image and the panchromatic image. Then, the SoftMax function is used to make the number in the convolution kernel less than 1, ensuring that it is a low-pass filter. The hyperspectral image and the panchromatic image are passed through the low-pass filter to obtain the low-frequency feature map of the image. Finally, the low-frequency feature map is subtracted from the original image to obtain the high-frequency feature map.

[0042] (2) This invention employs a Gabor filter-guided attention module, which uses an attention network to exchange information between high-frequency and low-frequency branches, capturing redundant or complementary information from the two branches. Simultaneously, a Gabor filter is used for guidance to enhance features in certain directions and strengthen the texture details of the image. It is noteworthy that in the low-frequency feature branch, the low-frequency feature map serves as the query vector for the attention network, while the high-frequency feature map serves as both the key and value vectors for the attention network; the reverse is true in the high-frequency branch.

[0043] (3) The present invention employs a feature fusion module, which fuses the low-frequency feature map and the high-frequency feature map after feature interaction and enhancement by the attention network and Gabor filter through multiple convolutional layers and residual convolutional layers to obtain the final sharpening result.

[0044] (4) The hyperspectral image panchromatic sharpening method based on Gabor filter-guided attention network designed in this invention mainly addresses the problem of current methods not fully utilizing the high and low frequency features of hyperspectral images and their corresponding panchromatic images. The network first generates low-pass filters (i.e., convolutional kernels with values ​​all less than 1) for each pixel of the hyperspectral and panchromatic images to separate the high and low frequency features. Then, the high and low frequency features are processed through a Gabor filter-guided attention module for feature interaction and enhancement. This module uses an attention network to interact information between two branches, capturing redundant or complementary information from both branches. Simultaneously, a Gabor filter is used for guidance to enhance features in certain directions, strengthening the texture details of the image. Finally, the features from the two branches are fused to obtain the final panchromatic sharpening result. Attached Figure Description

[0045] Figure 1 This is a flowchart of a hyperspectral image panchromatic sharpening method based on a Gabor filter-guided attention network, according to an embodiment of the present invention.

[0046] Figure 2 This is a structural diagram of the high- and low-frequency feature separation module in an embodiment of the present invention.

[0047] Figure 3 This is a structural diagram of the attention module guided by Gabor filtering in an embodiment of the present invention.

[0048] Figure 4 This is a structural diagram of the feature fusion module in an embodiment of the present invention.

[0049] Figure 5 This is a structural diagram of a hyperspectral image panchromatic sharpening network based on a Gabor filter-guided attention network, according to an embodiment of the present invention.

[0050] Figure 6(a) is a hyperspectral reference image from the University of Pavia, Figure 6(b) is an image processed by bicubic interpolation, Figure 6(c) is an image processed by matrix factorization algorithm, Figure 6(d) is an image processed by Bayesian algorithm, and Figure 6(e) is an image processed by the method described in this embodiment. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0052] Example

[0053] like Figures 1-5 As shown, a hyperspectral image panchromatic sharpening method based on Gabor filter-guided attention network includes the following steps:

[0054] S1 acquires hyperspectral images and their corresponding panchromatic images, and performs preprocessing. The preprocessed images constitute the training set and the test set.

[0055] Furthermore, the hyperspectral image and the panchromatic image registered with it. The size of the space satisfies the following quantitative relationship:

[0056] h2 = h1 × r

[0057] w2 = w1 × r

[0058] Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image, h1 and w1 are the height and width of the hyperspectral image, respectively, while h2 and w2 represent the height and width of the panchromatic image, and c represents the number of spectral channels contained in the hyperspectral image.

[0059] Furthermore, the preprocessing includes:

[0060] raw hyperspectral image data and panchromatic image data Construct a low-pass filter based on the sensor modulation transfer function and frequency response to smooth hyperspectral and panchromatic images;

[0061] Then, a downsampling method is used to further reduce the image resolution, resulting in a low-resolution hyperspectral image with a spatial resolution reduced by a factor of r. and panchromatic image

[0062] S2 constructs a hyperspectral image panchromatic sharpening network based on a Gabor filter-guided attention network, including a high- and low-frequency feature separation module, a Gabor filter-guided attention module, and a feature fusion module, specifically:

[0063] High- and low-frequency feature separation module: Input the upsampled hyperspectral image and panchromatic image, output the high-frequency and low-frequency feature maps of the upsampled hyperspectral image and panchromatic image, and separate the high- and low-frequency feature maps;

[0064] Gabor filter-guided attention module: Input high-frequency and low-frequency feature maps of the upsampled hyperspectral and panchromatic images, output high-frequency and low-frequency feature maps after feature enhancement;

[0065] Feature fusion module: Input high-frequency and low-frequency feature maps after feature enhancement, output hyperspectral image after full-color sharpening.

[0066] To further explain, the structure of the high- and low-frequency feature separation module is as follows:

[0067] The feature extraction convolutional layer Conv1 uses 64 3×3 convolutional kernels, and the input is an upsampled low-resolution hyperspectral image. and the corresponding panchromatic image Output feature map The operation process can be represented as follows: Where W1 and B1 represent the weights and biases of the convolution kernel, respectively, and cat(·) represents concatenation along the channel dimension. ReLU represents a non-linear activation function;

[0068] Low-pass filter layer (LPS): Uses 64×h1×w1 3×3 convolutional kernels and normalizes the vectors using the SoftMax function to ensure that the generated convolutions are low-pass filters. Feature map As input, generate different convolution kernels K for each pixel. n For the output, the convolution kernel K n Normalization is performed to make the generated convolution kernel a low-pass filter, and then it is combined with the feature map. Perform pixel-by-pixel convolution to output feature maps. Finally and Subtraction yields the feature map The operation process can be represented as follows: Where W2 and B2 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function, and σ(·) represents the non-linear activation function SoftMax. This represents a pixel-wise convolution operation, where i is the image index, n is the kernel index, and h1 and w1 are the length and width of the upsampled low-resolution hyperspectral image and panchromatic image, respectively.

[0069] Conv2 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output high-frequency feature map H (i) The operation process can be represented as follows: Where W3 and B3 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0070] Feature extraction convolutional layer Conv3: Uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output low-frequency feature map L (i) The operation process can be represented as follows: Where W4 and B4 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0071] The Gabor filter-guided attention module is specifically as follows:

[0072] Conv4, a convolutional layer for feature extraction, uses 64 3×3 convolutional kernels to extract high-frequency feature maps H. (i) Input: Feature map The operation process can be represented as follows: Where W5 and B5 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0073] Gabor convolutional layer (GAB): Uses 64 Gabor convolutional kernels of size 3×3 in four different directions, which are based on feature maps. Input: Feature map with enhanced directional features Feature map This refers to the query vector in the attention network; the operation process can be represented as follows: Where W6 and B6 represent the weights and biases of the convolution kernel, respectively. ReLU represents a non-linear activation function;

[0074] Feature extraction convolutional layer Conv5: Uses 64 3×3 convolutional kernels, which extract low-frequency feature maps L (i) Input: Feature map Feature map These are the key vectors and value vectors in the attention network, followed by the feature map. With feature map Multiply to obtain the feature map The operation process can be represented as follows: Where W7 and B7 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0075] SoftMax weight generation layer: based on feature maps As input, the feature map is processed using the Softmax function. After normalization, the weighted feature map S is output. (i) Then it was combined with Multiply and perform residual join, then combine with L (i) Add them together and output the low-frequency feature map after weight adjustment. The operation process can be represented as follows:

[0076] Where GP(·) represents global pooling, FC(·) represents a fully connected layer, and σ(·) represents the non-linear activation function Softmax.

[0077] Conv6 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract low-frequency feature maps. Input: Feature map Then it was combined with Perform residual connections to obtain the high-frequency feature map after high- and low-frequency feature interaction. The operation process can be represented as follows: Where W8 and B8 represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0078] Correspondingly, in the low-frequency feature branch, the low-frequency feature map is also obtained by sequentially passing through the above layers. It uses low-frequency features as query vectors and high-frequency features as key vectors and value vectors.

[0079] Furthermore, the structure of the feature fusion module is as follows:

[0080] Conv7 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract high-frequency feature maps. and low-frequency feature maps Input: Feature map The operation process can be represented as follows: Where W9 and B9 represent the weights and biases of the convolution kernel, respectively, and cat(·) represents concatenation along the channel dimension. ReLU represents the non-linear activation function.

[0081] Residual convolutional layer ResConv1: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: Among them W 10 W 10 and B 11 B 11 These represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function. This indicates a pixel-by-pixel addition operation.

[0082] Residual convolutional layer ResConv2: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map The operation process can be represented as follows: Among them W 12 W 12 and B 13 B13 These represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function. This indicates a pixel-by-pixel addition operation.

[0083] Conv8 feature-adjusted convolutional layer: uses c 3×3 convolutional kernels, which adjust the feature maps... As input, output is a panchromatic sharpened hyperspectral image I (i) The calculation process can be represented as follows: Among them W 14 and B 14 These represent the weights and biases of the convolution kernel, respectively. ReLU represents the non-linear activation function.

[0084] S3 sets the network's hyperparameters and initializes the weights and biases of the network's convolutions;

[0085] S4 uses the image data in the training set as input to the hyperspectral image panchromatic sharpening network. After forward propagation, it predicts the sharpening result, calculates the difference between the sharpening result and the true result, calculates the loss function, and continuously updates the weights and parameters of the network to obtain the optimal hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network.

[0086] S5 inputs the image data from the test set into the optimal hyperspectral image panchromatic sharpening network and outputs the predicted sharpening results for the test set.

[0087] Furthermore, the Gabor filter in the attention module guided by the Gabor filter in S3 can be expressed as:

[0088]

[0089] x′=xcosθ+ysinθ

[0090] y′=-xsinθ+ycosθ

[0091] Where λ is the wavelength of the sinusoidal factor, θ is the direction of the normal to the parallel fringe, ψ is the phase shift, σ is the standard deviation of the Gaussian sigma, and γ is the ellipticity.

[0092] Furthermore, the loss function in S5 adopts the mean absolute error loss function, the expression of which is:

[0093]

[0094] Where Ω represents the mapping relationship between the input and output of the full-color sharpening network, θ represents the network parameters, and N... b Represents the number of input samples in each iteration, ‖·‖F This represents the Frobenius norm.

[0095] This embodiment uses hyperspectral and panchromatic images of the University of Pavia captured by the ROSIS spectral imager to verify the effect of panchromatic sharpening. The hyperspectral image used covers the spectral range of 430-860 nm, with a total of 115 bands. In this experiment, the spatial resolution ratio between the hyperspectral and panchromatic images was set to 1:8.

[0096] Figure 6(a) shows the original hyperspectral image, serving as a reference image for the experiment. Figure 6(b) shows the hyperspectral image obtained by upsampling using bicubic interpolation. It can be observed that the sharpening result obtained by this method suffers from severe spatial spectral information distortion and blurred details. Figures 6(c) and 6(d) show the hyperspectral images processed by principal component analysis and adaptive Gram-Schmidt orthogonal transform, respectively. Although these methods show some improvement in sharpening effect compared to bicubic interpolation, blurring and distortion still exist in some areas rich in spatial detail, indicating that these methods have limitations in restoring high-frequency details. Finally, Figure 6(e) presents the hyperspectral image processed using the method described in this embodiment. The results show that this method performs excellently in reconstructing hyperspectral images, restoring most spatial details without significant spectral distortion. This is attributed to the use of a Gabor filter-guided attention network for texture enhancement and interaction between high and low frequency information. This method significantly improves sharpening effects and achieves good restoration results in both spatial and spectral dimensions. These results confirm the superiority and effectiveness of our method in panchromatic sharpening of hyperspectral images.

[0097] In summary, the hyperspectral image panchromatic sharpening method based on a Gabor filter-guided attention network designed in this invention primarily addresses the problem of incomplete utilization of high- and low-frequency features in hyperspectral and panchromatic images. This network first generates low-pass filters (convolutional kernels with values ​​all less than 1) for each pixel in both the hyperspectral and panchromatic images to separate their high- and low-frequency features. Subsequently, the high- and low-frequency features are processed and enhanced through a Gabor filter-guided attention module. This module uses an attention network to exchange information between two branches, capturing redundant or complementary information from both branches. Simultaneously, a Gabor filter is used to guide the enhancement of features in certain directions, strengthening the image's texture details. Finally, the features from the two branches are fused to achieve a panchromatic sharpening result with superior overall performance.

[0098] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A hyperspectral image panchromatic sharpening method based on Gabor filter-guided attention network, characterized in that, include: Acquire hyperspectral images and their corresponding panchromatic images, and preprocess them. The preprocessed images are then divided into training and test sets. A hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network is constructed, including a high- and low-frequency feature separation module, a Gabor filter-guided attention module, and a feature fusion module; The hyperspectral image panchromatic sharpening network was trained using image data from the training set, resulting in the optimal hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network. The image data in the test set is input into the optimal hyperspectral image panchromatic sharpening network, and the predicted sharpening result for the test set is output. The hyperspectral image panchromatic sharpening network includes a high- and low-frequency feature separation module, a Gabor filter-guided attention module, and a feature fusion module, specifically: High- and low-frequency feature separation module: Input the upsampled hyperspectral image and panchromatic image, output the high- and low-frequency feature maps of the upsampled hyperspectral image and panchromatic image, separate the high- and low-frequency feature maps to obtain high-frequency feature maps and low-frequency feature maps; The Gabor filter-guided attention module takes the high-frequency and low-frequency feature maps obtained from the high-frequency and low-frequency feature separation module as input and outputs the enhanced high-frequency and low-frequency feature maps as output. Feature fusion module: Input high-frequency and low-frequency feature maps after feature enhancement, output hyperspectral image after full-color sharpening; The Gabor filter-guided attention module is specifically as follows: Conv4 convolutional layer for feature extraction: uses 64 3×3 convolutional kernels to extract high-frequency feature maps. Input: Feature map ; Gabor convolutional layer (GAB): Uses 64 Gabor convolutional kernels of size 3×3 in four different directions, which are based on feature maps. Input: Feature map with enhanced directional features Feature map This refers to the query vector in the attention network; Conv5 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract low-frequency feature maps. Input: Feature map Feature map These are the key vectors and value vectors in the attention network, followed by the feature map. With feature map Multiply to obtain the feature map ; SoftMax weight generation layer: based on feature maps As input, the feature map is processed using the Softmax function. Normalization is performed, and the output is the weighted feature map. Then it was combined with Multiply and perform residual join, then combine with Add them together and output the low-frequency feature map after weight adjustment. ; Conv6 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract low-frequency feature maps. Input: Feature map Then it was combined with Perform residual connections to obtain the high-frequency feature map after high- and low-frequency feature interaction. .

2. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The hyperspectral image and its corresponding panchromatic image The size of the space satisfies the following quantitative relationship: Where r represents the ratio of the spatial resolution of the panchromatic image to that of the hyperspectral image. and These represent the height and width of the hyperspectral image, respectively. and This represents the height and width of the panchromatic image, while c represents the number of spectral channels contained in the hyperspectral image.

3. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The preprocessing includes: A low-pass filter based on the sensor modulation transfer function and frequency response is constructed for the acquired hyperspectral image and the corresponding panchromatic image to perform a smoothing operation on the hyperspectral image and the panchromatic image; By employing downsampling methods to reduce image resolution, a low-resolution hyperspectral image with a spatial resolution reduction of r times is obtained. and panchromatic images .

4. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The structure of the high- and low-frequency feature separation module is as follows: Feature extraction convolutional layer Conv1: Uses 64 3×3 convolutional kernels, inputting an upsampled low-resolution hyperspectral image. and the corresponding panchromatic image Output feature map ; Low-pass filter layer LPS: using 64× × Each 3×3 convolutional kernel, feature map For input, generate different convolution kernels for each pixel. For the output, the convolution kernel Normalization is performed to make the generated convolution kernel a low-pass filter, and then it is combined with the feature map. Perform pixel-by-pixel convolution to output feature maps. Finally and Subtraction yields the feature map ; Conv2 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output high-frequency feature map ; Feature extraction convolutional layer Conv3: Uses 64 3×3 convolutional kernels, which extract features from the convolutional layers. As input, output low-frequency feature map .

5. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The low-frequency feature map The low-frequency feature map is obtained by sequentially passing the attention module guided by Gabor filtering. It uses low-frequency features as query vectors and high-frequency features as key vectors and value vectors.

6. The hyperspectral image panchromatic sharpening method according to claim 1, characterized in that, The structure of the feature fusion module is as follows: Conv7 feature extraction convolutional layer: uses 64 3×3 convolutional kernels, which extract high-frequency feature maps. and low-frequency feature maps Input: Feature map ; Residual convolutional layer ResConv1: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map ; Residual convolutional layer ResConv2: Uses two sets of 64 3×3 convolutional kernels, which are based on feature maps. Input: Feature map ; Conv8 feature-adjusted convolutional layer: uses c 3×3 convolutional kernels, which adjust the feature maps... As input, output a pan-sharpened hyperspectral image. .

7. The hyperspectral image panchromatic sharpening method according to any one of claims 1-6, characterized in that, The hyperspectral image panchromatic sharpening network is trained using image data from the training set. Specifically, the sharpening result is predicted through forward propagation, the difference between the sharpened result and the true result is calculated, the loss function is calculated, and the weights and parameters of the network are continuously updated to obtain the optimal hyperspectral image panchromatic sharpening network based on Gabor filter-guided attention network.

8. The hyperspectral image panchromatic sharpening method according to claim 7, characterized in that, The loss function is the mean absolute error loss function, and the value of the loss function is optimized using the adaptive moment estimation algorithm.

Images