Quaternion-Based Super-Resolution Fusion Algorithm and System for Remote Sensing Hyperspectral Images

Abstract

The present invention relates to the technical field of remote sensing image processing, and specifically discloses a super-resolution fusion algorithm and system for remote sensing hyperspectral images based on quaternions. The method includes: using a panchromatic image quaternion feature extraction module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module to construct a dual U-Net structure network based on multi-scale residual links and skip connections; inputting the panchromatic image and the low-resolution hyperspectral image, and calculating to obtain a high-spatial-resolution hyperspectral image. The present invention embeds multiple spectral channels of the hyperspectral image and the high-resolution spatial information of the panchromatic image into the four components of the quaternion, and uses quaternion algebra to uniformly model the spectral and spatial characteristics; then performs feature extraction, fusion, and reconstruction through a deep learning network, and finally generates a fusion image with both hyperspectral fidelity and high spatial resolution, which can significantly improve the spatial resolution of the hyperspectral image while maintaining its spectral consistency.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and particularly to a remote sensing hyperspectral image super-resolution fusion algorithm and system based on quaternion. Background Technique

[0002] Hyperspectral remote sensing images are widely used in fields such as agricultural monitoring, ground object classification, and environmental protection due to the rich spectral information they contain. However, due to the physical limitations of imaging sensors, hyperspectral images usually have low spatial resolution and are difficult to meet the requirements of refined analysis. Therefore, through super-resolution fusion technology, combining the rich spectral information of hyperspectral images with the high spatial resolution information of panchromatic images or multispectral images to generate images with both hyperspectral fidelity and high spatial resolution has become a current research hotspot.

[0003] Traditional hyperspectral image super-resolution fusion algorithm methods mostly rely on technical frameworks based on matrix decomposition, sparse representation, and deep learning. These methods have significant limitations in modeling the complex spectral-spatial characteristics of hyperspectral data: on the one hand, matrix methods are difficult to fully capture the non-linear coupling relationships between multi-spectral channels, easily leading to spectral information distortion; on the other hand, existing deep learning methods often take scalar convolution as the core, and their expression ability for high-dimensional spectral information is limited, resulting in it being difficult to ensure both spectral consistency and spatial resolution in the fusion results. In addition, traditional methods perform poorly in terms of adaptability to multi-sensor and multi-scene data and are difficult to meet the actual application requirements.

[0004] Existing technologies still have significant deficiencies in spectral-spatial coupling modeling, multi-dimensional feature expression, cross-scene adaptability, and computational efficiency.

[0005] 1. Insufficient ability to model spectral-spatial characteristics

[0006] Linear hypothesis limitation: Traditional matrix decomposition methods (such as non-negative matrix decomposition, dictionary learning) are mostly based on linear hypotheses and can only handle the linear associations between spectral and spatial characteristics, and are difficult to effectively capture the non-linear coupling relationships commonly existing in hyperspectral images.

[0007] Spectral information distortion: In the fusion process, sparse representation or scalar-based deep learning methods often overemphasize the improvement of spatial details while ignoring the complete retention of spectral characteristics, resulting in deviations in the spectral fidelity of the fusion results.

[0008] 2. Limited ability to process multi-channel high-dimensional information

[0009] Limitations of scalar calculation: Existing deep learning frameworks (such as CNN, GAN) usually process multi-spectral channels in the way of scalar convolution. This method models each channel independently, ignoring the global correlation between spectral channels and making it difficult to express the multi-dimensional characteristics of hyperspectral images as a whole.

[0010] Information redundancy and fragmentation: The scalar method needs to process each channel separately, resulting in high computational complexity and easy fragmentation of the information correlation between channels, which limits the overall performance of the fusion result.

[0011] 3. Insufficient adaptability and robustness

[0012] Hyperspectral images usually come from different sensors or scenes, and their resolution, spectral range and noise characteristics vary greatly. However, existing methods have the following disadvantages in terms of adaptability and robustness:

[0013] Poor cross-sensor compatibility: Most methods rely on the image characteristics of specific sensors and it is difficult to achieve generality on multi-sensor data.

[0014] Insufficient adaptability to complex scenes: Existing models often show a decline in performance when facing complex ground object distributions or multi-scale features, unable to meet the requirements of practical applications.

[0015] 4. Computational complexity and efficiency issues

[0016] Due to the large number of spectral channels in hyperspectral images, the computational complexity of data processing is relatively high:

[0017] Efficiency bottleneck in large-scale data processing: Methods such as matrix decomposition and sparse representation usually involve complex iterative optimization processes, with high computational costs and difficult to meet the requirements of real-time processing.

[0018] Redundant calculations in deep learning models: Existing deep learning models based on scalar convolution have a large number of parameters, high resource requirements in the training and inference processes, high requirements for hardware performance, and are difficult to be applied to resource-constrained scenarios. Summary of the invention

[0019] The present invention aims to solve the problem that when processing multi-spectral data, the spectral correlation between channels cannot be effectively utilized, and the integrity of spectral information cannot be guaranteed while improving the spatial resolution. To this end, the present invention provides a quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm and system, which embeds multiple spectral channels of the hyperspectral image and the high-resolution spatial information of the panchromatic image into the four components of the quaternion, and uses quaternion algebra to uniformly model the spectral and spatial characteristics; then, through a deep learning network, feature extraction, fusion, and reconstruction are performed, and finally a fusion image with both hyperspectral fidelity and high spatial resolution is generated. The present invention can significantly improve the spatial resolution of hyperspectral images while maintaining their spectral consistency, providing technical support for the refined analysis and practical application of hyperspectral images.

[0020] The present invention provides a quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm, and the technical solution adopted is as follows: including:

[0021] Construct a panchromatic image quaternion feature extraction module, which is used to extract features from the panchromatic image to obtain a panchromatic image quaternion feature map;

[0022] Construct a hyperspectral image quaternion feature extraction module, which is used to represent the hyperspectral image in the form of quaternion, perform feature extraction, and obtain a hyperspectral image quaternion feature map;

[0023] Construct a self-attention fusion module, which is used to perform feature fusion and enhancement on the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through the self-attention mechanism;

[0024] Using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module, and the self-attention fusion module, construct a dual U-Net structure network based on multi-scale residual links and skip connections. The dual U-Net structure network includes a first U-Net branch and a second U-Net branch. Each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, with inter-layer skip connections; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module, with inter-layer skip connections;

[0025] Input the panchromatic image and the low-resolution hyperspectral image into the dual U-Net structure network, and calculate to obtain a high-spatial-resolution hyperspectral image.

[0026] Further, the specific process of the hyperspectral image quaternion feature extraction module representing the hyperspectral image in the form of quaternion, performing feature extraction, and obtaining a hyperspectral image quaternion feature map is as follows:

[0027] Combine three adjacent channels of the low-resolution hyperspectral image to serve as the three components of the imaginary part of the quaternion, and take the average of the pixel values at the same position as the real part value of the quaternion to obtain a hyperspectral image represented in quaternion form;

[0028] Input the hyperspectral image represented in quaternion form into a quaternion convolution kernel for feature extraction to obtain a quaternion feature map of the hyperspectral image.

[0029] Further, the representation form of the hyperspectral image represented in quaternion form is:

[0030] k

[0031] where, represents the pixel value of the hyperspectral image represented in quaternion form corresponding to the H-th, (H + 1)-th, and (H + 2)-th channels of the low-resolution hyperspectral image at the corresponding position; represents the pixel value of the H-th channel of the low-resolution hyperspectral image at the corresponding position; represents the pixel value of the (H + 1)-th channel of the low-resolution hyperspectral image at the corresponding position; represents the average value of the pixel values at the corresponding position; , k are three imaginary units, satisfying the following relationships:

[0032]

[0033]

[0034] ;

[0035] The quaternion convolution kernel is:

[0036]

[0037] where, is the first component of the quaternion convolution kernel, is the second component of the quaternion convolution kernel, is the third component of the quaternion convolution kernel, is the fourth component of the quaternion convolution kernel;

[0038] Based on quaternion multiplication for feature extraction, the calculation process of the quaternion feature map of the hyperspectral image is:

[0039]

[0040] Among them, The hyperspectral image represented in quaternion form.

[0041] Furthermore, the specific process of the panchromatic image quaternion feature extraction module extracting features from the panchromatic image to obtain the panchromatic image quaternion feature map is as follows:

[0042] The panchromatic image is respectively passed through convolution kernels with kernels of k1×k1, 1×k1, and k1×1 to extract features of the panchromatic image at different spatial scales in different directions. The three output feature maps are represented in quaternion form to obtain the panchromatic image represented in quaternion form;

[0043] The panchromatic image represented in quaternion form is input into a quaternion convolution kernel for feature extraction to obtain the panchromatic image quaternion feature map.

[0044] Furthermore, the representation form of the panchromatic image represented in quaternion form is:

[0045] k

[0046] Among them, Represents the Pixel value at the position of the panchromatic image represented in quaternion form; Represents the feature map output after the panchromatic image passes through the k1×k1 convolution kernel Pixel value at the position; Represents the feature map output after the panchromatic image passes through the 1×k1 convolution kernel Pixel value at the position; Represents the feature map output after the panchromatic image passes through the k1×1 convolution kernel Pixel value at the position; Represents The average value of the pixel values at the position; , k are three imaginary units, satisfying the following relationship:

[0047]

[0048]

[0049] .

[0050] Furthermore, the specific process of the self-attention fusion module performing feature fusion and enhancement on the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through the self-attention mechanism is as follows:

[0051] Perform a linear transformation on the quaternion feature map of the hyperspectral image to obtain the Query vector and the Key vector, and perform a linear transformation on the quaternion feature map of the panchromatic image to obtain the Value vector;

[0052] Calculate the attention weight A:

[0053]

[0054] where Q represents the Query vector, K represents the Key vector, T represents the transpose, represents the dimension of the Key vector, represents the Softmax operation;

[0055] Calculate the fused feature :

[0056]

[0057] where, represents the Value vector.

[0058] Furthermore, both the first U-Net branch and the second U-Net branch include five layers, where the first layer and the fifth layer are skip-connected, and the second layer and the fourth layer are skip-connected; the low-level features and the high-level features are fused through the skip connections.

[0059] The present invention also provides a quaternion-based remote sensing hyperspectral image super-resolution fusion system, and the technical solution adopted is as follows: including: a module construction unit, a network construction unit, and an input-output unit,

[0060] The module construction unit is used to construct a panchromatic image quaternion feature extraction module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module. The panchromatic image quaternion feature extraction module is used to extract features from the panchromatic image to obtain a panchromatic image quaternion feature map; the hyperspectral image quaternion feature extraction module is used to represent the hyperspectral image in the form of quaternions and perform feature extraction to obtain a hyperspectral image quaternion feature map; the self-attention fusion module is used to perform feature fusion and enhancement on the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through the self-attention mechanism;

[0061] The network construction unit is used to construct a dual U-Net structure network based on multi-scale residual links and skip connections by using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module, and the self-attention fusion module. The dual U-Net structure network includes a first U-Net branch and a second U-Net branch. Each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, with inter-layer skip connections; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module, with inter-layer skip connections;

[0062] The input-output unit is used to input the panchromatic image and the low-resolution hyperspectral image into the dual U-Net structure network, and output the hyperspectral image with high spatial resolution calculated by the dual U-Net structure network.

[0063] One or more of the above technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0064] The present invention captures the mutual relationship between adjacent channels of the low-resolution hyperspectral image through quaternion convolution, without losing the important information of each channel, so that the features of the spectral channels of each hyperspectral image are completely extracted; the feature maps obtained by quaternion integration of multi-directional features are used on the panchromatic image to more effectively extract the spatial detail information of the PAN image. The present invention has the characteristics of high efficiency, strong robustness, and good spectral fidelity, can significantly improve the spatial resolution of the hyperspectral image, and at the same time maintain its spectral consistency, providing technical support for the refined analysis and practical application of the hyperspectral image.

[0065] The additional aspects and advantages of the present invention will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present invention. Description of the Drawings

[0066] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0067] Figure 1 is the flowchart of the method provided by the present invention.

[0068] Figure 2 is the structural schematic diagram of the panchromatic image quaternion feature extraction module provided by the present invention.

[0069] Figure 3It is a schematic structural diagram of the hyperspectral image quaternion feature extraction module provided by the present invention.

[0070] Figure 4 It is a schematic structural diagram of the self-attention fusion module provided by the present invention.

[0071] Figure 5 It is a schematic structural diagram of the double U-Net structure network provided by the present invention.

[0072] Figure 6 It is a subjective comparison experimental result diagram of the Botswana dataset provided by the present invention. Detailed implementation manners

[0073] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope protected by the present invention. The following embodiments are used to illustrate the present invention but cannot be used to limit the scope of the present invention.

[0074] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art can combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0075] The following will be combined with Figures 1 to 6 to further elaborate on the present invention, and describe a quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm and system of the present invention:

[0076] In this embodiment, as Figure 1 shown, a quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm is provided, including the following steps:

[0077] Step 1: Construct a panchromatic image quaternion feature extraction module, and the panchromatic image quaternion feature extraction module is used to extract features from the panchromatic image to obtain a panchromatic image quaternion feature map.

[0078] As Figure 2As shown in the figure, the specific process of the panchromatic image quaternion feature extraction module extracting features from the panchromatic image to obtain the panchromatic image quaternion feature map is as follows:

[0079] The panchromatic image is respectively passed through three convolution kernels with different kernel sizes and shapes, as well as the LeakyReLU activation function. In this embodiment, three convolution kernels with kernels of k1×k1, 1×k1, and k1×1 are selected to extract the features of the panchromatic image in different directions of the spatial scale. The three output feature maps are subjected to a four-element transformation and represented in quaternion form to obtain a panchromatic image represented in quaternion form. .

[0080] Among them, the representation form of the panchromatic image represented in quaternion form is:

[0081] k

[0082] Among them, represents the pixel value at the position of the panchromatic image represented in quaternion form; represents the pixel value at the position of the feature map output after the panchromatic image passes through the k1×k1 convolution kernel; represents the pixel value at the position of the feature map output after the panchromatic image passes through the 1×k1 convolution kernel; represents the pixel value at the position of the feature map output after the panchromatic image passes through the k1×1 convolution kernel; represents the average value of the pixel values at the position; k are three imaginary units and satisfy the following relationships:

[0083]

[0084]

[0085] .

[0086] The panchromatic image represented in quaternion form is input into the quaternion convolution kernel for feature extraction to obtain the panchromatic image quaternion feature map.

[0087] Step 2: Construct a hyperspectral image quaternion feature extraction module. The hyperspectral image quaternion feature extraction module is used to represent the hyperspectral image in quaternion form, perform feature extraction, and obtain the hyperspectral image quaternion feature map.

[0088] As Figure 3As shown in the figure, the specific process of the hyperspectral image quaternion feature extraction module representing the hyperspectral image in quaternion form and performing feature extraction to obtain the hyperspectral image quaternion feature map is as follows:

[0089] Combine three adjacent channels of the low-resolution hyperspectral image to act as the three components of the imaginary part of the quaternion, and take the average of the pixel values at the same position as the real part value of the quaternion to perform a quaternion transformation to obtain a hyperspectral image represented in quaternion form.

[0090] The representation form of the hyperspectral image represented in quaternion form is:

[0091] k

[0092] where represents the pixel value of the hyperspectral image represented in quaternion form corresponding to the positions of the H-th, (H + 1)-th, and (H + 2)-th channels of the low-resolution hyperspectral image ; represents the pixel value at the position of the H-th channel of the low-resolution hyperspectral image ; represents the pixel value at the position of the (H + 1)-th channel of the low-resolution hyperspectral image ; represents the pixel value at the position of the (H + 2)-th channel of the low-resolution hyperspectral image ; represents the average value of the pixel values at the position; , k are three imaginary units and satisfy the following relationships:

[0093]

[0094]

[0095] .

[0096] Input the hyperspectral image represented in quaternion form into the quaternion convolution kernel for feature extraction to obtain the hyperspectral image quaternion feature map.

[0097] where the quaternion convolution kernel is:

[0098]

[0099] where is the first component of the quaternion convolution kernel, is the second component of the quaternion convolution kernel, is the third component of the quaternion convolution kernel, is the fourth component of the quaternion convolution kernel.

[0100] For quaternion convolution, this dot product is completed based on quaternion multiplication. Feature extraction is performed based on quaternion multiplication, and the quaternion feature map of the hyperspectral image The calculation process is as follows:

[0101]

[0102] Among them, The hyperspectral image represented in quaternion form.

[0103] Quaternion feature extraction is performed, and independent addition and multiplication operations are carried out for each channel (real part and imaginary part). Each convolution calculation will produce a quaternion feature map output, which contains four components (one real part and three imaginary parts).

[0104] Step 3: Construct a self-attention fusion module, which is used to perform feature fusion and enhancement on the quaternion feature map of the hyperspectral image and the quaternion feature map of the panchromatic image through the self-attention mechanism.

[0105] As Figure 4 shown, the specific process of the self-attention fusion module performing feature fusion and enhancement on the quaternion feature map of the hyperspectral image and the quaternion feature map of the panchromatic image through the self-attention mechanism is as follows:

[0106] The quaternion feature map of the hyperspectral image is sequentially reshaped and linearly transformed through a reshape layer and a linear layer to obtain a Query vector and a Key vector. The quaternion feature map of the panchromatic image is sequentially reshaped and linearly transformed through a reshape layer and a linear layer to obtain a Value vector. Thus, the three vectors required for the self-attention block are obtained.

[0107] By calculating the element-wise product of the Query vector and the Key vector, the attention score is obtained, and it is normalized through the Softmax function to calculate the attention weight A. The calculation formula is:

[0108]

[0109] Among them, Q represents the Query vector, K represents the Key vector, T represents the transpose, represents the dimension of the Key vector, represents the Softmax operation. The Softmax operation ensures that each row of the obtained weight matrix represents the correlation between different channels.

[0110] Apply the attention weight A to the Value vector, and thus calculate the fused feature through weighted summation :

[0111]

[0112] Among them, represents the Value vector.

[0113] Fusion feature After reshaping, a fusion graph is obtained.

[0114] Step 4: Using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module, and the self-attention fusion module, construct a dual U-Net structure network based on multi-scale residual links and skip connections, as Figure 5 shown.

[0115] The dual U-Net structure network includes two parallel branches, namely the first U-Net branch and the second U-Net branch. The first U-Net branch is used to process the panchromatic image, and the second U-Net branch is used to process the hyperspectral image. Each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, with inter-layer skip connections; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module, with inter-layer skip connections.

[0116] In this embodiment, both the first U-Net branch and the second U-Net branch include five layers. Among them, the first layer and the fifth layer are skip-connected, and the second layer and the fourth layer are skip-connected; the low-level features and high-level features are fused through the skip connections.

[0117] Step 5: Input the panchromatic image and the low-resolution hyperspectral image into the dual U-Net structure network, and through the calculation of the dual U-Net structure network, a high-spatial-resolution hyperspectral image is calculated.

[0118] In this embodiment, the panchromatic image is input into the first U-Net branch. In each layer of the first U-Net branch, it sequentially passes through the residual module and the panchromatic image quaternion feature extraction module to calculate the panchromatic image quaternion feature map, which is input to the next layer and the self-attention fusion module of the corresponding layer of the second U-Net branch. If skip connection is performed at this layer, the panchromatic image quaternion feature map also needs to be spliced onto the output of the panchromatic image quaternion feature extraction module of the layer skip-connected to it. The low-resolution hyperspectral image is input into the second U-Net branch. In each layer of the second U-Net branch, it sequentially passes through the residual module, the hyperspectral image quaternion feature extraction module, and the self-attention fusion module, and the output result is input to the next layer. If skip connection is performed at this layer, this output result also needs to be spliced onto the corresponding output. The panchromatic image and the low-resolution hyperspectral image pass through the entire dual U-Net structure network to calculate the high-spatial-resolution hyperspectral image.

[0119] To verify the effectiveness of this method, in this embodiment, quantitative index result comparison experiments and subjective comparison experiments are conducted on the public dataset Botswana with multiple internationally state-of-the-art algorithms. The multiple internationally state-of-the-art algorithms include GT, CNMF, DARN, DSNet, FPFNet, Hyperkite, Refiner, and TreeNet. To better observe the differences between the fusion results, the mean absolute error (MAE) graph between the fusion results and the ground truth (GTs) is calculated and shown, and the local area is magnified and shown in the lower right corner of the result, and the corresponding RGB image is shown in the lower left corner, as Figure 6 shown. These images show that there are more dark blue areas in the MAE graph of this method (ous), indicating the smallest fusion error. As shown in Table 1, it can also be seen from the quantitative index results that the quantitative result indexes of this method are better than those of internationally advanced algorithms in recent years.

[0120] Table 1 Comparison experiment results table of quantitative index results for Botswana dataset

[0121]

[0122] In the table, ↑ indicates that the larger the result, the better, and ↓ indicates that the smaller the result, the better. The indexes selected are linear correlation coefficient (CC), spectral angle mapping (SAM), root mean square error (RMSE), global relative error (ERGAS), and peak signal-to-noise ratio (PSNR).

[0123] This embodiment also provides a remote sensing hyperspectral image super-resolution fusion system based on quaternions, and the technical solution adopted is as follows: including: a module construction unit, a network construction unit, and an input-output unit connected in sequence.

[0124] The module construction unit is used to construct a panchromatic image quaternion feature extraction module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module. The panchromatic image quaternion feature extraction module is used to extract features from the panchromatic image to obtain a panchromatic image quaternion feature map; the hyperspectral image quaternion feature extraction module is used to represent the hyperspectral image in the form of quaternions and perform feature extraction to obtain a hyperspectral image quaternion feature map; the self-attention fusion module is used to perform feature fusion and enhancement on the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through the self-attention mechanism.

[0125] The network construction unit is used to construct a dual U-Net structure network based on multi-scale residual links and skip connections by using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module, and the self-attention fusion module. The dual U-Net structure network includes a first U-Net branch and a second U-Net branch. Each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, with an inter-layer skip connection; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module, and a self-attention fusion module, with an inter-layer skip connection.

[0126] The input-output unit is used to input the panchromatic image and the low-resolution hyperspectral image into the dual U-Net structure network and output the hyperspectral image with high spatial resolution calculated by the dual U-Net structure network.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm, characterized in that: include: Constructing a full-color image quaternion feature extraction module, wherein the full-color image quaternion feature extraction module is used to extract features from the full-color image to obtain a full-color image quaternion feature map; The full-color image quaternion feature extraction module extracts features from the full-color image, and the specific process of obtaining the full-color image quaternion feature map is as follows: The full-color image is passed through convolution kernels with kernels of k1×k1, 1×k1, and k1×1, respectively, to extract the features of the full-color image spatial scale in different directions, and the three output feature maps are expressed in the form of quaternions to obtain a full-color image expressed in the form of quaternions; Inputting the full-color image represented in quaternion form into a quaternion convolution kernel for feature extraction to obtain a full-color image quaternion feature map; Constructing a hyperspectral image quaternion feature extraction module, wherein the hyperspectral image quaternion feature extraction module is used to represent the hyperspectral image in the form of quaternion, perform feature extraction, and obtain a hyperspectral image quaternion feature map; The hyperspectral image quaternion feature extraction module represents the hyperspectral image in the form of quaternion and extracts the features. The specific process of obtaining the hyperspectral image quaternion feature map is as follows: The three adjacent channels of the low-resolution hyperspectral image are combined to serve as the three components of the imaginary part of the quaternion, and the average of the pixel values ​​at the same position is taken as the real part of the quaternion to obtain the hyperspectral image represented in the form of quaternion; Inputting the hyperspectral image represented in quaternion form into a quaternion convolution kernel for feature extraction to obtain a quaternion feature map of the hyperspectral image; Constructing a self-attention fusion module, wherein the self-attention fusion module is used to fuse and enhance the features of the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through a self-attention mechanism; Using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module and the self-attention fusion module, a dual U-Net structure network based on multi-scale residual links and jump connections is constructed, wherein the dual U-Net structure network includes a first U-Net branch and a second U-Net branch, wherein each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, and inter-layer jump connections; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module and a self-attention fusion module, and inter-layer jump connections; The panchromatic image and the low-resolution hyperspectral image are input into the dual U-Net structure network to calculate the hyperspectral image with high spatial resolution.

2. The quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm according to claim 1, characterized in that: The representation of a hyperspectral image in quaternion form is: k in, Represents the Hth, H+1th, and H+2th channels of the low-resolution hyperspectral image The pixel value of the hyperspectral image represented by quaternion corresponding to the position; Represents the Hth channel of the low-resolution hyperspectral image The pixel value of the position; Represents the H+1th channel of the low-resolution hyperspectral image The pixel value of the position; Represents the H+2th channel of the low-resolution hyperspectral image The pixel value of the position; express The average value of the pixel values ​​at the location; , k is three imaginary units, satisfying the following relationship: ； Quaternion convolution kernel for: in, is the first component of the quaternion convolution kernel, is the second component of the quaternion convolution kernel, is the third component of the quaternion convolution kernel, is the fourth component of the quaternion convolution kernel; Feature extraction based on quaternion multiplication, quaternion feature map of hyperspectral image The calculation process is: in, Hyperspectral imagery represented in quaternion form.

3. The quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm according to claim 1, characterized in that: The representation of a full-color image in quaternion form is: k in, Represents a full-color image represented in quaternion form The pixel value of the position; Represents the feature map output after the full-color image passes through the k1×k1 convolution kernel The pixel value of the position; Represents the feature map output after the full-color image passes through the 1×k1 convolution kernel The pixel value of the position; Represents the feature map output after the full-color image passes through the k1×1 convolution kernel The pixel value of the position; express The average value of the pixel values ​​at the location; , k is three imaginary units, satisfying the following relationship: 。 4. The quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm according to claim 1, characterized in that: The specific process of the self-attention fusion module to fuse and enhance the features of the hyperspectral image quaternion feature map and the panchromatic image quaternion feature map through the self-attention mechanism is as follows: Perform linear transformation on the quaternion feature map of the hyperspectral image to obtain the Query vector and the Key vector, and perform linear transformation on the quaternion feature map of the panchromatic image to obtain the Value vector; Calculate the attention weight A: Among them, Q represents the Query vector, K represents the Key vector, and T represents the transposition. Represents the dimension of the Key vector, Represents the Softmax operation; Calculate fusion features : in, Represents the Value vector.

5. The quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm according to claim 1, characterized in that: The first U-Net branch and the second U-Net branch each include five layers, wherein the first layer and the fifth layer are jump connected, and the second layer and the fourth layer are jump connected; the low-level features are fused with the high-level features through the jump connection.

6. A quaternion-based remote sensing hyperspectral image super-resolution fusion system, characterized in that: The method is used to execute the quaternion-based remote sensing hyperspectral image super-resolution fusion algorithm according to any one of claims 1 to 5, comprising: a module construction unit, a network construction unit and an input and output unit, The module construction unit is used to construct a full-color image quaternion feature extraction module, a hyperspectral image quaternion feature extraction module and a self-attention fusion module. The full-color image quaternion feature extraction module is used to extract features from the full-color image to obtain a full-color image quaternion feature map; the hyperspectral image quaternion feature extraction module is used to represent the hyperspectral image in the form of quaternions, perform feature extraction, and obtain a hyperspectral image quaternion feature map; the self-attention fusion module is used to perform feature fusion and enhancement on the hyperspectral image quaternion feature map and the full-color image quaternion feature map through a self-attention mechanism; The network construction unit is used to construct a dual U-Net structure network based on multi-scale residual links and jump connections by using the panchromatic image quaternion feature extraction module, the hyperspectral image quaternion feature extraction module and the self-attention fusion module, wherein the dual U-Net structure network includes a first U-Net branch and a second U-Net branch, wherein each layer of the first U-Net branch includes a residual module and a panchromatic image quaternion feature extraction module, and inter-layer jump connections; each layer of the second U-Net branch includes a residual module, a hyperspectral image quaternion feature extraction module and a self-attention fusion module, and inter-layer jump connections; The input-output unit is used to input the full-color image and the low-resolution hyperspectral image into the dual U-Net structure network, and output the high-spatial-resolution hyperspectral image calculated by the dual U-Net structure network.

Images