Hyperspectral image fusion method and system based on variance guidance and heavy tail estimation

Abstract

The application relates to the technical field of hyperspectral image processing, and particularly discloses a hyperspectral image fusion method and system based on variance guidance and heavy tail estimation, which comprises the following steps: acquiring a low spatial resolution hyperspectral image and a panchromatic image; performing up-sampling on the low spatial resolution hyperspectral image to obtain an up-sampled image; obtaining an absolute difference weight according to the channel-by-channel variance of the low spatial resolution hyperspectral image and the up-sampled image; performing channel-by-channel spatial fusion with the help of the absolute difference weight, injecting high-frequency information of the panchromatic image into the up-sampled image, and obtaining a preliminary fusion image; making the preliminary fusion image pass through an attention mechanism for spectral correlation correction learning to obtain a spectral mixing output; obtaining the up-sampled image subjected to variance guidance processing through a feedforward network; and obtaining a high spatial resolution hyperspectral image through a residual block and convolution. The application realizes accurate selection and fusion of spatial and spectral features.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image processing technology, and in particular to a hyperspectral image fusion method and system based on variance-guided and heavy-tailed estimation. Background Technology

[0002] The process of fusing low spatial resolution hyperspectral images with panchromatic images to reconstruct high spatial resolution hyperspectral images is commonly known as hyperspectral panchromatic sharpening (HS pansharpening). Deep learning-based methods currently hold great promise for HS pansharpening; however, deep learning networks often operate as "black boxes," as designers do not analyze the statistical features of the LRHS (low spatial resolution hyperspectral) image and the reference (GT) image, nor do they evaluate the heavy-tailed distribution in the fusion features to reduce spatial and spectral distortion in the fusion result.

[0003] Existing fusion networks do not analyze the statistical characteristics of PAN, LRHS, and GT images when fusing hyperspectral images, such as... Figure 4 As shown, during pan-color sharpening, LRHS often needs to be upsampled to achieve the same spatial size as the PAN image. By analyzing the variance values ​​of each channel in LRHS, upsampled LRHS, and GT images from the same scene, it can be observed that there is a difference in the variance values ​​of upsampled LRHS and LRHS in each channel (LRHS - upsampled LRHS). This difference is directly proportional to the difference between LRHS and GT (LRHS - GT), that is, when "LRHS - GT" increases, "LRHS - upsampled LRHS" also increases. Pixel value statistics for the same channel in PAN, upsampled LRHS, and GT images are shown below. Figure 5 As shown, the pixel distribution exhibits a heavy-tailed distribution characteristic.

[0004] Hyperspectral images (HS images) exhibit high spectral and spatial correlations in both the spectral and spatial dimensions. Therefore, HS images can be viewed as a sequence of grayscale image data connected by global spectral correlations. However, previous deep learning-based methods neglected the sequence-based data structure of hyperspectral images, leading to information loss.

[0005] In summary, the aforementioned design flaws lead to spatial and spectral variations in the fusion results, which restricts the promotion and performance improvement of hyperspectral image panchromatic sharpening technology in practical applications. Breakthroughs and innovations at both the theoretical and technical levels are urgently needed.

[0006] 1. Insufficient ability to model based on variance statistics: Previous deep learning-based methods neglected to analyze the variance statistics between images, resulting in mismatch in feature injection during the fusion process.

[0007] 2. Heavy-tailed distribution estimation of data was not considered: Previous deep learning-based methods ignored the heavy-tailed distribution characteristics of PAN, upsampled LRHS and GT image data when fusing HS and PAN images. The L1 loss of the sampled data cannot fit this data feature well, which limits the network performance.

[0008] 3. Insufficient sequence modeling capability: Previous methods built with deep learning failed to fully consider the sequential data structure characteristics of hyperspectral images, which ultimately led to information loss.

[0009] 4. Computational Complexity and Efficiency Issues: Hyperspectral images, due to their massive number of spectral channels, exhibit high computational complexity during data processing. Efficiency bottlenecks in large-scale data processing: Methods such as matrix factorization and sparse representation typically involve complex iterative optimization processes, resulting in high computational costs and difficulty in meeting real-time processing requirements. Redundant computations in deep learning models: Existing deep learning-based models often employ Transformer-based attention mechanisms, whose parameter count and computational complexity are quadratic with the data dimensionality. This leads to high resource requirements during training and inference, demanding high hardware performance and making them unsuitable for resource-constrained scenarios. Summary of the Invention

[0010] This invention aims to solve the aforementioned problems. To this end, this invention provides a hyperspectral image fusion method and system based on variance-guided and heavy-tailed estimation. Based on the variance differences between image data and the heavy-tailed characteristics of the data distribution, a high-efficiency fusion network based on the characteristics of image statistical information is constructed for hyperspectral panchromatic sharpening, achieving accurate selection and fusion of spatial and spectral features. In the cascaded multi-stage structure of the high-efficiency fusion network, each stage contains VG-RWKV. VG-RWKV first completes preliminary fusion using VGFB based on the variance difference between LRHS and upsampled LRHS, and then the result is fed into an improved recurrent memory attention structure. This structure consists of a spectral mixing module that adjusts spectral information and reduces fusion bias, and a feedforward network that enhances the network's nonlinear fitting ability. Furthermore, considering the heavy-tailed distribution characteristics of the data, a loss is constructed through heavy-tailed distribution estimation to constrain the fusion result and reduce spatial and spectral distortion.

[0011] This invention provides a hyperspectral image fusion method based on variance-guided and heavy-tailed estimation, employing the following technical solution: including the following steps:

[0012] S1: Acquire low spatial resolution hyperspectral and panchromatic images;

[0013] S2: Upsample the low spatial resolution hyperspectral image to obtain the upsampled image; obtain the absolute difference weight based on the channel-wise variance of the low spatial resolution hyperspectral image and the upsampled image;

[0014] S3: Input the upsampled image and the panchromatic image into the variance-guided fusion module. The variance-guided fusion module uses the absolute difference weight to perform channel-by-channel spatial fusion, injecting the high-frequency information of the panchromatic image into the upsampled image to obtain a preliminary fused image.

[0015] S4: Input the preliminary fused image and absolute difference weights into the spectral mixing module. The spectral mixing module uses the absolute difference weights to enable the preliminary fused image to learn spectral correlation correction through an attention mechanism, and obtain the spectral mixed output.

[0016] S5: The spectral mixing output is passed through a feedforward network to obtain an upsampled image that has undergone variance-guided processing;

[0017] S6: Repeat steps S3 to S5 to obtain the variance-guided fusion result;

[0018] S7: Variance-guided fusion results are used to obtain high spatial resolution hyperspectral images through residual blocks and convolution.

[0019] Furthermore, in step S2, the absolute values ​​of the channel-wise variances of the low spatial resolution hyperspectral image and the upsampled image are processed by the softmax activation function to obtain the absolute difference weights.

[0020] Furthermore, in step S3, the calculation formula for the preliminary fused image is expressed as follows:

[0021]

[0022]

[0023] in, Indicates full color weight. Indicates the absolute difference weight. Indicates hyperspectral weights, This indicates a depthwise separable convolution with a 1×1 kernel. This indicates pixel-by-pixel multiplication. This indicates a splicing operation at the channel level. express The Channel weights, express The Channel weights, Represents a panchromatic image. express The image of the first aisle, This indicates the initial fusion of images.

[0024] Furthermore, in step S4,

[0025] The initially fused image is patched along the channel dimension, then the patch migration operation is performed and linear interpolation is performed to obtain the mapping results of gating, loop, key and value;

[0026] Calculate the learnable fine-tuning weights based on the absolute difference weights;

[0027] Through a bidirectional recurrent attention mechanism, based on the mapping results of gating, recurrence, key, and value, learnable fine-tuning weights, and preliminary fusion images, a spectral mixture output is obtained.

[0028] Furthermore, in step S4, the fine-tuning weights can be learned. The calculation formula is:

[0029]

[0030] in, express The result of the linear interpolation shift operation, This represents the input patch for a linear interpolation shift operation. Indicates the absolute difference weight. This indicates a patch after a bidirectional offset. This indicates a splicing operation at the channel level. This indicates that the patch is executed after the forward offset. This indicates that the patch has been executed after the backward offset. This indicates that the input is adjusted by adding an offset. Functions that are lightweight and enhanced. For learnable vectors, This represents the tanh activation function. Represents matrix multiplication. This represents the first learnable matrix. This represents the second learnable matrix.

[0031] Furthermore, the formula for calculating the spectral mixing output is as follows:

[0032]

[0033]

[0034] in, This represents the result of global attention. This represents the SiLU activation function. Representation layer normalization, Indicates and Related linear layer weights, Indicates the absolute difference weight. Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the keys obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the values ​​obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The gating mapping results obtained by performing patch migration operation and linear interpolation operation respectively. This represents the global attention calculation function. Indicates spectral mixing output, This indicates the initial fusion of images. This indicates the patch after the segmentation.

[0035] Furthermore, in step S5, patch migration and linear interpolation operations are performed on the spectral mixing output, and then linear mapping and gating mechanisms are executed respectively to obtain the feedforward network output.

[0036] Furthermore, in step S5, the calculation formula for the feedforward network output is as follows:

[0037]

[0038] in, This represents the sigmoid activation function. This represents a multilayer perceptron. This indicates pixel-by-pixel multiplication. Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the keys obtained by performing patch migration operation and linear interpolation operation respectively. This represents the output of the feedforward network. This indicates a mixed spectral output.

[0039] Furthermore, the total loss function includes reconstruction loss and heavy-tailed estimation loss. The formula for calculating heavy-tailed estimation loss is as follows:

[0040]

[0041] in, This indicates the heavy-tailed estimated loss. Indicates the number of samples. Indicates the first A high spatial resolution hyperspectral image, Indicates the first One reference image, Indicates the first Each scale parameter Indicates the first One shape parameter, express The main part, This represents the Gamma function.

[0042] The present invention also provides a hyperspectral image fusion system based on variance-guided and heavy-tailed estimation, the technical solution of which is as follows: including: an image acquisition module, a weight calculation module, multiple variance-guided recurrent linear attention modules and an output module;

[0043] The image acquisition module is used to acquire low spatial resolution hyperspectral images and panchromatic images;

[0044] The weight calculation module is used to upsample the low spatial resolution hyperspectral image to obtain the upsampled image; and to obtain the absolute difference weight based on the channel-wise variance of the low spatial resolution hyperspectral image and the upsampled image.

[0045] A variance-guided recurrent linear attention module is used to calculate the variance-guided fusion result based on the upsampled image, the panchromatic image, and the absolute difference weights.

[0046] The output module is used to obtain a high spatial resolution hyperspectral image by passing the variance-guided fusion results through residual blocks and convolution.

[0047] The variance-guided recurrent linear attention module includes a variance-guided fusion module, a spectral mixing module, and a feedforward network;

[0048] The variance-guided fusion module is used to perform channel-by-channel spatial fusion by using absolute difference weights, injecting high-frequency information from the panchromatic image into the upsampled image to obtain a preliminary fused image.

[0049] The spectral mixing module is used to learn spectral correlation correction of the preliminary fused image through an attention mechanism by using absolute difference weights, so as to obtain the spectral mixed output.

[0050] A feedforward network is used to calculate the variance-guided upsampled image based on the spectral mixing output.

[0051] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0052] This invention introduces image variance differences and heavy-tailed distribution characteristics into hyperspectral image fusion. By using a variance-guided fusion module, it effectively avoids feature injection mismatch issues and reduces spatial and spectral distortion during the fusion process. Combined with heavy-tailed distribution modeling and loss function design, it improves robustness to outliers and uncertainties, thus ensuring more stable and reliable fusion results. Furthermore, this invention achieves sequential modeling in both the spectral and spatial dimensions, fully utilizing the global correlation of hyperspectral data to improve feature learning and fusion accuracy.

[0053] This invention guides PAN images to undergo feature fusion to varying degrees based on the magnitude of the variance differences by considering the relationships between variances, thereby avoiding incorrect information injection that could lead to spatial and spectral distortions in the fusion results. In fact, hyperspectral images exhibit strong correlations in both the spectral and spatial dimensions—possessing both significant spectral correlations and prominent spatial correlations. Based on this characteristic, hyperspectral images can be defined as a collection of sequential grayscale images linked by global spectral correlations.

[0054] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 This is a flowchart of the method provided by the present invention.

[0057] Figure 2 This is a system architecture diagram provided by the present invention.

[0058] Figure 3 This is a subjective comparison experiment diagram of the Pavia center dataset provided by this invention.

[0059] Figure 4 This is a variance statistics information graph provided by the present invention.

[0060] Figure 5 This is the pixel value distribution histogram provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0062] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0063] The following is combined Figures 1 to 3 The present invention will be further described in detail below, describing a hyperspectral image fusion method and system based on variance-guided and heavy-tailed estimation:

[0064] like Figure 2 As shown, this embodiment proposes a Variance Guidance and Heavy-tailed Estimation Network (VGHNet) with a cascaded multi-stage structure. Each stage constructs a variance-guided recurrent linear attention module (VG-RWKV). In VG-RWKV, a variance-guided fusion block (VGFB) is first designed to fuse the LRHS and upsampled LRHS images using variance difference information. After initial fusion, the fusion result is fed into an improved Circular-Memory-Attention Structure, which mainly consists of a spectral mixing module and a feedforward network. The spectral mixing module learns the spectral correlation between channels to adjust the spectral information of the fusion result, reducing fusion bias. The feedforward neural network is used to increase the network's nonlinear fitting capability.

[0065] Furthermore, based on the heavy-tailed distribution characteristics of the data, this embodiment performs heavy-tailed distribution estimation on the data distribution of the fusion result. By constructing a heavy-tailed estimation loss, the data information of the fusion result is further constrained, thereby further reducing the spatial and spectral distortion of the fusion result. This algorithm aims to provide an efficient HSpansharpening method based on the statistical information of hyperspectral images, which can significantly improve the spatial resolution of hyperspectral images while maintaining their spectral consistency, providing technical support for the refined analysis and practical application of hyperspectral images.

[0066] In this embodiment, as Figure 1 As shown, a hyperspectral image fusion method based on variance-guided and heavy-tailed estimation is provided, including the following steps:

[0067] S1: Acquire low spatial resolution hyperspectral images and panchromatic images.

[0068] S2: Upsample the low spatial resolution hyperspectral image to obtain the upsampled image; obtain the absolute difference weight based on the channel-by-channel variance of the low spatial resolution hyperspectral image and the upsampled image.

[0069] For low spatial resolution hyperspectral images Upsampling yields an upsampled image. , , Represents the real number field. express The number of channels, express height, express Image width, , express height, express The width. This embodiment uses a 4x upsampling.

[0070] After that and Perform channel-by-channel variance calculation and obtain the absolute difference between the two values. The specific operation is as follows:

[0071] (1)

[0072] in, Indicates the absolute difference weight. This represents the function for calculating the variance per channel. This represents the softmax activation function, which maps absolute difference data to the range [0,1].

[0073] Then, and panchromatic images The data is fed into the designed VG-RWKV for channel-by-channel spatial fusion and spectral correlation correction learning (steps S3-S5). .

[0074] S3: Input the upsampled image and the panchromatic image into the variance-guided fusion module. The variance-guided fusion module uses the absolute difference weight to perform channel-by-channel spatial fusion, injecting the high-frequency information of the panchromatic image into the upsampled image to obtain a preliminary fused image.

[0075] In VG-RWKV, channel-by-channel spatial fusion is first performed using the Variance Guidance Fusion Block (VGFB). High-frequency information is then injected into the PAN plot using variance difference information. The specific operation is as follows:

[0076] (2)

[0077] (3)

[0078] in, Indicates full color weight. Indicates hyperspectral weights, This indicates a depthwise separable convolution with a 1×1 kernel. This indicates pixel-by-pixel multiplication. This indicates a splicing operation at the channel dimension. express The Channel weights, express The Channel weights, Represents a panchromatic image. express The image of the first aisle, This indicates the initial fusion of images.

[0079] S4: Input the preliminary fused image and absolute difference weights into the spectral mixing module. The spectral mixing module uses the absolute difference weights to enable the preliminary fused image to learn spectral correlation correction through an attention mechanism, and obtain the spectral mixed output.

[0080] Will With weight In the input spectral mixing module, spectral correlation correction learning is performed. Within the spectral mixing module, global attention computation is performed in linear complexity as an attention mechanism.

[0081] First, The patch is segmented along the channel dimension, then a patch migration operation is performed followed by linear interpolation. The specific steps are as follows:

[0082] (4)

[0083] in, This represents the patch after splitting. This indicates that the data is grouped on an average basis along the channel dimension. This represents a spatial remodeling operation. Its dimensional changes are represented as follows: ,in, , This represents the number of channels in each patch after dividing the channel dimension into T groups. This represents the t-th patch after grouping. This represents the t-th patch after the split, where T is the total number of patches.

[0084] Perform patch migration and linear interpolation operations:

[0085] (5)

[0086] in, Indicates to The mapping results of gating, loop, key, and value obtained by performing patch migration operation and linear interpolation operation respectively. The dot (·) indicates a linear interpolation shift operation, and the dot (·) indicates pixel-by-pixel multiplication. Indicates the weights of the first linear layer. This is a mapping type identifier symbol, with values ​​G, R, K, and V. G represents gating, R represents looping, K represents key, and V represents value.

[0087] The specific steps are as follows:

[0088] (6)

[0089] in, For learnable weight vectors, This represents the input patch for a linear interpolation shift operation. This represents a patch after a bidirectional offset. These offset features enhance the interaction of spatial features, enabling attention to neighboring patches. Specifically, bidirectional offsets ensure the interaction between neighboring patches without adding additional computational complexity. This indicates that the patch is executed after the forward offset. This indicates that the patch has been executed after the backward offset. , .

[0090] Then, based on the weights associated with the input Constructing learnable fine-tuning weights :

[0091] (7)

[0092] in, express The result of the linear interpolation shift operation, This represents the tanh activation function. Represents matrix multiplication. For learnable vectors, This represents the first learnable matrix. This represents the second learnable matrix. This indicates that the input is adjusted by adding an offset. A function that performs lightweight enhancements. This process makes the weights... Each channel can be based on current and previous patches The mixture and variance-guided changes are used to make changes.

[0093] Subsequently, , , and It was used to compute global attention results through a bidirectional recurrent attention mechanism with linear complexity. The attention result is then compared with the gating. Multiplying yields the output:

[0094] (8)

[0095] (9)

[0096] in, This represents the SiLU activation function. Representation layer normalization, Indicates and Related linear layer weights, Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the keys obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the values ​​obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The gating mapping results obtained by performing patch migration operation and linear interpolation operation respectively. This represents the global attention calculation function. This indicates a mixed spectral output.

[0097] The calculation can be expressed as:

[0098] (10)

[0099] In the formula, This represents the matrix diagonalization operation. This represents the channel-by-channel learnable enhancement parameters. Indicates the dynamic decay factor. , express The component of the j-th channel, This represents an intermediate result of global attention. express The transpose of the t-th block, express The t-th block, express The t-th block, where i represents the cumulative index.

[0100] After the above operations, The spectral information is used for correction and learning, and then the output of the spectral mixing module is incorporated into the feedforward network to enhance the network's nonlinear fitting ability.

[0101] S5: The spectral mixing output is passed through a feedforward network to obtain an upsampled image that has undergone variance-guided processing.

[0102] In feedforward networks, patch migration and linear interpolation operations are performed first, and similarly... and pass get:

[0103] (11)

[0104] in, Indicates to The mapping results of loops and keys obtained by performing patch migration operations and linear interpolation operations respectively. This represents the weights of the second linear layer.

[0105] Then, linear mapping and gating mechanisms are executed separately to obtain the feedforward network output. That is, the upsampled image after variance-guided processing:

[0106] (12)

[0107] in, This represents the sigmoid activation function. This represents a multilayer perceptron. Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the keys obtained by performing patch migration operation and linear interpolation operation respectively.

[0108] S6: Repeat steps S3 to S5 to obtain the variance-guided fusion result.

[0109] The network undergoes N stages of spectral correlation correction and learning using VG-RWKV to achieve a good fusion result. The upsampled image calculated by the previous VG-RWKV and processed by variance guidance is used as the input for the next VG-RWKV; the output of the last VG-RWKV is the variance-guided fusion result.

[0110] S7: The variance-guided fusion results are processed through residual blocks and convolution to obtain a high spatial resolution hyperspectral (HRHS) image.

[0111] To further suppress spectral distortion, this embodiment constructs a heavy-tailed estimation structure and designs a heavy-tailed estimation loss for the heavy-tailed distribution of hyperspectral data.

[0112] In hyperspectral panchromatic sharpening, previous network... Given input and The network model will generate a pixel-by-pixel prediction result:

[0113] (13)

[0114] in, Represents any high spatial resolution hyperspectral image. Represents network parameters, Represents any low spatial resolution hyperspectral image. Represents any panchromatic image.

[0115] However, this point prediction cannot characterize the prediction distribution. ,in, The image represents a reference image, therefore it cannot capture the uncertainty in the output. To utilize this uncertainty to guide the network in finding optimal parameters... The model needs to estimate a conditional distribution. The parameters are used to capture the uncertainty in the predicted output and ultimately maximize the likelihood function through these parameters.

[0116] Therefore, this embodiment will use the network. parameters Represented as a set of trainable parameters, the task outputs a set of parameters:

[0117] (14)

[0118] in, These represent distribution parameters, which can be used to describe the predicted distribution. Then select distribution In order to use closed-form solutions that depend on the model's estimated parameters to predict uncertainties, while maximizing the likelihood function. To obtain the optimal parameters of the network This will lead to the best fusion result:

[0119] (15)

[0120] (16)

[0121] in, Represents an uncertain mapping function, This represents an uncertainty estimate, and N represents the total number of samples.

[0122] Figure 5 The pixel value distribution is shown, revealing that it follows a heavy-tailed distribution. Therefore, to make the model robust to various outliers, this embodiment uses a heteroscedastic generalized Gaussian distribution to simulate the heavy-tailed distribution of the output. When using a heavy-tailed distribution to model the fusion result, Designed as Scale parameters and shape parameters Scale parameters and shape parameters As trainable parameters, i.e. The optimization problem described above can be expressed as follows:

[0123] (17)

[0124] (18)

[0125] in, Represents the Gamma function. express The main part, Let represent the i-th reference image. By utilizing the parameters of these different prediction distributions, the uncertainty of the output can be calculated, and a heavy-tailed sub-estimation loss can be constructed. In the network, the scale parameter... and shape parameters The LRHS image is obtained by concatenated residual blocks and 3×3 convolutions, estimated by E 3×3 convolutions plus PReLU.

[0126] Based on the above, this embodiment proposes a joint loss function for the network, which includes reconstruction loss. As the main loss, the heavy-tailed estimated loss As an auxiliary loss function, the total loss It can be represented as:

[0127] (19)

[0128] (20)

[0129] (twenty one)

[0130] in, This represents the weight hyperparameter. Indicates the number of samples. Indicates the first A high spatial resolution hyperspectral image, Indicates the first One reference image, Indicates the first Each scale parameter Indicates the first One shape parameter, express The main part, This represents the L1 norm.

[0131] To verify the effectiveness of this embodiment, quantitative and subjective comparison experiments were conducted on the public dataset Pavia Center, comparing it with several state-of-the-art algorithms. The experimental results are shown in Table 1. Figure 3 As shown.

[0132] As shown in Table 1, the quantitative results demonstrate that the quantitative indicators of our method are superior to those of advanced international algorithms in recent years. Advanced algorithms used include Refiner, TreeNet, DFCFN, and FPFNet. The indicators used are SAM (spectral angle similarity), RMSE (root mean square error), ERGAS (relative global dimensionless composite error), and PSNR (peak signal-to-noise ratio). ↓ indicates a smaller value is better, and ↑ indicates a larger value is better.

[0133] Table 1

[0134]

[0135] like Figure 3As shown, to better observe the differences between the fusion results, the mean absolute error (MAE) plot between the fusion result and the true value (GT) is calculated and displayed. A magnified view of the local area is shown in the lower right corner of the result, and the corresponding RGB image is displayed in the lower left corner. These images show that the MAE plot of this method has more dark blue areas, indicating the smallest fusion error.

[0136] This embodiment also provides a hyperspectral image fusion system based on variance-guided and heavy-tailed estimation, and the technical solution adopted is as follows: including: an image acquisition module, a weight calculation module, multiple variance-guided recurrent linear attention modules and an output module.

[0137] The image acquisition module is used to acquire low spatial resolution hyperspectral images and panchromatic images;

[0138] The weight calculation module is used to upsample the low spatial resolution hyperspectral image to obtain the upsampled image; and to obtain the absolute difference weight based on the channel-wise variance of the low spatial resolution hyperspectral image and the upsampled image.

[0139] A variance-guided recurrent linear attention module is used to calculate the variance-guided fusion result based on the upsampled image, the panchromatic image, and the absolute difference weights.

[0140] The output module is used to obtain a high spatial resolution hyperspectral image by passing the variance-guided fusion results through residual blocks and convolution.

[0141] The variance-guided recurrent linear attention module includes a variance-guided fusion module, a spectral mixing module, and a feedforward network;

[0142] The variance-guided fusion module is used to perform channel-by-channel spatial fusion by using absolute difference weights, injecting high-frequency information from the panchromatic image into the upsampled image to obtain a preliminary fused image.

[0143] The spectral mixing module is used to learn spectral correlation correction of the preliminary fused image through an attention mechanism by using absolute difference weights, so as to obtain the spectral mixed output.

[0144] A feedforward network is used to calculate the variance-guided upsampled image based on the spectral mixing output.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral image fusion method based on variance-guided and heavy-tailed estimation, characterized in that, include: S1: Acquire low spatial resolution hyperspectral and panchromatic images; S2: Upsample the low spatial resolution hyperspectral image to obtain the upsampled image; obtain the absolute difference weight based on the channel-wise variance of the low spatial resolution hyperspectral image and the upsampled image; S3: Input the upsampled image and the panchromatic image into the variance-guided fusion module. The variance-guided fusion module uses the absolute difference weight to perform channel-by-channel spatial fusion, injecting the high-frequency information of the panchromatic image into the upsampled image to obtain a preliminary fused image. S4: Input the preliminary fused image and absolute difference weights into the spectral mixing module. The spectral mixing module uses the absolute difference weights to enable the preliminary fused image to learn spectral correlation correction through an attention mechanism, and obtain the spectral mixing output. S5: The spectral mixing output is passed through a feedforward network to obtain an upsampled image that has undergone variance-guided processing; S6: Repeat steps S3 to S5 to obtain the variance-guided fusion result; S7: Variance-guided fusion results are used to obtain high spatial resolution hyperspectral images through residual blocks and convolution.

2. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 1, characterized in that, In step S2, the absolute values ​​of the channel-wise variances of the low spatial resolution hyperspectral image and the upsampled image are processed by the softmax activation function to obtain the absolute difference weights.

3. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 1, characterized in that, In step S3, the calculation formula for the preliminary fused image is expressed as follows: in, Indicates full color weight. Indicates the absolute difference weight. Indicates hyperspectral weights, This indicates a depthwise separable convolution with a 1×1 kernel. This indicates pixel-by-pixel multiplication. This indicates a splicing operation at the channel level. express The Channel weights, express The Channel weights, Represents a panchromatic image. express The image of the first aisle, This indicates the initial fusion of images.

4. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 1, characterized in that, In step S4, The initially fused image is patched along the channel dimension, then the patch migration operation is performed and linear interpolation is performed to obtain the mapping results of gating, loop, key and value; Calculate the learnable fine-tuning weights based on the absolute difference weights; Through a bidirectional recurrent attention mechanism, based on the mapping results of gating, recurrence, key, and value, learnable fine-tuning weights, and preliminary fusion images, a spectral mixture output is obtained.

5. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 4, characterized in that, In step S4, the fine-tuning weights can be learned. The calculation formula is: in, express The result of the linear interpolation shift operation, This represents the input patch for a linear interpolation shift operation. Indicates the absolute difference weight. This indicates a patch after a bidirectional offset. This indicates a splicing operation at the channel level. This indicates that the patch is executed after the forward offset. This indicates that the patch has been executed after the backward offset. This indicates that the input is adjusted by adding an offset. Functions that are lightweight and enhanced. For learnable vectors, This represents the tanh activation function. Represents matrix multiplication. This represents the first learnable matrix. This represents the second learnable matrix.

6. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 4, characterized in that, The formula for calculating the spectral mixing output is: in, This represents the result of global attention. This represents the SiLU activation function. Representation layer normalization, Indicates and Related linear layer weights, Indicates the absolute difference weight. Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The key mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The mapping results of the values ​​obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The gating mapping results obtained by performing patch migration operation and linear interpolation operation respectively. This represents the global attention calculation function. Indicates spectral mixing output, This indicates the initial fusion of images. This indicates the patch after the segmentation.

7. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 1, characterized in that, In step S5, patch migration and linear interpolation operations are performed on the spectral mixing output, and then linear mapping and gating mechanisms are executed to obtain the feedforward network output.

8. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 7, characterized in that, In step S5, the formula for calculating the output of the feedforward network is: in, This represents the sigmoid activation function. This represents a multilayer perceptron. This indicates pixel-by-pixel multiplication. Indicates to The cyclic mapping results obtained by performing patch migration operation and linear interpolation operation respectively. Indicates to The key mapping results obtained by performing patch migration operation and linear interpolation operation respectively. This represents the output of the feedforward network. This indicates a mixed spectral output.

9. The hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in claim 1, characterized in that, The total loss function includes reconstruction loss and heavy-tailed estimation loss. The formula for calculating heavy-tailed estimation loss is as follows: in, This indicates the heavy-tailed estimated loss. Indicates the number of samples. Indicates the first A high spatial resolution hyperspectral image, Indicates the first One reference image, Indicates the first Each scale parameter Indicates the first One shape parameter, express The main part, This represents the Gamma function.

10. A hyperspectral image fusion system based on variance-guided and heavy-tailed estimation, characterized in that, A hyperspectral image fusion method based on variance-guided and heavy-tailed estimation as described in any one of claims 1 to 9, comprising: The system includes an image acquisition module, a weight calculation module, multiple variance-guided recurrent linear attention modules, and an output module. The image acquisition module is used to acquire low spatial resolution hyperspectral images and panchromatic images; The weight calculation module is used to upsample the low spatial resolution hyperspectral image to obtain the upsampled image; and to obtain the absolute difference weight based on the channel-wise variance of the low spatial resolution hyperspectral image and the upsampled image. A variance-guided recurrent linear attention module is used to calculate the variance-guided fusion result based on the upsampled image, the panchromatic image, and the absolute difference weights. The output module is used to obtain a high spatial resolution hyperspectral image by passing the variance-guided fusion results through residual blocks and convolution. The variance-guided recurrent linear attention module includes a variance-guided fusion module, a spectral mixing module, and a feedforward network; The variance-guided fusion module is used to perform channel-by-channel spatial fusion by using absolute difference weights, injecting high-frequency information from the panchromatic image into the upsampled image to obtain a preliminary fused image. The spectral mixing module is used to learn spectral correlation correction of the preliminary fused image through an attention mechanism by using absolute difference weights, so as to obtain the spectral mixed output. A feedforward network is used to calculate the variance-guided upsampled image based on the spectral mixing output.

Images