A dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion

Abstract

This invention discloses a dual-prior-guided cross-Mamba method for hyperspectral and multispectral image fusion. It includes the following steps: acquiring remote sensing images of the same region and extracting features from low-resolution and high-resolution multispectral images; then processing these features through modules 1, 2, and 3 to obtain the reconstructed high-resolution hyperspectral image. This invention enhances the spatial feature representation of low-resolution hyperspectral images, integrates multimodal semantics and edge structure prior information, and fully learns complementary local, regional, and global spatial-spectral features, thereby improving the effectiveness and robustness of hyperspectral and multispectral image fusion.

Description

Technical Field

[0001] This invention specifically relates to a dual-prior guided cross-Mamba method for the fusion of hyperspectral and multispectral images. Background Technology

[0002] Hyperspectral images capture a wide range of spectral bands, providing rich spectral information, which is crucial for remote sensing, medical imaging, and environmental monitoring applications. However, the practical application of hyperspectral images is often limited by low spatial or spectral resolution due to hardware constraints, thus limiting their effectiveness. Hyperspectral image super-resolution offers a promising solution by enhancing the reconstruction of high-resolution hyperspectral images through dual priors of semantic structure extracted from low-resolution images, thereby improving their spatial and spectral quality and enabling more accurate and reliable analysis.

[0003] Popular deep learning architectures in computer vision excel at focusing meaningful regions through local connectivity (e.g., Convolutional Neural Networks (CNNs), capturing region relationships using self-attention mechanisms (e.g., Transformers), and extracting long-range dependencies through state-space models (e.g., Mamba). However, designing an effective hybrid architecture of CNNs, transformers, and Mamba, guided by rich prior information, to efficiently learn complementary spatial-spectral feature information in complex hyperspectral images remains a challenge. Summary of the Invention

[0004] Purpose of the invention: To provide a dual-prior guided cross-Mamba method for the fusion of hyperspectral and multispectral images, which solves the above-mentioned problems existing in the prior art.

[0005] Technical solution: A dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion, comprising the following steps:

[0006] Obtain low-resolution hyperspectral images from remote sensing images of the same region. and high-resolution multispectral images Extracting low-resolution hyperspectral images Shallow features in Extracting high-resolution multispectral images Shallow features in ;

[0007] shallow features Feature maps of the same scale are obtained through convolutional downsampling. Same-scale feature map shallow features The same scale yields the same scale feature map. Shallow features and feature maps of the same scale As The module's input is used to extract global feature information. Shallow features Upsampled scale and shallow features The fused features are obtained by stitching along the channel dimension. ;

[0008] In three groups Modules are interconnected to obtain fusion features. , with fusion features The intermediate layer features at the corresponding scale are obtained as input. , , This enables the network to capture local, regional, and global feature information at the corresponding scale;

[0009] intermediate layer features After upsampling by a factor of two, and with intermediate layer features Adding them together yields the intermediate layer features. ,Will After upsampling by a factor of two, and with intermediate layer features Adding them together gives Then, features are obtained through convolution. , will feature Input to feedforward neural network In this process, a high-resolution hyperspectral image was obtained. .

[0010] Preferably, global feature information is extracted. The specific steps are as follows:

[0011] shallow features Mapped to query features respectively Key features Sum value characteristics In the middle, feature map of the same scale Mapping to query features Bond features Shallow features Dimensional transformation is performed to extract global feature information;

[0012] Along height and width The dimensions are averaged and pooled to generate two-dimensional compressed key features. , , key features , By stitching the images along the channel direction, a fused feature map integrating spatial dimension information is obtained. ;

[0013] Along height and width The dimensions are averaged and pooled to generate two-dimensional compressed key features. , , key features , By stitching the images along the channel direction, a fused feature map integrating spatial dimension information is obtained. The calculation formula is as follows:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] In the formula: Indicates height Average pooling operation, Indicates width Dimensional average pooling operation, This indicates a splicing operation along the channel direction;

[0021] query features With fusion map Perform matrix multiplication, then use The function is normalized to obtain the spatial attention weight matrix. Query features With fusion map Perform matrix multiplication, then use The function is normalized to obtain the spatial attention weight matrix. The calculation formula is as follows:

[0022] ;

[0023] ;

[0024] In the formula: This represents the softmax function;

[0025] After completing the spatial attention weight matrix Spatial attention weight matrix After calculation, the spatial attention weight matrix is... Spatial attention weight matrix After averaging and characteristic values Matrix multiplication is used to obtain global feature information. The calculation formula is as follows:

[0026] ;

[0027] ;

[0028] In the formula: Represents global feature information. Represents the spatial attention weight matrix Spatial attention weight matrix Take the average.

[0029] Preferred, fusion features The fusion steps are as follows:

[0030] Obtain global feature information global feature information Upsampling and shallow features The fused features can be obtained by stitching along the channel dimension. The calculation formula is as follows:

[0031] ;

[0032] In the formula: This indicates an upsampling operation of 4 times.

[0033] Preferred, intermediate layer features The processing steps are as follows:

[0034] Obtain fusion features Enter into the first group Simple convolutional branches and cross-modal modules Within the fusion branch, through convolutional branches and cross-modal... By fusion and intra-branch collaborative extraction of spatial and spectral features at corresponding scales, spatial features are obtained respectively. and spectral characteristics The calculation formula is as follows:

[0035] ;

[0036] ;

[0037] In the formula: Represents a simple convolution branch. Indicates cross-modal Semantic fusion branch;

[0038] In completing the features and spectral characteristics Then, the features and spectral characteristics After concatenation along the channel dimension, the number of channels is adjusted using a convolution with a kernel of 1, and then fed into the cross-modal Mamba fusion module along with a channel attention mechanism. The high-resolution multispectral image is then processed by the cross-modal Mamba fusion module. The global edge structure is integrated, and its calculation formula is as follows:

[0039] ;

[0040] ;

[0041] In the formula: Represents the characteristic tensor. This represents a convolution operation with a kernel size of 1. This indicates the channel attention mechanism operation; This indicates cross-modal Mamba fusion.

[0042] Preferred features The calculation steps are as follows:

[0043] Fusion features Generate by applying a convolution filter Filter feature map of the number of channels Then from the perspective of fusion characteristics Select the convolution filter to generate the corresponding number of feature channels. fusion filter feature map Then local spatial features With fusion filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. Again from the perspective of fusion characteristics Select the number of feature channels and the splicing and fusion features Matched filter feature map Then, the splicing and fusion features are combined. With filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. By analogy, the splicing and fusion features can be obtained. The specific formula is as follows:

[0044]

[0045]

[0046]

[0047]

[0048] In the formula: This represents the convolution filter operation. This represents the LeakyReLU activation function. Represents a positive integer.

[0049] Preferred spectral characteristics The calculation steps are as follows:

[0050] Fusion features The self-attention mechanism of the application window obtains global features of the intermediate layer. Then and high-resolution multispectral images The input is fed into the cross-semantic fusion module, which first performs... and high-resolution multispectral images At its height and width The main fusion features are obtained by performing average pooling operations on two dimensions. Main fusion features Main fusion features Main fusion features Subsequently, the main fusion features were analyzed. Main fusion features and main fusion features Main fusion features The global intermediate layer features are obtained by concatenating them. Global intermediate layer features , global intermediate layer features With global intermediate layer features Use after multiplication The correlation weight matrix is ​​obtained by function calculation. The correlation weight matrix With global intermediate layer features The product is multiplied to obtain the global semantic attention weight matrix. For the global semantic attention weight matrix height and width Two-dimensional The normalization operation is highly Global semantic awareness attention weights in dimensions and height Global semantic awareness attention weights in dimensions Then, the global semantic awareness attention weights are... Global semantic awareness attention weights and intermediate layer global features Multiplication yields spectral characteristics The calculation formula is as follows:

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] In the formula: This indicates the self-attention of the window. Represents a linear mapping function. Indicates at altitude Dimensional Activation function Indicates the width Dimensional Activation function.

[0064] Preferred, intermediate layer features The calculation process is as follows:

[0065] Using Laplace to analyze high-resolution multispectral images Processing yields the corresponding edge feature map spectral characteristics and edge feature map After undergoing layer normalization, linear mapping, convolution, and SiLU activation function processing, the data is input into the cross-modal Mamba fusion module to complete the interaction between the two modal variables and output intermediate features. .

[0066] Preferably, the complementary information of multiple modalities is obtained by averaging the transition matrix A after layer normalization, linear mapping, convolution, and SiLU activation function processing, and the complementary information of another modality is obtained by averaging the transition matrix B after layer normalization, linear mapping, convolution, and SiLU activation function processing. The calculation formula is as follows:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] In the formula: express The state transition matrix, Representation of edge feature map The state transition matrix, express The projection matrix, Representation of edge feature map The projection matrix, Indicates the time scale parameter. The input represents time t. This represents the hidden state at time t.

[0076] Preferred, intermediate layer features and The calculation steps are as follows:

[0077] Intermediate layer features Downsample by twice the input to the second In the middle, repeat the intermediate layer features The processing steps yield intermediate layer features. , intermediate layer features Downsample by twice the input to the third Middle repeating intermediate layer features The processing steps are obtained The specific formula is as follows:

[0078] ;

[0079] ;

[0080] In the formula: This indicates a downsampling operation of the scale size by a factor of 2. This represents the CNN-Transformer-Mamba model.

[0081] Preferably, the features Input to feedforward neural network The final high-resolution hyperspectral image was obtained from the image. The specific steps are as follows:

[0082] First, the features After inputting into the layer normalization process, normalized features are obtained. Then, the features are... Decompose into sub-features at the channel dimension and The model employs two parallel pointwise convolutions and depthwise convolutions for feature extraction, improving its feature representation capability. The GELU activation function is used in one branch, followed by sub-feature extraction. and Multiplication yields the final feature Sub-features With final features The images are stitched together along the channel dimension, and then convolution with a kernel size of 1 is used to obtain the final reconstructed high-resolution hyperspectral image. The calculation formula is as follows:

[0083] ;

[0084] ;

[0085] ;

[0086] In the formula: This indicates splitting along the channel dimension. and This represents a pointwise convolution filter. and This represents a depthwise convolutional filter with a kernel size of 3. This represents the GELU activation function. This represents a convolution with a kernel size of 1.

[0087] Beneficial effects: This invention relates to a dual-prior guided cross-Mamba method for fusion of hyperspectral and multispectral images, employing CANL, CTMB, and FFN modules. The CANL module incorporates high-resolution multispectral images. High-resolution spatial correlation to adjust low-resolution hyperspectral images Spatial attention weights better reflect low-resolution hyperspectral images. Spatial correlation; the CTMB module consists of a simple convolutional branch and a cross-modal Transformer semantic fusion branch in parallel. The cross-modal Transformer semantic fusion branch includes two attention mechanisms: a window-based self-attention mechanism to capture regional features within each window, and a cross-semantic fusion mechanism to facilitate feature interactions between windows, and leverages high-resolution multispectral images. Semantic priors enhance global semantic representation. The cascaded cross-modal Mamba semantic fusion module after dual-branch fusion effectively integrates low-resolution hyperspectral images from the fused model. Features and high-resolution multispectral images The complementary information of edge structure information; finally, the features are further optimized in the feedforward network FFN. That is, the present invention enhances the spatial feature representation of low-resolution hyperspectral images, integrates multimodal semantics and edge structure prior information, fully learns the complementary spatial-spectral features of local, regional and global, and improves the effectiveness and robustness of hyperspectral and multispectral image fusion. Attached Figure Description

[0088] Figure 1 This is a system block diagram of the present invention;

[0089] Figure 2 The high-resolution multispectral image and low-resolution hyperspectral image are given in this invention;

[0090] Figure 3 This is a pseudo-color image of the fusion result diagram of the present invention;

[0091] Figure 4 This is a pseudo-color image of the actual fusion result image given in this invention. Detailed Implementation

[0092] like Figures 1 to 4 As shown, this invention provides a technical solution: a dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion, comprising the following steps:

[0093] Obtain low-resolution hyperspectral images from remote sensing images of the same region. and high-resolution multispectral images Extracting low-resolution hyperspectral images Shallow features in Extracting high-resolution multispectral images Shallow features in ;

[0094] shallow features Feature maps of the same scale are obtained through convolutional downsampling. Same-scale feature map shallow features The same scale yields the same scale feature map Shallow features and feature maps of the same scale As The module's input is used to extract global feature information. Among them, global feature information is extracted. The specific steps are as follows:

[0095] shallow features Mapped to query features respectively Key features Sum value characteristics In the middle, feature map of the same scale Mapping to query features Bond features The shallow features x-dimensionally are transformed to extract global feature information;

[0096] Along height and width The dimensions are averaged and pooled to generate two-dimensional compressed key features. , , , key features , By stitching the images along the channel direction, a fused feature map integrating spatial dimension information is obtained. ;

[0097] Along height and width The dimensions are averaged and pooled to generate two-dimensional compressed key features. , , , key features , By stitching the images along the channel direction, a fused feature map integrating spatial dimension information is obtained. The calculation formula is as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] In the formula: Indicates height Average pooling operation, Indicates width Dimensional average pooling operation, This indicates a splicing operation along the channel direction;

[0105] query features With fusion map Perform matrix multiplication, then use The function is normalized to obtain the spatial attention weight matrix. Query features With fusion map Perform matrix multiplication, then use The function is normalized to obtain the spatial attention weight matrix. The calculation formula is as follows:

[0106] ;

[0107] ;

[0108] In the formula: express function;

[0109] After completing the spatial attention weight matrix Spatial attention weight matrix After calculation, the spatial attention weight matrix is... Spatial attention weight matrix After averaging and characteristic values Matrix multiplication is used to obtain global feature information. The calculation formula is as follows:

[0110] ;

[0111] ;

[0112] In the formula: Represents global feature information. Represents the spatial attention weight matrix Spatial attention weight matrix Take the average.

[0113] Shallow features Upsampled scale and shallow features The fused features are obtained by stitching along the channel dimension. Normally, images are 2D, with only height and width. Hyperspectral images, however, are 3D, with channels representing thickness and fusion features. Then through global feature information After upsampling, maintain the shallow features The same size, then combine it with shallow features The fused features are obtained by stitching along the channel dimension. Among them, fusion features The fusion steps are as follows:

[0114] Obtain global feature information global feature information Upsampling and shallow features The fused features can be obtained by stitching along the channel dimension. The calculation formula is as follows:

[0115] ;

[0116] In the formula: This indicates an upsampling operation of 4 times.

[0117] In three groups Modules are interconnected to obtain fusion features. , with fusion features The intermediate layer features at the corresponding scale are obtained as input. , , This enables the network to capture local, regional, and global feature information at the corresponding scale;

[0118] Among them, intermediate layer features The processing steps are as follows:

[0119] Obtain fusion features Enter into the first group Simple convolutional branches and cross-modal modules Within the fusion branch, through convolutional branches and cross-modal... By fusion and intra-branch collaborative extraction of spatial and spectral features at corresponding scales, spatial features are obtained respectively. and spectral characteristics The calculation formula is as follows:

[0120] ;

[0121] ;

[0122] In the formula: Represents a simple convolution branch. Indicates cross-modal Semantic fusion branch; among which, spectral features The calculation steps are as follows:

[0123] Fusion features The self-attention mechanism of the application window obtains global features of the intermediate layer. Then, the global features of the intermediate layer are... The high-resolution multispectral image Y is input into the cross-semantic fusion module to promote effective interaction between the high-resolution multispectral image and image reconstruction at different scales. The cross-semantic fusion module first processes the intermediate layer global features... The main fusion features are obtained by performing average pooling operations on the high-resolution multispectral image Y along its height H and width W. Main fusion features Main fusion features Main fusion features Subsequently, respectively , and , The main fusion features are obtained by splicing. Main fusion features , will integrate the main features With key fusion features Use after multiplication The correlation weight matrix is ​​obtained by function calculation. The correlation weight matrix With global intermediate layer features The product is multiplied to obtain the global semantic attention weight matrix. For the global semantic attention weight matrix height and width Two-dimensional The normalization operation achieves high efficiency. Global semantic awareness attention weights in dimensions and get high Global semantic awareness attention weights in dimensions Subsequently, it will receive a high degree of attention. Global semantic awareness attention weights in dimensions , obtain high Global semantic awareness attention weights in dimensions and intermediate layer global features Multiplication yields spectral characteristics The calculation formula is as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] ;

[0136] In the formula: This indicates the self-attention of the window. Represents a linear mapping function. Indicates at altitude Dimensional Activation function Indicates the width Dimensional Activation function.

[0137] In completing the features and Then, the features and After concatenation along the channel dimension, the number of channels is adjusted using a convolution with a kernel of 1, and then fed into the cross-modal Mamba fusion module along with a channel attention mechanism. The high-resolution multispectral image is then processed by the cross-modal Mamba fusion module. The global edge structure is integrated, and intermediate layer features are obtained through a cross-modal Mamba fusion module. The calculation process is as follows:

[0138] Laplace was used to process the high-resolution multispectral image Y to obtain the corresponding edge feature map. spectral characteristics and edge feature map After undergoing layer normalization, linear mapping, convolution, and SiLU activation function processing, the data is input into the cross-modal Mamba fusion module to complete the interaction between the two modal variables and output intermediate features. The complementary information of multiple modalities is obtained by averaging the transition matrix A after layer normalization, linear mapping, convolution, and SiLU activation function processing. Similarly, the complementary information of another modality is obtained by averaging the transition matrix B after layer normalization, linear mapping, convolution, and SiLU activation function processing. The calculation formulas are as follows:

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] In the formula: express The state transition matrix, Representation of edge feature map The state transition matrix, express The projection matrix, Representation of edge feature map The projection matrix, Indicates the time scale parameter. The input represents time t. Representing the hidden state at time t, after obtaining complementary information from multiple modalities, the intermediate layer features can be obtained through the following calculation method. The calculation formula is as follows:

[0148] ;

[0149] ;

[0150] In the formula: Represents the characteristic tensor. This represents a convolution operation with a kernel size of 1. This indicates the channel attention mechanism operation; Indicates cross-modal Mamba fusion;

[0151] Intermediate layer features and The calculation steps are as follows:

[0152] Intermediate layer features Downsample by twice the input to the second In the middle, repeat the intermediate layer features The processing steps yield intermediate layer features. , intermediate layer features Downsample by twice the input to the third Middle repeating intermediate layer features The processing steps are obtained The specific formula is as follows:

[0153] ;

[0154] ;

[0155] In the formula: This indicates a downsampling operation of the scale size by a factor of 2. This represents the CNN-Transformer-Mamba model.

[0156] In a further embodiment, the feature The calculation steps are as follows:

[0157] Fusion features Generate by applying a convolution filter Local spatial characteristics of the number of channels Then from the perspective of fusion characteristics Select the convolution filter to generate the corresponding number of feature channels. fusion filter feature map Then local spatial features With fusion filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. Again from the perspective of fusion characteristics Select the number of feature channels and the splicing and fusion features Matched filter feature map Then, the splicing and fusion features are combined. With filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. By analogy, the splicing and fusion features can be obtained. The specific formula is as follows:

[0158] ;

[0159] ;

[0160] ;

[0161] ;

[0162] In the formula: This represents the convolution filter operation. This represents the LeakyReLU activation function. Represents a positive integer.

[0163] intermediate layer features After upsampling by a factor of two, and with intermediate layer features Adding them together yields the intermediate layer features. , intermediate layer features After upsampling by a factor of two, and with intermediate layer features Adding them together gives Then, features are obtained through convolution. , will feature Input to feedforward neural network In this process, a high-resolution hyperspectral image was obtained. Among them, the features Input to feedforward neural network The final high-resolution hyperspectral image was obtained from the image. The specific steps are as follows:

[0164] First, the features After inputting into the layer normalization process, normalized features are obtained. Then, the features are... Decompose into sub-features at the channel dimension and The model employs two parallel pointwise convolutions and depthwise convolutions for feature extraction, improving its feature representation capability. The GELU activation function is used in one branch, followed by sub-feature extraction. and Multiplication yields the final feature Sub-features With final features The images are stitched together along the channel dimension, and then convolution with a kernel size of 1 is used to obtain the final reconstructed high-resolution hyperspectral image. The calculation formula is as follows:

[0165] ;

[0166] ;

[0167] ;

[0168] In the formula: This indicates splitting along the channel dimension. and This represents a pointwise convolution filter. and This represents a depthwise convolutional filter with a kernel size of 3. This represents the GELU activation function. This represents a convolution with a kernel size of 1.

[0169] Verification Experiment

[0170] like Figure 2 The pair of high-resolution multispectral images and low-resolution hyperspectral images shown are obtained through... The module introduces high-resolution multispectral images. High-resolution spatial correlation to adjust low-resolution hyperspectral images Spatial attention weights are used to reflect low-resolution hyperspectral images. Spatial correlation, reuse The module consists of a simple convolutional branch and a cross-modal branch. Semantic fusion branch parallelism, cross-modal The semantic fusion branch incorporates two attention mechanisms: a window-based self-attention mechanism to capture regional features within each window, and a cross-semantic fusion mechanism to facilitate feature interaction between windows. It also leverages the semantic prior of the high-resolution multispectral image Y to enhance the global semantic representation. After the two branches are fused, a concatenated cross-modal Mamba semantic fusion module effectively integrates complementary information from the features of the fused low-resolution hyperspectral image X and the edge structure information of the high-resolution multispectral image Y. Finally, a feedforward network... Further optimization of features yields the recovered high-resolution hyperspectral image. This improves the effectiveness and robustness of hyperspectral and multispectral image fusion.

[0171] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A dual-prior guided cross Mamba method for hyperspectral and multispectral image fusion, characterized in that, Includes the following steps: Acquiring low-resolution hyperspectral image in remote sensing image of same area and high-resolution multispectral image , extracting shallow features in low-resolution hyperspectral image , extracting shallow features in high-resolution multispectral image , extracting shallow features in high-resolution multispectral image , extracting shallow features in high-resolution multispectral image ; The shallow feature is obtained by convolutional down-sampling to obtain a same-scale feature map , the same-scale feature map has the same scale as the shallow feature , the shallow feature and the same-scale feature map are taken as inputs of a module to extract global feature information , the up-sampled scale of the shallow feature is lower than the shallow feature , and the fusion feature is obtained by concatenating the shallow feature along a channel dimension. With three groups The modules are connected in series to obtain fused features , with the fused features As input to obtain the corresponding scale intermediate layer features , , Make the network able to capture local, regional and global feature information at the corresponding scale; intermediate layer features up-sampled by two, and added to the intermediate layer features intermediate layer features up-sampled by two, and added to the intermediate layer features input to a feed-forward neural network , to obtain a restored high-resolution hyperspectral image .​​​​ 2. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 1, characterized in that, extracting global feature information The specific steps are as follows: map the shallow features to query features , key features and value features respectively, the same scale feature map is mapped to query features and key features , and the dimension of shallow features is transformed to extract global feature information; along the height and width dimensional average pooling operation, generating two types of dimensionally compressed key features 、 , the key features 、 along the channel direction, to obtain a fusion feature map that fuses spatial dimension information ; along the height and width dimensional pooling operation to generate two types of dimensionally compressed key features 、 , the key features 、 along the channel direction, to obtain a fusion feature map that fuses spatial dimension information ; the calculation formula is as follows: ； ； ； ； ； ； wherein: represents the height average pooling operation, represents the width dimension average pooling operation, represents a concatenation operation along the channel direction; query feature and the fusion feature map matrix multiplication operation, and then uses a normalization function to obtain a spatial attention weight matrix query feature and the fusion feature map matrix multiplication operation, and then uses a normalization function to obtain a spatial attention weight matrix The calculation formula is as follows: ； ； In the formulae: denotes function; After the calculation of the spatial attention weight matrix and the spatial attention weight matrix , the spatial attention weight matrix and the spatial attention weight matrix are averaged and matrix multiplied with the value feature to obtain the global feature information , whose calculation formula is as follows: ； ； In the formula: represents global feature information, represents a spatial attention weight matrix and the spatial attention weight matrix is averaged.

3. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 2, characterized in that, Fusion features The fusion step of the features of the invention is as follows: Obtaining global feature information Upsampling the global feature information and shallow layer features along the channel dimension to obtain fusion features The calculation formula is as follows: ； In the formulae: denotes an operation of up-sampling by 4.

4. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 1, characterized in that, Interlayer features The processing steps are as follows: Obtaining fused features , input into the first group Simple convolution branch and cross-modal fusion branch in the module, through the convolution branch and the cross-modal fusion branch cooperatively extracts spatial and spectral features of corresponding scales, respectively obtaining spatial features and spectral features , and the calculation formula is as follows: ； ； In the formula: denotes a simple convolution branch, denotes a cross-modal semantic fusion branch; After completing the feature and spectral features , the features and spectral features are spliced along the channel dimension, and the channel number is adjusted using a convolution kernel of 1, and input into the cross-modal fusion module through the channel attention mechanism. The cross-modal fusion module integrates the global edge structure of the high-resolution multispectral image, and its calculation formula is as follows: ； ； In the formula: represents the eigen tensor, represents a convolution operation with a kernel size of 1, represents a channel attention mechanism operation; represents cross-modal fusion.

5. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 4, characterized in that, Features The calculation step is as follows: Fusion features Generate by applying a convolution filter Local spatial characteristics of the number of channels Then from the perspective of fusion characteristics Select the convolution filter to generate the corresponding number of feature channels. fusion filter feature map Then local spatial features With fusion filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. Again from the perspective of fusion characteristics Select the number of feature channels and the splicing and fusion features Matched filter feature map Then, the splicing and fusion features are combined. With filter feature map The concatenated features are obtained by concatenating along the channel dimension and using them as input to the next convolutional layer. By analogy, the splicing and fusion features can be obtained. The specific formula is as follows: ； ； ； ； wherein: denotes a convolutional filter operation, denotes a LeakyReLU activation function, denotes a positive integer.

6. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 4, characterized in that, Spectral features The calculation steps are as follows: fused features Self-attention mechanism of the application window obtains intermediate layer global features , and the intermediate layer global features and the high-resolution multispectral image Y are input into a cross-semantic fusion module, in which the intermediate layer global features and the high-resolution multispectral image Y are first subjected to average pooling operation in the height H and width W dimensions to obtain main fused features , the main fused features , the main fused features , the main fused features , and then the main fused features , the main fused features and the main fused features , the main fused features are respectively spliced to obtain global intermediate layer features , the global intermediate layer features , the global intermediate layer features are multiplied with the global intermediate layer features , and then a correlation weight matrix is calculated using a function , the correlation weight matrix is multiplied with the global intermediate layer features to obtain a global semantic attention weight matrix , the height and width dimensions of the global semantic attention weight matrix are subjected to normalization operation to obtain global semantic perception attention weights in the height dimension and the height dimension , and the height dimension and the height dimension , the global semantic perception attention weights , the global semantic perception attention weights and the intermediate layer global features are multiplied to obtain spectral features , and the calculation formula is as follows: ； ； ； ； ； ； ； ； ； ； ； ； wherein: denotes a self-attention of the window, denotes a linear mapping function, denotes an activation function at the height dimension of activation function, denotes an activation function at the width dimension of activation function.

7. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 1, characterized in that, Intermediate layer features The calculation process is as follows: Using laplace on high resolution multispectral images The corresponding edge feature map is obtained by processing , the spectral feature and the edge feature map After layer normalization, linear mapping, convolution, SiLU activation function processing respectively, input into the cross-modal Mamba fusion module, complete the interaction between the two modal variables, output the intermediate features .

8. The dual-prior guided cross Mamba method for hyperspectral and multispectral image fusion according to claim 7, characterized in that, Complementary information of multiple modalities is obtained by averaging the transition matrix A after layer normalization, linear mapping, convolution, and SiLU activation function processing. Similarly, complementary information of another modality is obtained by averaging the transition matrix B after layer normalization, linear mapping, convolution, and SiLU activation function processing. The calculation formulas are as follows: ； ； ； ； ； ； ； ； In the formula: express The state transition matrix, Representation of edge feature map The state transition matrix, express The projection matrix, Representation of edge feature map The projection matrix, Indicates the time scale parameter. The input represents time t. This represents the hidden state at time t.

9. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 4, characterized in that, Intermediate layer features and The calculation step is as follows: intermediate layer features down-sampled by a factor of two into a second intermediate layer features intermediate layer features intermediate layer features down-sampled by a factor of two into a third intermediate layer features intermediate layer features where the specific formula is as follows: ； ； In the formula: denotes a 2x down-sampling operation of the scale size, denotes a CNN-Transformer-Mamba model.

10. The dual-prior guided cross-Mamba method for hyperspectral and multispectral image fusion according to claim 1, characterized in that, characteristics input to the feedforward neural network the final recovered high-resolution hyperspectral image The specific steps are as follows: First, the features are normalized after being input to the layer normalization to obtain normalized features, and then the features are split into sub-features in the channel dimension Two parallel pointwise convolution and depth convolution are used to extract features to improve the feature representation capability of the model. In one branch, a GELU activation function is used, and then the sub-features and are multiplied to obtain the final feature The sub-features and the final feature are spliced along the channel dimension, and then a convolution with a kernel size of 1 is used to obtain the final restored high-resolution hyperspectral image , and the calculation formula is as follows: ； ； ； wherein: represents a split along the channel dimension, and represents a point-wise convolutional filter, and represents a depthwise convolutional filter with a kernel size of 3, represents a GELU activation function, represents a convolution with a kernel size of 1.

Images