A hyperspectral and multispectral image fusion method based on attention mechanism

Abstract

The application discloses a hyperspectral and multispectral image fusion method based on an attention mechanism, which comprises the following steps: step 1, acquiring a low spatial resolution hyperspectral image and a high spatial resolution multispectral image, and constructing a training set and a test set; step 2, constructing a double-flow hyperspectral and multispectral image fusion model based on an attention mechanism; step 3, training the constructed image fusion model using the training set and an Adam optimizer, and obtaining an optimal model through loss function back propagation training; and step 4, inputting the low spatial resolution hyperspectral image and the high spatial resolution multispectral image to be fused in the test set into the trained image fusion model to obtain a high spatial resolution hyperspectral image. The method has good fusion performance when processing complex ground objects, can effectively reduce the overall error of the fusion result while paying attention to details, and can obtain a high-quality high-resolution hyperspectral image.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and deep learning technology, and in particular to a hyperspectral and multispectral image fusion method based on an attention mechanism. Background Technology

[0002] Due to the limitations of imaging sensors, the spectral and spatial resolutions of remote sensing images are mutually restrictive; no single sensor can simultaneously acquire data with high spatial, spectral, and temporal resolution. Hyperspectral images possess hundreds of spectral bands, providing rich spectral information and enabling more refined material identification and classification, but their spatial resolution is relatively low. Multispectral images offer higher spatial resolution, capturing clearer ground features, but their fewer spectral bands limit the detailed spectral information they can provide. By fusing hyperspectral and multispectral images, images with both high spatial and spectral resolution can be generated. This will significantly improve the ability to classify and identify ground objects and broaden the scope of image applications.

[0003] Currently, hyperspectral and multispectral image fusion methods can be divided into traditional methods and deep learning-based methods. Traditional fusion methods include those based on panchromatic sharpening, matrix factorization, and tensor representation. Deep learning-based methods can automatically learn and extract complex features from data, effectively processing high-dimensional data. However, due to the limitation of fixed convolution kernel size, they cannot model global semantic information. Therefore, attention mechanisms have been introduced into hyperspectral and multispectral fusion to capture long-range details in hyperspectral and multispectral images.

[0004] Existing hyperspectral and multispectral fusion methods have certain limitations, mainly in the following aspects:

[0005] 1) Most traditional methods are based on manually extracted features and rely on prior assumptions. They are usually very sensitive to parameter selection. If these assumptions are not applicable to the current problem, they may lead to quality degradation.

[0006] 2) There is a high correlation between the spectral bands of remote sensing images and non-local similarity in spatial location. Due to the limitation of the receptive field, convolutional neural networks extract local feature information within the window and do not make full use of the inherent features of remote sensing images.

[0007] 3) Since both hyperspectral and multispectral images contain spatial and spectral information, and there is redundant and complementary information between them, we should not only focus on how to extract spatial or spectral information separately, but also consider the information interaction between hyperspectral and multispectral images. Summary of the Invention

[0008] To address the aforementioned technical problems in existing technologies, this invention proposes a hyperspectral and multispectral image fusion method based on an attention mechanism. This method fully utilizes the strong correlation between spectral bands and the non-local similarity of remote sensing images in spatial location, capturing long-range dependency and self-similarity priors, and capturing redundant and complementary information from hyperspectral and multispectral images to achieve better fusion results. The specific technical solution is as follows:

[0009] A hyperspectral and multispectral image fusion method based on an attention mechanism includes the following steps:

[0010] Step 1: Obtain low spatial resolution hyperspectral images and high spatial resolution multispectral images to construct training and testing sets;

[0011] Step 2: Construct an attention-based dual-stream hyperspectral and multispectral image fusion model;

[0012] Step 3: Train the constructed image fusion model using the training set and the Adam optimizer, and obtain the optimal model through backpropagation of the loss function.

[0013] Step 4: Input the low spatial resolution hyperspectral image and the high spatial resolution multispectral image to be fused from the test set into the trained image fusion model to obtain the high spatial resolution hyperspectral image.

[0014] Furthermore, the image fusion model includes a dual-branch network, a feature fusion module, and an image reconstruction module;

[0015] The low spatial resolution hyperspectral image is upsampled to the same size as the high spatial resolution multispectral image and then input into a dual-branch network for feature extraction.

[0016] The feature fusion module fuses the extracted features, and then the image reconstruction module reconstructs a high spatial resolution hyperspectral image from the fused features.

[0017] Furthermore, the upsampling of the low spatial resolution hyperspectral image uses a bilinear interpolation algorithm.

[0018] Furthermore, the dual-branch network includes a spatial feature extraction branch and a spectral feature extraction branch;

[0019] The spatial feature extraction branch and the spectral feature extraction branch respectively perform shallow feature extraction and deep feature extraction on the high spatial resolution multispectral image and the upsampled low spatial resolution hyperspectral image, respectively.

[0020] The shallow feature extraction uses two independent shallow feature extraction modules to extract shallow spatial features and shallow spectral features from the high spatial resolution multispectral image and the upsampled low spatial resolution high spectral image, respectively, and then map the feature data to high-dimensional features.

[0021] The deep feature extraction employs two attention-guided cross-domain feature extraction modules, which alternately extract global feature domain information and cross-domain information, as well as cross-feature domain interactive feature information from shallow spatial features and shallow spectral features, to obtain deep spatial features and deep spectral features.

[0022] Furthermore, the shallow feature extraction module consists of two consecutive convolutional layers with a kernel size of 3×3 and a stride of 1. A parameter-corrected linear unit activation function is superimposed between the convolutional layers.

[0023] Furthermore, the attention-guided cross-domain feature extraction module consists of a cascaded Swin Transformer based on a self-attention mechanism and a Swin Transformer based on a cross-attention mechanism.

[0024] Furthermore, the Swin Transformer based on the self-attention mechanism includes a regular window layer at the top and a shifted window layer below it. The regular window layer includes a multi-head self-attention W-MSA module and a multilayer perceptron (MLP) module based on the window mechanism. Each module applies a residual connection, and there is an LN layer before the W-MSA and MLP. Corresponding to the regular window layer, the shifted window layer includes a multi-head self-attention SW-MSA module and a multilayer perceptron (MLP) module based on the shifted window mechanism. Each module applies a residual connection, and there is an LN layer before the W-MSA and MLP.

[0025] Furthermore, the specific process of feature data processing in the conventional window layer is as follows: Assuming the input is a feature F based on a self-attention mechanism of the Swin Transformer, the size of F is H×W×C, where C represents the number of channels. First, the input feature is divided into M×M non-overlapping local windows. The input will be reshaped into The characteristics, among which This represents the total number of windows; next, standard self-attention is performed for each window; for local window features... Three learnable weight matrices shared across different windows are used. , and It is projected onto the query Q, key K, and value V in the following way:

[0026]

[0027] The attention mechanism is defined as follows: By calculating the dot product of query Q and all keys K, and then normalizing using the softmax function, the corresponding attention score is obtained.

[0028]

[0029] B is the dimension of keys, and B is the learnable relative positional encoding.

[0030] Extending self-attention SA to multi-head self-attention MSA, therefore, local window features The entire process in the regular window layer can be formalized as follows:

[0031]

[0032]

[0033]

[0034] in Indicates The output of the regular window layer of the Swing Transformer, which serves as the input to the self-attention mechanism.

[0035] Furthermore, the specific process expression for the feature data processing of the shift window layer is as follows:

[0036]

[0037]

[0038] in Therefore The output of the Swing Transformer, which serves as the input to the self-attention mechanism, is expressed as follows in the multilayer perceptron (MLP) module:

[0039]

[0040] Where GELU is the Gaussian error linear unit, and These are the learnable weights of the fully connected layer, and and These are learnable bias parameters.

[0041] Furthermore, the Swin Transformer layer structure based on the cross-attention mechanism is similar to that based on the self-attention mechanism, wherein the attention module adopts a multi-head cross-attention (MCA) module. Therefore, the specific feature data processing process of the Swin Transformer based on the cross-attention mechanism is as follows:

[0042] For two local window features from feature domain 1 and feature domain 2 respectively, and The cross-attention mechanism is defined as:

[0043]

[0044] Where CA(·) is the calculation and Attention function for the relationship between features and Project it onto the query Q, key K, and value V in the following manner:

[0045]

[0046]

[0047] As shown in formulas (10) and (11), the features Used to generate keys Sum ,feature It is then used to generate queries. Then, attention weighting operations are performed using the generated keys, values, and queries to achieve cross-modal information fusion; at the same time, residual connection is used to preserve the original information in feature domain 1, realizing information storage and transmission, and the same processing is applied to feature domain 2.

[0048] Therefore, the entire process of the Swing Transformer based on the cross-attention mechanism is defined as follows:

[0049]

[0050]

[0051]

[0052]

[0053] in Representation of features After the output of the first layer MCA Representation of features After passing the output of the first MCA layer, the features are moved in two directions respectively. ( ) pixels, redivide the window, and calculate the attention within each window using the following formula:

[0054]

[0055]

[0056]

[0057]

[0058] in, Represents the features in feature domain 1 The output of the Swin Transformer based on the cross-attention mechanism, Features in feature domain 2 The output of the Swin Transformer based on the cross-attention mechanism.

[0059] Furthermore, the feature fusion module includes a Transformer-based deep feature fusion module and a CNN-based feature fusion module. The deep spectral features and deep spatial features are concatenated and then input into the Transformer-based deep feature fusion module to obtain fused deep spectral spatial features. Long skip connections are used to combine the fused deep spectral spatial features with shallow spatial features and shallow spectral features to obtain concatenated feature information. Finally, the concatenated feature information is input into the CNN-based feature fusion module to extract local information again from the features containing global information, so as to fuse local information in different domains and obtain fused spatial spectral features.

[0060] Furthermore, the image reconstruction module includes three convolutional layers, each with a filter size of 3×3 and a stride of 1. PReLU activation is applied after each convolutional layer. This module maps the fused deep spectral features and deep spatial features back to the image space and recovers the fused shallow spatial features and shallow spectral features to obtain a high spatial resolution hyperspectral image.

[0061] Compared with existing technologies, this invention uses the Swin Transformer to fully extract global information from hyperspectral and multispectral images, and employs a cross-attention mechanism to fully utilize global information within and between feature domains. The error distribution is uniform in the global range, exhibiting good fusion performance when processing complex regions. While focusing on details, it also effectively reduces the overall error of the fusion result, obtaining high-quality, high-resolution hyperspectral images. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the main process of a hyperspectral and multispectral image fusion method based on an attention mechanism according to the present invention;

[0063] Figure 2 This is a schematic diagram illustrating the specific process of image feature extraction, fusion, and image reconstruction using the image fusion model in an embodiment of the present invention.

[0064] Figure 3 This is a data processing flowchart of the image fusion model according to an embodiment of the present invention;

[0065] Figure 4 This is a data processing flowchart of the Swin Transformer based on the self-attention mechanism according to an embodiment of the present invention;

[0066] Figure 5 This is a data processing flowchart of the Swin Transformer based on the cross-attention mechanism according to an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0068] like Figure 1 As shown, the present invention provides a hyperspectral and multispectral image fusion method based on an attention mechanism, which mainly includes the following steps:

[0069] Step 1: Obtain low spatial resolution hyperspectral images and high spatial resolution multispectral images to construct training and testing sets;

[0070] Step 2: Construct an attention-based dual-stream hyperspectral and multispectral image fusion model;

[0071] Step 3: Train the constructed image fusion model using the training set and the Adam optimizer, and obtain the optimal model through backpropagation of the loss function.

[0072] Step 4: Input the low spatial resolution hyperspectral image and the high spatial resolution multispectral image to be fused from the test set into the trained image fusion model to obtain the high spatial resolution hyperspectral image.

[0073] Specifically, assuming This represents a w×h LR-HSI (Low Resolution Hyperspectral Image) with S spectral bands, where each spectral band has low spatial resolution. This represents a W×H HR-MSI (High Resolution Multispectral Image) with s spectral bands, but with high spatial resolution. Here, H and h represent the image height, W and w represent the image width, and S and s represent the number of spectral bands. .

[0074] like Figure 2 and Figure 3 As shown, in the image fusion model, firstly, bilinear interpolation is used to upsample the LR-HSI to the same size as the HR-MSI, resulting in... Then, the HR-MSI and the upsampled LR-HSI are input into the spatial feature extraction branch and the spectral feature extraction branch in the dual-branch network, respectively.

[0075] Then, shallow feature extraction is performed on HR-MSI and the upsampled LR-HSI respectively to obtain high-resolution shallow spatial and spectral features. The shallow feature extraction employs two independent shallow feature extraction modules to obtain spatial texture and spectral features from HR-MSI and the upsampled LR-HSI, respectively. Each of the two shallow feature extraction modules consists of two consecutive convolutional layers. Convolutional layers provide a simple and convenient way to increase feature dimensions. Then, a parameter-corrected linear unit (PReLU) activation function is used between the convolutional layers. The convolutional kernel size is 3×3, and the stride is 1. This extracts local semantic information from the shallow layers and maps the shallow feature data to high-dimensional features. The extracted shallow spectral features are... Shallow space features are C represents the number of channels, which is 192 in this embodiment.

[0076] Next, shallow spectral features and shallow spatial features are input into the deep feature extraction module. The module alternately extracts global feature domain information and cross-domain information, as well as cross-feature domain interactive feature information, from both the MSI (Multispectral Image) and HSI (Hyperspectral Image) to obtain deep spectral features and deep spatial detail features. The deep feature extraction module deploys two attention-guided cross-domain feature extraction (ACFE) modules. Each ACFE module consists of a cascaded self-attention Swin Transformer (SAST) and a cross-attention Swin Transformer (CAST), encouraging information exchange between the two features and performing interactive feature fusion extraction between the spatial detail branch and the spectral branch to capture redundant and complementary information from HSI and MSI.

[0077] Hyperspectral image (HSI) exhibits consistency or similarity in spectral properties and features across different regions within the entire image range. Due to the limited receptive field of CNNs, their ability to model long-range dependencies in images is limited; they can only model local information and cannot effectively capture global semantic information, nor do they possess the ability to filter and distinguish useful features from redundant information. Therefore, after initial shallow feature extraction, this invention uses a Swin Transformer to further extract features containing global information. Because the Swin Transformer has the ability to acquire long-range dependency information, each extracted feature incorporates global information.

[0078] Therefore, shallow spatial features and shallow spectral characteristics The input deep feature extraction module obtains deep spectral features through a self-attention-based Swing Transformer module and a cross-attention-based Swing Transformer module. Deep space details It includes global feature domain information and cross-feature domain global information of MSI and HSI.

[0079] Then, the shallow spatial features Shallow spectral characteristics Deep spectral characteristics and deep space details The input feature fusion module performs feature fusion to obtain spectral spatial fused features. The feature fusion module includes a Transformer-based deep feature fusion module and a CNN-based feature fusion module, which respectively fuse the extracted deep global features, and then fuse the fused deep global features with local features. Specifically, the deep spectral features... Deep space features After concatenation, the data is input into a Transformer-based deep feature fusion module to obtain fused depth spectral spatial features. Because Transformer self-attention computation requires stretching the patch into a one-dimensional token, spatial information within the patch is lost. Long-skip connections are used to fuse this information. shallow feature information , Combined to obtain the spliced ​​feature information While preserving local information, it also incorporates medium- to long-range information into the features. The stitched features... The input is a CNN-based feature fusion module, which extracts local information again from the features containing global information, in order to fuse local information from different domains and obtain fused spatial spectral features. .

[0080] Finally, the spatial spectral fusion features are input into the image reconstruction module to reconstruct the desired high spatial resolution hyperspectral image from the fused features. Specifically, the spatial spectral fusion features... The input image reconstruction module has three convolutional layers, each with a 3×3 filter size and a stride of 1. PReLU activation is applied after each convolutional layer. The fused deep features are mapped back to the image space, and the fused shallow features are recovered to obtain a high-resolution hyperspectral image. .

[0081] As described above, the attention-based dual-stream hyperspectral and multispectral image fusion method employs a dual-branch network, combining CNN and Swin Transformer to extract features containing both local and global information, extracting spatial details and spectral information from both multispectral and hyperspectral images. The introduction of a cross-attention mechanism allows for the interaction of features from the two branches of the hyperspectral and multispectral images, leveraging their similarity and redundancy for feature enhancement and better feature extraction from both types of images. The Swin Transformer is applied to merge spatial and spectral global features, which are then concatenated with the local information extracted by the CNN for better feature fusion. Finally, an image reconstruction network is used to reconstruct a high-spatial-resolution hyperspectral image from the fused features. By fully utilizing global information within and between feature domains, better fusion results can be achieved, yielding higher-quality, high-resolution hyperspectral images.

[0082] like Figure 4 As shown, the processing flow of the Swin Transformer based on the self-attention mechanism is the same as that of the Swin Transformer layer, including a window-based MSA (Multi-head Self Attention) module and a shift-window-based MSA module. The shift-window-based attention mechanism is a fundamental component in the design of SAST, capturing global information in the HR-MSI and LR-HSI feature extraction branches respectively to effectively integrate global information within the same feature domain. Specifically, assuming the input is a given feature F with a size of H×W×C, the input is first divided into M×M non-overlapping local windows, and the input is reshaped into... The characteristics, among which This represents the total number of windows. Next, standard self-attention is performed for each window. For local window features... Three learnable weight matrices shared across different windows are used. , and It is projected onto the query Q, key K, and value V in the following way:

[0083]

[0084] The attention mechanism essentially works by calculating the dot product of query Q and all keys K, then normalizing the result using the softmax function to obtain the corresponding attention score. The attention mechanism is defined as follows:

[0085]

[0086] Here, B is the dimension of the keys, and B is the learnable relative position encoding. Self-attention is extended to multi-head self-attention (MSA), enabling the attention mechanism to consider various attention distributions and allowing the model to capture information from different perspectives. In practice, the attention function is executed h times in parallel, and the results are concatenated to the multi-head self-attention function, where h is set to 8.

[0087] The Swin Transformer layer consists of a W-MSA module and a Multilayer Perceptron (MLP) module, each with a residual connection. An LN (LayerNorm) layer precedes both the W-MSA and MLP modules. Therefore, the entire process of the SAST module for local window features X is formulated as follows:

[0088]

[0089]

[0090]

[0091] in This represents the output of the first-layer Swing Transformer with X as input.

[0092] If the partitions of each layer remain fixed, connections cannot be established across local windows. To achieve cross-window connections, an alternating method of regular window and shifted window partitioning is used. In the first layer of regular windows, a standard window partitioning scheme is used, and self-attention is calculated within each window. Then, in the next layer of shifted window layers, new windows are created by shifting window partitions. Before performing this shifted window partitioning, features are moved in two directions (…). The window is redefined by dividing the space into smaller units of pixels (e.g., 10 pixels). Attention is calculated within each new window, allowing the new window's attention to cross the boundaries of the first-level window and establish connections between them. The calculation formula is:

[0093]

[0094]

[0095] in Therefore The output of SAST is used as input. The multilayer perceptron (MLP) layers are as follows:

[0096]

[0097] Where GELU is the Gaussian error linear unit, and These are the learnable weights of the fully connected layer, and and These are learnable bias parameters.

[0098] By using the SAST module to extract long-range dependency information of HSI and MSI, we can effectively learn the long-range spatial and spectral dependencies of MSI and HSI, thereby improving spatial and spectral quality.

[0099] like Figure 5 As shown, CAST, which is based on the cross-attention mechanism, and SAST both follow similar baselines, but have key differences. Specifically, CAST uses multi-head cross attention (MCA) instead of multi-head self-attention (MSA) to achieve global context exchange across feature domains.

[0100] A dual-branch network can extract spatial and spectral information from multispectral and hyperspectral images respectively. However, multispectral images also contain spectral information, and the dual-branch network ignores the complementarity between multispectral and hyperspectral images, which may lead to incomplete feature extraction and the reconstructed image still having spatial or spectral distortion problems. Therefore, this invention designs a Swing Transformer (CAST) based on a cross-attention mechanism to accurately model the cross-modal relationship between HSI and MSI.

[0101] In the deep feature extraction branch for HSI, HSI features are used to generate K and V, while MSI features are used to generate Q; for the deep feature extraction branch for MSI, MSI features are used to generate K and V, and HSI features are used to generate Q. Here, Q represents the query vector, K represents the keyword vector, and V represents the value vector.

[0102] Given two features respectively and The relationship between them can be modeled using an attention mechanism, defined as follows:

[0103]

[0104] Where CA(·) is the calculation and The attention function for the relationship between them. CA(·) is calculated using the same attention function as in Equation (2). Wherein, the features... and Project it onto the query Q, key K, and value V in the following manner:

[0105]

[0106]

[0107] As shown in formulas (10) and (11), the features in feature domain 1 Used to generate keys Sum In feature domain 2, the features It is then used to generate queries. Then, attention-weighted operations are performed using these generated keys, values, and queries to achieve cross-modal information fusion. Simultaneously, residual connections are used to preserve the original information in feature domain 1, achieving information storage and transfer; the same processing is applied to feature domain 2. This design effectively captures complementary information from the two feature domains and organically fuses them to enhance the model's expressive power.

[0108] Therefore, given two local window features from different domains and The entire CAST process is defined as follows:

[0109]

[0110]

[0111]

[0112]

[0113] in Represents the features in feature domain 1 After the output of the first layer MCA Represents the features in feature domain 2 After passing through the output of the first MCA layer, the feature is moved in two directions respectively. ( ) pixels, redivide the window, and calculate the attention within each window using the following formula:

[0114]

[0115]

[0116]

[0117]

[0118] in, Represents the features in feature domain 1 After CAST output Features in feature domain 2 The output after CAST.

[0119] Unlike MSA, where the inputs K, Q, and V all originate from the same image features, MCA's inputs K, Q, and V originate from features of different images. For example, K and V might come from HSI features, while Q might come from MSI features. After MCA computation, the HSI feature information will be influenced by the MSI feature information. Through the interaction of spatial and spectral attention, the feature redundancy between the two sub-networks is reduced, while also enhancing the complementarity between features, enabling the effective fusion of HSI and MSI feature information.

[0120] Cross-attention (HSI) is effectively applied to decouple spatial and spectral information, enhancing the complementarity of extracted features and reducing redundancy. Through the interaction of spatial and spectral attention, feature redundancy between the two sub-networks is reduced, while also increasing feature complementarity, thereby optimizing information integration and processing. This design helps to maximize the complementarity between HSI and MSI while ensuring the independence of feature extraction, thus achieving efficient feature fusion and optimizing the final image quality.

[0121] As described above, the attention-based hyperspectral and multispectral image fusion method of this invention first designs a dual-stream spatial-spectral fusion network with an attention mechanism, combining the advantages of CNN and Swin Transformer to fully exploit the local and global dependencies in HSI and MSI. Secondly, a cross-attention mechanism is introduced, which can model redundant and complementary information across HSI and MSI modes, capturing the complex correlations between HSI and MSI, and obtaining high correlations between spectral bands and non-local similarities in spatial location. Then, the Swin Transformer is used to fuse spatial details and global spectral features, which are then concatenated with the local information extracted by CNN for better feature fusion. Finally, an image reconstruction network is used to reconstruct the desired high spatial resolution hyperspectral image (HR-HSI) from the fused features.

[0122] While convolutional operations can properly fuse texture features in deep networks, they cannot adjust the value of each pixel in the global view. Compared with existing technologies, this invention introduces the Swing Transformer to capture global information and fully utilizes global information within and between feature domains through a cross-attention mechanism. The error distribution is uniform in the global scope, and it exhibits good fusion performance when processing complex features. While focusing on details, it also effectively reduces the overall error of the fusion result, and obtains higher quality, high-resolution hyperspectral images.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hyperspectral and multispectral image fusion method based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Obtain low spatial resolution hyperspectral images and high spatial resolution multispectral images to construct training and testing sets; Step 2: Construct an attention-based dual-stream hyperspectral and multispectral image fusion model; The image fusion model includes a two-branch network: a spatial feature extraction branch and a spectral feature extraction branch; The spatial feature extraction branch and the spectral feature extraction branch respectively perform shallow feature extraction and deep feature extraction on the high spatial resolution multispectral image and the upsampled low spatial resolution high spectral image, respectively. The shallow feature extraction uses two independent shallow feature extraction modules to extract shallow spatial features and shallow spectral features from the high spatial resolution multispectral image and the upsampled low spatial resolution high spectral image, respectively, and then map the feature data to high-dimensional features. The deep feature extraction employs two attention-guided cross-domain feature extraction modules, which alternately extract global feature domain information and cross-domain information, as well as cross-feature domain interactive feature information from shallow spatial features and shallow spectral features, to obtain deep spatial features and deep spectral features. The attention-guided cross-domain feature extraction module consists of a cascaded SwinTransformer based on a self-attention mechanism and a SwinTransformer based on a cross-attention mechanism. Step 3: Train the constructed image fusion model using the training set and the Adam optimizer, and obtain the optimal model through backpropagation of the loss function. Step 4: Input the low spatial resolution hyperspectral image and the high spatial resolution multispectral image to be fused from the test set into the trained image fusion model to obtain the high spatial resolution hyperspectral image.

2. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 1, characterized in that, The image fusion model also includes a feature fusion module and an image reconstruction module; The low spatial resolution hyperspectral image is upsampled to the same size as the high spatial resolution multispectral image and then input into a dual-branch network for feature extraction. The feature fusion module fuses the extracted features, and then the image reconstruction module reconstructs a high spatial resolution hyperspectral image from the fused features.

3. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 2, characterized in that, The upsampling of the low spatial resolution hyperspectral image uses a bilinear interpolation algorithm.

4. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 1, characterized in that, The shallow feature extraction module consists of two consecutive convolutional layers with a kernel size of 3×3 and a stride of 1. A parameter-corrected linear unit activation function is superimposed between the convolutional layers.

5. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 2, characterized in that, The self-attention-based Swing Transformer includes a regular window layer at the top and a shifted window layer below it. The regular window layer includes a multi-head self-attention W-MSA module and a multilayer perceptron (MLP) module based on the window mechanism. Each module applies a residual connection, and an LN layer precedes the W-MSA and MLP modules. Corresponding to the regular window layer, the shifted window layer includes a multi-head self-attention SW-MSA module and a multilayer perceptron (MLP) module based on the shifted window mechanism. Each module applies a residual connection, and an LN layer precedes the W-MSA and MLP modules.

6. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 5, characterized in that, The specific process of feature data processing in the conventional window layer is as follows: Assuming the input is a feature F based on the self-attention mechanism of the SwinTransformer, the size of F is H×W×C, where C represents the number of channels. First, the input feature is divided into non-overlapping M×M local windows. The input will be reshaped into The characteristics, among which This represents the total number of windows; then, standard self-attention (SA) is performed for each window; for local window features... Three learnable weight matrices shared across different windows are used. , and It is projected onto the query Q, key K, and value V in the following way: The attention mechanism is defined as follows: By calculating the dot product of query Q and all keys K, and then normalizing using the softmax function, the corresponding attention score is obtained. B is the dimension of keys, and B is the learnable relative positional encoding. Extending self-attention SA to multi-head self-attention MSA, therefore, local window features The entire process in the regular window layer can be formalized as follows: in Indicated by The output of the regular window layer of the Swing Transformer, which serves as the input to the self-attention mechanism.

7. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 6, characterized in that, The specific process expression for feature data processing of the shift window layer is as follows: in Therefore The output of the Swing Transformer, which serves as the input to the self-attention mechanism, is expressed as follows in the multilayer perceptron (MLP) module: Where GELU is the Gaussian error linear unit, and These are the learnable weights of the fully connected layer, and and These are learnable bias parameters.

8. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 5, characterized in that, The Swin Transformer layer structure based on the cross-attention mechanism is similar to the Swin Transformer based on the self-attention mechanism, wherein the attention module adopts a multi-head cross-attention (MCA) module. Therefore, the specific feature data processing process of the Swin Transformer based on the cross-attention mechanism is as follows: For two local window features from feature domain 1 and feature domain 2 respectively, and The cross-attention mechanism is defined as: Where CA(·) is the calculation and Attention function for the relationship between features and Project it onto the query Q, key K, and value V in the following manner: As shown in formulas (10) and (11), the features Used to generate keys Sum ,feature It is then used to generate queries. Then, attention weighting operations are performed using the generated keys, values, and queries to achieve cross-modal information fusion; at the same time, residual connections are used to preserve the original information in feature domain 1, realizing information storage and transmission, and the same processing is applied to feature domain 2. Therefore, the entire process of the Swing Transformer based on the cross-attention mechanism is defined as follows: in Representation of features After the output of the first layer MCA Representation of features The output after the first layer MCA; Move the feature in both directions respectively ( ( ) pixels, redivide the window, and calculate the attention within each window using the following formula: in, Represents the features in feature domain 1 The output of the Swin Transformer based on the cross-attention mechanism, Features in feature domain 2 The output of the Swin Transformer based on the cross-attention mechanism.

9. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 8, characterized in that, The feature fusion module includes a Transformer-based deep feature fusion module and a CNN-based feature fusion module. Deep spectral features and deep spatial features are concatenated and then input into the Transformer-based deep feature fusion module to obtain fused deep spectral spatial features. Long skip connections are used to combine the fused deep spectral spatial features with shallow spatial features and shallow spectral features to obtain concatenated feature information. Finally, the concatenated feature information is input into the CNN-based feature fusion module, which extracts local information again from the features containing global information to fuse local information from different domains and obtain fused spatial spectral features.

10. The hyperspectral and multispectral image fusion method based on an attention mechanism as described in claim 9, characterized in that, The image reconstruction module includes three convolutional layers, each with a filter size of 3×3 and a stride of 1. PReLU activation is applied after each convolutional layer. This module maps the fused deep spectral features and deep spatial features back to the image space and recovers the fused shallow spatial features and shallow spectral features to obtain a high spatial resolution hyperspectral image.

Images