A hyperspectral image fusion method based on multi-scale multi-head attention

Abstract

The application discloses a hyperspectral image fusion method based on multi-scale multi-head attention, comprising the following steps: step 1, data preprocessing of the LR-HSI image; step 2, connecting the RGB image and the preprocessed LR-HIS image in the channel dimension to obtain the connected features; step 3, extracting the shallow features from the connected features through the multi-scale convolution module; step 4, inputting the shallow features into the feature interaction module to realize the information fusion and interaction between the features; step 5, obtaining the residual of the HR-HSI through the information fusion module from the output of the feature interaction module; and step 6, adding the residual of the HR-HSI and the LR-HSI image to obtain the high-resolution hyperspectral image. The multi-scale feature extraction module is used to obtain rich and multi-scale feature information from the input data, the calculation resource demand is significantly reduced while the reconstruction quality is ensured, and a more practical and efficient technical path is provided for the hyperspectral image processing.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and remote sensing image processing, and particularly relates to a hyperspectral image fusion method based on multi-scale multi-head attention. BACKGROUND

[0002] The existing hyperspectral and multispectral image fusion methods mainly include transform domain-based methods, sparse representation-based methods and deep learning-based methods. The transform domain-based methods, such as wavelet transform and principal component analysis, realize feature decoupling and recombination by converting image features into the frequency domain, thereby enhancing image details to a certain extent. Although such methods have small computational overhead, they have limited feature extraction capability in complex scenes. The sparse representation method reconstructs the image and balances the spatial resolution and spectral fidelity through dictionary learning and feature selection, but these methods have strong dependence on image prior knowledge and relatively weak generalization performance.

[0003] In the field of hyperspectral image processing, traditional transform domain methods (such as wavelet transform and principal component analysis) have significant limitations in the separation and fusion of spectral and spatial information. These methods mainly rely on linear transformation, and the core problem is that: first, linear transformation is difficult to capture complex nonlinear features in the image, resulting in severe loss and distortion of spectral information. Second, these methods are usually based on simple mathematical transformations and lack a deep understanding of the intrinsic complex structure of the image. When facing the details and textures, edge information and spectral subtle differences in the hyperspectral image, these traditional methods often appear to be inadequate, and it is difficult to accurately restore the essential features of the image.

[0004] The fusion method of convolutional neural network (CNN) has a limited receptive field range due to the size of the convolution kernel, resulting in a lack of fusion effect. The fusion method based on VIT has a quadratic relationship between the computational complexity of its self-attention mechanism and the image size, which means that as the image size increases, the computational resources and storage requirements will increase dramatically. This characteristic seriously limits its practical application in resource-constrained scenarios, especially for real-time processing of hyperspectral image reconstruction tasks.

[0005] Swin Transformer alleviates the problem of computational complexity to a certain extent by designing a new window attention mechanism, making the computational complexity linearly related to the image size. Using a moving window strategy, it constructs cross-window information interaction, but this method of moving the window in a fixed direction by a fixed distance still ignores some dependency relationships between adjacent window patches. A hierarchical structure is used to generate multi-scale feature representations by gradually merging patches, but this approach has the drawback that the feature channel dimension is also multiplied after merging the patches, which leads to a sharp increase in network model parameters and computational complexity.

[0006] The PSRT constructs a fusion network on the basis of the Swin Transformer, introduces a window attention mechanism, and makes some improvements. Information interaction across windows is established through the Shuffle and Reshuffle strategies, and the window size is gradually reduced in each segment to obtain multi-scale information interaction. However, this method has certain defects;

[0007] Firstly, the Shuffle and Reshuffle strategies, although they can disrupt the positions of each patch through Shuffle and then establish information interaction across windows, the relative positions of each patch are completely disrupted, which will inevitably lead to learning of incorrect information when establishing the dependency relationship between each patch.

[0008] Secondly, the strategy of gradually reducing the window size in each segment to obtain multi-scale information interaction will cause the receptive field to gradually shrink, which limits the ability of the Transformer to establish long-distance dependency relationships and affects the improvement of the reconstruction effect.

[0009] On the other hand, the PSRT follows the default practice of using a 4*4 patch as the token input to the Swim Transformer calculation, which is feasible in natural images, but in hyperspectral images, this ignores the problem of spectral consistency. Finally, since the size of the features inside each layer of the Swin Transformer module is the same, the dependency relationship between multi-scale features is established between layers, and the dependency relationship between multi-scale features inside each layer is lacking. SUMMARY

[0010] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a multi-scale multi-head attention hyperspectral image fusion method, which obtains rich and multi-scale feature information from the input data through a multi-scale feature extraction module, significantly reduces the demand for computing resources while ensuring the reconstruction quality, and provides a more practical and efficient technical path for hyperspectral image processing.

[0011] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0012] A multi-scale multi-head attention hyperspectral image fusion method, comprising the following steps:

[0013] Step 1: data preprocessing of the LR-HSI image;

[0014] Step 2: connect the RGB image and the pre-processed LR-HSI image in the channel dimension to obtain the connected feature; the RGB and the LR-HSI are two modes of the same image, the RGB is a high-resolution multispectral image composed of 3 spectral bands, and the LR-HSI is a low-resolution hyperspectral image composed of multiple spectral bands (usually dozens to hundreds) ;

[0015] Step 3: extract multi-scale shallow features from the connected feature through a multi-scale convolution module (MS_CONV) ;

[0016] Step 4: input the multi-scale shallow features into a feature interaction module (FIM) to realize information fusion and interaction between the features;

[0017] Step 5: obtain the residual of the HR-HSI through an information fusion module (CONV_Relu) from the output of the feature interaction module;

[0018] Step 6: add the residual of the HR-HSI to the LR-HSI image to obtain a high-resolution hyperspectral image (HR-HIS).

[0019] The step 1 is specifically:

[0020] Step 1.1: first, perform Bicubic up-sampling processing on the LR-HSI image to make the LR-HSI image have the same resolution as the RGB image in the spatial dimension;

[0021] Step 1.2: after up-sampling, normalize the up-sampled LR-HSI to scale the pixel value to the range of [0, 1] ;

[0022] Step 1.3: perform data augmentation on the normalized LR-HSI to expand the data set.

[0023] The step 2 is specifically:

[0024] connect the input RGB image with the up-sampled and normalized LR-HSI image in the spectral channel dimension to obtain the feature input of the neural network

[0025] wherein C1 and C2 represent the feature channels of the RGB and the LR-HSI respectively, and H and W represent the height and width of the image;

[0026] The step 3 is specifically:

[0027] connect the The input multi-scale convolution module includes multiple parallel convolution layers of different sizes for simultaneously capturing spatial information of different scales.

[0028] The multi-scale convolution module specifically operates as follows:

[0029] First, a 1x1 convolution layer is used to preliminarily extract shallow features of I input The input I input is mapped from a low-dimensional space to a high-dimensional feature space, and high-dimensional embedding is realized for each pixel token.

[0030] F embed =Conv 1×1 (I input )

[0031] wherein, represents embedded features, C embed represents the number of feature channels of the output, Conv 1x1 represents a 1x1 convolution layer, represents an input three-dimensional data cube, and then the feature channels (C1+C2) of the three-dimensional data cube I input are divided into 3 heads, each head uses the same 3x3 convolution kernel, and each head has a different dilation rate (Dilation), different receptive fields are obtained under different Dilation, and three features of different scales are extracted.

[0032] F1,F2,F3=split(F embed )

[0033]

[0034]

[0035]

[0036] wherein, split(·) represents splitting channels, Conv 3x3 represents a 3x3 convolution, respectively represent intermediate feature data after splitting channels, respectively represent intermediate feature data after convolution, and Dilation represents a parameter dilation rate of Conv 3x3 convolution.

[0037] Finally, the extracted features of different scales are connected to obtain multi-scale shallow features

[0038]

[0039] wherein Cat(·) represents a concatenation channel.

[0040] The step 4 is specifically:

[0041] The multi-scale shallow features are input into a feature interaction module, and the feature interaction module enhances information exchange between features through an attention mechanism.

[0042] The feature interaction module comprises a multi-scale multi-head attention module (MSMH) and a center attention module (CAB).

[0043] The multi-scale multi-head attention module (MSMH) realizes feature fusion of the multi-scale shallow features F0 through window attention and head attention, and the center attention module (CAB) compensates for the limitation of insufficient interaction between adjacent windows in the multi-scale multi-head attention module.

[0044] The multi-scale multi-head attention is specifically:

[0045] The multi-scale multi-head attention module introduces a head attention module (Head Attention Block, HAB); the head attention module calculates attention weights between different heads to promote exchange of cross-head feature information. For a given input multi-scale shallow feature F0, the calculation process of the multi-scale multi-head attention module is

[0046] F N =LN(F0),

[0047] F M =MSA(F N )+αHAB(F N )+F0,

[0048] F out =MLP(LN(F M ))+F M ,

[0049] wherein F N and F M represent intermediate features, F out represents an output of the multi-scale multi-head attention module, L N represents a layer normalization operation, MLP represents a multi-layer perceptron, MSA represents a shift window multi-head self-attention module, HAB represents a head attention module, and a is a proportionality coefficient.

[0050] The window self-attention is specifically, for a given size of input multi-scale shallow feature F0, it is first divided into windows, each window size is S×S, and then calculate the self-attention in each window. After dividing the multi-scale shallow feature F0 into windows, the local window feature F of each window is obtained. W ∈R S×S×C , linearly mapped to Q, K, and V through three matrices respectively, and then the calculation method of self-attention is

[0051]

[0052] Where d represents the scaling factor and B represents the relative position encoding.

[0053] The specific calculation process of the head attention module (HAB) is:

[0054] For the input multi-scale shallow feature F0, the feature channel C is first divided and linearly mapped to obtain V in the self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two average pooling operations (Avgpool) are performed on the HW and C′ dimensions to obtain F avg ∈R N×1×1 , get Q and K through linear mapping, Q, K calculated to get the attention weight Attn∈R N×N , copy Attn multiple times to get Attn′∈R HW×N×N , and finally the self-attention calculation method is

[0055] Attention(Q,K,V)=Attn′×V

[0056] The central attention module (CAB) is introduced after the multi-scale multi-head attention module (MSMH).

[0057] The details of the central attention module are:

[0058] The output feature F of the multi-scale multi-head attention module out Through linear mapping, it is converted into three matrices: query (Q), key (K), and value (V); then, the features are divided into multiple attention heads, each of which processes information independently;

[0059] For each position (i, j) in the feature map, a center sampling strategy is adopted:

[0060] The K and V matrices are sampled in the area centered at (i, j), and the sampling range is a window of size W×W;

[0061] Different degrees of sparse sampling are applied to each head, and the feature map is traversed with a 3×3 window with a step size of 1, and the content within the window is sampled; the feature interaction module interactively fuses the multi-scale multi-head attention module and the central attention module;

[0062] The calculation process of the feature interaction module is as follows:

[0063] F m =CAB(MSMH(F0))

[0064] in represents the output of the feature interaction module, F0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

[0065] The step 5 is specifically as follows:

[0066] Will A nonlinear transformation is performed through a convolution layer with a convolution kernel size of 3x3 and a ReLU activation function to finally generate the residual information of a high-resolution hyperspectral image.

[0067] F r =Relu(Conv 3×3 (F m ))

[0068] The residual information contains high-frequency details that need to be supplemented into the upsampled LR-HSI.

[0069] The step 6 is specifically as follows:

[0070] The generated residual information is added element by element to the LR-HSI after upsampling in step 1 to obtain the final high-resolution hyperspectral image (HR-HSI);

[0071] I HR-HSI =I LR-HSI +F r .

[0072] Beneficial effects of the present invention:

[0073] (1) The present invention directly obtains multi-scale feature information from the input data through a multi-scale feature extraction module, rather than dividing the multi-scale feature information between each layer of attention modules. This avoids the sharp increase in model parameters caused by merging patches to generate multi-scale feature representations in the original Swin Transformer, and also avoids the reduction of the receptive field range due to the reduction of the window size like PSRT.

[0074] (2) The application adds a Head Attention module on the basis of the Swin Transformer, so that the dependency relationship between multi-scale features can also be established inside each layer of the self-attention module, and the interaction between multi-scale information is enhanced.

[0075] (3) The application adds a center attention module to calculate self-attention with each patch as the center, which can establish a dependency relationship between each adjacent patch, make up for the deficiency of the Swin Transformer in calculating attention with a fixed window, and avoid the Shuffle strategy of PSRT from learning incorrect information.

[0076] (4) The application uses a 1*1 size patch as a token to ensure spectral consistency. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The overall structure of the hyperspectral image fusion network of the application.

[0078] Figure 2 The structure of the multi-scale convolution module is shown in the figure.

[0079] Figure 3 The structure of the multi-scale multi-head attention module is shown in the figure.

[0080] Figure 4 The calculation process of the Head Attention is shown in the figure.

[0081] Figure 5 The calculation process of the center attention is shown in the figure.

[0082] Figure 6 The sparse sampling is shown in the figure. DETAILED DESCRIPTION

[0083] The application will be further described in detail below with reference to the accompanying drawings.

[0084] As shown in the figure, the application discloses a multi-scale multi-head attention hyperspectral image fusion method, which is based on the proposed hyperspectral and multispectral image fusion network, and the fusion network adopts an encoder-fusion layer-decoder architecture. Figure 1

[0085] As shown in the figure, the overall network structure includes the following steps: Figure 1 ​​

[0086] Step (1): Data preprocessing:

[0087] (1.1) Upsampling

[0088] First, the LR-HSI image is subjected to Bicubic upsampling processing so that the LR-HSI image has the same resolution as the RGB image in the spatial dimension, laying the foundation for subsequent feature extraction and fusion.

[0089] (1.2) Normalization

[0090] After upsampling, in order to improve the training effect of the model, the upsampled LR-HSI is normalized and the pixel values ​​are scaled to the range of [0, 1] to speed up the convergence.

[0091] (1.3) Data enhancement

[0092] Data augmentation techniques such as random cropping, rotation, and flipping are used on the normalized LR-HSI to increase the diversity of training samples and enhance the generalization ability of the model.

[0093] Step (2): Concatenate RGB and LR-HSI in the channel dimension:

[0094] The input RGB image I RGB ∈R C1×H×W and the upsampled LR-HSI image I LR-HSI ∈R C2×H×W Connect in the spectral channel dimension to get the input I of the neural network input ∈R (C1+C2)×H×W ;

[0095] Where C1 and C2 represent the feature channels of RGB and LR-HSI respectively, and H and W represent the height and width of the image.

[0096] This connection method can make full use of the high spatial resolution information of RGB images and the rich spectral information of LR-HSI. input Learning spatial and spectral correlation features reduces the complexity of processing the two images separately and then fusing them. At the same time, it enables RGB and LR-HSI information to interact in the early stages of the network, which is conducive to the network learning deeper feature correlations.

[0097] Step (3): Extract shallow features through the multi-scale convolution module (MS_CONV):

[0098] The connected features are input into a multi-scale convolution module, which contains multiple parallel convolution layers and can capture spatial information of different scales at the same time. Through this multi-scale feature extraction mechanism, the texture details and structural information of the image can be more comprehensively obtained.

[0099] The structure of the multi-scale convolution module is as shown in Figure 2 First, a 1x1 convolution layer is used for shallow feature extraction, and the input I input is mapped from a low-dimensional space to a high-dimensional feature space, and high-dimensional embedding is realized for each pixel token; 1x1 convolution is essentially a fully connected operation, which can capture the global correlation between pixels. In the case of keeping the spatial resolution unchanged, the feature channel number is increased through 1x1 convolution, which improves the feature representation ability of I input and lays a foundation for subsequent feature fusion.

[0100] F embed =Conv 1×1 (I input )

[0101] Wherein, represents the embedded feature, C embed represents the number of output feature channels, Conv 1x1 represents a 1x1 convolution layer, represents the input three-dimensional data cube, and then the three-dimensional data cube I input is divided into 3 heads, each head uses the same 3x3 convolution kernel, but each head has a different dilation rate (Dilation), and different receptive fields are obtained under different dilations to extract features of different scales

[0102] F1,F2,F3=split(F embed )

[0103]

[0104]

[0105]

[0106] Wherein, split(·) represents splitting the channel, Conv 3x3 represents a 3x3 convolution, respectively represent the intermediate feature data after splitting the channel, respectively represent the intermediate feature data after convolution, and dilation represents the parameter dilation rate of Conv 3x3 convolution. By adjusting the Conv 3x3The expansion rate of the module can capture both local and global features of the image with different receptive fields. Smaller expansion rates focus on local details, while larger expansion rates can capture more extensive contextual information.

[0107] Finally, the extracted features at different scales are connected to obtain a multi-scale shallow feature

[0108]

[0109] where Cat(·) represents the concatenation channel.

[0110] This step recombines the multi-scale features to generate a comprehensive multi-scale shallow feature representation. Channel connection ensures the integrity of the features, enabling subsequent network layers to obtain more rich feature information.

[0111] Step (4): Information exchange between features is achieved through a feature interaction module (FIM):

[0112] The shallow features extracted by the multi-scale convolution module are input into the feature interaction module. The feature interaction module enhances the information exchange between features through attention mechanisms.

[0113] The feature interaction module includes a multi-scale multi-head attention module (MSMH) and a center attention module (CAB).

[0114] The multi-scale multi-head attention module (MSMH) realizes feature fusion of multi-scale shallow features F0 through window attention and head attention, and the center attention module (CAB) compensates for the limitation of insufficient interaction between adjacent windows in the multi-scale multi-head attention module.

[0115] The structure of the multi-scale multi-head attention module is shown in Figure 3 The multi-scale multi-head attention module adopts a similar structure to the standard SwinTransformer block and retains the window-based self-attention mechanism. Similar to traditional attention mechanisms, the module divides the feature channels into multiple heads. The difference is that the feature information of each head is pre-divided by the previous multi-scale convolution module, which means that each head already contains feature information of different scales. This design provides the attention mechanism with a richer feature representation.

[0116] To further enhance the interaction between features of different scales, the module introduces a head attention module (HAB). This module calculates attention weights between different heads to facilitate cross-head feature information exchange. In general, for a given input multi-scale shallow feature F0, the calculation process of the multi-scale multi-head attention module is

[0117] F N =LN(F0),

[0118] F M =MSA(F N )+αHAB(F N )+F0,

[0119] F out =MLP(LN(F M ))+F M ,

[0120] Among them F N and F M represents the intermediate feature, F out represents the output of the multi-scale multi-head attention module, L N represents the layer normalization operation, MLP represents the multi-layer perceptron, MSA represents the shifted window multi-head self-attention module, HAB represents the head attention module, and α is the scaling coefficient.

[0121] When calculating the window self-attention module, for a given size of input multi-scale shallow feature F0, it is first divided into windows, each window size is S×S, and then calculate the self-attention in each window. The local window feature F of each window obtained after F0 is divided into windows is W ∈R S×S×C , the same as the traditional transformer, is linearly mapped to Q, K, and V through three matrices, and then the calculation method of its self-attention is

[0122]

[0123] Where d represents the scaling factor and B represents the relative position encoding.

[0124] In order to make full use of multi-scale feature information and strengthen the interaction between feature information of different scales, the present invention proposes a head attention module (HAB).

[0125] The calculation process of the head attention module (HAB) is as follows Figure 4 As shown,

[0126] For the input multi-scale shallow feature F0, the feature channel C is first divided and linearly mapped to obtain V in the self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two average pooling operations (Avgpool) are performed on the HW and C′ dimensions to obtain F avg ∈R N×1×1, get Q and K through linear mapping, Q, K calculate the attention weight Attn∈R N×N , copy Attn multiple times to get Attn′∈R HW×N×N , and finally the self-attention calculation method is

[0127] Attention(Q,K,V)=Attn′×V

[0128] The center attention module is introduced after the multi-scale multi-head attention module (MSMH).

[0129] The core design of the center attention module is to use the reconstruction point as the center when calculating self-attention, thereby making up for the shortcomings of window attention.

[0130] Specifically, the workflow of the central attention module includes several key steps.

[0131] like Figure 5 As shown, first, similar to the standard ViT, the output feature F of the multi-scale multi-head attention module out Through linear mapping, the features are converted into three matrices: query (Q), key (K), and value (V); these features are then divided into multiple attention heads, each of which processes information independently. However, unlike traditional methods, for each position (i, j) in the feature map, this paper adopts a center sampling strategy: the K and V matrices are sampled within a region centered at (i, j), with a sampling range of a W×W window. This way, when calculating self-attention, the query range of each query (Q) is a W×W window centered on itself. This ensures that every point in the feature map can establish a dependency relationship with surrounding feature points during feature interaction.

[0132] At the same time, in order to further expand the receptive field without increasing the computational complexity, the present invention adopts different degrees of sparse sampling for each head. The specific method of sparse sampling is as follows: Figure 6 As shown in the figure, a 3×3 window is traversed over the feature map with a step size of 1, and the content within the window is sampled. By changing the dilation rate during sampling, it is possible to sample farther points while maintaining the same number of parameters (9 sampling points), thus expanding the receptive field without increasing the computational complexity.

[0133] In general, the calculation process of the feature interaction module is as follows:

[0134] F m =CAB(MSMH(F0))

[0135] in represents the output of the feature fusion module, F0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

[0136] Step (5): Obtain the residual of HR-HSI through the information fusion module (CONV_Relu):

[0137] The output of the feature interaction module is transformed nonlinearly through a convolution layer with a convolution kernel size of 3x3 and a ReLU activation function to finally generate the residual information of a high-resolution hyperspectral image.

[0138] F r =Relu(Conv 3×3 (F m ))

[0139] The constructed neural network establishes a nonlinear mapping relationship from the upsampled LR-HSI to HR-HSI. Through the nonlinear mapping capability of the multi-layer network structure and nonlinear activation function, it learns the missing high-frequency detail information between LR-HSI and the target HR-HSI. The residual information contains the high-frequency details that need to be supplemented in the upsampled LR-HSI.

[0140] Step (6) Add the residual of HR-HSI to LR-HSI to obtain HR-HSI:

[0141] The generated residual information is added element-wise to the upsampled LR-HSI in step 1 to obtain the final high-resolution hyperspectral image (HR-HSI).

[0142] I HR-HSI =I LR-HSI +F r

[0143] In order to verify the effectiveness of the fusion network designed in the present invention, Table 1 shows the quantitative evaluation indicators of the fusion results of the present invention and other popular deep learning algorithms for multispectral and hyperspectral fusion on the CAVE dataset. The fusion effect is evaluated from the perspectives of PSNR, SAM, ERGAS, and SSIM, and the network parameter Params is also compared.

[0144] Table 1 Quantitative evaluation of fusion results of different algorithms

[0145]

[0146]

[0147] To evaluate the rationality of the various new modules proposed in this paper, multiple ablation experiments were conducted. Table 2 shows the results of the ablation experiments. Experiment 1 replaced the multi-scale convolution module with a convolution layer with a normal convolution kernel size of 3x3 to verify the effectiveness of the multi-scale convolution module. Experiment 2 removed the Head Attention module from the multi-scale multi-head attention module and adopted a standard Swin Transformer block to verify that the Head Attention module can enhance the interaction between features of different scales and improve the fusion effect. Experiment 3 replaced the center attention module with a standard Swin Transformer block to verify that the center attention module can make up for the shortcomings of the window-based attention Swin Transformer block.

[0148] Table 2 Ablation experiment

[0149]

[0150] This paper designs an innovative multi-scale convolution module (MS_CONV), which first implements high-dimensional feature embedding through 1×1 convolution, and then adopts a parallel multi-head strategy, with each head using 3×3 convolution with a different expansion rate to effectively capture feature information of different scales.

[0151] This paper improves the Swin Transformer structure, distributes multi-scale feature information to different attention heads, introduces the head attention module (HAB), and establishes the dependency relationship between multi-scale features within each attention layer.

[0152] The present invention proposes a sampling strategy centered on the reconstructed pixel, breaking through the fixed window limitation, so that all adjacent pixels can establish a dependency relationship, solving the problem of edge pixel information loss, and improving the reconstruction quality.

Claims

1. A multi-scale multi-head attention hyperspectral image fusion method, characterized by: The following steps are included: Step 1: Data preprocessing of LR-HSI images; Step 2: Concatenate the RGB image with the preprocessed LR-HSI image in the channel dimension to obtain the concatenated features; Step 3: extracting multi-scale shallow features from the concatenated features through a multi-scale convolution module; Step 4: Input the multi-scale shallow features into the feature interaction module to achieve information fusion interaction between features; Step 5: The output of the feature interaction module is passed through the information fusion module to obtain the residual of HR-HSI; Step 6: Add the HR-HSI residual to the LR-HSI image to obtain a high-resolution hyperspectral image; The step 4 is specifically as follows: The multi-scale shallow features are input into a feature interaction module, and the feature interaction module enhances information exchange between features through an attention mechanism; The feature interaction module includes a multi-scale multi-head attention module and a center attention module; The multi-scale multi-head attention module realizes the feature fusion of multi-scale shallow features F0 through window attention and head attention; The central attention module makes up for the limitation of insufficient interaction between adjacent windows in the multi-scale multi-head attention module; The multi-scale multi-head attention is specifically: The multi-scale multi-head attention module introduces the head attention module; the head attention module calculates the attention weights between different heads to promote the communication of cross-head feature information. For a given input multi-scale shallow feature F0, the calculation process of the multi-scale multi-head attention module is: F N =LN(F0), F M =MSA(F N )+αHAB(F N )+F0, F out =MLP(LN(F M ))+F M , Among them F N and F M represents the intermediate feature, F out represents the output of the multi-scale multi-head attention module, L N represents the layer normalization operation, MLP represents the multi-layer perceptron, MSA represents the shifted window multi-head self-attention module, HAB represents the head attention module, and α is the scaling factor; Specifically, for a given size of input multi-scale shallow feature F0, it is first divided into windows, each window size is S×S, and then calculate the self-attention in each window. After dividing the multi-scale shallow feature F0 into windows, the local window feature F of each window is obtained. W ∈R S×S×C , linearly mapped to Q, K, and V through three matrices respectively, and then the calculation method of self-attention is Where d represents the scaling factor and B represents the relative position encoding; The specific calculation process of the head attention module is: For the input multi-scale shallow feature F0, the feature channel C is first divided and linearly mapped to obtain V in the self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two mean pooling operations are performed on the HW and C′ dimensions to obtain F. avg ∈R N×1×1 , get Q and K through linear mapping, Q, K calculated to get the attention weight Attn∈R N×N , copy the attention weight Attn multiple times to get Attn′∈R HW ×N×N , and finally the self-attention calculation method is Attention(Q,K,V)=Attn′×V Introducing a central attention module after the multi-scale multi-head attention module; The details of the central attention module are: The output feature F of the multi-scale multi-head attention module out It is converted into three matrices Q, K and V through linear mapping; then, the features are divided into multiple attention heads, each of which processes information independently; For each position (i, j) in the feature map, a center sampling strategy is adopted. The K and V matrices are sampled in the area centered at (i, j), and the sampling range is a W×W window. Traverse the feature map with a 3×3 window with a step size of 1 and sample the content within the window; The feature interaction module interactively integrates the multi-scale multi-head attention module and the central attention module; The calculation process of the feature interaction module is as follows: F m =CAB(MSMH(F0)) in represents the output of the feature interaction module, F0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

2. The multi-scale multi-head attention hyperspectral image fusion method according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.1: First, perform Bicubic upsampling on the LR-HSI image so that the LR-HSI image has the same resolution as the RGB image in the spatial dimension; Step 1.2: After upsampling, the upsampled LR-HSI is normalized to scale the pixel values ​​to the range of [0, 1]. Step 1.3: Perform data augmentation on the normalized LR-HSI to expand the dataset.

3. The multi-scale multi-head attention hyperspectral image fusion method according to claim 2 is characterized in that: The step 2 is specifically as follows: RGB and LR-HSI are two modes of the same image. RGB is a high-resolution multispectral image composed of three spectral bands, and LR-HSI is a low-resolution hyperspectral image composed of multiple spectral bands. The input RGB image LR-HSI image after upsampling and normalization Connect in the spectral channel dimension to obtain the feature input of the neural network Where C1 and C2 represent the feature channels of RGB and LR-HSI respectively, and H and W represent the height and width of the image.

4. The multi-scale multi-head attention hyperspectral image fusion method according to claim 3 is characterized in that: The step 3 is specifically as follows: After connecting Enter the multi-scale convolution module, which contains multiple parallel convolution layers of different sizes to simultaneously capture spatial information of different scales; The specific operations of the multi-scale convolution module are: First, use a 1x1 convolutional layer to input Perform preliminary extraction of shallow features and input I input Mapping from low-dimensional space to high-dimensional feature space, while achieving high-dimensional embedding for each pixel token; F embed =Conv 1×1 (I input ) in, represents the embedded features, C embed Indicates the number of output feature channels, Conv 1x1 Represents a 1x1 convolutional layer, I input The feature channel (C1+C2) is divided into three heads. Each head uses the same 3x3 convolution kernel. Each head has a different expansion rate. Different receptive fields are obtained under different expansion rates, and three features at different scales are extracted. F1,F2,F3=split(F embed ) Among them, split(·) represents the split channel, Conv 3x3 represents a 3x3 convolution, Respectively represent the intermediate feature data after the segmentation channel, They represent the intermediate feature data after convolution, and Dilation represents Conv 3x3 The parameter expansion rate of the convolution; Finally, the features extracted at different scales are connected to obtain multi-scale shallow features Wherein, Cat(·) represents the connection channel.

5. The multi-scale multi-head attention hyperspectral image fusion method according to claim 1 is characterized in that: The step 5 is specifically as follows: Will A nonlinear transformation is performed through a convolution layer with a convolution kernel size of 3x3 and a ReLU activation function to finally generate the residual information of a high-resolution hyperspectral image. F r =Release(Conv 3×3 (F m ))。 6. The multi-scale multi-head attention hyperspectral image fusion method according to claim 5 is characterized in that: The step 6 is specifically as follows: The generated residual information is added element by element to the LR-HSI after upsampling in step 1 to obtain the final high-resolution hyperspectral image; I HR-HSI =I LR-HSI +F r 。

Images