Multi-scale multi-attention hyperspectral image fusion method

Abstract

The invention discloses a multi-scale multi-attention hyperspectral image fusion method. The method comprises the following steps: 1, carrying out multi-scale multi-attention hyperspectral image fusion; the method comprises the following steps of: 1, carrying out data preprocessing on an LR-HSI image; step 2, connecting the RGB image with the preprocessed LR-HIS image in a channel dimension to obtain a connected feature; 3, extracting shallow layer features from the connected features through a multi-scale convolution module; 4, inputting the shallow layer features into a feature interaction module to realize information fusion interaction between the features; step 5, enabling the output of the feature interaction module to pass through an information fusion module to obtain a residual error of HR-HSI; and step 6, adding the residual error of the HR-HSI and the LR-HSI image to obtain a high-resolution hyperspectral image. Rich and multi-scale feature information is obtained from input data through the multi-scale feature extraction module, the reconstruction quality is guaranteed, meanwhile, the computing resource requirement is remarkably reduced, and a more practical and efficient technical path is provided for hyperspectral image processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and remote sensing image processing, and in particular relates to a multi-scale multi-head attention hyperspectral image fusion method. Background Art

[0002] Existing hyperspectral and multispectral image fusion methods mainly include transform domain-based methods, sparse representation-based methods, and deep learning-based methods. Transform domain-based methods, such as wavelet transform and principal component analysis, achieve feature decoupling and reorganization by converting image features to the frequency domain, which enhances image details to a certain extent. Although these methods have low computational overhead, their feature extraction capabilities are limited in complex scenes. Sparse representation methods reconstruct images and balance spatial resolution and spectral fidelity through dictionary learning and feature selection, but these methods rely heavily on prior knowledge of images and have relatively weak generalization performance.

[0003] In the field of hyperspectral image processing, traditional transform domain methods (such as wavelet transform, principal component analysis, etc.) show significant limitations in the separation and fusion of spectral and spatial information. These methods mainly rely on linear transformations, and their core problems are: first, linear transformations are difficult to capture the complex nonlinear features in the image, resulting in serious loss and distortion of spectral information. Secondly, these methods are usually based on simple mathematical transformations and lack a deep understanding of the inherent complex structure of the image. When faced with detailed textures, edge information, and subtle spectral differences in hyperspectral images, these traditional methods often appear to be stretched to the limit and it is difficult to accurately restore the essential characteristics of the image.

[0004] The fusion method of convolutional neural network (CNN) has a poor fusion effect because the size of its convolution kernel limits the range of the receptive field. The computational complexity of the self-attention mechanism of the VIT-based fusion method is quadratic with the image size, which means that as the image size increases, the computing resources and storage requirements will increase sharply. This feature severely limits its practical application in resource-constrained scenarios, especially for hyperspectral image reconstruction tasks that require real-time processing.

[0005] Swin Transformer alleviates the computational complexity problem to a certain extent by designing a new window attention mechanism, making the computational complexity linearly related to the image size. Using the moving window strategy, cross-window information interaction is constructed, but this method of moving the window in a fixed direction and a fixed distance will still ignore the dependencies between some adjacent window patches. Multi-scale feature representations are generated by gradually merging the hierarchical structure of patches, but the drawback of this is that after merging the patches, the dimension of the feature channel also increases exponentially, which will cause the model parameters of the network to increase sharply and the computational complexity to increase.

[0006] PSRT builds a fusion network based on Swin Transformer, introduces the window attention mechanism and makes some improvements. Through the Shuffle and Reshuffle strategies, cross-window information interaction is established, and the window size is gradually reduced to obtain multi-scale information interaction. However, this method has certain defects;

[0007] The first is the Shuffle and Reshuffle strategy. Although Shuffle can disrupt the positions of patches and build cross-window information interaction on this basis, the relative positions of patches are completely disrupted, which will inevitably lead to learning wrong information when building the dependency relationship between patches.

[0008] Secondly, the strategy of gradually reducing the window size to obtain multi-scale information interaction will cause the receptive field to gradually shrink, which limits the Transformer's ability to establish long-distance dependencies and affects the improvement of the reconstruction effect.

[0009] On the other hand, PSRT uses a 4*4 patch as the token input to the default practice of Swim Transformer calculation, which is feasible in natural images, but in hyperspectral images, this ignores the problem of spectral consistency. Finally, since the scale of the features within each layer of the Swin Transformer module is the same, the dependency between multi-scale features is established between layers, and the dependency between multi-scale features within each layer is missing. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multi-scale multi-head attention hyperspectral image fusion method, which obtains rich and multi-scale feature information from the input data through a multi-scale feature extraction module, while ensuring the reconstruction quality, significantly reducing the demand for computing resources, and providing a more practical and efficient technical path for hyperspectral image processing.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A multi-scale multi-head attention hyperspectral image fusion method, comprising the following steps;

[0013] Step 1: Preprocess the LR-HSI image data;

[0014] Step 2: Connect the RGB image and the preprocessed LR-HSI image in the channel dimension to obtain the connected features; RGB and LR-HSI are two modes of the same image, RGB is a high-resolution multispectral image composed of three spectral bands, and LR-HSI is a low-resolution hyperspectral image composed of multiple spectral bands (generally dozens to hundreds);

[0015] Step 3: extracting multi-scale shallow features from the connected features through a multi-scale convolution module (MS_CONV);

[0016] Step 4: Input the multi-scale shallow features into a feature interaction module (FIM) to achieve information fusion interaction between features;

[0017] Step 5: The output of the feature interaction module is passed through the information fusion module (CONV_Relu) to obtain the residual of HR-HSI;

[0018] Step 6: Add the residual of HR-HSI to the LR-HSI image to obtain a high-resolution hyperspectral image (HR-HIS).

[0019] The step 1 is specifically as follows:

[0020] Step 1.1: First, the LR-HSI image is subjected to Bicubic upsampling so that the LR-HSI image has the same resolution as the RGB image in the spatial dimension;

[0021] Step 1.2: After upsampling, the upsampled LR-HSI is normalized to scale the pixel values ​​to the range of [0,1];

[0022] Step 1.3: Perform data augmentation on the normalized LR-HSI to expand the dataset.

[0023] The step 2 is specifically as follows:

[0024] The input RGB image LR-HSI image after upsampling and normalization Connect in the spectral channel dimension to get the feature input of the neural network

[0025] Among them C 1 and C 2 Represent the feature channels of RGB and LR-HSI respectively, H and W represent the height and width of the image;

[0026] The step 3 is specifically as follows:

[0027] After connecting Enter the multi-scale convolution module, which contains multiple parallel convolutional layers of different sizes to simultaneously capture spatial information of different scales.

[0028] The multi-scale convolution module specifically operates as follows:

[0029] First, use a 1x1 convolutional layer to input Perform preliminary extraction of shallow features and input I input Map from low-dimensional space to high-dimensional feature space, and implement high-dimensional embedding for each pixel token;

[0030] F embed =Conv 1×1 (I input )

[0031] in, represents the embedded feature, C embed Indicates the number of output feature channels, Conv 1x1 represents a 1x1 convolutional layer, Represents the input three-dimensional data cube, and then the three-dimensional data cube I input The characteristic channel (C 1 +C 2 ) is divided into three heads, each head uses the same 3x3 convolution kernel, each head has a different dilation rate, different receptive fields are obtained under different dilations, and three features at different scales are extracted;

[0032] F 1 ,F 2 ,F 3 =split(F embed )

[0033]

[0034]

[0035]

[0036] Among them, split(·) represents the split channel, Conv 3x3 represents a 3x3 convolution, Respectively represent the intermediate feature data after the segmentation channel, They represent the intermediate feature data after convolution, and Dilation represents Conv 3x3 The parameter dilation rate of the convolution;

[0037] Finally, the features extracted at different scales are connected to obtain multi-scale shallow features

[0038]

[0039] Wherein, Cat(·) represents the connection channel.

[0040] The step 4 is specifically as follows:

[0041] The multi-scale shallow features are input into a feature interaction module, and the feature interaction module enhances information exchange between features through an attention mechanism;

[0042] The feature interaction module includes a multi-scale multi-head attention module (MSMH) and a central attention module (CAB);

[0043] The multi-scale multi-head attention module (MSMH) realizes multi-scale shallow features F through window attention and head attention. 0 The feature fusion, the central attention module (CAB) makes up for the limitation of insufficient interaction between adjacent windows of window attention in the multi-scale multi-head attention module.

[0044] The multi-scale multi-head attention is specifically:

[0045] The multi-scale multi-head attention module introduces the head attention block (HAB); the head attention module calculates the attention weights between different heads to promote the communication of cross-head feature information. For a given input multi-scale shallow feature F 0 , the calculation process of the multi-scale multi-head attention module is

[0046] F N =LN(F 0 ),

[0047] F M =MSA(F N )+αHAB(F N )+F 0 ,

[0048] F out =MLP(LN(F M ))+F M ,

[0049] where F N and F M represents the intermediate feature, F out represents the output of the multi-scale multi-head attention module, L N represents layer normalization operation, MLP represents multi-layer perceptron, MSA represents shift window multi-head self-attention module, HAB represents head attention module, and α represents the scaling factor;

[0050] The window self-attention is specifically, for a given size of input multi-scale shallow feature F 0 , first divide it into windows, each of which is S×S in size, and then calculate self-attention in each window. 0 The local window feature F of each window obtained after dividing the window W ∈R S×S×C , are linearly mapped to Q, K, and V through three matrices respectively, and then the calculation method of self-attention is

[0051]

[0052] Where d represents the scaling factor and B represents the relative position encoding.

[0053] The specific calculation process of the head attention module (HAB) is:

[0054] For the input multi-scale shallow features F 0 , first divide and linearly map the feature channel C to obtain V in self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two average pooling operations (Avgpool) are performed on the HW and C′ dimensions to obtain F avg ∈R N×1×1 , Q and K are obtained through linear mapping, and the attention weight Attn∈R calculated by Q and K N×N , copy Attn multiple times to get Attn′∈R HW×N×N , and finally the self-attention calculation method is

[0055] Attention(Q,K,V)=Attn′×V

[0056] The central attention module (CAB) is introduced after the multi-scale multi-head attention module (MSMH).

[0057] The details of the central attention module are:

[0058] The output feature F of the multi-scale multi-head attention module out Through linear mapping, it is converted into three matrices: query (Q), key (K), and value (V); then, the features are divided into multiple attention heads, each of which processes information independently;

[0059] For each position (i, j) in the feature map, a center sampling strategy is adopted:

[0060] The K and V matrices are sampled in the area centered at (i, j), and the sampling range is a window of size W×W;

[0061] Different degrees of sparse sampling are used for each head, and the feature map is traversed with a window of 3×3 size, with a step size of 1, and the content in the window is sampled; the feature interaction module interactively fuses the multi-scale multi-head attention module and the central attention module;

[0062] The calculation process of the feature interaction module is as follows:

[0063] F m =CAB(MSMH(F 0 ))

[0064] in represents the output of the feature interaction module, F 0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

[0065] The step 5 is specifically as follows:

[0066] Will A convolution layer with a convolution kernel size of 3x3 and a ReLU activation function are used for nonlinear transformation to finally generate the residual information of a high-resolution hyperspectral image.

[0067] F r =Relu(Conv 3×3 (F m ))

[0068] The residual information contains high-frequency details that need to be supplemented into the upsampled LR-HSI.

[0069] The step 6 is specifically as follows:

[0070] The generated residual information is added element by element to the LR-HSI after upsampling in step 1 to obtain the final high-resolution hyperspectral image (HR-HSI);

[0071] I HR-HSI =I LR-HSI +F r .

[0072] Beneficial effects of the present invention:

[0073] (1) The present invention directly obtains multi-scale feature information from the input data through a multi-scale feature extraction module, rather than dividing the multi-scale feature information between each layer of attention modules. This avoids the sharp increase in model parameters caused by merging patches to generate multi-scale feature representations in the original Swin Transformer, and also avoids the reduction of the receptive field range due to the reduction of the window size like PSRT.

[0074] (2) The present invention adds a Head Attention module based on Swin Transformer, so that the dependency relationship between multi-scale features can be established within each layer of the self-attention module, thereby enhancing the interaction between multi-scale information.

[0075] (3) The present invention adds a central attention module to calculate self-attention with each patch as the center, which can not only establish a dependency relationship between each adjacent patch, making up for the deficiency of Swin Transformer in calculating attention with a fixed window, but also avoids the shuffle strategy of PSRT that leads to learning erroneous information.

[0076] (4) The present invention uses a 1*1 patch as a token to ensure the spectral consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is the overall structure of the hyperspectral image fusion network of the present invention.

[0078] Figure 2 Schematic diagram of the multi-scale convolution module structure.

[0079] Figure 3 Schematic diagram of the multi-scale multi-head attention module structure.

[0080] Figure 4 Schematic diagram of the Head Attention calculation process.

[0081] Figure 5 Schematic diagram of the central attention calculation process.

[0082] Figure 6 Schematic diagram of sparse sampling. DETAILED DESCRIPTION

[0083] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0084] like Figure 1As shown, the present invention discloses a multi-scale multi-head attention hyperspectral image fusion method. The fusion method is based on the proposed hyperspectral and multispectral image fusion network. The fusion network adopts an encoder-fusion layer-decoder architecture; the encoder is composed of a multi-scale convolution module (MS_CONV) to extract shallow features, the fusion layer is composed of a feature interaction module (FIM), and the attention mechanism is used to perform feature interaction fusion. The decoder is composed of a convolution layer and a Relu activation function to reconstruct the residual information of the high-resolution hyperspectral image.

[0085] like Figure 1 As shown, the overall network structure includes the following steps:

[0086] Step (1): Data preprocessing:

[0087] (1.1) Upsampling

[0088] First, the LR-HSI image is subjected to Bicubic upsampling processing so that the LR-HSI image has the same resolution as the RGB image in the spatial dimension, which lays the foundation for subsequent feature extraction and fusion.

[0089] (1.2) Normalization

[0090] After upsampling, in order to improve the training effect of the model, the upsampled LR-HSI was normalized and the pixel values ​​were scaled to the range of [0, 1] to speed up the convergence.

[0091] (1.3) Data enhancement

[0092] Data enhancement techniques such as random cropping, rotation, and flipping are used on the normalized LR-HSI to increase the diversity of training samples and enhance the generalization ability of the model.

[0093] Step (2): Connect RGB and LR-HSI in the channel dimension:

[0094] The input RGB image I RGB ∈R C1×H×W And the upsampled LR-HSI image I LR-HSI ∈R C2×H×W Connect in the spectral channel dimension to get the input I of the neural network input ∈R (C1+C2)×H×W ;

[0095] Among them C 1 and C 2 They represent the feature channels of RGB and LR-HSI respectively, and H and W represent the height and width of the image.

[0096] This connection method can make full use of the high spatial resolution information of RGB images and the rich spectral information of LR-HSI. input The spatial and spectral correlation features are learned in the network, which reduces the complexity of processing the two images separately and then fusing them. At the same time, the RGB and LR-HSI information begin to interact in the early stage of the network, which is conducive to the network learning deeper feature associations.

[0097] Step (3): Extract shallow features through the multi-scale convolution module (MS_CONV):

[0098] The concatenated features are input into the multi-scale convolution module, which contains multiple parallel convolution layers and can capture spatial information of different scales at the same time. Through this multi-scale feature extraction mechanism, the texture details and structural information of the image can be obtained more comprehensively.

[0099] The structure of the multi-scale convolution module is as follows: Figure 2 As shown, we first use a 1x1 convolutional layer to extract shallow features and then convert the input I input Mapping from low-dimensional space to high-dimensional feature space, while achieving high-dimensional embedding for each pixel token; 1x1 convolution is essentially a fully connected operation that can capture the global correlation between pixels. While keeping the spatial resolution unchanged, the number of feature channels is increased through 1x1 convolution to improve I input The feature representation capability lays the foundation for subsequent feature fusion.

[0100] F embed =Conv 1×1 (I input )

[0101] in, represents the embedded feature, C embed Indicates the number of output feature channels, Conv 1x1 represents a 1x1 convolutional layer, Represents the input three-dimensional data cube, and then the three-dimensional data cube I input The characteristic channel (C 1 +C 2 ) is divided into 3 heads, each head uses the same 3x3 convolution kernel, but each head has a different dilation rate (Dilation), different receptive fields are obtained under different dilations, and features at different scales are extracted

[0102] F 1 ,F 2 ,F 3 =split(F embed )

[0103]

[0104]

[0105]

[0106] Among them, split(·) represents the split channel, Conv 3x3 represents a 3x3 convolution, Respectively represent the intermediate feature data after the segmentation channel, They represent the intermediate feature data after convolution, and dilation represents Conv 3x3 The parameter expansion rate of the convolution. By adjusting Conv 3x3 The expansion rate of the module can capture the local and global features of the image with different receptive fields. A smaller expansion rate focuses on local details, while a larger expansion rate can capture broader contextual information.

[0107] Finally, the features extracted at different scales are connected to obtain multi-scale shallow features

[0108]

[0109] Wherein, Cat(·) represents the connection channel.

[0110] This step recombines multi-scale features to generate a comprehensive multi-scale shallow feature representation. Channel connections ensure the integrity of features, allowing subsequent network layers to obtain richer feature information.

[0111] Step (4): Realize information interaction between features through the feature interaction module (FIM):

[0112] The shallow features extracted by the multi-scale convolution module are input into the feature interaction module, which enhances the information exchange between features through the attention mechanism.

[0113] The feature interaction module includes a multi-scale multi-head attention module (MSMH) and a central attention module (CAB).

[0114] The multi-scale multi-head attention module (MSMH) realizes multi-scale shallow features F through window attention and head attention. 0 The feature fusion, the central attention module (CAB) makes up for the limitation of insufficient interaction between adjacent windows of window attention in the multi-scale multi-head attention module.

[0115] The structure of the multi-scale multi-head attention module is as follows Figure 3As shown in the figure, the multi-scale multi-head attention module adopts a similar structure to the standard SwinTransformer block and retains the window-based self-attention mechanism. Similar to the traditional attention mechanism, the module divides the feature channel into multiple heads. The difference is that the feature information of each head is pre-divided by the previous multi-scale convolution module, which means that each head already contains feature information of different scales. This design provides a richer feature representation for the attention mechanism.

[0116] In order to further enhance the interaction between features of different scales, the module introduces the Head Attention Block (HAB). This module calculates the attention weights between different heads and promotes the communication of cross-head feature information. In general, for a given input multi-scale shallow feature F 0 , the calculation process of the multi-scale multi-head attention module is

[0117] F N =LN(F 0 ),

[0118] F M =MSA(F N )+αHAB(F N )+F 0 ,

[0119] F out =MLP(LN(F M ))+F M ,

[0120] where F N and F M represents the intermediate feature, F out represents the output of the multi-scale multi-head attention module, L N represents layer normalization operation, MLP represents multi-layer perceptron, MSA represents shift window multi-head self-attention module, HAB represents head attention module, and α represents the scaling coefficient.

[0121] When calculating the window self-attention module, for a given size of input multi-scale shallow features F 0 , first divide it into windows, each of which is S×S in size, and then calculate self-attention in each window. 0 The local window feature F of each window obtained after dividing the window W ∈R S×S×C , the same as the traditional transformer, is linearly mapped to Q, K, and V through three matrices, and then the self-attention is calculated as follows

[0122]

[0123] Where d represents the scaling factor and B represents the relative position encoding.

[0124] In order to make full use of multi-scale feature information and strengthen the interaction between feature information of different scales, the present invention proposes a head attention module (HAB).

[0125] The calculation process of the head attention module (HAB) is as follows Figure 4 As shown,

[0126] For the input multi-scale shallow features F 0 , first divide and linearly map the feature channel C to obtain V in self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two average pooling operations (Avgpool) are performed on the HW and C′ dimensions to obtain F avg ∈R N×1×1 , Q and K are obtained through linear mapping, Q, K calculate the attention weight Attn∈R N×N , copy Attn multiple times to get Attn′∈R HW×N×N , and finally the self-attention calculation method is

[0127] Attention(Q,K,V)=Attn′×V

[0128] The center attention module is introduced after the multi-scale multi-head attention module (MSMH).

[0129] The core design of the center attention module is to use the reconstruction point as the center when calculating self-attention, thereby making up for the shortcomings of window attention.

[0130] Specifically, the workflow of the central attention module includes several key steps.

[0131] like Figure 5 As shown, first, similar to the standard ViT, the output feature F of the multi-scale multi-head attention module outThrough linear mapping, it is converted into three matrices: query (Q), key (K), and value (V); then, these features are divided into multiple attention heads, each of which processes information independently. However, unlike traditional methods, for each position (i, j) in the feature map, the present invention adopts a central sampling strategy: the K and V matrices are sampled in the area centered on (i, j), and the sampling range is a W×W window. In this way, when calculating self-attention, the query range of each query (Q) is a W×W window centered on itself, which ensures that every point at each position in the feature map can establish a dependency relationship with the surrounding feature points when performing feature interaction.

[0132] At the same time, in order to further expand the receptive field without increasing the computational complexity, the present invention adopts different degrees of sparse sampling for each head. The specific method of sparse sampling is as follows: Figure 6 As shown in the figure, the feature map is traversed with a window of size 3×3, and the step size is 1, and the content in the window is sampled. By changing the dilation rate during sampling, it is possible to collect farther points while maintaining the same number of parameters (9 sampling points), so that the receptive field can be expanded without increasing the computational complexity.

[0133] In general, the calculation process of the feature interaction module is as follows:

[0134] F m =CAB(MSMH(F 0 ))

[0135] in represents the output of the feature fusion module, F 0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

[0136] Step (5): Obtain the residual of HR-HSI through the information fusion module (CONV_Relu):

[0137] The output of the feature interaction module is transformed nonlinearly through a convolution layer with a convolution kernel size of 3x3 and a ReLU activation function to finally generate the residual information of a high-resolution hyperspectral image.

[0138] F r =Relu(Conv 3×3 (F m ))

[0139] The constructed neural network establishes a nonlinear mapping relationship from the upsampled LR-HSI to HR-HSI. Through the nonlinear mapping capability of the multi-layer network structure and the nonlinear activation function, it learns the missing high-frequency detail information between LR-HSI and the target HR-HSI. The residual information contains the high-frequency details that need to be supplemented to the upsampled LR-HSI.

[0140] Step (6) Add the residual of HR-HSI to LR-HSI to obtain HR-HSI:

[0141] The generated residual information is added element-by-element to the upsampled LR-HSI in step 1 to obtain the final high-resolution hyperspectral image (HR-HSI).

[0142] I HR-HSI =I LR-HSI +F r

[0143] In order to verify the effectiveness of the fusion network designed in the present invention, the quantitative evaluation indicators of the fusion results of the present invention and other popular deep learning algorithms for multispectral and hyperspectral fusion on the CAVE dataset are shown in Table 1. The fusion effect is evaluated from PSNR, SAM, ERGAS, and SSIM, and the network parameter Params are also compared.

[0144] Table 1 Quantitative evaluation of fusion results of different algorithms

[0145]

[0146]

[0147] In order to evaluate the rationality of each new module proposed in the present invention, multiple ablation experiments were conducted, and the results of the ablation experiments are shown in Table 2. Experiment 1 uses a convolution layer with a common convolution kernel size of 3x3 to replace the multi-scale convolution module to verify the effectiveness of the multi-scale convolution module; Experiment 2 removes the Head Attntion module in the multi-scale multi-head attention module and uses a standard Swin Transformer block to verify that the Head Attntion module can enhance the interaction between features of different scales and improve the fusion effect; Experiment 3 replaces the center attention module with a standard Swin Transformer block to verify that the center attention module can make up for the defects of the Swin Transformer block based on window attention.

[0148] Table 2 Ablation experiment

[0149]

[0150] The present invention designs an innovative multi-scale convolution module (MS_CONV), which first implements high-dimensional feature embedding through 1×1 convolution, and then adopts a parallel multi-head strategy, in which each head uses 3×3 convolution with different dilation rates to effectively capture feature information of different scales.

[0151] The present invention improves the Swin Transformer structure, distributes multi-scale feature information to different attention heads, introduces a head attention module (HAB), and establishes the dependency relationship between multi-scale features within each attention layer.

[0152] The present invention proposes a sampling strategy centered on the reconstruction pixel, breaking through the fixed window limitation, so that all adjacent pixels can establish a dependency relationship, solving the problem of edge pixel information loss, and improving the reconstruction quality.

Claims

1. A multi-scale multi-head attention hyperspectral image fusion method, characterized in that: The steps include: Step 1: Preprocess the LR-HSI image data; Step 2: Connect the RGB image and the preprocessed LR-HSI image in the channel dimension to obtain the connected features; Step 3: extracting multi-scale shallow features from the connected features through a multi-scale convolution module; Step 4: Input the multi-scale shallow features into a feature interaction module to achieve information fusion interaction between features; Step 5: The output of the feature interaction module is passed through the information fusion module to obtain the residual of HR-HSI; Step 6: Add the residual of HR-HSI to the LR-HSI image to obtain a high-resolution hyperspectral image.

2. The multi-scale multi-head attention hyperspectral image fusion method according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.1: First, the LR-HSI image is subjected to Bicubic upsampling so that the LR-HSI image has the same resolution as the RGB image in the spatial dimension; Step 1.2: After upsampling, the upsampled LR-HSI is normalized to scale the pixel values ​​to the range of [0,1]; Step 1.3: Perform data augmentation on the normalized LR-HSI to expand the dataset.

3. The multi-scale multi-head attention hyperspectral image fusion method according to claim 2 is characterized in that: The step 2 is specifically as follows: RGB and LR-HSI are two modes of the same image. RGB is a high-resolution multispectral image composed of three spectral bands, and LR-HSI is a low-resolution hyperspectral image composed of multiple spectral bands. The input RGB image LR-HSI image after upsampling and normalization Connect in the spectral channel dimension to get the feature input of the neural network Where C1 and C2 represent the feature channels of RGB and LR-HSI respectively, and H and W represent the height and width of the image.

4. The multi-scale multi-head attention hyperspectral image fusion method according to claim 3 is characterized in that: The step 3 is specifically as follows: After connecting Enter the multi-scale convolution module, which contains multiple parallel convolutional layers of different sizes to capture spatial information of different scales simultaneously; The multi-scale convolution module specifically operates as follows: First, use a 1x1 convolutional layer to input Perform preliminary extraction of shallow features and input I input Map from low-dimensional space to high-dimensional feature space, and implement high-dimensional embedding for each pixel token; F embed =Conv 1×1 (I input ) in, represents the embedding feature, C embed Indicates the number of feature channels of the output, Conv 1x1 Represents a 1x1 convolutional layer, I input The feature channel (C1+C2) is divided into three heads. Each head uses the same 3x3 convolution kernel. Each head has a different dilation rate. Different receptive fields are obtained under different dilation rates, and three features at different scales are extracted. <h2 style=";text-align:left;direction:ltr">F1,F2,F3=split(F<h2 style=";text-align:left;direction:ltr"> embed <h2 style=";text-align:left;direction:ltr"> ) Among them, split(·) represents the split channel, Conv 3x3 represents a 3x3 convolution, Respectively represent the intermediate feature data after the segmentation channel, They represent the intermediate feature data after convolution, and Dilation represents Conv 3x3 The parameter dilation rate of the convolution; Finally, the features extracted at different scales are connected to obtain multi-scale shallow features Wherein, Cat(·) represents the connection channel.

5. The multi-scale multi-head attention hyperspectral image fusion method according to claim 4 is characterized in that: The step 4 is specifically as follows: The multi-scale shallow features are input into a feature interaction module, and the feature interaction module enhances information exchange between features through an attention mechanism; The feature interaction module includes a multi-scale multi-head attention module and a central attention module; The multi-scale multi-head attention module realizes the feature fusion of multi-scale shallow features F0 through window attention and head attention; The central attention module makes up for the limitation of insufficient interaction between adjacent windows of window attention in the multi-scale multi-head attention module.

6. The multi-scale multi-head attention hyperspectral image fusion method according to claim 5 is characterized in that: The multi-scale multi-head attention is specifically: The multi-scale multi-head attention module introduces the head attention module; the head attention module calculates the attention weights between different heads to promote the communication of cross-head feature information. For a given input multi-scale shallow feature F0, the calculation process of the multi-scale multi-head attention module is: F N =LN(F0), F M =MSA(F N )+αHAB(F N )+F0, F out =MLP(LN(F M ))+F M , where F N and F M represents the intermediate feature, F out represents the output of the multi-scale multi-head attention module, L N represents layer normalization operation, MLP represents multi-layer perceptron, MSA represents shift window multi-head self-attention module, HAB represents head attention module, and α represents the scaling factor; Specifically, for a given size of input multi-scale shallow feature F0, it is first divided into windows, each of which is S×S in size, and then calculate self-attention in each window. After dividing the multi-scale shallow feature F0 into windows, the local window feature F of each window is obtained. W ∈R S×S×C , are linearly mapped to Q, K, and V through three matrices respectively, and then the calculation method of self-attention is Where d represents the scaling factor and B represents the relative position encoding.

7. The multi-scale multi-head attention hyperspectral image fusion method according to claim 6 is characterized in that: The specific calculation process of the head attention module is: For the input multi-scale shallow feature F0, the feature channel C is first divided and linearly mapped to obtain V in the self-attention, where V∈R HW×N×C′ , Q and K are obtained by first dividing the feature channel C into N heads Then, two mean pooling operations are performed on the HW and C′ dimensions to obtain F. avg ∈R N×1×1 , Q and K are obtained through linear mapping, and the attention weight Attn∈R calculated by Q and K N×N , copy the attention weight Attn multiple times to get Attn′∈R HW ×N×N , and finally the self-attention calculation method is Attention(Q,K,V)=Attn′×V A central attention module is introduced after the multi-scale multi-head attention module.

8. The multi-scale multi-head attention hyperspectral image fusion method according to claim 7 is characterized in that: The details of the central attention module are: The output feature F of the multi-scale multi-head attention module out Through linear mapping, it is converted into three matrices Q, K and V. Then, the features are divided into multiple attention heads, and each attention head processes information independently; For each position (i, j) in the feature map, a central sampling strategy is adopted. The K and V matrices are sampled in the area centered at (i, j), and the sampling range is a window of size W×W. Traverse the feature map with a window of size 3×3, with a step size of 1, and sample the content within the window; The feature interaction module interactively integrates the multi-scale multi-head attention module and the central attention module; The calculation process of the feature interaction module is as follows: F m =CAB(MSMH(F0)) in represents the output of the feature interaction module, F0 represents the multi-scale shallow features of the input, MSMH(·) represents the multi-scale multi-head attention module, and CAB(·) represents the central attention module.

9. The multi-scale multi-head attention hyperspectral image fusion method according to claim 8, characterized in that: The step 5 is specifically as follows: Will A convolution layer with a convolution kernel size of 3x3 and a ReLU activation function are used for nonlinear transformation to finally generate the residual information of a high-resolution hyperspectral image. F r =Release(Conv 3×3 (F m ))。 10. The multi-scale multi-head attention hyperspectral image fusion method according to claim 9, characterized in that: The step 6 is specifically as follows: The generated residual information is added element by element to the LR-HSI after upsampling in step 1 to obtain a final high-resolution hyperspectral image; I HR-HSI =I LR-HSI +F r 。

Images