Hyperspectral and multispectral image fusion method and system based on improved Transform

Abstract

The invention discloses a hyperspectral and multispectral image fusion method and system based on an improved Transform, and the method comprises the steps: carrying out the up-sampling of a hyperspectral image, and extracting the potential features of the hyperspectral image; multi-scale features of the hyperspectral image and the multi-spectral image are respectively extracted by using a double-branch structure, and global feature extraction is carried out on feature maps through a multi-layer deformable convolution crossover Transform; and adding the extracted features according to the same direction, splicing the features in a channel dimension, and gradually reconstructing a multispectral image through cascade up-sampling. The invention provides a new method based on an up-sampling recursive sub-network and a multilayer deformable convolutional cross Transformer (URSMDCT) for hyperspectral and multispectral image fusion, potential features and multiscale features of the image can be fully extracted, and spectral information and spatial information are effectively fused.

Description

Technical Field

[0001] The present invention belongs to the technical field of image reconstruction, and the present invention relates to a hyperspectral and multispectral image fusion method and system based on an improved Transformer. Background Art

[0002] The fusion of hyperspectral and multispectral images has experienced a long period of development, and many methods have emerged, which can be roughly divided into four types: methods based on pan-sharpening, methods based on matrix decomposition, methods based on tensor decomposition, and methods based on deep learning. Based on the pan-sharpening method, the high-resolution multispectral image HrMSI is fused into an image with higher spectral resolution than the PAN image, showing better spatial resolution. In order to adapt this method to classical pan-sharpening, it is necessary to solve the band allocation problem, that is, to select the appropriate HrMSI band on the given low-resolution hyperspectral image LrHSI band to improve the spatial resolution. Decomposition-based methods include matrix decomposition and tensor decomposition. Matrix decomposition first uses the LrHSI learning dictionary, and then combines the transformed HrHSI dictionary with the sparse representation coefficient to obtain HrHSI; tensor decomposition uses the characteristics of the three-dimensional cube structure of HrMSI and LrHSI that contain both spatial and spectral information to achieve high-resolution hyperspectral image HrHSI reconstruction. The above two methods have high data requirements, large computational complexity, and require certain image prior knowledge. With the rapid development of deep learning, CNN has made great achievements in the field of image fusion, and representative methods such as MHF-net, SSR-NET, MoG-DCN, and MSDCNN have emerged, all of which have achieved good fusion effects. However, CNN-based methods cannot extract long-range dependencies in images due to the limited receptive field of the convolution kernel.

[0003] At the same time, Transformer is widely used to extract global features of source images due to its powerful long-range modeling capabilities. Vibashan et al. proposed an image fusion Transformer (IFT) method that combines local information with global information to improve fusion performance. Hu et al. proposed the Fusformer method, which was the first to introduce Transformer into the field of hyperspectral and multispectral image fusion. Jia et al. proposed a multi-scale spatial-spectral transformer network MSST-Net, which fully utilized the self-attention mechanism of Transformer to extract spectral and spatial features of hyperspectral images. These methods use Transformer as the backbone network, use the self-similarity measurement matrix to dynamically model the correlation between sequences, effectively extract long-distance dependencies, and better express the characteristics of the image. However, most of the current Transformer-based fusion methods only extract a single modality of HSI and MSI, and finally simply splice and fuse the features of the two modalities, thereby ignoring the cross-modal information interaction between the two images in the spatial modality and spectral modality, lacking the effective use of spatial position information, and not conducive to the completion of the fusion task. Summary of the invention

[0004] The purpose of the present invention is to provide a hyperspectral and multispectral image fusion method and system based on an improved Transformer. A new upsampling recursive subnetwork and multi-layer deformable convolution cross Transformer (URSMDCT) is proposed for the fusion of hyperspectral and multispectral images, which can fully extract the potential features of the image and the multi-scale features of the image, and effectively fuse the spectral information and spatial information.

[0005] The technical solution to achieve the purpose of the present invention is:

[0006] A hyperspectral and multispectral image fusion method based on an improved Transformer comprises the following steps:

[0007] S01: Upsample the hyperspectral image to extract the potential features of the hyperspectral image;

[0008] S02: Use a dual-branch structure to extract multi-scale features of hyperspectral images and multispectral images respectively, and perform global feature extraction on feature maps through multi-layer deformable convolution cross Transformer;

[0009] S03: Add the extracted features in the same direction, concatenate them in the channel dimension, and gradually reconstruct the multispectral image through cascade upsampling.

[0010] In the preferred technical solution, the method for upsampling the hyperspectral image in step S01 includes:

[0011] The hyperspectral image is initialized through the convolution layer and the ReLU nonlinear activation layer to obtain a series of feature maps;

[0012] Through two branches, one branch is directly transmitted to the end of the network through the global residual connection, and the other branch performs deep feature extraction through multiple consecutive recursive blocks. The features output by each recursive block are connected with the features output by the last recursive block through the residual connection to obtain the second feature;

[0013] The deconvolution layer is used to reorganize the extracted image features to obtain the upsampled image.

[0014] In the preferred technical solution, the method for extracting global features from the feature map in step S02 includes:

[0015] First, linear projection is performed on the input hyperspectral image feature map F1 and multispectral image feature map F2 to obtain the feature maps after dimensionality reduction. The position information of the feature maps F1 and F2 is encoded using learnable position embedding, and then fused with the feature maps after dimensionality reduction. Then, the features are layer-normalized to obtain two sequences y1 and y2.

[0016] The core module Multi-head Fusion Attention (MHFA) is used to model the long-range dependency between hyperspectral images and multispectral images. MHFA obtains the attention map by obtaining the correlation between the query vector Query and the key vector Key, and then obtains the eigenvalues ​​in the value vector Value based on another attention map.

[0017] After the feature map output by MHFA is normalized and fed-forwarded, the feature mapping module is used to convert the output sequence of MHFA into a standard feature map. After the cross attention, the deformable convolution module is introduced to adaptively determine the scale and adjust the receptive field size according to the objects of different scales corresponding to different positions, so as to further refine the features of the fused image.

[0018] In the preferred technical solution, in step S03, feature aggregation and reconstruction are performed through a double-layer asymmetric convolution module, the input features and output features of the multi-layer deformable convolution cross Transformer are jump-connected, and the potential features of the image are utilized through a double-layer asymmetric convolution module and a cascade upsampling operation to restore the spatial information.

[0019] In the preferred technical solution, the double-layer asymmetric convolution module includes strip convolution. The input feature map is input into four convolution paths of different shapes after 1×1 convolution. It captures long-distance context information from four directions: horizontal, vertical, left diagonal and right diagonal, and extracts texture details to obtain four feature maps. The four feature maps are respectively subjected to the second layer of strip convolution to further extract edge features. The extracted features are added in the same direction, spliced ​​in the channel dimension, and finally reduced in dimension through upsampling and convolution.

[0020] In the preferred technical solution, step S03 further includes: optimizing the reconstructed image using a loss function, where the loss function is:

[0021]

[0022] Among them, L fus is the loss function of the fused image, Z is the reference image, is the reconstructed image, where H, W, and C are the height, width, and number of channels.

[0023] The present invention also discloses a hyperspectral and multispectral image fusion system based on an improved Transformer, comprising:

[0024] The upsampling recursive sub-network module upsamples the hyperspectral image and extracts the potential features of the hyperspectral image;

[0025] The multi-layer cross-modal interaction module uses a dual-branch structure to extract multi-scale features of hyperspectral images and multispectral images respectively, and extracts global features from feature maps through a multi-layer deformable convolution cross Transformer.

[0026] The feature aggregation reconstruction module adds the extracted features in the same direction, concatenates them in the channel dimension, and gradually reconstructs the multispectral image through cascade upsampling.

[0027] In the preferred technical solution, the upsampling recursive subnetwork module includes a feature initialization layer, 3 recursive blocks and a deconvolution layer. The recursive block consists of two groups of 3×3 convolution and ReLU activation layers. The input and output are residually connected to fuse several consecutive residual blocks with deep features.

[0028] In the preferred technical solution, the multi-layer cross-modal interaction module linearly projects the input hyperspectral image feature map F1 and the multispectral image feature map F2 to obtain the feature map after dimensionality reduction, and uses the learnable position embedding to encode the position information of the feature maps F1 and F2, which are fused with the feature map after dimensionality reduction, and then the features are layer-normalized to obtain two sequences y1 and y2;

[0029] The core module Multi-head Fusion Attention (MHFA) is used to model the long-range dependency between hyperspectral images and multispectral images. MHFA obtains the attention map by obtaining the correlation between the query vector Query and the key vector Key, and then obtains the eigenvalues ​​in the value vector Value based on another attention map.

[0030] After the feature map output by MHFA is normalized and fed-forwarded, the feature mapping module is used to convert the output sequence of MHFA into a standard feature map. After the cross attention, the deformable convolution module is introduced to adaptively determine the scale and adjust the receptive field size according to the objects of different scales corresponding to different positions, so as to further refine the features of the fused image.

[0031] The present invention further discloses a computer storage medium on which a computer program is stored. When the computer program is executed, the hyperspectral and multispectral image fusion method based on the improved Transformer is implemented.

[0032] Compared with the prior art, the present invention has the following significant advantages:

[0033] The present invention proposes a new upsampling recursive subnetwork and multi-layer deformable convolution cross Transformer (URSMDCT) for the fusion of hyperspectral and multispectral images. URSMDCT consists of three parts: (1) Upsampling recursive subnetwork module: LrHSI is upsampled by using the powerful nonlinear ability of neural network to fully extract the potential features of hyperspectral images; (2) Multi-layer cross-modal interaction module: Multi-scale features of LrHSI and HrMS are extracted respectively by using dual branches, and global features are extracted from feature maps by using multi-layer deformable convolution cross Transformer, thus realizing cross-modal information interaction between LrHSI and HrMSI at the same scale, making up for the lack of spectral information in HrMSI and the lack of spatial information in LrHSI. (3) Feature aggregation reconstruction module: Double-layer asymmetric convolution module is used to further restore edge features, and HrHSI is gradually reconstructed by cascade upsampling.

[0034] 1. This paper designs an improved upsampling recursive sub-network, which enhances feature propagation and stimulates feature reuse by aggregating residual blocks with deep features, and fully extracts the potential features of the image by utilizing the powerful nonlinear representation capability of CNN.

[0035] 2. This paper designs a multi-layer cross-modal interaction module based on a dual-branch structure to fully extract the multi-scale features of the image, and extracts the global features of two feature maps of the same scale through a multi-layer deformable convolutional cross Transformer, effectively integrating spectral information and spatial information.

[0036] 3. This paper designs a feature aggregation reconstruction module, which uses two-layer asymmetric convolution to refine the spatial edge features and texture features of the fused image, and captures long-range context information from two levels and four directions (horizontal, vertical, left diagonal and right diagonal) of the feature map through strip convolution.

[0037] 4. The method proposed in this paper is experimented on five public datasets: Pavia, Washington, Houston, Chikusei, and Xiongan New Area, and compared with 10 most advanced algorithms. The experimental results verify the effectiveness of the method proposed in this paper. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the overall block diagram of the upsampling recursive sub-network and multi-layer deformable convolution cross Transformer of this embodiment;

[0039] Figure 2 This is the structural diagram of the upsampling recursive sub-network module;

[0040] Figure 3 This is the overall structure diagram of the multi-layer deformable convolution cross Transformer;

[0041] Figure 4 This is the structure diagram of multi-head fusion attention MHFA;

[0042] Figure 5 a-5c is the structural diagram of the feature aggregation reconstruction module;

[0043] Figure 6 The fusion results of different methods on the Pavia Center dataset; the first row shows the pseudo RGB result image after fusion, the second row shows the local magnification of the fused image, and the third row shows the difference image after pseudo color processing;

[0044] Figure 7 The fusion results of different methods on the Washington DC Mall dataset; the first row shows the pseudo RGB result image after fusion, the second row shows the local magnification of the fused image, and the third row shows the difference image after pseudo color processing;

[0045] Figure 8 The fusion results of different methods on the Houston dataset; the first row shows the pseudo RGB result image after fusion, the second row shows the local magnification of the fused image, and the third row shows the difference image after pseudo color processing;

[0046] Fig. 9The fusion results of different methods on the Chikusei dataset; the first row shows the pseudo RGB result image after fusion, the second row shows the local magnification of the fused image, and the third row shows the difference image after pseudo color processing;

[0047] Fig.10 The fusion results of different methods on the Xiongan New Area dataset; the first row shows the pseudo-RGB result image after fusion, the second row shows the local magnification of the fused image, and the third row shows the difference image after pseudo-color processing;

[0048] Fig.11 The effect diagram of different modules on the ablation experiment; the first row shows the pseudo RGB result image after fusion, and the second row shows the difference image after pseudo color processing; (a) removing the upsampling recursive subnetwork, (b) removing the multi-layer deformable convolution cross Transformer, (c) removing the double-layer asymmetric convolution, (d) URSMDCT, (e) GT. DETAILED DESCRIPTION

[0049] The principle of the invention is: a new upsampling recursive subnetwork and multi-layer deformable convolutional cross Transformer (URSMDCT) is proposed for the fusion of hyperspectral and multispectral images, which can fully extract the potential features of the image and the multi-scale features of the image, and effectively integrate the spectral information and spatial information.

[0050] Embodiment 1:

[0051] In order to promote the effective fusion of LrHSI and HrMSI, this paper proposes a new upsampling recursive subnetwork and multi-layer deformable convolution cross Transformer (URSMDCT) for the fusion of hyperspectral and multispectral images. URSMDCT consists of three parts: (1) Upsampling recursive subnetwork module: LrHSI is upsampled by using the powerful nonlinear ability of neural network to fully extract the potential features of hyperspectral images; (2) Multi-layer cross-modal interaction module: The multi-scale features of LrHSI and HrMS are extracted by using dual branches, and the feature map is extracted globally by using multi-layer deformable convolution cross Transformer, which realizes the cross-modal information interaction between LrHSI and HrMSI at the same scale, making up for the lack of spectral information in HrMSI and the lack of spatial information in LrHSI. (3) Feature aggregation reconstruction module: The edge features are further restored by using a double-layer asymmetric convolution module, and HrHSI is gradually reconstructed by cascade upsampling.

[0052] The overall structure of the URSMDCT model is as follows: Figure 1 As shown in the figure, the network adopts a dual-branch structure, which is mainly composed of an upsampling recursive sub-network module, a multi-layer cross-modal interaction module, and a feature aggregation and reconstruction module.

[0053] The upsampling recursive subnetwork module uses CNN to fully extract the potential features of the image, fully solving the problem of spatial-spectral information loss in the process of upsampling LrHSI to HrHSI through simple linear interpolation. Considering that remote sensing images have a large scale space, features of different scales have different effects on reconstructing images. The low-level detail feature map contains rich spatial information, and the high-level semantic feature map reflects detailed location information. For this reason, a multi-layer cross-modal interaction module is designed. Based on a dual-branch structure, this module extracts multi-scale information from the input image, and extracts the global features of two feature maps of the same scale through multi-layer cross Transformers, fully integrating spectral information and spatial information. At the same time, deformable convolution is introduced to adaptively adjust the shape of the convolution kernel to better adapt to features of different scales. In the feature aggregation reconstruction module, double-layer asymmetric convolution is used to restore the spatial detail features of the fused image, and high-resolution hyperspectral images are reconstructed through cascade upsampling. The contributions of this paper are as follows:

[0054] 1. This paper designs an improved upsampling recursive sub-network, which enhances feature propagation and stimulates feature reuse by aggregating residual blocks with deep features, and fully extracts the potential features of the image by utilizing the powerful nonlinear representation capability of CNN.

[0055] 2. This paper designs a multi-layer cross-modal interaction module based on a dual-branch structure to fully extract the multi-scale features of the image, and extracts the global features of two feature maps of the same scale through a multi-layer deformable convolutional cross Transformer, effectively integrating spectral information and spatial information.

[0056] 3. This paper designs a feature aggregation reconstruction module, which uses two-layer asymmetric convolution to refine the spatial edge features and texture features of the fused image, and captures long-range context information from two levels and four directions (horizontal, vertical, left diagonal and right diagonal) of the feature map through strip convolution.

[0057] The upsampling recursive subnetwork module consists of a feature initialization layer, three recursive blocks, and a deconvolution layer. Figure 2 As shown in the figure, the recursive block consists of two groups of 3×3 convolution and ReLU activation layers, and the input and output are connected with residuals. Fusing several consecutive residual blocks with deep features can alleviate the problem of gradient disappearance and generate more representative features. The specific process is as follows:

[0058] After the LR-HSI image is input into the upsampling recursive sub-network, it is first initialized by a 3×3 convolutional layer and a ReLU nonlinear activation layer to obtain a series of feature maps L1∈R H×w×CThen, through two branches, one branch is directly transmitted to the end of the network through the global residual connection. The function of this residual connection is to directly give the low-frequency information contained in the LR-HSI image that has not been processed too much to the image to be generated. The other branch performs deeper feature extraction through 3 consecutive recursive blocks. The features output by each recursive block are connected with the features output by the last recursive block through the residual connection to obtain L2∈R with richer features. H×w×C Finally, the deconvolution layer is used to reorganize the extracted image features to obtain the upsampled image LR0∈R H×W×C The process is shown in Formula 1-3.

[0059] L1=ReLU(Conv 3×3 (LR-HSI))(1)

[0060] L2=R(R(R(L1))) (2)

[0061] LR0=Dec(L1+L2)(3)

[0062] Where R represents the recursive block and Dec represents the deconvolution layer.

[0063] In the multi-layer cross-modal interaction module, the design of the multi-layer deformable convolution cross Transformer module realizes the cross-modal information fusion of spectral modality and spatial modality, and models the long-range dependency relationship of feature maps. Figure 3 As shown in the figure, the multi-layer deformable convolution cross Transformer adopts a dual-branch structure, including linear projection, feedforward network, layer normalization, deformable convolution, and the core module Multi Head Fusion Attention (MHFA).

[0064] The specific process is as follows:

[0065] (1) First, linear projection is performed on the input LR-HSIF1 and HR-MSIF2 to reduce the channel dimension and remove the adverse effects of redundant information to obtain the reduced-dimensional feature map. At the same time, the position information of the feature maps F1 and F2 is encoded using learnable position embedding, and then fused with the reduced-dimensional feature map. The features are then layer-normalized to obtain two d×N (N=H×W) sequences y1, y2. d represents the channel dimension of the spectrum, and N is the height×width of the spectral image. The process is shown in Formula 4.

[0066] y i =LN(LP(F i )+PE) (4)

[0067] Among them, i can be 1 or 2, LP represents the linear projection operation, PE represents position embedding, and LN represents layer normalization.

[0068] (2) Considering that the two branches extract modal information on different feature maps respectively, in order to make the two cross-modal information interact more fully and realize the fusion of spatial and spectral information, the core module multi-head fusion attention MHFA is then used to model their long-range dependencies. Different from the traditional self-attention mechanism, MHFA obtains the attention map by obtaining the correlation between the query vector Query and the key vector Key, and then obtains the feature value in the value vector Value based on another attention map. Figure 4 As shown in the figure, y1 and y2 are used as inputs to split n attention heads in the spatial domain. Linear transformation is performed on y1 and y2, and Q, K and V are generated by multiplying them with different weight matrices. Then, after a series of dot product operations and activation functions, the sequences Z1 and Z2 are finally obtained respectively. The process is as follows:

[0069]

[0070] Where Attention() is the scaled dot product attention function, d k is the vector dimension of the query vector Q and the key vector K in the self-attention mechanism, and softmax() is the activation function.

[0071] (3) After the feature map is normalized and fed-forwarded, in order to adapt its dimension to the next network structure, the feature mapping module is used to convert the output sequence of MHFA into a standard H×W×C feature map (height, width, number of channels) k1, k2. The result of adding k1 and k2 is output through deformable convolution to make its dimension the same as the input feature map. Among them, the feed-forward network contains a two-layer multilayer perceptron with a hidden layer expansion rate of r. In order to impose greater weight on more meaningful positions, a deformable convolution module is introduced after the cross attention. According to the different scales of objects corresponding to different positions, the scale and receptive field size can be adaptively determined to further refine the features of the fused image. The process is shown in Formula 7.

[0072] Output=DeConv(k3) (7)

[0073] k3 is the result image generated by adding the two feature maps k1 and k2 element by element on the pixel, and DeConv() represents the deformable convolution operation.

[0074] like Figure 5As shown in a-5c, the feature aggregation and reconstruction module designs a double-layer asymmetric convolution module for feature aggregation and reconstruction, jump-connects the input features and output features of the multi-layer deformable convolution cross Transformer, and fully utilizes the potential features of the image through operations such as double-layer asymmetric convolution modules and cascade upsampling to gradually restore the spatial information.

[0075] In the figure, C stands for concat, which means that the two images are concatenated in the channel dimension without changing the width and height of the image. The core of the double-layer asymmetric convolution is the strip convolution. Compared with the traditional convolution, the strip convolution has fewer parameters and higher accuracy. In the strip convolution block, the input feature map is input into 4 convolution paths of different shapes after 1×1 convolution. It captures long-distance context information from four directions: horizontal, vertical, left diagonal, and right diagonal, and extracts texture details to obtain four feature maps. These four maps are then further extracted by the second layer of strip convolution. Edge features. In order to make the feature map fuse feature information from different levels and directions, the extracted features are added in the same direction, concatenated in the channel dimension, and finally reduced in dimension through upsampling and convolution. The process is as follows: In the first layer of convolution, the input is F, which is convolved horizontally with H, vertically with V, left diagonally with L, and right diagonally with R, and the output is F. H , F V , F L , F R The second convolution layer takes as input the output F of the first convolution layer X , where X can be in four directions: H, V, L, and R, and the output is F XH , F XV , F XL , F XR Finally, the result of the second convolution layer F is respectively HX , F VX , F LX , F RX Add element by element, and the result is the same as the F generated by the first convolution layer. X Perform splicing to generate Fxs.

[0076] (F H , F V , F L , F R )=StripConv(F) (8)

[0077] (F XH , F XV , F XL , F XR )=StripConv(F X ) (9)

[0078]

[0079] Among them, F XS They are the outputs of convolution in four directions respectively.

[0080] In the super-resolution image reconstruction task, since the difference between the predicted value and the true value is not large, the MSE loss function is more effective. Therefore, URSMDCT uses MSE to reconstruct HrHSI. Optimize as shown in formula 11:

[0081]

[0082] Where L fus is the loss function of the fused image, Z∈R H×W×C is the reference HrHSI, It is the reconstructed HrHSI.

[0083] Experimental results and analysis

[0084] This section first introduces the datasets and evaluation metrics used in the experiments, followed by the experimental settings. Finally, we compare the URSMDCT fusion results with 10 state-of-the-art algorithms on four hyperspectral datasets and conduct ablation experiments to verify the effectiveness of different modules in our method.

[0085] In order to effectively verify the superiority of the proposed method, five public datasets were used for experiments, including Pavia Center, Washington, Houston, Chikusei, and Xiongan New Area datasets.

[0086] (i) Pavia Center: The Pavia Center dataset is obtained by the ROSIS sensor, which has a total of 115 bands. After processing, the Pavia Center dataset actually has 102 bands used for training, with a size of 1096×715.

[0087] (ii) Washington DC Mall: The Washington DC National Mall (WDCM) dataset was collected by the Hyperspectral Digital Imagery Acquisition Experiment (HYDICE) sensor over the National Mall in Washington DC in 1995. WDCM has a total of 210 bands, and after removing the water absorption band, the remaining 191 bands are used. The size of WDCM is 1280×307, and the spatial resolution is 2.5 meters.

[0088] (iii) Houston: The Houston dataset is provided by the Hyperspectral Image Analysis Group and the NSF-funded Center for Airborne Laser Mapping (NCALM) at the University of Houston. It contains 144 bands, the size of Houston is 349×1905, and the spatial resolution is 2.5 meters.

[0089] (iv) Chikusei: This dataset was taken by the Headwall HyperSpec-VNIR-C sensor in Tsukushi, Japan, and contains 128 spectral bands. To ensure the fusion quality, 110 spectral bands are retained after discarding the bands contaminated by noise. The size of Chikusei is 2354×2185, and the spatial resolution is 2.5 meters.

[0090] (v) Xiongan New Area: The Xiongan (Ma Ti Wan Village) dataset is collected by the GF-MS full-spectrum multi-modal imaging spectrometer. The spectral range is 400-1000nm, the number of bands is 250, and the size is 3750*1580.

[0091] Evaluation indicators

[0092] The experiment uses a combination of subjective and objective evaluation to verify the effectiveness of URSMDCT. In the objective evaluation, four evaluation indicators are used for comparison: root mean square error RMSE, peak signal-to-noise ratio PSNR, relative global dimensionless error ERGAS, and spectral angle mapping SAM. In the subjective evaluation, not only the RGB image and detail magnification of the fused image are compared, but also the difference between the fusion result and the reference HrHSI is shown.

[0093] Among the following objective evaluation indicators, Z k (i,j) represents the reference HrHSI value at the k-th band position (i,j), Represents the estimated HrHSI value of the kth band.

[0094] (i) RMSE is used to measure the difference between the reference image and the reconstructed image. The smaller the RMSE, the better the quality of the reconstructed image. RMSE is defined as:

[0095]

[0096] (ii) PSNR reflects the ratio of peak power to noise power in the fused image, and can reflect the distortion during the fusion process at the pixel level. The larger the PSNR, the better the quality of the reconstructed image. The PSNR of the i-th band is defined as:

[0097]

[0098] Where ||·||2 is the two-norm, max is the maximum possible value of the pixel value, and HW is the total number of pixels in the image.

[0099] (iii) ERGAS is mainly used to evaluate spectral distortion and can reflect the overall situation of spectral changes. By weighted summing the MSE of each band and dividing it by the spatial resolution of the image and the square root of the band mean, a single value reflecting the global spectral distortion is obtained. The smaller the ERGAS, the better the spectral quality of the fused image.

[0100]

[0101] Among them, h is the spatial resolution of HrMSI, l is the spatial resolution of LrHSI, N is the number of bands, B i is the HrMSI of the ith band, M i is the pixel mean of the i-th band of the reconstructed image.

[0102] (iv) SAM measures the similarity between spectra by calculating the angle between spectral data, which is a measure of the spectral quality of the reconstructed image. The smaller the SAM value, the better the spectral quality of the fused image.

[0103]

[0104] in,<i,j> Represents the inner product operation.

[0105] Comparison Methods

[0106] In this paper, the proposed method is compared with 10 state-of-the-art fusion methods. The 10 fusion algorithms are CNMF

[20] , CSU

[30] , SSE

[31] , TFNet

[26] , ResTFNet

[26] , SSFCNN

[24] , ConSSFCNN

[24] , MSDCNN

[25] , SSR-NET[8] and MCT

[29] .

[0107] Experimental setup

[0108] LrHSI is obtained by Gaussian blurring and four-fold downsampling of the HrHSI image. The experiment extracts five equally spaced bands from the HrHSI to form the HrMSI, crops a 128×128 sub-region in the center of the image for testing, and the rest is used for training. In each iteration, a 128×128 image is randomly cropped from the training area for training. The operating system used in this experiment is Liunx system ubantu20.04, NVIDIA2080Ti, the deep learning framework used is PyTorch 1.10.0, the programming language is Python 3.8, the optimizer is Adam, and the learning rate is set to 0.0001.

[0109] Experimental Results

[0110] Table 1 Experimental results on Pavia. The best and suboptimal results are marked in red and blue respectively.

[0111]

[0112] Table 2 Experimental results on Washington DC Mall. The best and second-best results are marked in red and blue respectively.

[0113]

[0114] Table 3 Experimental results on Houston. The best and suboptimal results are marked in red and blue respectively.

[0115]

[0116] Table 4 Experimental results on Chikusei. The best and suboptimal results are marked in red and blue respectively.

[0117]

[0118] Table 5 Experimental results on Xiongan New Area. The best and suboptimal results are marked in red and blue respectively.

[0119]

[0120] (i) On the Pavia Center dataset: The results of various evaluation indicators in the experiment are shown in Table 1, and the subjective results are shown in Figure 6 . Objectively, the method in this paper performs best in all four evaluation indicators. Compared with the suboptimal method, it shows the strong ability of URSMDCT in detail recovery and spectral preservation. Compared with the suboptimal result MCT, RMSE decreased by 14.74%, PSNR increased by 3.78%, ERGAS decreased by 11.7%, and SAM decreased by 12.38%. The advantages of the indicators are obvious. Subjectively, it can be seen from the observation of the local magnified image that the two buildings of the reference real HrHSI present a darker red. Except for MCT and URSMDCT, the other methods are very light red, which produces spectral distortion; compared with MCT, URSMDCT is better in processing yellow light strips, and the results are brighter and clearer. From the difference map after pseudo-color processing, the error of URSMDCT is obviously smaller than that of MCT, and it is closer to the real HrHSI. Therefore, from the overall analysis, compared with other methods, the HrHSI image reconstructed by URSMDCT has clear edges and better spectral information retention, which is enough to prove that URSMDCT can well extract and retain spatial detail information.

[0121] (ii) On the Washington DC Mall dataset: The results of various evaluation indicators in the experiment are shown in Table 2, and the subjective results are shown in Figure 7 Objectively, although the traditional methods also achieved good results, URSMDCT performed better in all four indicators. Compared with the suboptimal results, RMSE decreased by 21.74%, PSNR increased by 4.58%, ERGAS decreased by 21.89%, and SAM decreased by 19.91%. Subjectively, as Figure 7 As shown in the figure, from the comparison of pseudo-RGB images, it can be seen that all CNN-based methods have blurred edges; from the comparison of local magnification images, it can be seen that TFNet and ConSSFCNN are overall blurred, the details are not processed properly, and the spectrum is seriously distorted. Compared with MCT, URSMDCT is closer to the real image, and the local details of the black block are clearer; from the difference comparison of pseudo-color processing, it can be seen that compared with other methods, although MCT performs well, URSMDCT has the smallest error. Although the number of bands in this dataset is large, URSMDCT still achieves very competitive reconstruction results.

[0122] (iii) On the Houston dataset: The results of various evaluation indicators in the experiment are shown in Table 3; the subjective results are shown in Figure 8 . Objectively, URSMDCT achieved the best results in all four indicators. Compared with the suboptimal results, RMSE decreased by 24.86%, PSNR increased by 5.78%, ERGAS decreased by 30.03%, and SAM decreased by 30.06%. Subjectively, from the local magnification of the fused image, ConSSFCNN has a higher overall brightness than other methods, the texture details of the flower are lost, and the spectral distortion is large. The center of the flower of the real HrHSI is brighter, and the brightness of the middle of the flower is missing for methods other than URSMDCT and ConSSFCNN. From the difference comparison of pseudo-color processing, although SSFCNN and SSR-Net also have good error performance, URSMDCT has the smallest error, showing a pure dark blue color, which is closest to the reference image. This fully demonstrates the effectiveness of URSMDCT in maintaining spectral and spatial information.

[0123] (iv) On the Chikusei dataset: The results of various evaluation indicators in the experiment are shown in Table 4; the subjective results are shown in Fig. 9. Objectively, the method we proposed achieved the best results in all four indicators. Compared with the suboptimal results, RMSE decreased by 20.39%, PSNR increased by 4.43%, ERGAS decreased by 22.35%, and SAM decreased by 15.14%. From the comparison of subjective fusion results, the CNN-based method has problems in spectral preservation, especially the SSFCNN method has the most serious spectral distortion. In the local magnification image, the MSDCNN, TFNet, and ConSSFCNN methods have edge blurring. In contrast, URSMDCT is closer to the reference HrHSI, and is better than the CNN-based method in both local magnification comparison and pseudo-color processing difference value comparison, indicating that URSMDCT can improve the overall image fusion quality on this dataset.

[0124] On the Xiongan New Area dataset: The results of the various evaluation indicators in the experiment are shown in Table 5; the subjective results are shown in Fig.10 Objectively, the method we proposed achieved the best results in all four indicators. Compared with the suboptimal results, RMSE decreased by 6.04%, PSNR increased by 2.03%, ERGAS decreased by 11.24%, and SAM decreased by 6.10%. From the comparison of subjective fusion results, the spectrum of the ConSSFCNN method was severely distorted, and the details of the local magnification images processed by MSDCNN, SSFCNN, and ConSSFCNN were blurred, and the edges were not clear enough. From the difference comparison of pseudo-color processing, compared with the real HrHSI, other methods have errors, and the result of URSMDCT is basically the same as the real value, which fully demonstrates the excellent performance of URSMDCT on the Xiongan New Area dataset.

[0125] Ablation experiment

[0126] In order to verify the influence of each module in URSMDCT on the fusion results, we conducted ablation experiments on the upsampling recursive sub-network module, the multi-layer deformable convolution cross Transformer module, and the double-layer asymmetric convolution module. The ablation experiment results are shown in Table 5 and Fig.11 As shown. In order to verify the effect of the upsampling recursive subnetwork module, this paper uses traditional bilinear interpolation upsampling to replace the upsampling recursive subnetwork for ablation experiments. From the objective indicators, without the upsampling recursive subnetwork, compared with URSMDCT, RMSE increased by 11.23%, PSNR decreased by 19.03%, EGRGAS increased by 11.29%, and SAM increased by 5.19%. From the subjective graph, the network has serious spectral distortion, the image details are blurred, and the error is large compared with Figure (d) and Figure (e). This shows that the upsampling recursive subnetwork can improve the performance of the network.

[0127] Since the cross transformer can better capture the long-range context information in the image features by processing the semantic relationship between two different sequences, when the transformer is used for long-term training, almost uniform attention is used on the feature map, and no greater weight is applied to the more meaningful positions. The deformable convolution makes up for this defect. In order to verify the effect of the multi-layer deformable convolution cross transformer module, this paper replaces it with a cross transformer. From the objective indicators, without the multi-layer deformable convolution cross transformer, RMSE increased by 0.47%, PSNR decreased by 0.29%, EGRGAS increased by 1.24%, SAM increased by 0.33%, and all indicators decreased. From a subjective point of view, the spectral distortion is slightly better than that of Figure (a). From the comparison of the difference map, it can be seen that although Figure (b) performs well, it is slightly inferior to Figure (d) and Figure (e). The results prove the effectiveness of this module in improving fusion performance.

[0128] Since the spatial edge of the image contains high-frequency features, which is crucial for spatial reconstruction, and asymmetric convolution can capture the long-term dependencies of the region for irregular objects such as rivers and roads, this paper designs a double-layer asymmetric convolution module to restore the spatial information of the image in feature reconstruction. In order to verify the effectiveness of the double-layer asymmetric convolution, ordinary convolution is used to replace it for ablation experiments. The experimental results are shown in the third row of Table 5. Compared with the original results, the double-layer asymmetric convolution module is removed, and the RMSE increases by 8.23%, the PSNR decreases by 0.81%, the EGRGAS increases by 5.52%, and the SAM increases by 2.43%. All indicators decrease. From a subjective point of view, consistent with the above two network analyses, Figure (d) has a better effect. This shows that the double-layer asymmetric convolution module can effectively improve the performance of fusion.

[0129] Table 5 Ablation experiment results on the Washington DC Mall dataset

[0130]

[0131] in conclusion

[0132] This paper proposes an upsampling recurrent subnetwork and multi-layer deformable convolution cross Transformer for LrHSI and HrMSI image fusion. URSMDCT mainly consists of three parts: upsampling recurrent subnetwork module, multi-layer cross-modal interaction module, and feature aggregation and reconstruction module. In particular, a multi-layer deformable convolution cross Transformer module is designed in the multi-layer cross-modal interaction module, and deformable convolution is introduced to automatically adjust the receptive field to adapt to features of different scales. Experiments are conducted on five public datasets and compared with 10 state-of-the-art methods. The results show that URSMDCT performs best in RMSE, PSNR, ERGAS and SAM indicators. On the Pavia Center dataset, compared with the second-best result MCT, RMSE decreases by 14.74%, PSNR increases by 3.78%, ERGAS decreases by 11.7%, and SAM decreases by 12.38%. At the same time, the implementation of ablation experiments also verifies the importance of upsampling recurrent subnetwork, multi-layer deformable convolution cross Transformer and double-layer asymmetric convolution for preserving spectral information and extracting spatial information. In the future, we will explore extending this approach to other fields such as medical image processing, biological monitoring, and military defense, and improve the structure of the network to achieve a trade-off between performance and efficiency for hyperspectral data.

[0133] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. A hyperspectral and multispectral image fusion method based on improved Transformer, characterized in that: The following steps are involved: S01: Upsample the hyperspectral image to extract the potential features of the hyperspectral image; S02: Use a dual-branch structure to extract multi-scale features of hyperspectral images and multispectral images respectively, and perform global feature extraction on feature maps through multi-layer deformable convolution cross Transformer; S03: Add the extracted features in the same direction, concatenate them in the channel dimension, and gradually reconstruct the multispectral image through cascade upsampling.

2. The hyperspectral and multispectral image fusion method based on improved Transformer according to claim 1, characterized in that: The method for upsampling the hyperspectral image in step S01 includes: The hyperspectral image is initialized through the convolution layer and the ReLU nonlinear activation layer to obtain a series of feature maps; Through two branches, one branch is directly transmitted to the end of the network through the global residual connection, and the other branch performs deep feature extraction through multiple consecutive recursive blocks. The features output by each recursive block are connected with the features output by the last recursive block through the residual connection to obtain the second feature; The extracted image features are reorganized using the deconvolution layer to obtain the upsampled image.

3. The hyperspectral and multispectral image fusion method based on improved Transformer according to claim 1, characterized in that: The method for extracting global features from the feature map in step S02 includes: First, linear projection is performed on the input hyperspectral image feature map F1 and multispectral image feature map F2 to obtain the feature maps after dimensionality reduction. The position information of the feature maps F1 and F2 is encoded using learnable position embedding, and then fused with the feature maps after dimensionality reduction. Then, the features are layer-normalized to obtain two sequences y1 and y2. The core module Multi-head Fusion Attention (MHFA) is used to model the long-range dependency between hyperspectral images and multispectral images. MHFA obtains the attention map by obtaining the correlation between the query vector Query and the key vector Key, and then obtains the eigenvalues ​​in the value vector Value based on another attention map. The feature maps output by MHFA are output into two sequences after layer normalization and feed-forward network. The sequences output by the feature mapping module are converted into standard feature maps. A deformable convolution module is introduced after cross attention to adaptively determine the scale and adjust the receptive field size according to objects of different scales corresponding to different positions, so as to further refine the features of the fused image.

4. The hyperspectral and multispectral image fusion method based on improved Transformer according to claim 3 is characterized in that: In step S03, feature aggregation and reconstruction are performed through a double-layer asymmetric convolution module, and the input features and output features of the multi-layer deformable convolution cross Transformer are jump-connected. The potential features of the image are utilized through a double-layer asymmetric convolution module and a cascade upsampling operation to restore the spatial information.

5. The hyperspectral and multispectral image fusion method based on improved Transformer according to claim 4 is characterized in that: The double-layer asymmetric convolution module includes strip convolution. The input feature map is input into four convolution paths of different shapes after 1×1 convolution. The long-distance context information is captured from four directions: horizontal, vertical, left diagonal and right diagonal, to extract texture details, and four feature maps are obtained. The four feature maps are further subjected to the second layer of strip convolution to further extract edge features. The extracted features are added in the same direction and spliced ​​in the channel dimension. Finally, the dimension is reduced by upsampling and convolution.

6. The hyperspectral and multispectral image fusion method based on improved Transformer according to claim 1, characterized in that: Step S03 also includes: optimizing the reconstructed image using a loss function, where the loss function is: Among them, L fus is the loss function of the fused image, Z is the reference image, is the reconstructed image, where H, W, and C are the height, width, and number of channels.

7. A hyperspectral and multispectral image fusion system based on improved Transformer, characterized in that: include: The upsampling recursive sub-network module upsamples the hyperspectral image and extracts the potential features of the hyperspectral image; The multi-layer cross-modal interaction module uses a dual-branch structure to extract multi-scale features of hyperspectral images and multispectral images respectively, and extracts global features from feature maps through a multi-layer deformable convolution cross Transformer. The feature aggregation reconstruction module adds the extracted features in the same direction, concatenates them in the channel dimension, and gradually reconstructs the multispectral image through cascade upsampling.

8. The hyperspectral and multispectral image fusion system based on improved Transformer according to claim 7, characterized in that: The upsampling recursive subnetwork module includes a feature initialization layer, three recursive blocks and a deconvolution layer. The recursive block consists of two groups of 3×3 convolution and ReLU activation layers. The input and output are residually connected to fuse several consecutive residual blocks with deep features.

9. The hyperspectral and multispectral image fusion system based on improved Transformer according to claim 7, characterized in that: The multi-layer cross-modal interaction module linearly projects the input hyperspectral image feature map F1 and multispectral image feature map F2 to obtain the feature map after dimensionality reduction. The learnable position embedding is used to encode the position information of the feature maps F1 and F2, which are then fused with the feature maps after dimensionality reduction. The features are then layer-normalized to obtain two sequences y1 and y2. The core module Multi-head Fusion Attention (MHFA) is used to model the long-range dependency between hyperspectral images and multispectral images. MHFA obtains the attention map by obtaining the correlation between the query vector Query and the key vector Key, and then obtains the eigenvalues ​​in the value vector Value based on another attention map. The feature maps output by MHFA are normalized and fed-forwarded into two sequences. The feature mapping module is used to convert the output sequence into a standard feature map. After the cross attention, a deformable convolution module is introduced to adaptively determine the scale and adjust the receptive field size according to the objects of different scales corresponding to different positions, so as to further refine the features of the fused image.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the hyperspectral and multispectral image fusion method based on the improved Transformer according to any one of claims 1 to 6 is implemented.

Images