A remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer

Abstract

The present invention discloses a remote sensing image spatial-spectral fusion method based on a denoising diffusion probability model and a Transformer. The method comprises designing a three-branch encoder-decoder network structure for spatial-spectral fusion, and respectively utilizing a denoising diffusion probability model (DDPM) and a Transformer network to extract spatial feature information of a panchromatic image and spectral feature information of a multispectral image, and integrating the extracted spatial information and spectral information into the encoder branch layer by layer to realize the interaction and effective fusion of the spatial information of the panchromatic remote sensing image and the spectral information of the multispectral remote sensing image, and finally obtaining a fused image with rich spatial information and spectral information. A large number of experimental results on public data sets show that, compared with other representative comparison methods, the method proposed in the present invention achieves an overall superior fusion effect in both subjective visual effect comparison and objective index evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image space-spectrum fusion method based on a denoising diffusion probability model and a Transformer. Background Art

[0002] The remote sensing image fusion task aims to fuse the features and information of remote sensing images collected by different sensor devices, so as to obtain a fused image with the advantages of different modal remote sensing images. Panchromatic (PAN) and multispectral (MS) remote sensing images are two common remote sensing images in heterogeneous remote sensing images. PAN has high spatial resolution but lacks spectral information; on the contrary, MS has rich spectral information but low spatial resolution. By using image fusion technology to fuse panchromatic and multispectral remote sensing images, a fused image with both high spatial resolution and high spectral resolution can be obtained, which is beneficial to downstream visual tasks such as semantic segmentation of images.

[0003] In recent years, deep learning network models have shown excellent performance in the field of remote sensing image fusion due to their powerful feature extraction and feature integration capabilities. At present, the deep learning network models favored by researchers in the field of remote sensing image fusion mainly include Convolutional Neural Network (CNN), Generative Adversarial Network (GAN), Transformer network and Denoising Diffusion Probabilistic Model (DDPM). Remote sensing image fusion methods based on CNN usually show strong local information extraction capabilities; remote sensing image fusion methods based on GAN often have good feature information integration capabilities; remote sensing image fusion methods based on Transformer network have strong global information extraction capabilities. In view of the unique advantages of different deep learning network models, in recent years, researchers tend to combine different deep learning network models to achieve complementary advantages between different deep learning network models. By combining CNN with GAN, researchers have proposed many remote sensing image fusion methods with good feature extraction and feature integration capabilities. As one of the first attempts in the field of remote sensing image fusion, the deep learning model PSGAN proposed by Liu et al., which combines the advantages of CNN and GAN, has achieved good fusion effects; Zhao et al. proposed the FGF-GAN model based on fast guided filtering, which has strong spatial information retention ability; the MDSSC-GAN model proposed by Gastineau et al., after decomposing the source image in color space, adopted different fusion strategies for different channel components and achieved good fusion effects. Although the deep learning model combining CNN and GAN has achieved good results in remote sensing image fusion tasks, the hybrid model combining CNN and GAN often has problems such as unstable training and gradient disappearance. The hybrid model combining CNN and Transformer network is also one of the hot directions in the field of remote sensing image fusion in recent years. By effectively combining the powerful local information extraction ability of CNN with the powerful global information extraction ability of Transformer network, a deep learning network model with good local and global information extraction and integration capabilities can be realized. The TRRNet model proposed by Zhang et al. extracts local information of the source image through convolution blocks and global information of the source image through Swin Transformer blocks, achieving good fusion results. The DBCT-Net model proposed by Wang et al. extracts and integrates local information and global information respectively through a dual-branch CNN and Transformer hybrid model.The network model that combines CNN and Transformer has achieved good fusion effect in remote sensing image fusion tasks, but the hybrid deep learning model that combines CNN and Transformer networks often has shortcomings such as too many training parameters and long training time.

[0004] At present, the denoising diffusion probability model has attracted more and more attention from researchers in the field of remote sensing image fusion due to its advantages such as stable training process, high image generation quality and strong interpretability, and has shown excellent performance and great potential. As one of the first attempts to apply DDPM to the field of remote sensing image fusion, the PanDiff model proposed by Meng et al. has achieved good fusion effect by extracting and integrating features of source images using DDMP; the self-supervised CrossDiff model proposed by Xing et al. extracts spatial and spectral features from source images in stages through a cross-prediction diffusion model, showing good fusion effect. In the existing remote sensing image fusion methods based on deep learning, how to further combine multiple deep learning network models with the denoising diffusion probability model to achieve a hybrid model with the complementary advantages of multiple networks is still a problem to be studied. Summary of the invention

[0005] The present invention proposes a method for spatial-spectral fusion of multispectral and panchromatic remote sensing images based on DDPM and Transformer. The method includes designing a three-branch encoder-decoder network structure for spatial-spectral fusion, and respectively using DDPM and Transformer networks to extract features of panchromatic remote sensing images and multispectral remote sensing images, and integrating the extracted spatial information of the panchromatic image and the spectral information of the multispectral image into the encoder branch layer by layer to achieve deep interaction and effective fusion of spatial information and spectral information, and finally obtaining a fused image with rich spatial information and spectral information.

[0006] To achieve the above object, the present invention adopts the following technical solution: a remote sensing image spatial-spectral fusion method based on a denoising diffusion probability model and a Transformer, characterized in that the method comprises the following steps:

[0007] S1. Input the panchromatic image PAN into DDPM for feature extraction, input the multispectral image MS into the Transformer network for feature extraction, and input PAN and MS into the encoder after concatenating them in the channel dimension for feature extraction;

[0008] S2. The spatial information features of the panchromatic image and the spectral information features of the multispectral image extracted by the DDPM and Transformer network are integrated layer by layer into the encoder network for feature integration;

[0009] S3. The integrated features are input into the decoder, and the features integrated and integrated layer by layer in the encoder are further integrated layer by layer into the decoder for feature fusion and reconstruction;

[0010] S4. Input the reconstructed features into the fusion module to obtain a fused image.

[0011] As a preferred technical solution of the present invention: the network structures of DDPM, Transformer and encoder in S1 are as follows:

[0012] S1-1: The main structure of the DDPM branch network is a "U"-shaped network. The convolution layer consists of two layers of convolution. The downsampling layer and the upsampling layer each consist of a convolution layer for downsampling or upsampling and an attention layer.

[0013] S1-2: The Transformer branch network structure consists of a spectral attention layer and five Transformer layers; the network structure of the spectral attention layer mainly consists of an average pooling module, a maximum pooling module and a multi-layer perceptron;

[0014] S1-3: The encoder network consists of a convolutional module and five serially connected encoder modules, where the convolutional module consists of a convolutional layer, a batch normalization layer, and an activation function layer.

[0015] As a preferred technical solution of the present invention: before downsampling, the feature map of each layer in the DDPM branch network structure in S1-1 is connected to the feature map of the previous layer after jump connection and the feature map of the corresponding layer. Figure 1 The mixed feature maps are stacked in the channel dimension and input into the upsampling convolution layer of the corresponding level. During the upsampling process, the feature map output of each layer of DDPM is integrated into the encoder branch to enhance the spatial feature information of the encoder branch.

[0016] As a preferred technical solution of the present invention: the operation process of the Transformer branch network structure in S1-2 is:

[0017] Step 1: After the multispectral image is input into the spectral attention layer, it is processed by the average pooling module and the maximum pooling module respectively, and the results after the two modules are connected in the channel dimension;

[0018] Step 2: Input the result of channel dimension concatenation into the multi-layer perceptron, and perform a dot multiplication operation between the output of the multi-layer perceptron and the source multispectral image;

[0019] Step 3: Add the pixel matrix value of the result after the dot multiplication operation to the source multispectral image to obtain the output result of the spectral attention layer;

[0020] Step 4: Input the output of the spectral attention layer into five serially connected Transformer modules.

[0021] As a preferred technical solution of the present invention: in the Transformer module in step 4, the input feature map first passes through a convolution module composed of a convolution layer, a batch normalization layer and an activation function layer to extract features and keep the size of the feature map consistent with the size required for subsequent processing; then the feature map is input into the Transformer network, wherein each layer of Transformer includes layer normalization, window-based multi-head self-attention or moving window-based multi-head self-attention, and multi-layer perception; next, the feature map output by each Transformer layer is retained separately and integrated into the encoder branch layer by layer to enhance the spectral feature information of the encoder branch.

[0022] As a preferred technical solution of the present invention: in the encoder network in S1-3, the network parameters of the convolution layer in the convolution module are: the convolution kernel size is 3, the step size is 1, and the number of convolution kernels is 64. The feature map output by the convolution module is connected with the enhanced spatial feature map output by the DDPM branch and the enhanced spectral feature map output by the Transformer branch, and then input into the encoder module.

[0023] As a preferred technical solution of the present invention: the decoder network in S3 is composed of five convolution modules and one fusion module. Except for the first convolution module, the inputs of the subsequent four convolution modules are all the output feature maps of the decoder convolution module of the previous layer, and the feature maps are connected with the enhanced feature maps output by the encoder module of the corresponding layer to realize the interaction of shallow and deep feature information. The feature map output by the last decoder convolution module will be input into the fusion module, and the output result of the fusion module is the final fused image.

[0024] As a preferred technical solution of the present invention: the fusion module in S4 is composed of four convolution blocks, wherein the first three convolution blocks each include a convolution layer, a batch normalization layer, and an activation function; and the last convolution block is composed of a convolution layer and an activation function layer.

[0025] As a preferred technical solution of the present invention: the encoder-decoder network model in S2 to S3 is optimized by a loss function, wherein the loss function consists of three parts: intensity loss, gradient loss and spectral loss.

[0026] Furthermore, the specific form of the loss function is shown in the following formula (1):

[0027] L=aL int +bL grad +cL spectrum (5)

[0028] Where L is the total loss function, L int is the strength loss, L grad is the gradient loss, L spectrum is the spectral loss; a, b, c are the coefficients of balanced intensity loss, gradient loss, and spectral loss, respectively;

[0029] The specific definition of strength loss is shown in formula (2):

[0030]

[0031] Where H and W are the height and width of the image, respectively, i and j represent the pixel coordinates of the image; F represents the fused image, I MS represents the source multispectral image; |·| represents the absolute value;

[0032] The gradient loss is defined as shown in formula (3):

[0033]

[0034] Among them, unlike the strength loss, I F-AP represents the fused image after average pooling, I PAN represents the source full-color image; represents the gradient operator;

[0035] The spectral loss is shown in formula (4):

[0036]

[0037] Where M1 represents a matrix of all 1s, <·> represents an inner product operation, and ‖·‖2 represents the 2-norm;

[0038] In short, the loss function is used to constrain the spectral consistency of the fused image by calculating the intensity loss and spectral loss of the fused image and the source multispectral image; at the same time, the loss function is used to constrain the spatial consistency of the fused image by calculating the gradient loss of the fused image and the source panchromatic image.

[0039] The following beneficial effects can be obtained through the above technical solution:

[0040] The main body of the proposed method is a three-branch encoder-decoder network structure. On this basis, the spatial information of the panchromatic image and the spectral information of the multispectral image are extracted through the DDPM branch and the Transformer branch respectively, and the extracted spatial information and spectral information are integrated layer by layer into the intermediate branch, namely the encoder branch, to enhance the spatial feature information and spectral feature information of the encoder branch, thereby realizing the interaction of spatial information and spectral information, and the interaction of deep information and shallow information. The experimental results show that compared with the comparison method, the texture information of the fusion result obtained by the algorithm of the present invention is richer and the subjective visual effect is better; in addition, the results of objective evaluation show that the algorithm of the present invention is superior to the average value of the comparison method in five indicators, namely information entropy, root mean square error, general image quality index, spectral distortion index and peak signal-to-noise ratio, by about 1.28%, 5.18%, 48.58%, 12.61% and 22.90%, respectively, which further illustrates that the proposed method can better fuse the spatial texture information of the panchromatic image while effectively retaining the spectral information of the multispectral image, thereby improving the performance of the existing multispectral and panchromatic remote sensing image fusion algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall fusion framework diagram of the method proposed in the present invention.

[0042] Figure 2 This is the DDPM branch network structure diagram.

[0043] Figure 3 This is the Transformer branch network structure diagram.

[0044] Figure 4 This is the network structure diagram of the encoder module.

[0045] Figure 5 It is a fusion module network structure. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0047] like Figure 1-5 As shown, the present invention provides a technical solution: a remote sensing image spatial-spectral fusion method based on a denoising diffusion probability model and a Transformer, which is specifically performed according to S1 to S4 as follows to obtain a fused image;

[0048] S1. Input the panchromatic image PAN into DDPM for feature extraction, input the multispectral image MS into the Transformer network for feature extraction, and input PAN and MS into the encoder after concatenating them in the channel dimension for feature extraction;

[0049] S2. The spatial information features of the panchromatic image and the spectral information features of the multispectral image extracted by the DDPM and Transformer network are integrated layer by layer into the encoder network for feature integration;

[0050] S3. The integrated features are input into the decoder, and the features integrated and integrated layer by layer in the encoder are further integrated layer by layer into the decoder for feature fusion and reconstruction;

[0051] S4. Input the reconstructed features into the fusion module to obtain a fused image.

[0052] The overall framework of the proposed method is as follows Figure 1 As shown. Specifically, the convolutional neural network is used to extract features (encoder part) and fusion reconstruction (decoder part) of the joint image of the panchromatic image and the multispectral image; at the same time, in view of the different characteristics of the panchromatic image and the multispectral image, which contain clear spatial texture information and rich spectral information respectively, the DDPM and Transformer network are used to further extract the spatial information of the panchromatic image and the spectral information of the multispectral image, and the extracted spatial and spectral information are integrated into the network (encoder and decoder) layer by layer. Figure 1 As shown in the figure, in order to realize the interaction between deep information and shallow information in the fusion process, the features integrated by the encoder are further integrated into the decoder network layer by layer through skip connections, thereby enhancing the spatial texture information and spectral information of the fused image.

[0053] In the present invention, a three-branch encoder-decoder (Encoder-Decoder) network model is designed to realize the spatial-spectral fusion of remote sensing images in view of the different characteristics of panchromatic remote sensing images and multispectral remote sensing images and the characteristics of various neural networks (CNN, Transformer and DDPM); the three-branch encoder-decoder network structure inputs the panchromatic image and the multispectral image into a spatial branch based on DDPM and a spectral branch based on Transformer for feature extraction, and performs layer-by-layer interaction and fusion of the three branch information in a fusion branch based on CNN; and the proposed network model is optimized through a mixed loss function.

[0054] DDPM has demonstrated powerful feature extraction capabilities in many tasks in the field of computer vision; the Transformer network has excellent global feature information expression capabilities. The method proposed in the present invention uses DDPM to further extract the spatial information of full-color images, uses Transformer to further extract the spectral information of multispectral images, and further integrates the extracted spatial information and spectral information into the encoder-decoder network layer by layer. Next, the DDPM branch network structure, Transformer branch network structure, encoder-decoder network structure, and fusion module network structure in S1 to S4 will be introduced in detail.

[0055] S1-1: DDPM branch network structure

[0056] Figure 2 (a) shows the specific network structure of DDPM. Its main structure is a "U"-shaped network (U-Net), in which the convolutional layer consists of two layers of convolution; the downsampling layer and the upsampling layer are both composed of a convolution layer for downsampling or upsampling and an attention layer. Figure 2 (b) It should be noted that before downsampling, the feature map of each layer is connected to the feature map of the previous layer after skip connection and the feature map of the corresponding layer. Figure 1 The mixed feature maps are stacked in the channel dimension and input into the upsampling convolution layer of the corresponding level. In addition, during the upsampling process, the feature map output (spatial feature map) of each layer of DDPM is integrated into the encoder branch to enhance the spatial feature information of the encoder branch.

[0057] S1-2: Transformer branch network structure

[0058] like Figure 1 As shown in FIG. 1 , in the method proposed in the present invention, the Transformer branch consists of a spectral attention layer and five Transformer layers. The network structure of the spectral attention layer is mainly composed of an average pooling module (Avg Pool), a maximum pooling module (Max Pool) and a multi-layer perceptron (MLP), as shown in FIG. Figure 3(a) As shown. Specifically, after the multispectral image is input into the spectral attention layer, it is first processed by the average pooling module and the maximum pooling module respectively, and the results after the processing of the two modules are connected in the channel dimension and then input into the multilayer perceptron; then, the output of the multilayer perceptron is multiplied by the source multispectral image; then, the pixel matrix of the result after the dot multiplication operation is added to the source multispectral image to obtain the output result of the spectral attention layer; finally, the output result of the spectral attention layer is input into five series-connected Transformer modules, where the network structure of each Transformer module is as follows: Figure 3 (b), its specific network parameters are shown in Table 1:

[0059] Table 1 Network parameters of the five Transformer layers

[0060]

[0061] like Figure 3 As shown in (b), in the network structure of the Transformer layer, the input feature map first passes through a convolution module consisting of a convolution layer (CONV), a batch normalization layer (BN) and an activation function layer (LRELU) to extract features and keep the size of the feature map consistent with the size required for subsequent processing; then the feature map is input into the Transformer network, where each layer of Transformer contains layer normalization (Layer Normalization, LN), window-based multi-head self-attention (Window-based Multi-head Self-Attention, WMSA) or shifted window-based multi-head self-attention (Shifted Window-based Multi-head Self-attention, SWMSA), and multi-layer perception (Multi-layerPerceptron, MLP); next, the feature map output by each Transformer layer is retained separately and integrated into the encoder branch layer by layer to enhance the spectral feature information of the encoder branch.

[0062] S1-3: Encoder network structure

[0063] like Figure 1As shown in the figure, the source panchromatic image and the source multispectral image are concatenated in the channel dimension and then input into the encoder network. The encoder network consists of a convolutional module and five serially connected encoder modules (Encoder Block), where the convolutional module consists of a convolutional layer, a batch normalization layer, and an activation function layer. The network parameters of the convolutional layer in the convolutional module are: convolution kernel size is 3, step size is 1, and number of convolution kernels is 64. The feature map output by the convolutional module is concatenated with the enhanced spatial feature map output by the DDPM branch and the enhanced spectral feature map output by the Transformer branch, and then input into the encoder module. The network structure of the encoder module is shown in Figure 4 shown.

[0064] Each encoder module consists of two convolutional layers to avoid the problem of information loss due to large convolution span. The feature maps output by all encoder modules are concatenated with the enhanced spatial feature maps output by the DDPM branch and the enhanced spectral feature maps output by the Transformer branch and then input into the next encoder module. The network parameter settings of the five encoder modules are shown in Table 2:

[0065] Table 2 Network parameters of five encoder modules

[0066]

[0067] S3: Decoder network structure

[0068] like Figure 1 As shown in , the feature maps output by each encoder module are output through a convolution block and integrated into the decoder network as enhanced features. The feature map output by the last encoding module of the encoder network is directly input into the first convolution module of the decoder network. The decoder network consists of five convolution modules and a fusion module, as shown in Figure 5 The network parameter settings of the five convolution modules are shown in Table 3:

[0069] Table 3 Network parameters of the five decoder convolutional modules

[0070]

[0071] In the decoder network, except for the first convolution module, the inputs of the following four convolution modules are all the output feature maps of the decoder convolution module of the previous layer, which are connected with the enhanced feature maps output by the encoder module of the corresponding layer to realize the interaction of shallow and deep feature information. The feature map output by the last decoder convolution module will be input into the fusion module.

[0072] S4: Fusion module network structure

[0073] like Figure 5 As shown in the figure, the fusion module consists of four convolution blocks, of which the first three convolution blocks include convolution layers, batch normalization layers, and activation functions (LRELU); the last convolution block consists of a convolution layer and an activation function layer (TANH). In addition, the convolution kernel size of the four convolution layers is 3, the step size is 1, and the number of convolution kernels is 64, 32, 16, and 4 respectively. The output result of the fusion module is the final fused image.

[0074] In the method proposed by the present invention, the network model is further optimized by a mixed loss function:

[0075] The loss function of the algorithm proposed in the present invention consists of three parts: intensity loss, gradient loss and spectral loss. The specific form is shown in the following formula (1):

[0076] L=aL int +bL grad +cL spectrum (9)

[0077] Where L is the total loss function, L int is the strength loss, L grad is the gradient loss, L spectrum is the spectral loss; a, b, c are the coefficients of balanced intensity loss, gradient loss, and spectral loss, respectively;

[0078] The specific definition of strength loss is shown in formula (2):

[0079]

[0080] Where H and W are the height and width of the image, respectively, i and j represent the pixel coordinates of the image; F represents the fused image, I MS represents the source multispectral image; |·| represents the absolute value;

[0081] The gradient loss is defined as shown in formula (3):

[0082]

[0083] Among them, unlike the strength loss, I F-AP represents the fused image after average pooling, I PAN represents the source full-color image; represents the gradient operator;

[0084] The spectral loss is shown in formula (4):

[0085]

[0086] Where M1 represents a matrix of all 1s, <·> represents an inner product operation, and ‖·‖2 represents the 2-norm;

[0087] In short, the constraint of the loss function on the spectral consistency of the fused image is realized by calculating the intensity loss and spectral loss of the fused image and the source multispectral image; at the same time, the constraint of the loss function on the spatial consistency of the fused image is realized by calculating the gradient loss of the fused image and the source panchromatic image.

[0088] The above are all preferred embodiments of the present invention. For ordinary technicians in this technical field, without departing from the principle of the present invention, various equivalent modifications to the present invention belong to the protection scope of the claims attached to this application.

Claims

1. A remote sensing image spatial-spectral fusion method based on a denoising diffusion probability model and Transformer, characterized by: The method comprises the following steps: S1. Input the panchromatic image PAN into DDPM for feature extraction, input the multispectral image MS into the Transformer network for feature extraction, and input PAN and MS into the encoder after concatenating them in the channel dimension for feature extraction; S2. The spatial information features of the panchromatic image and the spectral information features of the multispectral image extracted by the DDPM and Transformer network are integrated layer by layer into the encoder network for feature integration; S3. The integrated features are input into the decoder, and the features integrated and integrated layer by layer in the encoder are further integrated layer by layer into the decoder for feature fusion and reconstruction; S4. inputting the reconstructed features into a fusion module to obtain a fused image; The network structures of DDPM, Transformer, and encoder in S1 are as follows: S1-1: The main structure of the DDPM branch network is a "U"-shaped network. The convolution layer consists of two layers of convolution. The downsampling layer and the upsampling layer each consist of a convolution layer for downsampling or upsampling and an attention layer. S1-2: The Transformer branch network structure consists of a spectral attention layer and five Transformer layers; the network structure of the spectral attention layer consists of an average pooling module, a maximum pooling module and a multi-layer perceptron; S1-3: The encoder network consists of a convolutional module and five serially connected encoder modules, where the convolutional module consists of a convolutional layer, a batch normalization layer, and an activation function layer.

2. According to claim 1, a remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer is characterized by: The feature map of each layer in the DDPM branch network structure in S1-1 is stacked in the channel dimension together with the feature map of the previous layer after jump connection and the feature map of the corresponding layer before downsampling, and the stacked mixed feature map is input into the upsampling convolution layer of the corresponding layer; During the upsampling process, the feature map output of each layer of DDPM is integrated into the encoder branch to enhance the spatial feature information of the encoder branch.

3. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 1, characterized in that: The operation process of the Transformer branch network structure in S1-2 is as follows: Step 1: After the multispectral image is input into the spectral attention layer, it is processed by the average pooling module and the maximum pooling module respectively, and the results after the two modules are connected in the channel dimension; Step 2: Input the result of channel dimension concatenation into the multi-layer perceptron, and perform a dot multiplication operation between the output of the multi-layer perceptron and the source multispectral image; Step 3: Add the pixel matrix value of the result after the dot multiplication operation to the source multispectral image to obtain the output result of the spectral attention layer; Step 4: Input the output of the spectral attention layer into five serially connected Transformer modules.

4. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 3 is characterized by: In the Transformer module in step 4, the input feature map first passes through a convolution module consisting of a convolution layer, a batch normalization layer, and an activation function layer to extract features and keep the size of the feature map consistent with the size required for subsequent processing; then the feature map is input into the Transformer network, wherein each layer of Transformer includes layer normalization, window-based multi-head self-attention or moving window-based multi-head self-attention, and multi-layer perception; next, the feature map output by each Transformer layer is retained separately and integrated into the encoder branch layer by layer to enhance the spectral feature information of the encoder branch.

5. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 1, characterized in that: In the encoder network in S1-3, the network parameters of the convolution layer in the convolution module are: the convolution kernel size is 3, the step size is 1, and the number of convolution kernels is 64. The feature map output by the convolution module is connected with the enhanced spatial feature map output by the DDPM branch and the enhanced spectral feature map output by the Transformer branch, and then input into the encoder module.

6. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 1, characterized in that: The decoder network in S3 is composed of five convolution modules and one fusion module. Except for the first convolution module, the inputs of the subsequent four convolution modules are all the output feature maps of the decoder convolution module of the previous layer, which are connected with the enhanced feature maps output by the encoder module of the corresponding layer to realize the interaction of shallow and deep feature information. The feature map output by the last decoder convolution module will be input into the fusion module, and the output result of the fusion module is the final fused image.

7. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 1, characterized in that: The fusion module in S4 is composed of four convolution blocks, wherein the first three convolution blocks each include a convolution layer, a batch normalization layer, and an activation function; and the last convolution block is composed of a convolution layer and an activation function layer.

8. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 1, characterized in that: The encoder-decoder network model is optimized through the loss function, where the loss function consists of three parts: intensity loss, gradient loss, and spectral loss.

9. The remote sensing image spatial-spectral fusion method based on denoising diffusion probability model and Transformer according to claim 8, characterized in that: The specific form of the loss function is shown in the following formula (1): <h2 style=";text-align:left;direction:ltr">L=aL<h2 style=";text-align:left;direction:ltr"> int <h2 style=";text-align:left;direction:ltr"> +bL<h2 style=";text-align:left;direction:ltr"> grad <h2 style=";text-align:left;direction:ltr"> +cL<h2 style=";text-align:left;direction:ltr"> spectrum <h2 style=";text-align:left;direction:ltr"> (1) Where L is the total loss function, L int is the strength loss, L grad is the gradient loss, L spectrum is the spectral loss; a, b, c are the coefficients of balanced intensity loss, gradient loss, and spectral loss, respectively; The specific definition of strength loss is shown in formula (2): Where H and W are the height and width of the image, respectively, i and j represent the pixel coordinates of the image; F represents the fused image, I MS represents the source multispectral image; |·| represents the absolute value; The gradient loss is defined as shown in formula (3): Among them, unlike the strength loss, I F-AP represents the fused image after average pooling, I PAN represents the source full-color image; represents the gradient operator; The spectral loss is shown in formula (4): Where M1 represents a matrix of all 1s, <·> represents an inner product operation, and ||·||2 represents the 2-norm; In short, the constraint of the loss function on the spectral consistency of the fused image is realized by calculating the intensity loss and spectral loss of the fused image and the source multispectral image; at the same time, the constraint of the loss function on the spatial consistency of the fused image is realized by calculating the gradient loss of the fused image and the source panchromatic image.

Images