Panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer

Abstract

The present invention proposes a panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer to solve the problems of loss of detail information such as texture and spectral distortion. This method adopts a specific generative adversarial network structure and designs parallel base layer and detail layer branches to effectively extract the texture and spectral features of the image. The source image is decomposed by guided filters. By using convolutional neural networks and Transformer networks, this method can effectively extract the local texture and global spectral features of the image respectively. The discriminator part contains two specially designed sub-networks to discriminate the base layer and detail layer respectively. This method meets the high-quality requirements of remote sensing image fusion to ensure that in the fusion process, the texture information of the panchromatic image with high spatial resolution is retained, and the spectral information of the multispectral image with high spectral resolution is transmitted as much as possible, providing a powerful tool for remote sensing image analysis.

Description

Technical field:

[0001] The present invention belongs to remote sensing image fusion, and in particular to a panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer. Background technology:

[0002] With the development of remote sensing technology, a large number of remote sensing images collected by satellites and other long-distance sensors have important application value in geological exploration, environmental monitoring, urban planning, agricultural management, disaster assessment and other fields because of their advantages such as high efficiency and wide coverage. However, due to the limitations of acquisition equipment, it is often difficult for the same sensor to collect remote sensing images with the same resolution for different bands. For example, when obtaining remote sensing images with higher spectral resolution, the sensor needs to collect band information with a narrower width, but this will result in a reduction in the amount of radiation it receives, which limits the spatial resolution of the collected image. Therefore, in practical applications, the same sensor needs to make a trade-off between the spectral resolution and spatial resolution of the collected image, and collect images with different information according to the needs of different tasks, such as multi-spectral images with rich spectral information and panchromatic images with clear spatial texture information. Remote sensing image fusion can extract the required information from the input images with complementary information about the same scene, and retain it in the generated fused image as much as possible. For example, by fusing a multi-spectral image with a panchromatic image, a fused image with both rich spectral information and clear spatial texture information can be finally obtained.

[0003] The Chinese patent closest to the present invention is CN201910681109.9. This solution discloses a multispectral remote sensing image fusion method and device based on residual learning. First, the interpolation image of the multispectral image, the gradient image of the panchromatic image and the differential image are calculated, and then a convolutional neural network model is used for training. However, this method mainly relies on residual learning to achieve fusion, and it does not take into account more advanced feature extraction technology, multi-scale fusion or Transformer-based global information extraction technology.

[0004] The Chinese patent is CN202110431506.8, which proposes a remote sensing image fusion method based on adaptive multi-scale residual convolution. This method adaptively matches multispectral images and panchromatic images and uses a multi-scale deep residual convolutional neural network for image fusion. However, although this method can more fully extract the spatial features of the image, it does not take into account image decomposition strategies such as guided filter-based, dual-branch feature extraction, or advanced discriminator structures.

[0005] Although existing multispectral and panchromatic image fusion technologies have achieved good results in some aspects, they still have some obvious limitations and problems. First, traditional fusion methods such as IHS transform and PCA transform may cause color distortion or loss of spatial details. Second, although deep learning methods provide more accurate fusion effects, they often require a large amount of training data and have high model complexity, resulting in low computational efficiency. In addition, existing technologies have not fully considered the difference between the spectral resolution of multispectral images and the spatial resolution of panchromatic images, which may affect the quality of the fused image.

[0006] In view of the above problems, the technical solution of this application aims to solve the following main technical problems: how to effectively extract the spectral information of multispectral images and the spatial information of panchromatic images; how to design a fusion network that can capture local texture details and retain spectral features; how to ensure that the fusion image is optimized in both spectral and spatial quality while maintaining computational efficiency. To this end, the present invention proposes a novel fusion method that combines guided filters, convolutional neural networks and Transformer networks to meet the above technical requirements and provide an efficient and high-quality solution for remote sensing image fusion. Summary of the invention:

[0007] The purpose of the present invention is to provide a panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer to solve the problems raised by the above background technology in view of the defects of the prior art.

[0008] To achieve the above purpose, the present invention provides the following technical solution: a panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer. The algorithm proposed in the present invention can not only fully retain the rich spectral information in the multispectral image, but also effectively reproduce the spatial texture and other detail information in the panchromatic image; the specific steps are:

[0009] S1: Use guided filters to decompose the source image to obtain a multispectral base layer, a multispectral detail layer, a panchromatic base layer, and a panchromatic detail layer, and use different decomposition strategies according to the characteristics of the panchromatic image and the multispectral image;

[0010] S2: stack the multispectral base layer and the panchromatic base layer in the channel dimension to obtain an input base layer image; stack the multispectral detail layer and the panchromatic detail layer in the channel dimension to obtain an input detail layer image;

[0011] S3: For the detail layer image containing the main texture information, a convolutional neural network is used to extract the features of the texture information;

[0012] S4: For the base layer image containing some texture detail information and more spectral information, the Transformer network is used to extract features, and the features extracted in S3 are fused with the features extracted in this step. Then, the image is reconstructed based on the fused features, so as to obtain a fused image with rich texture information and spectral information as the output result of the generator;

[0013] S5: Perform guided filtering decomposition on the fusion image output by the generator to obtain a base layer image and a detail layer image of the fusion image, and input the base layer image and the source image base layer calculated by the average value, and the detail layer image and the source image detail layer calculated by the maximum value into the base layer discriminator and the detail layer discriminator respectively, and discriminate the input fusion image and the source image from the two levels of base layer information and detail layer information;

[0014] S6: The generator is continuously confronted and optimized with the base layer discriminator and the detail layer discriminator until the base layer discriminator and the detail layer discriminator are unable to distinguish the fused image generated by the generator, thereby obtaining the final fusion result.

[0015] As a technical preferred solution of the present invention, different decomposition strategies are adopted in S1: the spatial resolution of the full-color image is relatively high, and it itself is used as the guiding image of the guiding filter; and since the multispectral image has high spectral resolution and rich spectral information but fuzzy texture features, the multispectral image is subjected to principal component analysis, and its principal component image is used as the guiding image.

[0016] As a technical preferred solution of the present invention, the network structure of the generator in S4 is composed of the following parts:

[0017] S4-1: Base layer branch: used to extract spectral information features from the base layer image;

[0018] S4-2: Detail layer branch: used to extract detail features such as texture from the detail layer image.

[0019] As a technical preferred solution of the present invention, the training of the generator in S6 uses a loss function composed of adversarial loss, content loss and spectral loss;

[0020] S6-1-1: Adversarial loss is defined as the adversarial relationship between the generator and the base layer discriminator and the detail layer discriminator;

[0021] S6-1-2: Content loss includes intensity loss, gradient loss and structural similarity loss, where intensity loss involves the comparison between the intensity of the fused image and the intensity of the source image, gradient loss involves the comparison between the gradient of the fused image and the gradient of the source image, and structural similarity loss involves the structural similarity between the fused image and the source image;

[0022] S6-1-3: Spectral loss involves the spectral similarity between the multispectral image and the fused image;

[0023] S6-1-4: The calculation formula of the loss function is: L G =L adv +λ1L content +λ2L spectrum

[0024] Among them, L G represents the total loss of the generator G, L adv Represents the generator G and the base layer discriminator D base , detail layer discriminator D detail The adversarial loss between them, L content represents the content loss of the fused image, L spectrum represents spectral loss, λ1 and λ2 are weight coefficients for adjusting the proportion of content loss and spectral loss respectively.

[0025] As a technical preferred solution of the present invention, the training of the discriminator in S6 uses a loss function, which consists of two parts: the base layer discriminator loss and the detail layer discriminator loss, and its calculation formula is:

[0026] S6-2-1: Loss of base layer discriminator for:

[0027]

[0028] Among them, D base represents the base layer discriminator, D base (·) represents the judgment value of the base layer discriminator on the authenticity of the input image; I base-mean It represents the image obtained by averaging the multispectral image base layer and the panchromatic image base layer. is the base layer of the fused image; E(·) represents the mathematical expectation;

[0029] S6-2-2: Detail layer discriminator loss for:

[0030]

[0031] Among them, D detail Denotes the detail layer discriminator, D detail (·) represents the judgment value of the detail layer discriminator on the authenticity of the input image; I detail-max It represents the image obtained by performing maximum processing on the detail layer of the multispectral image and the detail layer of the panchromatic image. is the detail layer of the fused image; E(·) represents the mathematical expectation value.

[0032] As a technical preferred solution of the present invention, the base layer branch in S4-1 is composed of two Conv Blocks and three Transformer Blocks;

[0033] S4-1-1: The first Conv Block is used to preliminarily extract shallow information. Its input is a base layer image including 4 multispectral channels and 1 panchromatic channel. The convolution kernel size is set to 3×3, the step size is 1, and the number of convolution kernels is 96;

[0034] S4-1-2: The obtained feature map is input into three Transformer Blocks with the same structure, and residual connections are used between multiple Transformer Blocks;

[0035] S4-1-3: Each Transformer Block consists of two identical Swin Transformer Layers, where the Swin Transformer Layer contains two multi-layer perceptrons MLP, a window-based multi-head self-attention mechanism W-MSA and a moving window-based multi-head self-attention mechanism SW-MSA, and a layer normalization LN is added before each multi-head self-attention mechanism and each multi-layer perceptron, and a residual connection is used after each module.

[0036] As a technical preferred solution of the present invention, the detail layer branch in S4-2 uses a convolutional neural network to extract features of clear texture information in the detail layer image, jump connections are used between each convolution layer, each convolution block contains batch normalization and uses the ReLU function as an activation function; the output feature map of the detail layer branch and the base layer branch are the same size, the output feature maps of the two branches are added and sent to an image reconstruction layer with a convolution kernel size of 3×3, a step size of 1, and a number of convolution kernels of 4, and the Tanh activation function is used to obtain the final output fused image of the generator.

[0037] As a technical preferred solution of the present invention, the network structure of the Transformer Block in S4-1-2 includes the following features:

[0038] S4-1-2-1: The input of each Transformer Block is a 128×128×96 feature map with a fixed shape. The feature map is first split into 256 8×8×96 feature maps by an 8×8 local window, and then each window feature map F windowPerform multi-head self-attention calculations respectively; calculate the attention matrix of the local window through the self-attention mechanism; then send the attention matrix output by the multi-head self-attention into the layer normalization, and then send it to the multi-layer perceptron for nonlinear transformation of position encoding and feature mapping, and finally obtain a feature map with global features;

[0039] S4-1-2-2: The query feature matrix Q, key feature matrix K and value feature matrix V in the calculation process of multi-head self-attention calculation are expressed as:

[0040] Q=F window ·M Q

[0041] K=F window ·M K

[0042] V=F window ·M V

[0043] Among them, M Q 、M K 、M V is the projection matrix;

[0044] S4-1-2-3: The attention matrix of the local window is calculated through the self-attention mechanism. The calculation process is expressed as:

[0045]

[0046] where S(·) represents the normalized exponential function; d represents the dimension; and p represents the learnable relative position encoding.

[0047] As a technical preferred solution of the present invention, the base layer discriminator and detail layer discriminator of S5 constitute a discriminator, and the discriminator consists of the following parts:

[0048] S5-1: The network structure of the base layer discriminator and the detail layer discriminator is the same, and each discriminator contains five convolutional layers;

[0049] S5-2: The kernel size of each of the five convolutional layers is 3×3, the number of input channels is 4, 16, 32, 64, and 128, respectively, and the step size is 2. A fully connected layer is connected after the five convolutional layers and the Tanh activation function is used;

[0050] S5-3: The input of the base layer discriminator is the base layer image of the source multispectral image base layer and the source panchromatic image base layer after averaging and the base layer of the fused image;

[0051] S5-4: The input of the detail layer discriminator is the detail layer image of the source multispectral image detail layer and the source panchromatic image detail layer after maximization processing and the detail layer of the fused image.

[0052] As a preferred technical solution of the present invention, the anti-loss L in S6-1-4 adv The specific calculation formula is:

[0053]

[0054] in, represents the base layer discriminator, D detail represents the detail layer discriminator; Represents the fused image I generated by the generator F The base layer image, I F Detail layer image; a is the balance D base With D detail The weight coefficient of

[0055] The content loss L in S6-1-4 content The specific calculation formula is:

[0056] L content =αL int +βL grad +γL SSIM

[0057] Among them, L int is the strength loss, L grad is the gradient loss, L SSIM is the structural similarity loss, α, β, γ are the weight coefficients for balancing the three, and L int is defined as follows:

[0058]

[0059] Among them, H and W represent the height and width of the input image, ω is the weight coefficient, and L int-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer strength loss between int-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer strength loss between is defined as:

[0060]

[0061]

[0062] Among them, b is the weight coefficient of the two balance items, ||·|| F is the F norm, content loss L content The second gradient loss L grad The specific definition of is:

[0063]

[0064] Among them, L grad-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer gradient loss between grad-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The gradient loss of the detail layer between is specifically defined as:

[0065]

[0066]

[0067] Among them, the content loss L content The third structural similarity loss L SSIM The specific definition of is:

[0068] L SSIM =ωL SSIM-base +(1-ω)L SSIM-detail

[0069] Among them, L SSIM-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer structural similarity loss between SSIM-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer structural similarity loss between , is specifically defined as:

[0070]

[0071]

[0072] Among them, L SSIM (·) indicates the structural similarity of two items;

[0073] The spectral loss L in S6-1-4 spectrumThe specific calculation formula is:

[0074] L spectrum =L spectrum-base +L spectrum-detail

[0075] Where L spectrum-base represents the base layer spectral loss between the multispectral image base layer and the fused image base layer, L spectrum-detail It represents the spectral loss of the detail layer between the multispectral image detail layer and the fused image detail layer, which is specifically defined as:

[0076]

[0077]

[0078] Among them, I is an all-one matrix; <·> is the inner product of two terms; ||·||2 is the 2-norm.

[0079] Compared with the relevant existing technologies, this application proposal has the following main technical advantages:

[0080] 1) Efficient image decomposition and reconstruction: By adopting the guided filter, the present invention can effectively decompose the source image to obtain a multispectral base layer, a multispectral detail layer, a panchromatic base layer and a panchromatic detail layer. This decomposition strategy can fully consider the high spatial resolution of the panchromatic image and the spectral characteristics of the multispectral image, thereby providing a more reasonable data representation for subsequent feature extraction and fusion.

[0081] 2) Deep feature extraction and fusion: This paper uses convolutional neural networks and Transformer networks to ensure that both texture feature extraction at the detail layer and global information extraction at the base layer are optimized. The dual-branch feature extraction strategy can better capture the spatial and spectral information of the image, thereby obtaining a richer fusion map.

[0082] 3) Enhanced discrimination capability: By designing a base layer discriminator and a detail layer discriminator, the present invention can discriminate the fused image and the source image from different levels, thereby ensuring that the generated fused image has high fidelity in both spectrum and space.

[0083] 4) Detailed loss function design: The present invention comprehensively considers adversarial loss, content loss and spectral loss to ensure that the quality of the fused image can be fully optimized during the training process while maintaining spectral and spatial details.

[0084] 5) High-quality fusion results: Compared with existing technologies, the present invention can generate fused images with higher spectral and spatial quality, meeting the high-quality requirements of remote sensing image fusion.

[0085] The present invention proposes a panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer. The proposed method is an "end-to-end" network model composed of a generator and two discriminators. Experiments were conducted using public panchromatic and multispectral remote sensing image datasets. The experimental results show that compared with the existing methods, the fusion results obtained by the algorithm of the present invention have richer texture information and better subjective visual effects; in addition, the results of objective evaluation show that the algorithm of the present invention is superior to the average values ​​of the comparison methods in terms of information entropy, spectral angle, spectral distortion index and peak signal-to-noise ratio by about 6.82%, 1.51%, 27.76% and 26.07%, respectively, which further illustrates that the method proposed in the present invention can better fuse the spatial texture information of the panchromatic image while effectively retaining the spectral information of the multispectral image, thereby improving the performance of the existing multispectral and panchromatic remote sensing image fusion algorithm. Description of the drawings:

[0086] Figure 1 is a flow chart of the method of the present invention;

[0087] Figure 2 It is the overall fusion framework of the method proposed in the present invention;

[0088] Figure 3 It is a network structure diagram of the generator provided by the present invention;

[0089] (a) Generator base layer network structure; (b) Generator detail layer network structure;

[0090] Figure 4 This is the discriminator network structure provided by the present invention. Specific implementation method:

[0091] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0092] Regarding the current panchromatic and multispectral remote sensing image fusion problem based on the Transformer network, considering the different characteristics of panchromatic images with clear spatial texture information and multispectral images with rich spectral information, how to effectively combine the better global information extraction ability of Transformer with the better local information extraction ability of CNN in the network to achieve better fusion effect is still a problem to be solved. Based on the above analysis, in order to further improve the effect of panchromatic and multispectral remote sensing image fusion, the present invention proposes a new unsupervised panchromatic and multispectral remote sensing image fusion method based on a dual-branch generative adversarial network and Transformer. The proposed network model adopts a single generator-dual discriminator structure to extract and integrate features from the base layer and detail layer of the source image. First, the generator adopts a dual-branch structure, and different branches extract features at different levels, and then the fusion layer reconstructs the fused image according to the fusion features. Secondly, the present invention adopts guided filtering decomposition processing on the source image to highlight the unique information of the base layer and detail layer of the source image. Finally, in order to further constrain the generator and the two discriminators to target different characteristics of different images, loss functions are designed from three perspectives: content loss, adversarial loss, and spectral loss.

[0093] Embodiment 1: A panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer, the method comprising the following steps:

[0094] S1: Use guided filters to decompose the source image to obtain a multispectral base layer, a multispectral detail layer, a panchromatic base layer, and a panchromatic detail layer, and use different decomposition strategies according to the characteristics of the panchromatic image and the multispectral image;

[0095] Different decomposition strategies are adopted: the panchromatic image has a higher spatial resolution and is used as the guiding image of the guided filter; while the multispectral image has a high spectral resolution and rich spectral information but fuzzy texture features, so principal component analysis is performed on the multispectral image and its principal component image is used as the guiding image.

[0096] S2: stack the multispectral base layer and the panchromatic base layer in the channel dimension to obtain an input base layer image; stack the multispectral detail layer and the panchromatic detail layer in the channel dimension to obtain an input detail layer image;

[0097] S3: For the detail layer image containing the main texture information, a convolutional neural network is used to extract the features of the texture information;

[0098] S4: For the base layer image containing some texture detail information and more spectral information, the Transformer network is used to extract features, and the features extracted in S3 are fused with the features extracted in this step. Then, the image is reconstructed based on the fused features, so as to obtain a fused image with rich texture information and spectral information as the output result of the generator;

[0099] The network structure of the generator consists of the following parts:

[0100] S4-1: Base layer branch: used to extract spectral information features from the base layer image;

[0101] The base layer branch consists of two Conv Blocks and three Transformer Blocks;

[0102] S4-1-1: The first Conv Block is used to preliminarily extract shallow information. Its input is a base layer image including 4 multispectral channels and 1 panchromatic channel. The convolution kernel size is set to 3×3, the step size is 1, and the number of convolution kernels is 96;

[0103] S4-1-2: The obtained feature map is input into three Transformer Blocks with the same structure, and residual connections are used between multiple Transformer Blocks;

[0104] The network structure of Transformer Block includes the following features:

[0105] S4-1-2-1: The input of each Transformer Block is a 128×128×96 feature map with a fixed shape. The feature map is first split into 256 8×8×96 feature maps by an 8×8 local window, and then each window feature map F window Perform multi-head self-attention calculations respectively; calculate the attention matrix of the local window through the self-attention mechanism; then send the attention matrix output by the multi-head self-attention into the layer normalization, and then send it to the multi-layer perceptron for nonlinear transformation of position encoding and feature mapping, and finally obtain a feature map with global features;

[0106] S4-1-2-2: The query feature matrix Q, key feature matrix K and value feature matrix V in the calculation process of multi-head self-attention calculation are expressed as:

[0107] Q=F window ·M Q

[0108] K=F window ·M K

[0109] V=F window ·M V

[0110] Among them, M Q 、M K 、M V is the projection matrix;

[0111] S4-1-2-3: The attention matrix of the local window is calculated through the self-attention mechanism. The calculation process is expressed as:

[0112]

[0113] where S(·) represents the normalized exponential function; d represents the dimension; and p represents the learnable relative position encoding.

[0114] S4-1-3: Each Transformer Block consists of two identical Swin Transformer Layers, where the Swin Transformer Layer contains two multi-layer perceptrons MLP, a window-based multi-head self-attention mechanism W-MSA and a moving window-based multi-head self-attention mechanism SW-MSA, and a layer normalization LN is added before each multi-head self-attention mechanism and each multi-layer perceptron, and a residual connection is used after each module.

[0115] S4-2: Detail layer branch: used to extract detail features such as texture from the detail layer image.

[0116] The detail layer branch uses a convolutional neural network to extract the features of clear texture information in the detail layer image. Skip connections are used between each convolution layer. Each convolution block contains batch normalization and uses the ReLU function as the activation function. The output feature maps of the detail layer branch and the base layer branch are the same size. The output feature maps of the two branches are added and sent to the image reconstruction layer with a convolution kernel size of 3×3, a step size of 1, and a number of convolution kernels of 4. The Tanh activation function is used to obtain the final output fused image of the generator.

[0117] S5: Perform guided filtering decomposition on the fusion image output by the generator to obtain a base layer image and a detail layer image of the fusion image, and input the base layer image and the source image base layer calculated by the average value, and the detail layer image and the source image detail layer calculated by the maximum value into the base layer discriminator and the detail layer discriminator respectively, and discriminate the input fusion image and the source image from the two levels of base layer information and detail layer information;

[0118] The base layer discriminator and the detail layer discriminator form the discriminator, which consists of the following parts:

[0119] S5-1: The network structure of the base layer discriminator and the detail layer discriminator is the same, and each discriminator contains five convolutional layers;

[0120] S5-2: The kernel size of each of the five convolutional layers is 3×3, the number of input channels is 4, 16, 32, 64, and 128, respectively, and the step size is 2. A fully connected layer is connected after the five convolutional layers and the Tanh activation function is used;

[0121] S5-3: The input of the base layer discriminator is the base layer image of the source multispectral image base layer and the source panchromatic image base layer after averaging and the base layer of the fused image;

[0122] S5-4: The input of the detail layer discriminator is the detail layer image of the source multispectral image detail layer and the source panchromatic image detail layer after maximization processing and the detail layer of the fused image.

[0123] S6: The generator is continuously confronted and optimized with the base layer discriminator and the detail layer discriminator until the base layer discriminator and the detail layer discriminator are unable to distinguish the fused image generated by the generator, thereby obtaining the final fusion result;

[0124] The generator is trained using a loss function composed of adversarial loss, content loss, and spectral loss;

[0125] S6-1-1: Adversarial loss is defined as the adversarial relationship between the generator and the base layer discriminator and the detail layer discriminator;

[0126] S6-1-2: Content loss includes intensity loss, gradient loss and structural similarity loss, where intensity loss involves the comparison between the intensity of the fused image and the intensity of the source image, gradient loss involves the comparison between the gradient of the fused image and the gradient of the source image, and structural similarity loss involves the structural similarity between the fused image and the source image;

[0127] S6-1-3: Spectral loss involves the spectral similarity between the multispectral image and the fused image;

[0128] S6-1-4: The calculation formula of the loss function is: L G =L adv +λ1L content +λ2L spectrum

[0129] Among them, L G represents the total loss of the generator G, L adv Represents the generator G and the base layer discriminator D base , detail layer discriminator D detail The adversarial loss between them, L content represents the content loss of the fused image, Lspectrum represents the spectral loss, λ1 and λ2 are the weight coefficients for adjusting the proportion of content loss and spectral loss respectively;

[0130] Adversarial loss L adv The specific calculation formula is:

[0131]

[0132] Among them, D base represents the base layer discriminator, D detail represents the detail layer discriminator; Represents the fused image I generated by the generator F The base layer image, I F Detail layer image; a is the balance D base With D detail The weight coefficient of

[0133] Content loss L content The specific calculation formula is:

[0134] L content =αL int +βL grad +γL SSIM

[0135] Among them, L int is the strength loss, L grad is the gradient loss, L SSIM is the structural similarity loss, α, β, γ are the weight coefficients for balancing the three, and L int is defined as follows:

[0136]

[0137] Among them, H and W represent the height and width of the input image, ω is the weight coefficient, and L int-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer strength loss between int-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer strength loss between is defined as:

[0138]

[0139]

[0140] Among them, b is the weight coefficient of the two balance items, ||·||F is the F norm, content loss L content The second gradient loss L grad The specific definition of is:

[0141]

[0142] Among them, L grad-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer gradient loss between grad-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The gradient loss of the detail layer between is specifically defined as:

[0143]

[0144]

[0145] Among them, the content loss L content The third structural similarity loss L SSIM The specific definition of is:

[0146] L SSIM =ωL SSIM-base +(1-ω)L SSIM-detail

[0147] Among them, L SSIM-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer structural similarity loss between SSIM-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer structural similarity loss between , is specifically defined as:

[0148]

[0149]

[0150] Among them, L SSIM (·) indicates the structural similarity of two items;

[0151] Spectral loss L spectrum The specific calculation formula is:

[0152] L spectrum =Lspectrum-base +L spectrum-detail

[0153] Where L spectrum-base represents the base layer spectral loss between the multispectral image base layer and the fused image base layer, L spectrum - detail It represents the spectral loss of the detail layer between the multispectral image detail layer and the fused image detail layer, which is specifically defined as:

[0154]

[0155]

[0156] Among them, I is an all-one matrix; <·> is the inner product of two terms; ||·||2 is the 2-norm.

[0157] The training of the discriminator uses a loss function, which consists of two parts: the base layer discriminator loss and the detail layer discriminator loss. The calculation formula is:

[0158] S6-2-1: Loss of base layer discriminator for:

[0159]

[0160] Among them, D base represents the base layer discriminator, D base (·) represents the judgment value of the base layer discriminator on the authenticity of the input image; I base-mean It represents the image obtained by averaging the multispectral image base layer and the panchromatic image base layer. is the base layer of the fused image; E(·) represents the mathematical expectation;

[0161] S6-2-2: Detail layer discriminator loss for:

[0162]

[0163] Among them, D detail Denotes the detail layer discriminator, D detail (·) represents the judgment value of the detail layer discriminator on the authenticity of the input image; I detail-max It represents the image obtained by performing maximum processing on the detail layer of the multispectral image and the detail layer of the panchromatic image. is the detail layer of the fused image; E(·) represents the mathematical expectation value;

[0164] Example 2: According to Figure 1 As shown, the workflow of the present invention includes:

[0165] 1) Data preprocessing: Obtain a set of remote sensing image data, including multispectral images with high spectral resolution (including four channels: red, green, blue, and near-infrared) and panchromatic images with high spatial resolution. These two sets of images are registered according to their geographic coordinates to ensure that they are spatially aligned.

[0166] 2) Image decomposition: Use guided filters to decompose the source image. Obtain a multispectral base layer, a multispectral detail layer, a panchromatic base layer, and a panchromatic detail layer. Stack the multispectral base layer and the panchromatic base layer in the channel dimension to obtain the input base layer image. At the same time, stack the multispectral detail layer and the panchromatic detail layer to obtain the input detail layer image.

[0167] 3) Feature extraction: Use convolutional neural networks to extract features from detail layer images. This step is to capture clear texture information. Use the Transformer network with strong global information extraction capabilities to extract features from the base layer image. The features extracted by the convolutional neural network and the Transformer network are fused to obtain a fused feature map.

[0168] 4) Image synthesis: Use the generator network to convert the fused feature maps into a fused image.

[0169] 5) Discriminator training: Use the base layer and detail layer of the source image and the base layer and detail layer of the fused image output by the generator to train the base layer discriminator and detail layer discriminator.

[0170] 6) Adversarial training between generator and discriminator: Use the generator and discriminator to perform adversarial training until both the base layer discriminator and the detail layer discriminator cannot distinguish between the fused image generated by the generator and the real source image.

[0171] 7) Result evaluation: The quality of the generated fused image and the source image is evaluated using a set of standard metrics (e.g., peak signal-to-noise ratio, structural similarity index, etc.).

[0172] Example 3: According to Figure 2 As shown, in this embodiment, based on the given network structure and parameters, it will be described in detail how to train the model and finally use it to fuse panchromatic and multispectral remote sensing images. Parameter and data settings:

[0173] 1) Training parameters: learning rate = 0.0001, batch size = 32, optimizer = RMSprop.

[0174] 2) Dataset: Assume that the present invention has a dataset containing a panchromatic image and its corresponding multispectral image. These images should have been geo-calibrated to ensure that they are spatially aligned.

[0175] 3) Preprocessing: All images are normalized to the range [0,1]. Data augmentation techniques such as random cropping, rotation, and flipping are used to increase the diversity of the data.

[0176] 4) Model initialization: Initialize the weights of the generator network and the discriminator network. You can use a pre-trained model or random initialization.

[0177] 5) Adversarial training: For each batch of data: Use the generator to generate a fused image. Use the discriminator to evaluate the similarity between the generated fused image and the true source image. Calculate the loss based on the output of the generator and discriminator, including adversarial loss, content loss, and spectral loss. Use backpropagation to update the weights of the generator and discriminator. Repeat the above steps until the model converges or the preset number of training iterations is reached.

[0178] 6) Model evaluation and saving: Use the validation dataset to evaluate the performance of the model. Save the model weights, structure, and training parameters as needed.

[0179] 7) Model usage: Load the trained model. Input the panchromatic image and multispectral image into the model to obtain a fused image. Perform post-processing as needed, such as color correction.

[0180] Through the above embodiments, the remote sensing image fusion method based on guided filtering, convolutional neural network and Transformer network proposed in the present invention is fully demonstrated and verified. This method makes full use of the high spatial resolution characteristics of panchromatic images and the rich spectral information of multispectral images to achieve effective fusion of the two images. Compared with the prior art, the method of the present invention significantly improves the spatial clarity of the fused image while maintaining high spectral quality. In addition, by adopting a dual discriminator structure, the method further enhances the ability to discriminate the base layer and detail layer information of the generated image, thereby achieving a more accurate and robust fusion result. Overall, the present invention provides an efficient and reliable solution for remote sensing image fusion, and has a wide range of application value for practical remote sensing image processing applications.

[0181] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer, characterized in that: The method comprises the following steps: S1: Use guided filters to decompose the source image to obtain a multispectral base layer, a multispectral detail layer, a panchromatic base layer, and a panchromatic detail layer, and use different decomposition strategies according to the characteristics of the panchromatic image and the multispectral image; S2: stack the multispectral base layer and the panchromatic base layer in the channel dimension to obtain an input base layer image; stack the multispectral detail layer and the panchromatic detail layer in the channel dimension to obtain an input detail layer image; S3: For the detail layer image containing the main texture information, a convolutional neural network is used to extract the features of the texture information; S4: For the base layer image containing some texture detail information and more spectral information, the Transformer network is used to extract features, and the features extracted in S3 are fused with the features extracted in this step. Then, the image is reconstructed based on the fused features, so as to obtain a fused image with rich texture information and spectral information as the output result of the generator; S5: Perform guided filtering decomposition on the fused image output by the generator to obtain a base layer image and a detail layer image of the fused image, and input the base layer image and the source image base layer calculated by the average value, and the detail layer image and the source image detail layer calculated by the maximum value into the base layer discriminator and the detail layer discriminator respectively, and discriminate the input fused image and the source image from the two levels of base layer information and detail layer information; S6: The generator is continuously confronted and optimized with the base layer discriminator and the detail layer discriminator until the base layer discriminator and the detail layer discriminator are unable to distinguish the fused image generated by the generator, thereby obtaining the final fusion result.

2. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 1, characterized in that: Different decomposition strategies are adopted in S1: the panchromatic image has a higher spatial resolution and is used as the guiding image of the guiding filter; while the multispectral image has a high spectral resolution and rich spectral information but fuzzy texture features, so the multispectral image is subjected to principal component analysis and its principal component image is used as the guiding image.

3. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 1, characterized in that: The network structure of the generator in S4 consists of the following parts: S4-1: Base layer branch: used to extract spectral information features from the base layer image; S4-2: Detail layer branch: used to extract texture detail features from the detail layer image.

4. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 1, characterized in that: The training of the generator in S6 uses a loss function composed of adversarial loss, content loss and spectral loss; S6-1-1: Adversarial loss is defined as the adversarial relationship between the generator and the base layer discriminator and the detail layer discriminator; S6-1-2: Content loss includes intensity loss, gradient loss and structural similarity loss, where intensity loss involves the comparison between the intensity of the fused image and the intensity of the source image, gradient loss involves the comparison between the gradient of the fused image and the gradient of the source image, and structural similarity loss involves the structural similarity between the fused image and the source image; S6-1-3: Spectral loss involves the spectral similarity between the multispectral image and the fused image; S6-1-4: The calculation formula of the loss function is: L G =L adv +λ1L content +λ2L spectrum Among them, L G represents the total loss of the generator G, L adv Represents the generator G and the base layer discriminator D base , detail layer discriminator D detail The adversarial loss between them, L content represents the content loss of the fused image, L spectrum represents spectral loss, λ1 and λ2 are weight coefficients for adjusting the proportion of content loss and spectral loss respectively.

5. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 1, characterized in that: The training of the discriminator in S6 uses a loss function, which consists of two parts: the base layer discriminator loss and the detail layer discriminator loss. The calculation formula is: S6-2-1: Loss of base layer discriminator for: Among them, D base represents the base layer discriminator, D base (·) represents the judgment value of the base layer discriminator on the authenticity of the input image; I base-mean It represents the image obtained by averaging the multispectral image base layer and the panchromatic image base layer. is the base layer of the fused image; E(·) represents the mathematical expectation; S6-2-2: Detail layer discriminator loss for: Among them, D detail Denotes the detail layer discriminator, D detail (·) represents the judgment value of the detail layer discriminator on the authenticity of the input image; I detail-max It represents the image obtained by performing maximum processing on the detail layer of the multispectral image and the detail layer of the panchromatic image. is the detail layer of the fused image; E(·) represents the mathematical expectation value.

6. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 3, characterized in that: The base layer branch in S4-1 is composed of two Conv Blocks and three Transformer Blocks; S4-1-1: The first Conv Block is used to preliminarily extract shallow information. Its input is a base layer image including 4 multispectral channels and 1 panchromatic channel. The convolution kernel size is set to 3×3, the step size is 1, and the number of convolution kernels is 96; S4-1-2: The obtained feature map is input into three Transformer Blocks with the same structure, and residual connections are used between multiple Transformer Blocks; S4-1-3: Each Transformer Block consists of two identical Swin Transformer Layers, where the Swin Transformer Layer contains two multi-layer perceptrons MLP, a window-based multi-head self-attention mechanism W-MSA and a moving window-based multi-head self-attention mechanism SW-MSA, and a layer normalization LN is added before each multi-head self-attention mechanism and each multi-layer perceptron, and a residual connection is used after each module.

7. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 3, characterized in that: The detail layer branch in S4-2 uses a convolutional neural network to extract the features of clear texture information in the detail layer image. Skip connections are used between each convolution layer. Each convolution block contains batch normalization and uses the ReLU function as the activation function. The output feature map of the detail layer branch and the base layer branch are the same size. The output feature maps of the two branches are added and sent to an image reconstruction layer with a convolution kernel size of 3×3, a step size of 1, and a number of convolution kernels of 4. The Tanh activation function is used to obtain the final output fused image of the generator.

8. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 6, characterized in that: The network structure of the Transformer Block in S4-1-2 includes the following features: S4-1-2-1: The input of each Transformer Block is a 128×128×96 feature map with a fixed shape. The feature map is first split into 256 8×8×96 feature maps by an 8×8 local window, and then each window feature map F window Perform multi-head self-attention calculations respectively; calculate the attention matrix of the local window through the self-attention mechanism; then send the attention matrix output by the multi-head self-attention into the layer normalization, and then send it to the multi-layer perceptron for nonlinear transformation of position encoding and feature mapping, and finally obtain a feature map with global features; S4-1-2-2: The query feature matrix Q, key feature matrix K and value feature matrix V in the calculation process of multi-head self-attention calculation are expressed as: Q=F window ·M Q K=F window ·M K V=F window ·M V Among them, M Q 、M K 、M V is the projection matrix; S4-1-2-3: The attention matrix of the local window is calculated through the self-attention mechanism. The calculation process is expressed as: where S(·) represents the normalized exponential function; d represents the dimension; and p represents the learnable relative position encoding.

9. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 1, characterized in that: The base layer discriminator and detail layer discriminator of S5 constitute a discriminator, and the discriminator consists of the following parts: S5-1: The network structure of the base layer discriminator and the detail layer discriminator is the same, and each discriminator contains five convolutional layers; S5-2: The kernel size of each of the five convolutional layers is 3×3, the number of input channels is 4, 16, 32, 64, and 128, respectively, and the step size is 2. A fully connected layer is connected after the five convolutional layers and the Tanh activation function is used; S5-3: The input of the base layer discriminator is the base layer image of the source multispectral image base layer and the source panchromatic image base layer after averaging and the base layer of the fused image; S5-4: The input of the detail layer discriminator is the detail layer image of the source multispectral image detail layer and the source panchromatic image detail layer after maximization processing and the detail layer of the fused image.

10. The panchromatic and multispectral remote sensing image fusion method based on GAN and Transformer according to claim 4, characterized in that: The loss L in S6-1-4 adv The specific calculation formula is: Among them, D base represents the base layer discriminator, D detail represents the detail layer discriminator; Represents the fused image I generated by the generator F The base layer image, Indicates I F Detail layer image; a is the balance D base With D detail The weight coefficient of The content loss L in S6-1-4 content The specific calculation formula is: L content =αL int +βL grad +γL SSIM Among them, L int is the strength loss, L grad is the gradient loss, L SSIM is the structural similarity loss, α, β, γ are the weight coefficients for balancing the three, and L int is defined as follows: Among them, H and W represent the height and width of the input image, ω is the weight coefficient, and L int-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer strength loss between int-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer strength loss between is defined as: Among them, b is the weight coefficient of the two balance items, ||·|| F is the F norm, content loss L content The second gradient loss L grad The specific definition of is: Among them, L grad-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer gradient loss between grad-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The gradient loss of the detail layer between is specifically defined as: Among them, the content loss L content The third structural similarity loss L SSIM The specific definition of is: L SSIM =ωL SSIM-base +(1-ω)L SSIM-detail Among them, L SSIM-base Represents the fused image base layer With full color image base layer Multispectral image base layer The base layer structural similarity loss between SSIM-detail Represents the fused image detail layer With full color image detail layer Multispectral image detail layer The detail layer structural similarity loss between , is specifically defined as: Among them, L SSIM (·) indicates the structural similarity of two items; The spectral loss L in S6-1-4 spectrum The specific calculation formula is: L spectrum =L spectrum-base +L spectrum-detail Where L spectrum-base represents the base layer spectral loss between the multispectral image base layer and the fused image base layer, L spectrum-detail It represents the spectral loss of the detail layer between the multispectral image detail layer and the fused image detail layer, which is specifically defined as: Among them, I is an all-one matrix; <·> is the inner product of two terms; ||·||2 is the 2-norm.

Images