A remote sensing image fusion method based on a generative adversarial network and multiple loss functions

Abstract

The application provides a remote sensing image fusion method based on a generative adversarial network and a multi-loss function, comprising the following steps: constructing a data set based on a panchromatic image and a multispectral image in a source image; constructing a generator network, a spatial discriminator network and a spectral discriminator network; inputting an image in the data set into the generator network to generate a pseudo image; judging the similarity of the pseudo image based on the spatial discriminator network and the spectral discriminator network; training the generator network, the spatial discriminator network and the spectral discriminator network based on a minimum loss function; and generating a fusion image by fusing the panchromatic image and the multispectral image in the source image based on the trained generator network. The fusion effect of the application is better than that of a traditional fusion algorithm and a general fusion algorithm based on deep learning, the performance on a full-size image is better, the generalization ability is stronger, and the remote sensing image can be efficiently fused.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image fusion technology, and particularly relates to a remote sensing image fusion method based on multiple loss functions of generative adversarial networks. Background Technology

[0002] Due to the inherent limitations of imaging technology, satellites typically carry two types of cameras: one for acquiring panchromatic images (PAN images), which contain information from only one spectral band, offering high spatial resolution and rich detail; and another for acquiring multispectral images (MS images), which cover multiple spectral bands, offering high spectral resolution and rich color information. However, in practical applications, images often require both high spatial and spectral resolution. This "full-size multispectral" image is obtained through remote sensing image fusion technology.

[0003] Remote sensing image fusion is an important branch of the field of image fusion, widely used in important areas such as agricultural vegetation surveys, land resource monitoring, water resource protection, and national defense science and technology construction. Existing methods can be classified into the following three categories:

[0004] The first category is spatial domain-based methods: these methods typically perform operations on the source images in the spatial domain to construct the final fused image. The simplest model for this method is weighted averaging. However, this model often leads to severe boundary effects and, due to differences in the size of the remote sensing images, the final fused image may lose crucial information from the source images.

[0005] The second category is transformation domain-based methods. These algorithms typically involve three steps: image decomposition, coefficient fusion, and image reconstruction. Multiscale transform (MST) is a classic method in this category, with many common variations such as Laplacian pyramid, hierarchical pyramid, discrete wavelet transform (DWT), non-subsampled contourlet transform (NSCT), and dual-tree complex wavelet transform (DTCWT). These methods can effectively improve upon the artifacts and boundary effects present in spatial domain methods. Unfortunately, these methods require manually defined fusion rules during reconstruction, and the formulation of these rules relies heavily on domain knowledge, significantly limiting the generalization ability of these algorithms.

[0006] The third type of algorithm is based on deep learning. In recent years, with the development of deep learning technology, remote sensing image fusion algorithms based on convolutional neural networks have gradually emerged. These algorithms often achieve good results on downsampled training sets and do not rely on manually defined fusion rules. However, despite the progress made by convolutional neural network-based algorithms, their performance is poor when applied to full-size images because their training process is based on downsampled training sets, resulting in the loss of some information from the source images.

[0007] Therefore, there is an urgent need to propose a remote sensing image fusion method based on multiple loss functions of generative adversarial networks to improve the problems existing in deep learning-based methods. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a remote sensing image fusion method based on multiple loss functions of generative adversarial networks. This method is capable of fusing panchromatic and multispectral images to obtain a "panchromatic multispectral" image. Its fusion effect is superior to traditional fusion algorithms and general deep learning-based fusion algorithms. It performs better on full-size images, has stronger generalization ability, and can efficiently fuse remote sensing images.

[0009] To achieve the above objectives, this invention provides a remote sensing image fusion method based on multiple loss functions of generative adversarial networks, comprising: constructing a dataset based on panchromatic and multispectral maps in the source images;

[0010] A generator network, a spatial discriminator network, and a spectral discriminator network are constructed. Images from the dataset are input into the generator network to generate pseudo images. The spatial discriminator network and the spectral discriminator network are then used to verify the pseudo images.

[0011] The generator network, spatial discriminator network, and spectral discriminator network are trained based on minimizing the loss function.

[0012] Based on the trained generator network, the panchromatic image and multispectral image in the source image are fused to generate a fused image.

[0013] Furthermore, constructing the dataset includes:

[0014] Bicubic interpolation downsampling is used to downsample the panchromatic and multispectral images in the source image;

[0015] Use the multispectral map in the source image as a label;

[0016] The dataset is constructed based on the sampled panchromatic and multispectral images and the labels.

[0017] Furthermore, the generator network includes an encoder and a decoder; the encoder is connected to the decoder.

[0018] The encoder is used to extract semantic information from the input image, reduce the spatial scale of the image, and increase the number of channels in the image;

[0019] The decoder is used to restore the image information and output the pseudo image.

[0020] Furthermore, the encoder includes: a plurality of downsampling modules, each of the downsampling modules being stacked with a convolutional layer and a batch normalization layer, and the output of the batch normalization layer being processed by a ReLU activation function, wherein the number of each convolutional kernel is set based on a first preset value.

[0021] Furthermore, the decoder includes: a plurality of upsampling modules, each of the upsampling modules being stacked with a transposed convolutional layer and a batch normalization layer, and the output of the batch normalization layer being processed by the LeakyReLU activation function, wherein the number of each convolutional kernel is set based on a second preset value;

[0022] The upsampling modules and the downsampling modules are connected along the channel dimension.

[0023] Furthermore, the decoder is also connected to several additional convolutional layers at its tail.

[0024] Furthermore, the verification of the fake image using the spatial discriminator network and the spectral discriminator network includes:

[0025] The spatial discriminator network is used to determine the similarity between the pseudo-image and the panchromatic image in the source image;

[0026] The spectral discriminator is used to determine the similarity between the multispectral maps of the pseudo-image and the source image.

[0027] The verification of the pseudo-image is completed based on the complete similarity between the pseudo-image and the source image.

[0028] Furthermore, the spatial discriminator network and the spectral discriminator network adopt the same network structure, both including several convolutional modules. Each convolutional module is stacked with a convolutional layer and a batch normalization layer, and the output of the batch normalization layer is processed by the LeakyReLU activation function. The number of each convolutional kernel is set based on a third preset value.

[0029] Furthermore, the minimized loss function includes generator loss, spatial discriminator adversarial loss, and spectral discriminator adversarial loss;

[0030] The generator loss includes: content loss, parameterless loss, and adversarial loss;

[0031] The expression for the content loss is:

[0032]

[0033] Where N is the total number of registered images in the dataset, i is the image number in the dataset, F represents the pseudo image generated by the generator, and T represents the gold standard obtained via Wald's protocol;

[0034] The expression for the parameterless loss is:

[0035] L Q =1-QNR

[0036] Where QNR is an image quality evaluation index without reference, and its expression is:

[0037] QNR=(1-D λ (1-D) s )

[0038]

[0039]

[0040]

[0041] Where QNR represents (1-D λ ) and (1-D s The product of the two terms, D λ D is the spectral distortion evaluation index. s Here, Q(x,y) is the spatial quality evaluation index, where i and j are two distinct image indices, K is the number of bands, M and P are the multispectral and panchromatic images respectively, Q(x,y) is the image quality evaluation formula, and σ is the spatial quality evaluation index. xy The covariance of the input image. and This represents the mean of the input image. and Represents the variance of the input image;

[0042] The expression for the adversarial loss is:

[0043]

[0044] Where D1 and D2 represent the spatial discriminator and the spectral discriminator, respectively;

[0045] The expression for the adversarial loss of the spatial discriminator is:

[0046]

[0047] in, Let F be the mean value of each channel of the pseudo-image F;

[0048] The expression for the spectral discriminator's resistance loss is:

[0049]

[0050] in, This is the image obtained by downsampling the pseudo-image F.

[0051] Compared with the prior art, the present invention has the following advantages and technical effects:

[0052] 1. This invention obtains a clearer fused image through a novel remote sensing image fusion algorithm based on generative adversarial networks;

[0053] 2. This invention designs a fusion-based generator network structure, eliminating the need for manual design of fusion rules;

[0054] 3. This invention designs a discriminator network group to improve the performance of fused images on full-size images;

[0055] 4. This invention can process remote sensing images well and can overcome the limitations of satellite imaging technology to a certain extent. The resulting fused image can have both high spatial and spectral resolution. Attached Figure Description

[0056] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0057] Figure 1 This is a schematic diagram of a remote sensing image fusion method based on multiple loss functions of generative adversarial networks according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the generator network structure according to an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the discriminator network according to an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the panchromatic PAN (panchromatic image) according to an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of a multispectral image (MS) according to an embodiment of the present invention;

[0062] Figure 6This is a schematic diagram of the fused image F obtained in an embodiment of the present invention. Detailed Implementation

[0063] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0064] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0065] Example

[0066] This embodiment provides a remote sensing image fusion method based on multiple loss functions of generative adversarial networks, including:

[0067] A dataset is constructed based on the panchromatic and multispectral maps of the source images;

[0068] A generator network, a spatial discriminator network, and a spectral discriminator network are constructed. Images from the dataset are input into the generator network to generate pseudo images. The spatial discriminator network and the spectral discriminator network are then used to verify the pseudo images.

[0069] The generator network, spatial discriminator network, and spectral discriminator network are trained based on minimizing the loss function.

[0070] Based on the trained generator network, the panchromatic image and multispectral image in the source image are fused to generate a fused image.

[0071] Furthermore, constructing the dataset includes:

[0072] Bicubic interpolation downsampling is used to downsample the panchromatic and multispectral images in the source image;

[0073] Use the multispectral map in the source image as a label;

[0074] The dataset is constructed based on the sampled panchromatic and multispectral images and the labels.

[0075] Furthermore, the generator network includes an encoder and a decoder; the encoder is connected to the decoder.

[0076] The encoder is used to extract semantic information from the input image, reduce the spatial scale of the image, and increase the number of channels in the image;

[0077] The decoder is used to restore the image information and output the pseudo image.

[0078] Furthermore, the encoder includes: a plurality of downsampling modules, each of the downsampling modules being stacked with a convolutional layer and a batch normalization layer, and the output of the batch normalization layer being processed by a ReLU activation function, wherein the number of each convolutional kernel is set based on a first preset value.

[0079] Furthermore, the decoder includes: a plurality of upsampling modules, each of the upsampling modules being stacked with a transposed convolutional layer and a batch normalization layer, and the output of the batch normalization layer being processed by the LeakyReLU activation function, wherein the number of each convolutional kernel is set based on a second preset value;

[0080] The upsampling modules and the downsampling modules are connected along the channel dimension.

[0081] Furthermore, the decoder is also connected to several additional convolutional layers at its tail.

[0082] Furthermore, the verification of the fake image using the spatial discriminator network and the spectral discriminator network includes:

[0083] The spatial discriminator network is used to determine the similarity between the pseudo-image and the panchromatic image in the source image;

[0084] The spectral discriminator is used to determine the similarity between the multispectral maps of the pseudo-image and the source image.

[0085] The verification of the pseudo-image is completed based on the complete similarity between the pseudo-image and the source image.

[0086] Furthermore, the spatial discriminator network and the spectral discriminator network adopt the same network structure, both including several convolutional modules. Each convolutional module is stacked with a convolutional layer and a batch normalization layer, and the output of the batch normalization layer is processed by the LeakyReLU activation function. The number of each convolutional kernel is set based on a third preset value.

[0087] In this embodiment, the remote sensing image fusion method based on multiple loss functions of generative adversarial networks adopts the following technical solution: (e.g.) Figure 1 As shown;

[0088] Step 1: Following Wald's protocol, use bicubic interpolation downsampling to process the panchromatic area (PAN) in the source image (e.g., ...). Figure 4 (as shown) and multispectral MS (e.g.) Figure 5As shown), the original MS image is used as the gold standard T (i.e., the label) to obtain the training set {P,M|T}. That is:

[0089]

[0090]

[0091] MS → T

[0092] Where P is the downsampled PAN image, M is the downsampled MS image, and T is the MS image in the source image, used as the gold standard for training. This gold standard will be used to measure the quality of the pseudo-images generated by the generator network.

[0093] Step 2, construct the generator network G (e.g., Figure 2 (As shown). Its construction process is as follows:

[0094] The network uses UNet as its backbone and is mainly divided into two parts: an encoder and a decoder. The encoder is used to extract semantic information from the input image and continuously reduce the spatial scale of the image by stacking convolutional layers and increasing the number of channels in the image to expand the overall receptive field of the model and enrich the semantic information in a unit space. The decoder is used to restore the image information and makes the output as close as possible to the gold standard by stacking transposed convolutional layers.

[0095] Specifically, the encoder contains four downsampling modules. Each downsampling module stacks a convolutional layer and a batch normalization layer, and uses the ReLU activation function to process the output of the batch normalization layer. The number of convolutional kernels are 64, 128, 256, and 512, respectively. The kernel size is designed to be 4*4 with a stride of 2. After an image passes through a downsampling module, its spatial size is halved, and the number of channels changes to the number of convolutional kernels in that layer.

[0096] In the decoder, the convolutional layer contains four upsampling modules. Each upsampling module stacks a transposed convolutional layer and a batch normalization layer, and uses the LeakyReLU activation function to process the output of the batch normalization layer, with a slope of 0.2. The number of convolutional kernels are 512, 1024, 512, and 256, respectively. The kernel size is designed to be 4*4 with a stride of 2. Unlike the encoder, in the decoder, the i-th layer needs to receive the input of the (i-1)-th layer and the output of the ni-th layer of the encoder. These two layers are concatenated along the channel dimension and used together as the input to the i-th layer of the decoder. This operation ensures that the semantic information of the shallow network in the encoder is preserved to the greatest extent in the deep network of the decoder. After the image passes through the upsampling module, the spatial size will double, and the number of channels will change to the number of convolutional kernels in that layer.

[0097] In addition, the generator network is connected to four extra convolutional layers at the end, with kernel numbers of 128, 64, 32, and 4 respectively. The first convolutional layer has a 3x3 kernel, while the others have 1x1 kernels. Each layer has a stride of 1. After passing through each layer, the spatial dimensions of the entire image remain unchanged, while the number of channels gradually decreases to the number of kernels in that layer. These four extra convolutional layers allow for a smooth decrease in the number of image channels and further enhance the fusion effect.

[0098] During training, the generator takes P and M as network inputs, and after forward propagation, obtains a pseudo-image F. The network parameters are then progressively optimized through adversarial training, allowing the pseudo-image F to approximate the gold standard T. Formally, the formula for generating the pseudo-image F is as follows:

[0099] F = G(P, M)

[0100] The generated pseudo-image F will be used as a reference to optimize the generator network parameters, with the gold standard T as the reference.

[0101] Step 3: After constructing the generator network G, construct the spatial discriminator network D1 and the spectral discriminator network D2. For example... Figure 3 As shown, the spatial discriminator network D1 and the spectral discriminator network D2 use the same architecture. The spatial discriminator network D1 uses the mean value of each channel of the pseudo-image F. As input, determine Spatial similarity to P (Sim) P ,Right now:

[0102]

[0103] On the other hand, the spectral discriminator network D2 obtains the image after downsampling the pseudo-image F. As input, determine Spectral similarity to M (Sim) M .

[0104]

[0105] The construction process of the two networks is as follows:

[0106] The network employs a fully convolutional structure with four layers. Each layer uses one convolutional layer and one batch normalization layer stacked together. The LeakyReLU activation function is used to process the output of the batch normalization layer, with a slope of 0.2. The number of convolutional kernels is 64, 128, 256, and 512, respectively. The kernel size is 4x4 with a stride of 2. The difference lies in the processing order: for the spatial discriminator network D1, the pseudo-image F needs to be averaged along its channels to obtain a single-pass image input. Furthermore, D1 needs to determine the similarity between the network output and P. For the spectral discriminator network D2, no additional processing is required on the pseudo-image F; it only needs to determine the similarity between the network output and M. The spatial and spectral discriminators can influence the generator's pseudo-image generation performance through adversarial training. In theory, after reaching Nash equilibrium, the spatial discriminator and the spectral discriminator will be unable to distinguish the pseudo-image F from the panchromatic image P and the multispectral image M. That is, the pseudo-image F generated by the generator is indistinguishable from the panchromatic image P in the spatial domain and from the multispectral image M in the spectral domain.

[0107] Step 4: Minimize the loss function. The loss function mainly includes the generator loss L. G Spatial discriminator to combat loss Spectral discriminator against loss Among them, the generator loss L G The formula is as follows:

[0108]

[0109] Specifically, the generator loss L G The first term L in the formula con The formula for content loss is as follows:

[0110]

[0111] Where N represents the total number of registered images in the dataset.

[0112] Generator loss L G The second term L in the formula Q For parameter-free loss, the formula is as follows:

[0113] L Q =1-QNR

[0114] The specific formula for QNR is as follows:

[0115] QNR=(1-D λ (1-D) s )

[0116]

[0117]

[0118]

[0119] Where QNR represents (1-D λ ) and (1-D s The product of the two terms, D λ D is the spectral distortion evaluation index. s Here, Q(x,y) is the spatial quality evaluation index, where i and j are two distinct image indices, K is the number of bands, M and P are the multispectral and panchromatic images respectively, Q(x,y) is the image quality evaluation formula, and σ is the spatial quality evaluation index. xy The covariance of the input image. and This represents the mean of the input image. and This represents the variance of the input image.

[0120] Generator loss L G The third term in the formula To counteract losses, the formula is as follows:

[0121]

[0122] The discriminator's loss is the Wasserstein loss, specifically:

[0123] Spatial discriminator to combat loss The formula is as follows:

[0124]

[0125] Spectral discriminator against loss The formula is as follows:

[0126]

[0127] During training, the generator loss L needs to be optimized alternately. G With discriminator loss This is to achieve the goal of adversarial training. During training, the Adam optimizer is used to alternately optimize the generator loss L. G With discriminator loss The Adam optimizer hyperparameters β1 and β2 are set to 0.9 and 0.999, respectively. Ultimately, the network converges to the Nash equilibrium point, yielding the optimal generator parameters G. * .

[0128] Step 5: After obtaining the optimal generator network parameters G... * Then, this network can be used to generate the required fused image on the existing dataset. That is:

[0129] F = G * (PAN,MS)

[0130] The generator's network output F is the desired fused image (e.g., Figure 6 (As shown).

[0131] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A remote sensing image fusion method based on a generative adversarial network and a multi-loss function, characterized in that, The method comprises the following steps: constructing a data set based on a panchromatic image and a multispectral image in a source image; constructing a generator network, a spatial discriminator network and a spectral discriminator network, inputting an image in the data set into the generator network to generate a pseudo image, and verifying the pseudo image by using the spatial discriminator network and the spectral discriminator network; training the generator network, the spatial discriminator network and the spectral discriminator network based on a minimized loss function; the minimized loss function comprises a generator loss, a spatial discriminator adversarial loss and a spectral discriminator adversarial loss; the generator loss comprises a content loss, a parameter-free loss and an adversarial loss; an expression of the content loss is as follows: wherein N is the total number of images that have been registered in the data set, i is the image serial number in the data set, F represents the pseudo image generated by the generator, and T represents a gold standard obtained via Wald's protocol; an expression of the parameter-free loss is as follows: wherein QNR is a no-reference image quality evaluation index, and the expression is as follows: where QNR represents the product of two terms, is the spectral distortion evaluation index; is the spatial quality evaluation index, i and j are two different picture serial numbers, and K is the number of bands, the product of is a multispectral image and a panchromatic image, respectively, is the image quality evaluation formula, is the covariance of the input image, the product of represents the mean of the input image, the product of represents the variance of the input image; an expression of the adversarial loss is as follows: wherein, and represent a spatial discriminator and a spectral discriminator, respectively; an expression of the spatial discriminator adversarial loss is as follows: wherein is a pseudo image the mean of each channel; an expression of the spectral discriminator adversarial loss is as follows: wherein is a pseudo image an image obtained after downsampling based on the trained generator network, fusing the panchromatic image and the multispectral image in the source image to generate a fused image.

2. The method of claim 1, wherein the method is based on a generative adversarial network and a multi-loss function. The method comprises the following steps: using a bicubic interpolation downsampling method to downsample the panchromatic image and the multispectral image in the source image; taking the multispectral image in the source image as a label; based on the sampled panchromatic image and multispectral image and the label, the data set is constituted.

3. The method of claim 1, wherein the method is based on a generative adversarial network and a multi-loss function. The generator network comprises an encoder and a decoder; the encoder is connected with the decoder, the encoder is used to extract semantic information of an input image, reduce the spatial scale of the image, and increase the channel number of the image; the decoder is used to restore picture information and output the pseudo image.

4. The method of claim 3, wherein the method is based on a generative adversarial network and a multi-loss function. The encoder comprises a plurality of down-sampling modules; each down-sampling module is stacked with one convolutional layer and one batch normalization layer, and the output of the batch normalization layer is processed by using a ReLU activation function, wherein the number of each convolutional kernel is set based on a first preset value.

5. The method of claim 4, wherein the method is based on a generative adversarial network and a multi-loss function. The decoder comprises a plurality of up-sampling modules; each up-sampling module is stacked with one transposed convolutional layer and one batch normalization layer, and the output of the batch normalization layer is processed by using a LeakyReLU activation function, wherein the number of each convolutional kernel is set based on a second preset value; a plurality of the up-sampling modules are connected with a plurality of the down-sampling modules in the channel dimension.

6. The method of claim 5, wherein the method is based on a generative adversarial network and a multi-loss function. The tail of the decoder is further connected with a plurality of additional convolutional layers.

7. The remote sensing image fusion method based on a multi-loss function of a generative adversarial network according to claim 1, wherein verifying the pseudo image by using the spatial discriminator network and the spectral discriminator network comprises the following steps: judging the similarity between the pseudo image and the panchromatic image in the source image by using the spatial discriminator network; judging the similarity between the pseudo image and the multispectral image in the source image by using the spectral discriminator. ​ Based on the overall similarity of the pseudo image and the source image, verification of the pseudo image is completed.

8. The method of claim 1, wherein the method is based on a generative adversarial network and a multi-loss function. The spatial discriminator network and the spectral discriminator network adopt the same network structure and each include a plurality of convolution modules, each convolution module stack adopts a convolution layer and a batch normalization layer, and the output of the batch normalization layer is processed by a LeakyReLU activation function, wherein the number of each convolution kernel is set based on a third preset value.

Images