Multi-level feature spatiotemporal remote sensing image fusion method based on generative adversarial network

Abstract

The present invention provides a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network. The generative adversarial network in the present invention is composed of a conditional generator network and a discriminator network. The conditional generator network uses an image feature extraction network, an image feature fusion network, and an image reconstruction network to extract and fuse multi-level features, and relies on multi-level features to reconstruct images to handle the huge difference between high-resolution images and low-resolution images. In the feature fusion stage, an adaptive instance normalization module is designed to handle the global change relationship between multi-phase images, and an attention module is designed to handle local change relationships. After alternating training of the conditional generator network and the discriminator network, the trained conditional generator network can be used to solve the remote sensing spatiotemporal image fusion problem and generate a fused predicted time high-resolution image.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network. Background Art

[0002] For remote sensing imaging satellites, there are two important properties, temporal resolution and spatial resolution, which greatly affect the scope of application of remote sensing observations. Spatial resolution determines the minimum size of objects that can be identified by remote sensing images; temporal resolution refers to the time required for a satellite to return to an area for observation again during its orbit. Due to the limitations of satellite technology and economic conditions, the products captured by a single satellite sensor cannot take into account both spatial resolution and temporal resolution at the same time. However, with the increasing demand for earth observation, many remote sensing applications have a strong demand for such high spatial resolution and high temporal resolution images.

[0003] Through image fusion technology, images with high temporal resolution and low spatial resolution and images with low temporal resolution and high spatial resolution can be fused to obtain composite images with high temporal and spatial resolution, which can meet the corresponding remote sensing application requirements to a certain extent. In the process of image fusion, it is necessary to consider important factors such as the differences between satellite sensors, changes in land object types over time, and changes in spatial resolution scale.

[0004] At present, scholars at home and abroad have achieved certain results in spatiotemporal fusion algorithms. Generally speaking, when synthesizing high spatiotemporal resolution images, one or more pairs of real high spatial resolution and low spatial resolution images at the same time are required, as well as a low spatial resolution image at the current prediction time. Through such a prediction model, researchers have developed remote sensing spatiotemporal fusion algorithms based on weight allocation, pixel unmixing, machine learning, etc., as well as hybrid algorithms based on the above principles. However, the current algorithms still have problems that are difficult to synthesize images in areas where the type of land objects mutates, the images generated after upsampling are not realistic enough, and the overall differences of sensors are difficult to solve. Summary of the invention

[0005] The present invention provides a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network, which is used to solve the defects in the prior art of remote sensing spatiotemporal fusion problems, such as the difficulty in predicting the sudden change area of ​​the type of land objects, the large overall difference caused by the sensor, and the insufficient reality of the generated image. The method realizes the generation of more realistic remote sensing images by utilizing a multi-level feature fusion strategy and adopting a generative adversarial network to generate images.

[0006] The present invention provides a multi-level feature spatiotemporal remote sensing image fusion based on a generative adversarial network, comprising:

[0007] Acquire a high-resolution real image of the target reference time, a low-resolution real image of the target reference time, and a low-resolution real image of the target prediction time;

[0008] Based on the target reference time high-resolution real image, the target reference time low-resolution real image, the target prediction time low-resolution real image and the trained conditional generator network, multi-level feature spatiotemporal remote sensing image fusion is performed to generate a target prediction time high-resolution image;

[0009] Wherein, the condition generator network includes an image feature extraction network, an image feature fusion network and an image reconstruction network;

[0010] The trained conditional generator network is obtained by a trained conditional generative adversarial network, and the trained conditional generative adversarial network is obtained by training through the following steps:

[0011] Constructing a sample set, wherein the sample set includes a high-resolution real image at a sample reference time, a low-resolution real image at a sample reference time, a low-resolution real image at a sample prediction time, and a high-resolution real image at a sample prediction time;

[0012] Through the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the sample prediction time high-resolution real image, the image feature extraction network, image feature fusion network and image reconstruction network of the conditional generator network in the conditional generative adversarial network, and the discriminator network are trained for multi-level feature spatiotemporal remote sensing image fusion according to the loss function to obtain a trained conditional generative adversarial network, which includes a trained conditional generator network and a trained discriminator network.

[0013] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention, the image feature extraction network, the image feature fusion network and the image reconstruction network of the conditional generator network in the conditional generative adversarial network, and the discriminator network are trained for multi-level feature spatiotemporal remote sensing image fusion according to the loss function through the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the sample prediction time high-resolution real image, to obtain a trained conditional generative adversarial network, including:

[0014] Generate a sample prediction time high-resolution estimation image based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the image feature extraction network, image feature fusion network and image reconstruction network of the conditional generator network;

[0015] generating a discrimination result based on the sample prediction time high-resolution estimated image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image and the discriminator network;

[0016] Calculate a loss function based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image, the sample prediction time high-resolution estimated image and the identification result;

[0017] The parameters of the conditional generator network and the parameters of the discriminator network are updated according to the loss function, and the process of generating a sample prediction time high-resolution estimated image based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the conditional generator network is returned and re-executed;

[0018] When a preset condition is met, the conditional generator network is determined as a trained conditional generator network, and the discriminator network is determined as a trained discriminator network, so as to obtain a trained conditional generative adversarial network.

[0019] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention, the image feature extraction network, image feature fusion network and image reconstruction network based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the conditional generator network include:

[0020] Based on the image feature extraction network, multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are respectively extracted;

[0021] Based on the image feature fusion network, the multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are fused to obtain fused multi-level features;

[0022] Based on the image reconstruction network, upsampling and reconstruction are performed according to the fused multi-level features and the multi-level features of the low-resolution real image at the sample prediction time to obtain a high-resolution estimated image at the sample prediction time.

[0023] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention, the image feature fusion network includes an adaptive instance normalization module and an attention mechanism module;

[0024] The multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the high-resolution real image at the sample reference time are globally fused by the adaptive instance normalization module to obtain an initial fused multi-level feature;

[0025] The adaptive instance normalization module performs global fusion through the following formula:

[0026]

[0027] Wherein, AdaIN(·) represents the initial fused multi-level features, σ(·) represents variance operation, μ(·) represents mean operation, x represents the multi-level features of the high-resolution real image at the sample reference time, and y represents the multi-level features of the low-resolution real image at the sample prediction time;

[0028] The attention mechanism module generates a saliency map based on the multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the low-resolution real image at the sample reference time, and local fusion is performed on the saliency map, the low-resolution real image at the sample prediction time and the initial fusion features to obtain a fused multi-level feature.

[0029] The attention mechanism module performs local fusion through the following formula:

[0030] C=MA+(1-M)B

[0031] Wherein, C is a single-layer feature constituting the fused multi-level feature, M is a single-layer saliency map, A is a single-layer feature of the low-resolution real image at the sample prediction time, and B is a single-layer feature in the initial fused multi-level feature.

[0032] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention, the loss function includes an overall accuracy loss function, a spectral loss function, a structural loss function, a conditional generative adversarial loss function and a discrimination loss function;

[0033] The loss function is calculated according to the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image, the sample prediction time high-resolution estimated image and the identification result, including:

[0034] The overall accuracy loss function, the spectral loss function, the structural loss function, the conditional generative adversarial loss function and the identification loss function are calculated based on the following formulas:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040] Among them, G(F t-1 ,C t-1 ,C t ) is the sample prediction time high-resolution estimation image, F t Predict the temporal high-resolution real image for the sample, F t-1 is the high-resolution real image of the sample reference time, F t Predict the temporal high-resolution real image for the sample, C t-1 is the low-resolution real image of the sample reference time, C t Predict the temporal low-resolution real image for the sample, K represents the number of pixels in the image, I represents the unit matrix, and v N , j and μ j represents the weight coefficient, l m ,c j and j denote brightness, contrast and structure comparison respectively, E denotes expectation, D denotes the discriminator network, D(·) denotes the identification result, G denotes the conditional generator network, represents the overall accuracy loss function, represents the spectral loss function, represents the structural loss function, represents the conditional generative adversarial loss function of the conditional generator network, represents the discrimination loss function of the discriminator network. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of multi-level feature spatiotemporal remote sensing image fusion based on a generative adversarial network provided by the present invention;

[0043] Figure 2 It is a schematic diagram of the conditional generative adversarial network training process provided by the present invention;

[0044] Figure 3 is a schematic diagram of a discriminator network provided by the present invention;

[0045] Figure 4 is a schematic diagram of a feature extraction network provided by the present invention;

[0046] Figure 5 is a schematic diagram of an adaptive instance normalization module provided by the present invention;

[0047] Figure 6 is a schematic diagram of the attention mechanism module provided by the present invention;

[0048] Figure 7 It is a schematic diagram of the image reconstruction network provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention provides a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network, comprising:

[0051] S11, obtaining a target reference time high-resolution real image, a target reference time low-resolution real image and a target prediction time low-resolution real image.

[0052] S12, input the target reference time high-resolution real image, the target reference time low-resolution real image, and the target prediction time low-resolution real image into the trained conditional generator network to perform multi-level feature spatiotemporal remote sensing image fusion to generate a target prediction time high-resolution image.

[0053] Wherein, the condition generator network includes an image feature extraction network, an image feature fusion network and an image reconstruction network.

[0054] Specifically, in one example, Figure 1As shown, the acquired reference time high-resolution image, reference time low-resolution image, and prediction time low-resolution image are input into a trained conditional generator network composed of an image feature extraction network, an image feature fusion network, and an image reconstruction network to perform multi-level feature spatiotemporal remote sensing image fusion. The process includes: inputting the reference time high-resolution image into the image feature extraction network to obtain the corresponding multi-level features, globally fusing the multi-level features of the prediction time low-resolution image and the multi-level features of the reference time high-resolution image through an adaptive instance normalization module to obtain initial fused multi-level features, locally fusing the multi-level features of the prediction time low-resolution image, the multi-level features of the reference time low-resolution image, and the initial fused features through an attention mechanism module to obtain fused features, and inputting the fused features into the image reconstruction network to obtain a fused prediction time high-resolution image.

[0055] The trained conditional generator network is obtained by a trained conditional generative adversarial network, and the trained conditional generative adversarial network is obtained by training through the following steps:

[0056] S21. Construct a sample set, wherein the sample set includes a high-resolution real image at a sample reference time, a low-resolution real image at a sample reference time, a low-resolution real image at a sample prediction time, and a high-resolution real image at a sample prediction time.

[0057] Specifically, the sample set includes high-resolution remote sensing satellite images and low-resolution remote sensing satellite images, wherein the high-resolution satellite images and the low-resolution satellite images need to correspond to each other in time, and the low-resolution images are upsampled to the same scale as the high-resolution images by bilinear interpolation to obtain the upsampled low-resolution remote sensing images. The high-resolution remote sensing satellite images and the upsampled low-resolution remote sensing images are taken out of two pairs of images at adjacent times to form a group of training images. A group of training images has four images, namely: a high-resolution real image at the sample reference time, a low-resolution real image at the sample reference time, a high-resolution real image at the sample prediction time, and a low-resolution real image at the sample prediction time. The training sample set is obtained after being cropped to a size that can be stored and operated by a computer.

[0058] In one example, the raw data of the high- and low-resolution remote sensing image sample sets are surface reflectance data from MODIS and Landsat, where MODIS surface reflectance data is used as a low-resolution image and Landsat surface reflectance data is used as a high-resolution image. The data set is divided into the CIA data set and the LGC data set according to the experimental range. The CIA data set is a summer crop growing area in Australia, with a range of 43 kilometers in the north-south direction and 51 kilometers in the east-west direction. The data pairs in the data set are Landsat-7 ETM+ and MODIS's MODTRAN4 products. The pixel size of Landsat-7 ETM+ is 1720×2040. In the image, all MODIS images are upsampled to the same size as Landsat. The CIA data set has a total of 17 pairs of images, and the first 11 pairs are used as training data in the present invention. The LGC data set is a flood area located in the back of New Wales, 80 kilometers in the north-south direction and 68 kilometers in the east-west direction. The data pairs in the data set are Landsat-5 TM images and MODIS MOD09GA product images. The pixel size of Landsat-5 TM is 3200×2720. Similarly, all MODIS images are upsampled to the same size as Landsat. The LGC dataset has a total of 14 pairs of images, and the present invention uses the first 9 pairs as training data. The input image size of the conditional generator network is set to 256×256 according to computer performance, and the images of the CIA dataset and the LGC dataset are divided into two pairs of high and low resolution images with adjacent time pairs as a group, and the images are cropped at an interval of 200 pixels. The cropped image pair set is the high and low resolution remote sensing image sample set.

[0059] S22. Through the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the sample prediction time high-resolution real image, the image feature extraction network, the image feature fusion network and the image reconstruction network of the conditional generator network, and the discriminator network are trained with multi-level feature spatiotemporal remote sensing image fusion according to the loss function to obtain a trained conditional generative adversarial network, which includes a trained conditional generator network and a trained discriminator network.

[0060] The present invention utilizes a generative adversarial network to synthesize higher spatial resolution images that lack temporal resolution. The introduction of a discriminator can produce more realistic generation results that are more similar to real Landsat images.

[0061] In the embodiment of the present invention, the generative adversarial network is composed of a conditional generator network and a discriminator network. The conditional generator network uses an image feature extraction network, an image feature fusion network, and an image reconstruction network to extract and fuse multi-level features, and relies on multi-level features to reconstruct images to handle the huge difference between high-resolution images and low-resolution images. In the feature fusion stage, an adaptive instance normalization module is designed to handle the global change relationship between multi-phase images, and an attention module is designed to handle local change relationships. After alternating training of the conditional generator network and the discriminator network, the trained conditional generator network can be used to solve the remote sensing spatiotemporal image fusion problem and generate a fused predicted time high-resolution image.

[0062] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention, step S22 specifically includes:

[0063] S221, based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the image feature extraction network, image feature fusion network and image reconstruction network of the conditional generator network, generate a sample prediction time high-resolution estimation image.

[0064] S222, generating an identification result based on the sample prediction time high-resolution estimated image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image and the discriminator network.

[0065] S223, calculate the loss function based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image, the sample prediction time high-resolution estimated image and the identification result.

[0066] S224 updates the parameters of the conditional generator network and the parameters of the discriminator network according to the loss function, and returns to re-execute the step of inputting the sample reference time high-resolution real image, the sample reference time low-resolution real image, and the sample prediction time low-resolution real image into the conditional generator network to generate a sample prediction time high-resolution estimated image.

[0067] S225 determines the conditional generator network as a trained conditional generator network and the discriminator network as a trained discriminator network when the preset conditions are met, so as to obtain a trained conditional generative adversarial network.

[0068] In an example, Figure 2As shown, the training set is divided into an input training set and an output training set. The input training set consists of a sample reference time high-resolution real image, a sample reference time low-resolution real image, and a sample prediction time low-resolution real image. The output training set consists of a sample real reference time low-resolution image and a sample real reference time high-resolution image. The images in the input training set are input into the conditional generator network to obtain a fused image, i.e., a sample prediction time high-resolution estimated image. The images in the output training set are taken as the true result, the fused image and the sample prediction time low-resolution real image in the output training set are taken as a set of input data, the sample reference time low-resolution real image and the sample reference time high-resolution real image in the output training set are taken as a set of input data, and the two sets of input data are input into the discriminator network to obtain the identification result.

[0069] Furthermore, the loss function may include an overall accuracy loss function, a spectral loss function, a structural loss function, a conditional generative adversarial loss function, and a discriminative loss function. The training process is as follows:

[0070] The overall accuracy loss function, spectral loss function and structural loss function can be calculated by the sample prediction time high-resolution estimation image and the sample prediction time high-resolution true image. The overall accuracy loss function, spectral loss function and structural loss function are used to represent the similarity between the generated sample prediction time high-resolution estimation image and the actual sample prediction time high-resolution true image.

[0071] The overall accuracy loss function, spectral loss function, and structural loss function can be determined by the following formulas 1-3.

[0072]

[0073]

[0074]

[0075] Among them, G(F t-1 ,C t-1 ,C t ) is the sample prediction time high-resolution estimation image, F t For the sample prediction time, high resolution real image, F t-1 is the high-resolution real image at the sample reference time, F t For sample prediction time, high resolution real image, C t-1 is the low-resolution real image at the sample reference time, C t is the real low-resolution image for sample prediction time, K represents the number of pixels in the image, I represents the unit matrix, and v N , j and μ jrepresents the weight coefficient, l m ,c j and j They represent brightness, contrast, and structure comparison, respectively.

[0076] The sample prediction time high-resolution estimated image and the sample prediction time high-resolution real image are input into the discriminator network to obtain the identification result, and the conditional generative adversarial loss function is determined according to the identification result. The conditional generative adversarial loss function represents the ability of the conditional generator network to generate images that can mislead the discriminator network to make wrong judgments.

[0077] The conditional generative adversarial loss function can be determined by the following formula 4.

[0078]

[0079] in, represents the conditional generative adversarial loss function for the conditional generator network, F t-1 is the high-resolution real image of the sample reference time, F t Predict the temporal high-resolution real image for the sample, C t-1 is the low-resolution real image of the sample reference time, C t Predict a temporal low-resolution real image for the sample, E represents the expectation, D represents the discriminator network, D(·) represents the identification result, and G represents the conditional generator network. G(F t-1 ,C t-1 ,C t ) represents the sample prediction time high-resolution estimated image obtained by inputting the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image into the conditional generator network. D(G(F t-1 ,C t-1 ,C t ),C t ) represents the identification result obtained after the sample prediction time high-resolution estimated image and the sample prediction time high-resolution true image are input into the discriminator network.

[0080] Update the parameters of the conditional generator network according to the overall accuracy loss function, spectral loss function, structural loss function and conditional generative adversarial loss function.

[0081] After that, a set of images in the input training set is input into the condition generator network to obtain a fused image, i.e., a sample prediction time high-resolution estimated image, the fused image and the sample prediction time low-resolution real image are input into the discriminator network to obtain a first identification result, and the sample prediction time low-resolution real image and the sample prediction time high-resolution real image are input into the discriminator network to obtain a second identification result. The identification loss function is determined according to the first identification result and the second identification result. The parameters of the discriminator network are updated according to the identification loss function.

[0082] The identification loss function can be determined by the following formula 5.

[0083]

[0084] in, represents the identification loss function for the discriminator network, F t-1 is the high-resolution real image at the sample reference time, F t For sample prediction time, high resolution real image, C t-1 is the low-resolution real image at the sample reference time, C t is the real low-resolution image for sample prediction time, K represents the number of pixels in the image, I represents the unit matrix, ν N , j and μ j represents the weight coefficient, l m ,c j and j They represent brightness, contrast and structure comparison respectively, E represents expectation, D represents the discriminator network, G represents the conditional generator network, G(F t-1 ,C t-1 ,C t ) represents the sample prediction time high-resolution estimated image obtained by inputting the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image into the conditional generator network. D(G(F t-1 ,C t-1 ,C t ),C t ) represents the first identification result obtained after inputting the sample prediction time high-resolution estimated image and the sample prediction time high-resolution real image into the discriminator network, D(F t ,C t ) represents the second identification result obtained after the sample prediction time high-resolution real image and the sample prediction time low-resolution real image are input into the discriminator network.

[0085] In an example, Figure 3As shown in the figure, the discriminator network is a binary classification network consisting of a four-layer convolutional block network, two layers of convolution operations, a layer of LeakyReLU activation function, and a layer of Sigmoid activation function. The convolution block consists of a convolution layer with a convolution kernel size of 4, a step size of 2, an edge padding width of 1, a normalization layer, and a LeakyReLU activation function with a negative slope of 0.01; the convolution consists of a convolution layer with a convolution kernel size of 4, a step size of 2, an edge padding width of 1, and a LeakyReLU activation function with a negative slope of 0.01. The discriminator divides the image to be identified into blocks and identifies the true probability of each part. The input is a low-resolution real image at the sample prediction time and a high-resolution real image at the sample prediction time, or the input is a low-resolution real image at the sample prediction time and a high-resolution estimated image at the sample prediction time. The output result is a 0-1 matrix that identifies whether the local image pair is true and whether the image pair is from the same area. The local range in the local image pair is determined by the receptive field of the discriminator network, and the image pair refers to a pair of high- and low-resolution images at the same time, which is also the image data for training the discriminator network. The 0-1 matrix is ​​a probability evaluation matrix, 0 represents that the discriminator network believes that the partial image pair is not a pair of high- and low-resolution images at the same time, and 1 represents that the partial image pair is a pair of high- and low-resolution images at the same time.

[0086] After completing the update of the parameters of the conditional generator network and the discriminator network once through the above steps, the above training process can be cyclically trained according to the overall accuracy loss function, the spectral loss function, the structural loss function, the conditional generative adversarial loss function and the discrimination loss function, and the parameters of the conditional generator network and the discriminator network are continuously updated. When the preset conditions are met, it is determined that the conditional generator network and the discriminator network have reached a stable result, and the training is terminated to obtain a trained conditional generator network and a trained discriminator network. The trained conditional generator network and the trained discriminator network constitute a trained conditional generative adversarial network.

[0087] In one example, training a conditional generative adversarial network using a sample set includes fixing network training parameters and setting preset conditions. For the conditional generator network, when training based on the loss function, the weight of the conditional generative adversarial loss function is set to 1×10 -2 , the weights of the overall accuracy loss function, spectral loss function, and structural loss function are all 1; for the discriminator network, training is performed based on the discrimination loss function. During the training process, the learning rates of the conditional generative network and the discriminator network are both 2×10 -4 The preset condition is that the number of generations of training for the entire training set is 200. After 200 generations of training, a conditional generative adversarial network with fixed parameters is obtained.

[0088] In an embodiment of the present invention, a loss function is calculated based on multiple images, a condition generator network, and a discriminator network in a sample set, and the condition generator network and the discriminator network are trained and updated based on the loss function. Through the above steps, the condition generator network and the discriminator network can be fully trained to obtain a condition generator network that can generate relatively realistic predicted images and a discriminator network with good discrimination ability, which facilitates the subsequent multi-level feature spatiotemporal remote sensing image fusion based on the trained condition generator network to generate relatively realistic predicted images.

[0089] According to a multi-level feature spatiotemporal remote sensing image fusion method based on a generative adversarial network provided by the present invention,

[0090] Step S221 specifically includes:

[0091] S31. Based on the image feature extraction network, multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are respectively extracted.

[0092] Specifically, in one example, Figure 4 As shown in Fig. 1, the image feature extraction network can be composed of residual convolution blocks, down-sampling residual convolution blocks and convolution operations. The input is an image and the output is a multi-level feature of the image. The network structure is as follows: Figure 3 As shown. The residual convolution block is divided into four layers, namely: convolution layer with kernel size of 3, stride of 1, and edge padding width of 1; LeakyReLU activation function with negative slope of 0.01; convolution layer with the same parameters; activation function with the same parameters. The downsampled residual convolution block is based on the residual convolution block, and adds two layers at the head of the residual convolution block, namely: convolution layer with kernel size of 3, stride of 2, and edge padding width of 1; LeakyReLU activation function with negative slope of 0.01. The convolution operation is mainly placed before the image input process to preliminarily extract image features, and consists of a convolution layer with kernel size of 7, stride of 1, and edge padding width of 3. The feature extraction module consists of convolution operation, residual convolution block, and four-layer downsampled residual convolution block. In the feature extraction module, the residual convolution block and the four-layer downsampled residual convolution block output five single-layer features of different scales, and the five single-layer features constitute the multi-level features of the corresponding image. The image feature extraction network is divided into an image feature extraction network for high-resolution images and an image feature extraction network for low-resolution images. The network structures are the same but the parameters are different during the training process.

[0093] S32. Based on the image feature fusion network, the multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are fused to obtain fused multi-level features.

[0094] Furthermore, the image feature fusion network includes an adaptive instance normalization module and an attention mechanism module, and step S32 may specifically include steps S321-S322.

[0095] S321. Globally fuse the multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the high-resolution real image at the sample reference time through the adaptive instance normalization module to obtain initial fused multi-level features.

[0096] S322. Generate a saliency map according to the multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the low-resolution real image at the sample reference time through the attention mechanism module, and perform local fusion on the saliency map, the low-resolution real image at the sample prediction time and the initial fusion features to obtain fused multi-level features.

[0097] Specifically, the feature fusion network may include an adaptive instance normalization module and an attention mechanism module, the input of which is multi-level features of different images, and each layer of features in the multi-level features extracted by the feature extraction network is operated separately, and the fused multi-level features are output.

[0098] Adaptive instance normalization modules, such as Figure 5 As shown in the figure, the input is a single-layer feature in the multi-level feature of the high-resolution real image at the sample reference time and a single-layer feature in the multi-level feature of the low-resolution real image at the sample prediction time. The statistical features of the latter are superimposed on the former, and global fusion is performed to obtain the initial fused single-layer feature. Multiple initial fused single-layer features constitute the initial fused multi-level feature.

[0099] The Adaptive Instance Normalization (AdaIN) module transfers the global information in the low-resolution real image features at the sample prediction time to the high-resolution real image features at the sample reference time, so as to unify the global features of the high-resolution real image at the sample reference time with more spatial information and the low-resolution real image at the sample prediction time. The Adaptive Instance Normalization module performs global fusion through the following formula 6:

[0100]

[0101] Wherein, AdaIN(·) represents the initial fused multi-level features, σ(·) represents variance operation, μ(·) represents mean operation, x represents the high-resolution real image at the sample reference time, and y represents the low-resolution real image at the sample prediction time.

[0102] Attention mechanism modules, such as Figure 6As shown in the figure, the attention mechanism module consists of two convolution layers with a convolution kernel size of 3, a stride of 1, and an edge filling width of 1; the LeakyReLU activation function with a negative slope of 0.01 is used as a convolution block to process the single-layer features in the multi-level features of the reference time low-resolution image and the single-layer features in the multi-level features of the predicted time low-resolution image, merge the two single-layer features, put them into the convolution block of the same structure again, connect the Sigmoid layer, and output the saliency map. The saliency map guides the fusion of the single-layer features in the initial fusion multi-level features and the single-layer features in the multi-level features of the predicted time low-resolution image.

[0103] The features of the low-resolution real image at the sample prediction time and the features of the low-resolution real image at the sample reference time are compared, and a saliency map guiding fusion is output through convolution operation; the saliency map is used to provide fusion of the fusion features obtained after adaptive instance normalization and the features of the low-resolution real image at the sample prediction time. The attention mechanism module generates a saliency map based on the multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the low-resolution real image at the sample reference time, and local fusion is performed based on the saliency map, the low-resolution real image at the sample prediction time and the initial fusion features to obtain a fused multi-level feature. The attention mechanism module performs local fusion through the following formula 7:

[0104] C=MA+(1-M)B (7)

[0105] Where C is the single-layer feature that constitutes the fused multi-level feature, M is the single-layer saliency map, A is the single-layer feature of the low-resolution real image at the sample prediction time, and B is the single-layer feature in the initial fused multi-level feature obtained after adaptive instance normalization.

[0106] The present invention designs an adaptive instance normalization (AdaIN) module and an attention mechanism module for the spatiotemporal fusion problem, which can more effectively solve the spatiotemporal fusion problem in terms of time span, changes in ground object type and spectrum. The AdaIN module solves this problem as a whole, while the attention mechanism module focuses more on local problems.

[0107] S33, based on the image reconstruction network, upsampling and reconstruction are performed according to the fused multi-level features and the multi-level features of the low-resolution real image at the sample prediction time to obtain a high-resolution estimated image at the sample prediction time.

[0108] Specifically, the image reconstruction network is symmetrical to the feature extraction network structure, and is composed of convolution blocks, convolution operations, and up-sampled residual convolution blocks. The input is the largest-scale layer of features in the low-resolution real image at the sample prediction time output by the feature extraction network and the fused multi-level features output by the feature fusion network. The output is the generated high-resolution image at the prediction time, that is, the high-resolution estimated image at the sample prediction time. The convolution blocks, convolution operations, and feature extraction network structures are the same. The up-sampled residual convolution is a down-sampled residual convolution block that replaces the convolution layer with a convolution kernel size of 3, a step size of 2, and an edge padding width of 1 with a deconvolution layer with a convolution kernel size of 3, a step size of 2, and an edge padding width of 1. The structure of the image reconstruction network consists of four up-sampled convolution modules, a residual module, a convolution operation, a residual module, and a convolution operation. In an example, Figure 7 As shown, the image reconstruction network can be a decoder. The decoder and the encoder have basically symmetrical structures, including four layers of upsampling convolution modules, each layer consisting of multiple residual modules, and the output end of each layer is a deconvolution layer with a step size of 2. Each layer outputs a scale feature corresponding to the encoder, which is combined with the fused multi-level features to output the generated prediction time high-resolution image, that is, the sample prediction time high-resolution estimation image.

[0109] Based on the above steps S31-S33, the conditional generator network extracts multi-level features based on the sample reference time high-resolution real image, the sample prediction time low-resolution real image and the sample reference time low-resolution real image input into the image feature network, inputs the multi-level features into the corresponding adaptive instance normalization module and the attention mechanism module output in the image feature fusion network to obtain the fused multi-level features, and the image reconstruction network generates the sample prediction time high-resolution estimated image based on the fused multi-level features and the multi-level features of the sample prediction time low-resolution real image.

[0110] In the disclosed embodiment, the condition generator network includes an image feature extraction network, an image feature fusion network and an image reconstruction network. The image feature fusion network includes an adaptive instance standardization module and an attention mechanism module. The multi-level features of the input image are obtained through the image feature extraction network. The adaptive instance standardization module and the attention mechanism module are used to solve the time span, the type of ground objects and the spectral changes in the spatiotemporal fusion problem, realize the fusion of image features, and generate predicted high-resolution images through the image reconstruction network. In this way, a relatively realistic remote sensing image is generated by using a multi-level feature fusion strategy and a generative adversarial network to generate an image technology.

[0111] Furthermore, steps S11-S12 are steps for actually applying the trained conditional generator network, which are similar to the method flow of steps S31-S33 for training the conditional generator network and will not be repeated here. Please refer to the description of steps S31-S33.

[0112] In order to further illustrate the feasibility and effectiveness of the method of the present invention, commonly used MAE, RMSE, SAM and SSIM are usually used to evaluate the quantitative results of the generated images.

[0113] Table 1 shows the calculation results of MAE (mean absolute error), RMSE (root mean square error), SAM (spectral angle) and SSIM (structural similarity) of the six methods for the CIA dataset. Table 2 shows the calculation results of MAE, RMSE, SAM and SSIM of the six methods for the LGC dataset. The five methods are: flexible spatiotemporal fusion (FSDAF), spatiotemporal fusion enhanced deep convolutional model (EDCSTFN), flexible remote sensing image spatiotemporal fusion model based on conditional generative adversarial network (GAN-STFM), explicit multi-stage model of remote sensing image spatiotemporal fusion based on deep neural network (SSTSTF), sensor bias-driven spatiotemporal fusion model based on convolutional neural network (BiaSTF) and the multi-level feature fusion remote sensing spatiotemporal fusion method based on generative adversarial network (MLFF-GAN) of the present invention.

[0114] Table 1 Comparison of results with other algorithms on CIA dataset

[0115]

[0116] Among them, the invention marked with * in the upper right corner has different prediction dates. The input of BiaSTF is 5 images, and the input of other methods is 3 images. The predicted date is one day less than that of other images, so the results are different.

[0117] Table 2 Comparison of results with other algorithms on the LGC dataset

[0118]

[0119]

[0120] Among them, the invention marked with * in the upper right corner has different prediction dates. The input of BiaSTF is 5 images, and the input of other methods is 3 images. The predicted date is one day less than that of other images, so the results are different.

[0121] Table 1 and Table 2 both show the performance comparison of different methods with four evaluation indicators. In general, these indicators prove that the method proposed in the present invention can further improve the performance, especially the unique feature processing method, which makes the present method more competitive. MAE and RMSE values ​​show that these methods can roughly predict the generated image, but the present method significantly improves the overall generation effect through the processing of global features and local features by the adaptive instance normalization module and the attention mechanism module, and obtains the best indicator value. The SAM value shows that the present method performs better in spectral preservation, thanks to the processing of global features by the present method in the adaptive instance normalization module. The SSIM value shows that the present method performs better in structural preservation, and has experimental results that are basically the same as those of real remote sensing images, thanks to the use of conditional generative adversarial networks in the present method to generate images. The present invention has certain advantages in the current spatiotemporal fusion algorithm, and has relative advantages in the four evaluation indicators of root mean square error, mean absolute error, spectral angle and structural similarity.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-level feature spatiotemporal remote sensing image fusion method based on generative adversarial network, characterized in that: include: Acquire a high-resolution real image of the target reference time, a low-resolution real image of the target reference time, and a low-resolution real image of the target prediction time; Input the target reference time high-resolution real image, the target reference time low-resolution real image, and the target prediction time low-resolution real image into the trained conditional generator network to perform multi-level feature spatiotemporal remote sensing image fusion to generate a target prediction time high-resolution image; Wherein, the condition generator network includes an image feature extraction network, an image feature fusion network and an image reconstruction network; The trained conditional generator network is obtained by a trained conditional generative adversarial network, and the trained conditional generative adversarial network is obtained by training through the following steps: Constructing a sample set, wherein the sample set includes a high-resolution real image at a sample reference time, a low-resolution real image at a sample reference time, a low-resolution real image at a sample prediction time, and a high-resolution real image at a sample prediction time; Through the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the sample prediction time high-resolution real image, according to the loss function, the image feature extraction network, the image feature fusion network and the image reconstruction network of the conditional generator network, and the discriminator network are trained with multi-level feature spatiotemporal remote sensing image fusion to obtain a trained conditional generative adversarial network, wherein the trained conditional generative adversarial network includes a trained conditional generator network and a trained discriminator network; The method uses the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the sample prediction time high-resolution real image to perform multi-level feature spatiotemporal remote sensing image fusion training on the image feature extraction network, the image feature fusion network and the image reconstruction network of the conditional generator network and the discriminator network according to the loss function to obtain a trained conditional generative adversarial network, including: Generate a sample prediction time high-resolution estimation image based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image, and the image feature extraction network, image feature fusion network, and image reconstruction network of the conditional generator network; generating a discrimination result based on the sample prediction time high-resolution estimated image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image and the discriminator network; Calculate a loss function based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image, the sample prediction time high-resolution real image, the sample prediction time high-resolution estimated image and the identification result; The parameters of the condition generator network and the parameters of the discriminator network are updated according to the loss function, and the process of inputting the sample reference time high-resolution real image, the sample reference time low-resolution real image, and the sample prediction time low-resolution real image into the condition generator network is returned and re-executed to generate a sample prediction time high-resolution estimated image; When a preset condition is met, the conditional generator network is determined as a trained conditional generator network, and the discriminator network is determined as a trained discriminator network, so as to obtain a trained conditional generative adversarial network.

2. The multi-level feature spatiotemporal remote sensing image fusion method based on generative adversarial network according to claim 1 is characterized in that: The image feature extraction network, image feature fusion network and image reconstruction network based on the sample reference time high-resolution real image, the sample reference time low-resolution real image, the sample prediction time low-resolution real image and the conditional generator network include: Based on the image feature extraction network, multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are respectively extracted; Based on the image feature fusion network, the multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image are fused to obtain fused multi-level features; Based on the image reconstruction network, upsampling and reconstruction are performed according to the fused multi-level features and the multi-level features of the low-resolution real image at the sample prediction time to obtain a high-resolution estimated image at the sample prediction time.

3. The multi-level feature spatiotemporal remote sensing image fusion method based on generative adversarial network according to claim 2 is characterized in that: The image feature fusion network includes an adaptive instance normalization module and an attention mechanism module; The method of fusing the multi-level features of the sample reference time high-resolution real image, the sample reference time low-resolution real image and the sample prediction time low-resolution real image based on the image feature fusion network to obtain the fused multi-level features includes: The multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the high-resolution real image at the sample reference time are globally fused by the adaptive instance normalization module to obtain an initial fused multi-level feature; The adaptive instance normalization module performs global fusion through the following formula: Wherein, AdaIN(·) represents the initial fused multi-level features, σ(·) represents variance operation, μ(·) represents mean operation, x represents the multi-level features of the high-resolution real image at the sample reference time, and y represents the multi-level features of the low-resolution real image at the sample prediction time; Generate a saliency map according to the multi-level features of the low-resolution real image at the sample prediction time and the multi-level features of the low-resolution real image at the sample reference time through the attention mechanism module, and locally fuse the saliency map, the low-resolution real image at the sample prediction time and the initial fused multi-level features to obtain a fused multi-level feature; The attention mechanism module performs local fusion through the following formula: C=MA+(1-M)B In the formula, C is a single-layer feature constituting the fused multi-level feature, M is a single-layer saliency map, A is a single-layer feature of the low-resolution real image at the sample prediction time, and B is a single-layer feature in the initial fused multi-level feature.

Images