A few-shot remote sensing spatio-temporal fusion method and system based on degenerate fusion cascade network

Abstract

The application discloses a few-shot remote sensing spatio-temporal fusion method and system based on a degenerate fusion cascade network. In the first stage, a degenerate network is designed to learn the complex degenerate relationship between the high spatial resolution image to be fused and the low resolution image, instead of constructing a training set by traditional interpolation down-sampling. The high spatial resolution image is taken as input, and the low spatial resolution image is taken as a label to train the degenerate network. After the training is completed, the low resolution image is input into the degenerate network to obtain a low spatial resolution degenerate image. Then, a fusion network based on a cycle consistency generative adversarial network is trained by taking the low resolution image and the low resolution degenerate image as training data. The low resolution degenerate image is taken as input, and the low resolution image is taken as a label to train the fusion network. After the training is completed, the low resolution image is input into the fusion network to obtain a final fused high resolution image. The method has a lower requirement for the number of training samples, high fusion precision, and certain universality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and particularly relates to a few-sample remote sensing spatio-temporal fusion method based on a degeneration fusion cascade network. BACKGROUND

[0002] Due to the limitation of the physical characteristics of satellite sensors, the time resolution and the spatial resolution of single satellite remote sensing images are often mutually restricted. The so-called time resolution refers to the minimum time interval between two adjacent remote sensing observations of the same place on the earth by a satellite. The spatial resolution refers to the size or size of the smallest unit that can be distinguished in detail on a remote sensing image, which is an index for representing the details of the ground target distinguished by the image. In addition, the imaging process is easily affected by external factors such as clouds and self factors such as damage to the sensor itself, resulting in a limited number of available high-resolution images, which greatly affects the application such as dynamic monitoring, change detection, and ground feature type identification. In view of the above problems, the spatio-temporal fusion method of remote sensing images can be used to fuse the high spatial and low temporal resolution images of different satellite sensors with the low spatial and high temporal resolution images to obtain high spatial and high temporal resolution images, thereby fully improving the spatio-temporal availability of high-quality remote sensing images and fully utilizing and mining the useful information contained in remote sensing images.

[0003] At present, the existing spatio-temporal fusion methods can be basically divided into traditional methods and deep learning methods. The former includes methods based on weight functions and methods based on mixed pixel decomposition. These methods have simple algorithms and fast calculation speeds, but have limitations such as linear hypothesis. The deep learning method has the advantages of strong non-linear feature extraction and expression, and its fusion effect is better than that of most traditional fusion methods. The deep learning spatio-temporal fusion method is mainly divided into supervised learning, self-supervised learning or unsupervised learning methods. The supervised learning method needs a large amount of data for training to ensure the universality of the model, but when there is a large difference between the training data and the features of the data to be fused, i.e. the network trained is not suitable for the data to be fused, the fusion effect is poor. The self-supervised learning method generally reduces the sampling of the original image by linear interpolation method and then constructs a training set with the original image as the label data for training. However, due to the large difference in image style between high-resolution satellite images and low-resolution satellite images from different satellite sensors, the ordinary linear interpolation method cannot well simulate the complex degeneration relationship between the two, which also affects the fusion effect, resulting in a large difference between the fusion result and the reference result. In addition, most deep learning spatio-temporal fusion methods require more samples or multiple reference time data for training, which has a high requirement for data. The present method is a few-sample spatio-temporal fusion method, which only needs a pair of high and low resolution images at one reference time, has a lower requirement for data and has a higher fusion precision, and can better serve practical applications. SUMMARY

[0004] To solve the problems in the prior art, the application provides a few-shot remote sensing spatio-temporal fusion method based on a degradation fusion cascade network. Firstly, in view of the problem that the interpolation downsampling mode used in the existing deep learning spatio-temporal fusion method based on a self-supervised learning strategy may not be able to well simulate the complex degradation relationship between high spatial resolution remote sensing images and low spatial resolution remote sensing images from different remote sensors, a degradation network is designed in the first stage to learn the complex degradation relationship between the high spatial resolution image to be fused and the low resolution image by means of the powerful nonlinear learning ability of a deep neural network, so as to replace the traditional interpolation downsampling to construct a training set. The high spatial resolution image is taken as input, and the low spatial resolution image is taken as a label to train the degradation network. After the training is completed, the low resolution image is input into the degradation network to obtain a low spatial resolution degraded image. Then, a fusion network based on a cycle consistency generative adversarial network is trained by taking the low resolution image and the low resolution degraded image as training data, and the low resolution degraded image is taken as input and the low resolution image is taken as a label to train the fusion network. After the training is completed, the low resolution image is input into the fusion network to obtain a final fused high resolution image. The method only needs a pair of high and low spatial resolution images at a single reference time, has a low requirement on the number of training samples, has high fusion precision, and has certain universality. The method comprises the following steps:

[0005] Step 1, first, a degradation network is constructed and trained, and the degradation network is composed of a feature extraction and coding module and a degradation decoding module; in the degradation network training stage, the high spatial resolution image at the reference time t0 and the low spatial resolution image are taken as training data;

[0006] Step 2, in the degradation network test stage, the high resolution image at the reference time, the low resolution image at the reference time and the low resolution image at the predicted time are input into the trained degradation network to obtain respective degraded images The degraded images and the low resolution image at the predicted time jointly constitute a training set of a main fusion network in the next stage, and t1 is the predicted time;

[0007] Step 3, the main fusion network is built based on a cycle consistency generative adversarial network (Cycle-GAN) framework, and the main fusion network comprises a forward generator a backward feedback generator and two corresponding discriminators ​In the main fusion network training stage, the forward generator generates the high-resolution image at the prediction time by fusing the high-resolution image at the reference time and the low-resolution image at the prediction time, the backward feedback generator is used to ensure that the high-resolution image generated by the forward generator contains information constraints on the original input image, and the high-resolution image generated by the forward generator is used to obtain the original input high-resolution image, and the discriminator is used to extract the input image features to determine the authenticity of the input image, and the generator is used to improve the performance of each other.

[0008] Step 4, in the main fusion network test stage, the final prediction time fusion image is obtained by inputting the original to-be-fused image into the trained forward generator.

[0009] Further, in step 1, the feature extraction and encoding module first extracts shallow features through the first layer of convolution, and then extracts multi-level features through a plurality of activation, convolution and normalization combined layers, and the degradation decoding module generates degradation results step by step through a plurality of activation, deconvolution and normalization combined layers, and the training set required by the fusion network is constructed by data enhancement, and the multi-layer convolution and deconvolution are composed.

[0010] Further, in step 1, the specific process of the degradation network training stage is as follows: the high-resolution image at the reference time is taken as the input, the low-resolution image at the reference time is taken as the label data for training, and the process is represented by the following formula:

[0011]

[0012] Among them, is the false reference time low image generated by the network in the training process, N D represents the degradation network generation process, θ D represents the trainable parameters of the degradation network; after the training stage, a trained degradation network capable of expressing the degradation relationship of high and low spatial resolution images from different sensors is obtained, wherein M, N and B represent the length, width and band number of the image, and s is the scale factor of the high-resolution image and the low-resolution image.

[0013] Further, the implementation process of the degradation network test stage in step 2 is represented by the following formula:

[0014]

[0015] Among them, is the reference time high-resolution image is the degraded image obtained after the reference time high-resolution image is input into the degradation network, is the reference time and prediction time low-resolution image ​​The degraded images obtained after being input into the degradation network are shown below, where M, N, and B represent the image length, width, and number of bands, and s is the scale factor for high-resolution and low-resolution images.

[0016] Furthermore, in step 3, the two generators It consists of a shallow feature extraction module, a deep residual module, and a fusion generation module. The shallow feature extraction module consists of multiple convolutional layers, padding layers, and activation layers to extract shallow features. The deep residual module consists of multiple residual blocks to extract deep features. The fusion module also consists of convolutional layers, activation layers, and padding layers to generate the fused result; two discriminators are also included. It consists of multiple convolutional layers, activation layers, and normalization layers.

[0017] Furthermore, the specific implementation method of step 3 is as follows:

[0018] Step 3.1, obtain the degenerate network The input is fed into the forward generator to obtain the corresponding predicted high-resolution image at time t1. It can be expressed as the following formula:

[0019]

[0020] in, Fusion images of targets at predicted time points generated during training. These are the trainable parameters of the forward generator;

[0021] Step 3.2, take the generated in step 3.1 Its reference label image Input to forward discriminator Generated by the discriminator constraint The quality;

[0022] Step 3.3, the product generated in step 3.1 and The original high-resolution image is fed into the backfeedback generator to obtain the prediction. To constrain the consistency between the spatial information of the generated fused image and the spatial information of the original high spatial resolution image, it is expressed as:

[0023]

[0024] in, The original high-resolution imagery is generated as the prediction source during the training process. These are the trainable parameters for the backfeedback generator;

[0025] Step 3.4, the product generated in step 3.3 Its reference label image input to the backward feedback discriminator , the quality of the generated is constrained by the discriminator.

[0026] Further, in the degenerated network training stage, the target loss function adopted is as follows:

[0027]

[0028] Further, in the main fusion network training stage, the total target loss function adopted is:

[0029] L F = L adv + λL content + δL consist (18)

[0030] wherein, L adv represents the generative adversarial loss, L content represents the content loss, i.e. the difference loss between the generated image and the reference image, L consist represents the consistency constraint loss, which is used to constrain the fusion image to retain more spatial features of the input image, λ and δ represent weight parameters for weighing the contributions of the content loss and the consistency constraint loss, and the generative adversarial loss is defined as follows:

[0031] L F = L adv + λL content + δL consist (19)

[0032] wherein each term is defined as:

[0033]

[0034]

[0035]

[0036] wherein, and respectively represent the generative adversarial loss of the fusion process from the reference time to the prediction time and the process of generating the original input image from the fusion image at the prediction time to the reference time in reverse;

[0037] The content loss and the consistency constraint loss are defined as follows:

[0038]

[0039]

[0040] The application also provides a few-sample remote sensing space-time fusion system based on a degenerative fusion cascade network, comprising the following modules.

[0041] A degenerative network construction module, which firstly constructs and trains a degenerative network, wherein the degenerative network is jointly constituted by a feature extraction and coding module and a degenerative decoding module; in the degenerative network training stage, high spatial resolution images of a reference time t0 and low spatial resolution images are taken as training data;

[0042] A degenerative network testing module, which is used for inputting high resolution images and low resolution images of the reference time and low resolution images of a prediction time into the trained degenerative network to obtain respective degenerative images in the degenerative network testing stage, wherein the degenerative images and the low resolution images of the prediction time jointly constitute a training set of a main fusion network in the next stage, and t1 is the prediction time;

[0043] A main fusion network construction module, wherein the main fusion network is built based on a Cycle-GAN framework, and the main fusion network comprises a forward generator , a backward feedback generator and two corresponding discriminators In the main fusion network training stage, the forward generator is used to generate high resolution images of the prediction time by fusing the high resolution images of the reference time and the low resolution images of the prediction time, the backward feedback generator is used to ensure that the high resolution images generated by the forward generator contain information constraints on the original input images, the original input high resolution images are obtained by using the high resolution images generated by the forward generator, and the discriminators are used to extract input image features to determine the authenticity of the input images, so as to improve the performance of the generators.

[0044] A main fusion network testing module, which is used for inputting original to-be-fused images into the trained forward generator to obtain final prediction time fusion images in the main fusion network testing stage.

[0045] Compared with the prior art, the application has the following advantages and beneficial effects:

[0046] 1. The application provides a space-time fusion method based on a deep learning degenerative network, which firstly learns the complex mapping relationship between high and low spatial resolution images of different remote sensors by deep learning to replace the interpolation down-sampling method in traditional self-supervised learning to construct training data, so that the training data is more consistent with the actual test data, and the fusion quality is improved.

[0047] ​​2, adopt the cycle consistency generative adversarial network as the basic framework of the main fusion network, can better keep the spatial feature information of the generated high-resolution image. In terms of loss function, in addition to the basic generative adversarial loss, the content loss and consistency constraint loss are added to further constrain the fusion result;

[0048] 3, the spatio-temporal fusion method based on small sample of the application, the requirement of sample number is lower, can be applied to the spatio-temporal fusion in more extreme cases (such as only three images of reference time high resolution, low resolution image and predicted time low resolution image), has higher practical application value;

[0049] 4, through visual, quantitative comparison experiment shows that, compared with other representative methods of spatio-temporal fusion at present stage, the application has higher fusion precision. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the total flow chart of the application.

[0051] Figure 2 is the training and testing flow chart of the degradation network stage of the application.

[0052] Figure 3 is the training flow chart of the main fusion network of the application.

[0053] Figure 4 is the main fusion network structure diagram of the application.

[0054] Figure 5 is the fusion result display diagram of the application. DETAILED DESCRIPTION

[0055] In order to facilitate those skilled in the art to understand and implement the application, the application will be further described in detail below in combination with the drawings and implementation examples, and it should be understood that the implementation examples described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0056] The technical scheme adopted by the application is: a small sample remote sensing spatio-temporal fusion method based on degradation fusion cascade network, for a given single reference time t0 high resolution image and low resolution image , and predicted time t1 low resolution image , wherein M, N and B represent the length, width and band number of the image, s is the scale factor of the high resolution image and the low resolution image, and the predicted time high resolution image is obtained by fusion, comprising the following steps:

[0057] I. Degradation network stage

[0058] 1、Image degradation network uses the powerful feature extraction and learning advantage of deep learning to simulate the complex degradation relationship between high spatial resolution images and low spatial resolution images to replace the traditional simple interpolation downsampling method. The training set required by the fusion network is constructed by data enhancement. The network structure is shown in Figure 2 The image degradation network is composed of a feature extraction and coding module and a degradation decoding module. The former extracts shallow features through the first layer of convolution, and then extracts multi-level features through multiple activation, convolution, and normalization combination layers. Correspondingly, the latter gradually generates degradation results through multiple activation, deconvolution, and normalization combination layers.

[0059] In the degradation network training phase, a degradation network that can better simulate the degradation relationship of high and low spatial resolution images from different remote sensors is trained. The specific implementation is as follows: the high-resolution image at the reference time is input, and the low-resolution image at the reference time is used as label data for training. The process can be represented by the following formula:

[0060]

[0061] wherein, is the false reference time low-resolution image generated by the network during training, N D represents the degradation network generation process, θ D represents the trainable parameters of the degradation network. After the training phase is completed, a trained degradation network that can express the degradation relationship of high and low spatial resolution images from different sensors is obtained. This network can replace the traditional linear interpolation method to realize data enhancement and provide training data for the next phase (main fusion phase).

[0062] 2、In the degradation network test phase, the input is input into the trained degradation network, and the corresponding degraded low-resolution image can be obtained. The above process can be represented by the following formula:

[0063]

[0064] wherein, is the reference time high-resolution image input into the degradation network to obtain the degraded image, is the reference time and the predicted time low-resolution image input into the degradation network to obtain the degraded image, respectively. The generated degraded image will be used as part of the training set in the subsequent main fusion network module.

[0065] II. Main fusion network phase

[0066] 1. The main fusion network is built on the Cycle-GAN framework, and its network structure is as follows: Figure 4 As shown, the network contains a forward generator. Backfeedback generator and the corresponding two discriminators and Two generators It mainly consists of a shallow feature extraction module, a deep residual module, and a fusion generation module. The shallow feature extraction module, composed of multiple convolutional layers, filler layers, and activation layers, is used to extract shallow features. The deep residual module, composed of multiple residual blocks (ResBlocks), is used to extract deep features. The fusion module, also composed of convolutional layers, activation layers, and filler layers, is mainly used to generate the fused result. Two discriminators are also included. It consists of multiple convolutional layers, activation layers, and normalization layers, used to extract features from the input image to determine the authenticity of the input image, and it works in competition with the generator to improve the performance of each other.

[0067] The forward generator fuses a high-spatial-resolution image at the reference time and a low-spatial-resolution image at the prediction time to generate a high-spatial-resolution image at the prediction time. The backward feedback generator ensures that the spatiotemporal information contained in the high-spatial-resolution fused image generated by the forward generator is constrained to the original input image. The high-spatial-resolution fused image generated by the forward generator is input into the network to obtain the original high-spatial-resolution image. During the main fusion network training phase, the steps are as follows:

[0068] 1) Obtained from the degenerate network The training data is input into the forward generator to obtain the corresponding predicted high-resolution image at time t1. This process can be expressed as follows:

[0069]

[0070] in, Fusion images of targets at predicted time points generated during training. These are the trainable parameters of the forward generator;

[0071] 2) The product generated in step 1) Its reference label image Input to forward discriminator Generated by constraints from the forward discriminator Similarity with reference label images is used to improve fusion quality.

[0072] 3) The product generated in step 1) and The original high-spatial-resolution image is fed into the backfeed generator to obtain the predicted image, thus constraining the consistency between the spatial information of the generated fused image and the spatial information of the original high-spatial-resolution image. This process can be represented as:

[0073]

[0074] in, The original high-resolution imagery is generated as the prediction source during the training process. These are the trainable parameters for the backfeedback generator;

[0075] 4) The product generated in step 3) Its reference label image Input to the backfeedback discriminator Generated by the discriminator constraint The quality makes it consistent with the reference label image.

[0076] 2. During the main fusion network testing phase, the original images to be fused were tested. and The input is fed into a trained forward generator to obtain the final predicted time-fused image. The process is as follows:

[0077]

[0078] As a preferred option:

[0079] 1. During the training phase of the degenerate network, the target loss function used is as follows:

[0080]

[0081] 2. During the training phase of the main fusion network, the overall objective loss function used is:

[0082] L F =L adv +λL content +δL consist (31)

[0083] Among them, L adv L represents the generation of adversarial loss. content This represents the content loss, specifically the difference loss between the fused image generated by the forward generator and the reference labeled image. L consist The consistency constraint loss is used to ensure that the fused image retains more spatial detail features from the original input image. λ and δ represent weighting parameters used to balance the contributions of content loss and consistency constraint loss. Generative adversarial loss L... adv The definition is as follows:

[0084]

[0085]

[0086]

[0087] wherein, and respectively represent the generation of adversarial loss of the fusion process (i.e. the generation of the fusion image) from the reference time to the prediction time and the generation of the fusion image from the prediction time to the reference time. The original input high spatial resolution image generation process (i.e. the original input high spatial resolution image generation process) is reversed.

[0088] The content loss and the consistency constraint loss are defined as follows:

[0089]

[0090]

[0091] In view of the problem that the interpolation downsampling method used in the existing deep learning spatio-temporal fusion method based on self-supervised learning strategy may not be able to better simulate the complex degradation relationship (nonlinear relationship) between high spatial resolution and low spatial resolution remote sensing images from different remote sensors, a degradation network is designed to learn the spatial degradation relationship between the high resolution and low resolution images to be fused by relying on the powerful nonlinear learning ability of the deep neural network. The implementation process is divided into two stages, as shown in the flowchart of Figure 1 , the first stage is to learn the degradation relationship based on the degradation network, and the second stage is the main fusion stage. Taking the fusion of Sentinel-2 true color (spatial resolution: 10 meters) images and Landsat8 satellite (spatial resolution: 30 meters) true color images as an example.

[0092] Firstly, all the obtained Sentinel-2 and Landsat8 images are preprocessed, including coordinate system registration, pixel value normalization and cropping, etc. Then, the network training and prediction stage is entered, and the process is as follows:

[0093] 1) In the degradation network training stage, the Sentinel-2 and Landsat8 true color images at the reference time are used as training data, the former is used as input and the latter is used as label data for training, and the training process is as shown in Figure 2 . The designed degradation network structure is shown in the figure, which is composed of a feature extraction and coding module and a degradation decoding module. The former is used to extract multi-level features through multiple convolutional layers, and the latter is used to generate degradation results through multiple deconvolutional layers. The loss function used in this stage is shown in formula (13), and the interpolation two-norm is adopted for the degraded image and the reference label image (Landsat8 true color image);

[0094] 2) In the degradation network test stage, the reference time Sentinel-2, Landsat8 true color image and the predicted time Landsat8 true color image are input into the degradation network to obtain the respective degradation images, such as Figure 2 as shown, as part of the main fusion network training data;

[0095] 3) In the main fusion network training stage, the whole process is based on the cycle consistency generative adversarial network (Cycle-GAN) framework, as shown in Figure 3 , the network contains a forward generator, a backward feedback generator and two corresponding discriminators. The forward generator is used to generate high-resolution fusion images at the predicted time by fusing high-resolution images at the reference time and low-resolution images at the predicted time, realizing the prediction process from the reference time to the predicted time. The backward generator is to ensure that the spatial feature information of the high-resolution fusion image generated by the forward generator is constrained to the spatial feature of the original input high spatial resolution image, realizing the process of returning from the predicted time to the reference time.

[0096] The specific steps of the main fusion network training stage are: input the reference time Sentinel-2 degradation image and the predicted time Landsat8 degradation image into the forward generator to obtain the corresponding fusion image at the predicted time; input the generated fusion image and its label image Landsat8 original image into the discriminator corresponding to the forward generator for discrimination, improve the discrimination ability of the forward discriminator and the quality of the image generated by the forward generator; input the generated fusion image and the reference time Landsat8 degradation image into the backward generator to generate false predicted time Landsat8 degradation image; input the generated false predicted time Landsat8 degradation image and its label image predicted time Landsat8 degradation image into the discriminator corresponding to the backward feedback generator for discrimination, improve the discrimination ability of the backward discriminator and the quality of the image generated by the backward feedback generator. The above steps are shown in Figure 3 . The network structure of the generator and the discriminator is shown in Figure 4 . Since the forward generator is the main generator to generate the fusion result, its structure is composed of three parts: a shallow feature extraction module, a deep residual module and a fusion generation module. The shallow feature extraction module extracts shallow features by multiple convolution layers, the deep residual module is composed of multiple residual blocks for further extracting deep features, and the fusion module is mainly used to generate the fusion result.

[0097] 4) In the main fusion network test stage, the reference time Sentinel-2 image and the predicted time Landsat8 image are input into the trained forward generator to obtain the final fusion image. The experimental results are shown in Figure 5As shown, the contrast method adopted is the classic spatio-temporal fusion method ESTARFM, SPSTFM, FSDAF and Fit-fc. The quantitative evaluation results are shown in Table 1, and the quantitative evaluation indexes of image quality adopted are RMSE (root mean square error, the smaller the better), SSIM (structural similarity index, the larger the better), ERGAS (relative global dimensionless error, the smaller the better) and PSNR (peak signal-to-noise ratio, the larger the better). It can be seen from the visual effect and the quantitative evaluation results that the present application has greater advantages compared with the other four methods. It can be obviously seen from the enlarged area of the visual results that the experimental results of the present application are closest to the reference images. Compared with the traditional remote sensing image fusion method, the present application makes full use of the powerful nonlinear feature extraction capability of the neural network, better simulates the relationship between the input images and the fusion images, and effectively improves the phenomena of spectral distortion and partial loss of spatial details in the traditional method. In the quantitative evaluation results, the present application achieves the best results in each evaluation index.

[0098] Table 1 shows the quantitative evaluation results

[0099]

[0100] The present application also provides a few-sample remote sensing spatio-temporal fusion system based on a degenerative fusion cascade network, comprising the following modules:

[0101] A degenerative network construction module, which firstly constructs and trains a degenerative network, wherein the degenerative network is jointly composed of a feature extraction and coding module and a degenerative decoding module; in the degenerative network training stage, the high spatial resolution image of the reference time t0 and the low spatial resolution image are taken as training data;

[0102] A degenerative network testing module, which is used for inputting the high resolution image and the low resolution image of the reference time and the low resolution image of the predicted time to the trained degenerative network in the degenerative network testing stage to obtain the respective degenerative images , wherein the degenerative images and the low resolution image of the predicted time jointly constitute the training set of the next stage main fusion network, and t1 is the predicted time;

[0103] A main fusion network construction module, which is built based on the Cycle-GAN framework of the cycle consistency generative adversarial network, and the main fusion network comprises a forward generator , a backward feedback generator and two corresponding discriminators During the main fusion network training phase, the following steps are taken: the forward generator uses the high-resolution image at the reference time and the low-resolution image at the prediction time to fuse and generate the high-resolution image at the prediction time; the backward feedback generator is used to ensure that the high-resolution image generated by the forward generator contains information constrained to the original input image; the original high-resolution input image is obtained by using the high-resolution image generated by the forward generator; the discriminator is used to extract the features of the input image to determine the authenticity of the input image, and it competes with the generator to improve their respective performance.

[0104] The main fusion network testing module is used to obtain the final predicted time-of-fuse fused image by inputting the original image to be fused into the trained forward generator during the main fusion network testing phase.

[0105] The specific implementation methods of each module are the same as those of each step, and will not be described in this invention.

[0106] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network, characterized in that, The method comprises the following steps: Step 1: First, construct and train the degenerate network, which consists of a feature extraction encoding module and a degenerate decoding module; during the training phase of the degenerate network, at reference time... High spatial resolution images Low spatial resolution images As training data; Step 2, Degradation Network Testing Phase: High-resolution and low-resolution images at the reference time and low-resolution images at the prediction time are used ( , , The data is input into the trained degradation network to obtain the corresponding degraded images. The degraded images and the low-resolution images at the predicted time together constitute the training set for the main fusion network in the next stage. For the predicted time; Step 3, a main fusion network is built based on a cycle consistency generative adversarial network (Cycle-GAN) framework, and the main fusion network comprises a forward generator , a backward feedback generator , and two discriminators corresponding to the forward generator and the backward feedback generator , ; in a training stage of the main fusion network, the forward generator is used to generate high-resolution images at a prediction time by fusing high-resolution images at a reference time and low-resolution images at the prediction time, the backward feedback generator is used to ensure that the high-resolution images generated by the forward generator are information-constrained to original input images, and the original input high-resolution images are obtained by using the high-resolution images generated by the forward generator, and the discriminators are used to extract input image features to determine the authenticity of the input images, and the discriminators and the generators are mutually played against each other to improve their respective performances; The specific implementation of step 3 is as follows: Step 3.1, the degraded network obtained in step 1 is input into the forward generator to obtain the corresponding predicted high-resolution image at the time instant, denoted as follows: , Step 3.2, the predicted high-resolution image obtained in step 3.1 is input into the backward generator to obtain the corresponding predicted low-resolution image at the time instant, denoted as follows: , Step 3.3, the predicted low-resolution image obtained in step 3.2 is input into the degradation network to obtain the corresponding (1) wherein, is a predicted time instant target fused image generated during the training process, is a trainable parameter of the forward generator; Step 3.2, the generated with its reference label image input to the forward discriminator quality of the generated by the discriminator Step 3.3, the generated with to the backward feedback generator to obtain the predicted original input high-resolution image to constrain the consistency of the spatial information of the generated fused image and the spatial information of the original input high spatial resolution image, expressed as: (2) wherein, is a predicted original input high resolution image generated during the training process, is a trainable parameter of the backward feedback generator; Step 3.4, the generated with its reference label image input to the backward feedback discriminator quality by the discriminator constraint generated ; Step 4, in the main fusion network test stage, the final predicted time fusion image is obtained by inputting the original image to be fused into the trained forward generator.

2. The few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network according to claim 1, characterized in that: In step 1, the feature extraction and encoding module first extracts shallow features through the first layer of convolution, and then extracts multi-level features through a plurality of activation, convolution and normalization combined layers, and the degradation decoding module generates degradation results through a plurality of activation, deconvolution and normalization combined layers, and the training set required by the fusion network is constructed through data enhancement, and the training set is composed of a plurality of convolution and deconvolution.

3. The few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network according to claim 1, characterized in that: In Step 1, the specific process of the degenerated network training phase is as follows: high-resolution images at the reference time are taken as input Low-resolution images at the reference time are taken as input Training is performed as label data, and the process is represented by the following formula: (3) wherein, a low-scored image generated by the network in the training process for a fake reference time point, represents a degradation network generation process, represents a trainable parameter of the degradation network; after the training phase ends, a trained degradation network capable of expressing the degradation relationship of high and low spatial resolution images from different sensors is obtained, wherein M, N, and B represent the length, width, and band number of the image, and s is the scale factor of the high-resolution image and the low-resolution image.

4. The few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network according to claim 1, characterized in that: The implementation process of the degradation network test stage in step 2 is represented by the following formula: (4) wherein, is the high-resolution image at the reference time is the degraded image obtained after inputting the high-resolution image into the degradation network, is the low-resolution image at the reference time and the predicted time , are the degraded images obtained after inputting the low-resolution images into the degradation network, respectively, wherein M, N, and B represent the length, width, and band number of the image, and s is the scale factor of the high-resolution image and the low-resolution image.

5. The few-shot remote sensing spatio-temporal fusion method based on degenerated fusion cascade network according to claim 1, characterized in that: In step 3, two generators 、 The shallow feature extraction module is composed of multiple convolution layers, padding layers and activation layers for extracting shallow features, the deep residual module is composed of multiple residual blocks for extracting deep features, and the fusion module is also composed of convolution layers, activation layers and padding layers for generating fusion results; two discriminators 、 composed of multiple convolution layers, activation layers and normalization layers.

6. The few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network according to claim 1, characterized in that: In the degradation network training stage, the target loss function adopted is as follows: (5)。 7. The few-shot remote sensing spatio-temporal fusion method based on degenerate fusion cascade network according to claim 1, characterized in that: In the main fusion network training stage, the total target loss function adopted is as follows: (6) wherein, represents a generative adversarial loss, represents a content loss, i.e., a difference loss between the image generated by the generator and the reference image, represents a consistency constraint loss, used to constrain the fused image to retain more spatial features of the input image, and represents a weight parameter used to weigh the contributions of the content loss and the consistency constraint loss, and the generative adversarial loss is defined as follows: (7) Wherein each term is defined as follows: (8) (9) (10) wherein, with respectively represent the generation adversarial loss of the fusion process from the reference time to the prediction time and the inverse generation of the original input image from the prediction time to the reference time. The content loss and consistency constraint loss are defined as follows: (11) (12)。 8. A few-shot remote sensing spatio-temporal fusion system based on degenerated fusion cascade network, characterized in that, The method comprises the following modules: The degradation network construction module first constructs and trains a degradation network, which consists of a feature extraction encoding module and a degradation decoding module. During the degradation network training phase, a reference time is used... High spatial resolution images Low spatial resolution images As training data; The degradation network testing module is used to test the degradation network by combining high-resolution and low-resolution images at the reference time with low-resolution images at the predicted time. , , The data is input into the trained degradation network to obtain the corresponding degraded images. The degraded images and the low-resolution images at the predicted time together constitute the training set for the main fusion network in the next stage. For the predicted time; The main fusion network is built based on a cycle consistency generative adversarial network (Cycle-GAN) framework, and the main fusion network comprises a forward generator , a backward feedback generator , and two corresponding discriminators , ; in the training stage of the main fusion network, the forward generator is used to generate high-resolution images at a prediction time by fusing reference high-resolution images and low-resolution images at the prediction time, the backward feedback generator is used to ensure that the high-resolution images generated by the forward generator are information-constrained to original input images, the high-resolution images generated by the forward generator are used to obtain original input high-resolution images, and the discriminators are used to extract input image features to determine the authenticity of the input images, and the discriminators and the generators are used to improve the performance of each other. The specific implementation is as follows: Step 3.1, the degraded network obtained in step 2.1 is input into the forward generator to obtain the corresponding predicted high-resolution image at the time instant, denoted as the following formula: , , represents the corresponding predicted high-resolution image at the time instant, denoted as the following formula:​​ (13) wherein, is a predicted time instant target fused image generated during the training process, is a trainable parameter of the forward generator; Step 3.2, the generated with its reference label image input to the forward discriminator by the discriminator quality of the generated Step 3.3, the generated with to the backward feedback generator to obtain the predicted original input high-resolution image to constrain the consistency of the spatial information of the generated fusion image and the spatial information of the original input high spatial resolution image, expressed as: (14) wherein, is a predicted original input high resolution image generated during the training process, is a trainable parameter of the backward feedback generator; Step 3.4, the generated with its reference label image input to the backward feedback discriminator , the quality of the generated by the discriminator; The main fusion network test module is used for, in the main fusion network test stage, inputting the original image to be fused into the trained forward generator to obtain the final predicted time fusion image.

Images