Remote sensing image restoration method based on time series images

Abstract

The invention discloses a remote sensing image restoration method based on time-series images, which comprises the following steps: S1: obtaining a training sample set, including training sample groups of different regions, each training sample group comprising an information-complete image, a time-series image, an information-missing image, a foreground mask and a background mask; S2: constructing a multi-temporal image generation network, the generation network comprising a domain code extractor, a multi-temporal generator and a discriminator; S3: using the training sample set to train the multi-temporal image generation network to obtain a trained multi-temporal image generation network; S4: obtaining a remote sensing image to be restored and a time-series image of the same region to generate a remote sensing image group to be restored; S5: inputting the image group to be restored into the trained multi-temporal image generation network to obtain a restored remote sensing image. The method adopts a style transfer method in the image restoration process to ensure that the style of the area to be restored is consistent with that of the existing area, and to avoid splicing traces at the boundary of the restoration area.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a remote sensing image restoration method based on time-series images. Background Art

[0002] Due to sensor defects or atmospheric conditions, the acquired remote sensing data contains a large amount of missing information (such as dead pixels, light pollution, and cloud pollution). In the prior art, the missing information can be reconstructed through remote sensing data acquired at different time periods at the same location of the image to be repaired. This method currently has two problems: 1. In order to avoid large differences in images, the time interval generally needs to be small enough, which results in a large overlap between the acquired image and the repaired image. 2. When acquiring images of different time periods with less missing information, the time interval for acquiring images may be long, and the style difference between the images may be large. The image obtained after repairing the image to be repaired is very different from the original image style. Therefore, it is necessary to provide an image repair method that can use images of multiple different time periods to repair an image and ensure that the color and other styles of the repaired image do not change significantly. Summary of the invention

[0003] In view of the above technical problems, the purpose of the present invention is to provide a remote sensing image restoration method based on time-series images, and its specific technical solution is as follows:

[0004] A remote sensing image restoration method based on time series images, comprising:

[0005] S1. Obtain a training sample set, wherein the training sample set includes training sample groups of different regions, each training sample group includes a remote sensing image A with complete information, a remote sensing image A' with missing information, a first time series image group {B1, B2, . . . , B N}, a foreground mask M f and a background mask M b ;

[0006] S2. construct a multi-temporal image generation network, wherein the multi-temporal image generation network includes a domain code extractor E, a multi-temporal generator G and a discriminator D;

[0007] S3, training the multi-temporal image generation network using the acquired training sample set to obtain a trained multi-temporal image generation network;

[0008] S4, acquiring the remote sensing image to be restored and the time series images of the same area as the remote sensing image to be restored, and generating a remote sensing image data set to be restored;

[0009] S5. Input the image group to be restored into the trained multi-temporal image generation network to obtain the restored remote sensing image.

[0010] The first time-sequential image group {B1, B2, . . . , B N} is a remote sensing image of the remote sensing image A with complete information in the same area and at different phases.

[0011] The remote sensing image A' with missing information in step S1 is a simulated remote sensing image with missing information of the remote sensing image A with complete information. The acquisition method is: using a random number generator to generate a background mask M of a certain shape and size on the remote sensing image A with complete information. b , applying the background mask to the remote sensing image A with complete information, and obtaining the remote sensing image A' with missing information.

[0012] The foreground mask M in step S1 f By the background mask M b Complementary generation, where M f =1-M b .

[0013] The domain code extractor E in step S2 includes domain code sub-extractor E1, domain code sub-extractor E2, domain code sub-extractor E3 and domain code sub-extractor E4; each domain code sub-extractor includes a plurality of local convolution layer (PartialConv) structures.

[0014] The multi-phase generator G in step S2 includes a content generator G content and style generator G style ; The content generator G content It includes an initial feature extractor, a feature fusion module and an intermediate result output module.

[0015] Furthermore, the style generator G style It includes an initial feature extractor, a feature fusion module and a final result output module.

[0016] The discriminator D in step S2 includes a content discriminator D content and style discriminator D style The content discriminator D content is the PatchGAN network structure; the style discriminator D style It is the PixelGAN network structure.

[0017] Furthermore, the specific process of training the multi-temporal image generation network in step S3 includes:

[0018] S31 The remote sensing image A' with missing information in the training sample group and the first time series image group {B1, B2, . . . , B N}、Foreground mask M f and background mask M bInput the multi-temporal image generation network to obtain the repaired image C;

[0019] The specific steps of generating the repaired image C in step S31 are:

[0020] (1) The remote sensing image A' with missing information in the training sample group and the first time series image group {B1, B2, ..., B N}, input the content generator G of the multi-temporal image generation network content , and obtain the intermediate result image I0;

[0021] (2) The intermediate result image I0 and the foreground mask M f After splicing in the spectral channel dimension, it is input into the domain code sub-extractor E1 to obtain the foreground code of the intermediate result image;

[0022] (3) Combine the information-missing remote sensing image A' with the background mask M b The foreground encoding of the intermediate result image is concatenated in the spectral channel dimension and input into the style generator G style Finally, the repaired image C is obtained.

[0023] S32 obtains the restored image C and the remote sensing data A' with missing information, the first time series image group {B1, B2, . . . , B N} to obtain the merged false data, and at the same time, the remote sensing sample A with complete information and the remote sensing data A' with missing information, the first time series image group {B1, B2, ..., B N} are merged to obtain merged true data, and the merged false data and the merged true data are respectively input into the discriminator to obtain the discriminant value of the merged false data and the discriminant value of the merged true data.

[0024] Furthermore, the discrimination of the merged false data and true data in step S32 includes two branches, image style discrimination and image content discrimination; wherein, the image style discrimination discriminates the background coding of the remote sensing image A' with missing information, the foreground coding of the remote sensing image A with complete information, and the foreground coding input of the restored image C; and the image content discrimination discriminates the image input in the merged false data and true data.

[0025] The background encoding of the information-missing remote sensing image A' is composed of the information-missing remote sensing data A' and the background mask M b Obtained through domain code sub-extractor E2.

[0026] The foreground encoding of the complete remote sensing image A consists of the complete remote sensing sample A and the foreground mask M f Obtained through domain code sub-extractor E3.

[0027] The foreground encoding of the repaired image C is composed of the repaired image C and the foreground mask M f Obtained through domain code sub-extractor E4.

[0028] S33 calculates a comprehensive loss function of the multi-temporal image generation network, and updates parameters of the multi-temporal image generation network through the comprehensive loss function;

[0029] The comprehensive loss function includes a first loss function L1, a second loss function L2, and a third loss function L3.

[0030] The first loss function L1 is the reconstruction loss, which is obtained by calculating the image difference between the restored image C obtained through the multi-temporal image generation network and the remote sensing image A with complete information.

[0031] The second loss function L2 is the triplet loss, which is obtained by calculating the similarity of the domain coding between the intermediate result image I0, the restored image C and the remote sensing image A with complete information.

[0032] The third loss function L3 is obtained by calculating the difference between the discriminant value of the merged false data and the discriminant value of the merged true data.

[0033] The comprehensive loss function is calculated based on the first loss function, the second loss function and the third loss function:

[0034] L=ΔL1+β L2 +γL3

[0035] Among them, L1 is the first loss function, L2 is the second loss function, L3 is the third loss function, α, β, and γ represent the weight parameters of the first loss function, the second loss function, and the third loss function, respectively.

[0036] S34 obtains a new training sample group, and repeats steps S31 to S33 to perform iterative training until the comprehensive loss function converges, thereby obtaining a trained multi-temporal image generation network.

[0037] The image group to be restored in step S4 includes: a remote sensing image to be restored, a time series image, a background mask, and a foreground mask.

[0038] The remote sensing image to be repaired is a missing image, including any one or more combinations of a remote sensing image with shadows, a remote sensing image covered by clouds, and a remote sensing image with dead pixels.

[0039] The beneficial effects produced by the present invention are:

[0040] 1. The present invention constructs a multi-temporal image generation network, including multiple feature extractors and feature fusion modules, which can use multiple time series images to repair remote sensing images with a high degree of effective domain loss. There are no quality and source requirements for the selected time series remote sensing images. The selected time series images can be remote sensing images with missing information, and only need to contain regional images to be repaired, which greatly reduces the difficulty and threshold of obtaining time series images.

[0041] 2. The present invention constructs a multi-temporal image generation network, including a domain code extractor, a content generator and a style generator. By using style transfer technology in the restoration process, it ensures that the style of the area to be restored in the image is consistent with that of the existing area, avoiding splicing marks on the boundary of the restoration area.

[0042] 3. The present invention adopts three loss functions to conduct comprehensive evaluation from multiple aspects, optimizes the multi-temporal image generation network, and improves the quality of generated restoration images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of a remote sensing image restoration method based on time-series images provided by the present invention;

[0045] Figure 2 A schematic diagram of a process of generating a restored image C using the multi-temporal image generation network provided by the present invention;

[0046] FIG3( a ) is a schematic diagram of the structure of a generator G in a multi-temporal image generation network provided by the present invention;

[0047] FIG3(b) is a style discriminator D in the multi-temporal image generation network provided by the present invention. style Schematic diagram of the structure;

[0048] FIG. 3( c ) is a content discriminator D in the multi-temporal image generation network provided by the present invention. content Schematic diagram of the structure;

[0049] Figure 4 A schematic diagram of the specific structure of each module in the multi-temporal image generation network provided by the present invention;

[0050] FIG5( a ) is a schematic diagram of generating a remote sensing image A′ with missing information in the present invention;

[0051] FIG. 5( b ) is a foreground mask M in the present invention. f Schematic diagram of the generation of . DETAILED DESCRIPTION

[0052] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0053] refer to Figure 1 The present invention provides a remote sensing image restoration method based on time-series images, the method comprising:

[0054] S1. Obtain a training sample set, wherein the training sample set includes training sample groups of different regions, each training sample group includes: a remote sensing image A with complete information, a remote sensing image A' with missing information, and a time series image set with N (N≥2) time series images {B1, B2, . . . , B N}, a foreground mask M f and a background mask M b .

[0055] The method for obtaining the training sample group is:

[0056] (1) Obtaining remote sensing image A with complete information;

[0057] (2) Obtain a set of remote sensing images of the same area and different phases in remote sensing image A with complete information as the first time series image group {B1, B2, ..., B N};

[0058] The time series image is a remote sensing image of the same area obtained by expanding the remote sensing image A in time. The expansion time range is 15 days before and after.

[0059] (3) Use a random number generator to randomly select a remote sensing image A∈R with complete information. H×W×3 (H represents the image height, W represents the image width) to generate a background mask M of a certain shape and size b ∈R H×W×1 , apply the background mask to the remote sensing image A, and obtain the remote sensing image A'∈R with missing information H×W×3 The mask is a binary matrix, 1 represents the area to be covered, and 0 represents the area to be retained, as shown in FIG5(a).

[0060] (4) Based on the background mask M b Generate a complementary foreground mask M f ∈RH×W×1 (M f =1-M b ), the specific situation is shown in Figure 5(b).

[0061] (5) The remote sensing image A with complete information, the image A' with missing information, and the first time series image group {B1, B2, ..., B N}, background mask M b and the foreground mask M f Multiple sample data constitute a training sample group.

[0062] S2. construct a multi-temporal image generation network, wherein the multi-temporal image generation network includes a domain code extractor E, a multi-temporal generator G and a discriminator D;

[0063] The domain code extractor E (domain code extractor) includes domain code sub-extractor E1, domain code sub-extractor E2, domain code sub-extractor E3 and domain code sub-extractor E4. Each domain code sub-extractor includes a plurality of partial convolution layer (Partial Conv) structures (partial convolution layers ≥ 4).

[0064] The multi-phase generator G includes a content generator G content and style generator G style The content generator G content It includes an initial feature extractor (model_initial), a feature fusion module (model_intermediate) and an intermediate result output module (model_final_0). style It includes the initial feature extractor (model_initial), the feature fusion module (model_intermediate) and the final result output module (model_final_1).

[0065] The discriminator D includes a content discriminator D content and style discriminator D style The content discriminator D content The PatchGAN network structure may include ≥3 convolutional layers; the style discriminator D style The PixelGAN network structure is adopted, which contains 3 convolutional layers.

[0066] Furthermore, the initial feature extractor adopts a convolutional layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer structure.

[0067] The feature fusion module includes multiple convolution blocks (convolution blocks ≥ 2) and multiple residual blocks (residual blocks ≥ 6) structures, and each convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer.

[0068] The result output module includes multiple convolution blocks (convolution blocks ≥ 2), each convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer.

[0069] S3, training the multi-temporal image generation network using the acquired training sample set to obtain a trained multi-temporal image generation network;

[0070] The specific process of training the multi-temporal image generation network using the training sample set includes:

[0071] S31 The remote sensing image A' with missing information in the training sample group and the first time series image group {B1, B2, . . . , B N}、Foreground mask M f and background mask M b Input the multi-temporal image generation network to obtain the repaired image C;

[0072] Furthermore, the specific steps of generating the repaired image C are as follows:

[0073] (1) The remote sensing image A' with missing information in the training sample group and the first time series image group {B1, B2, ..., B N}, input the content generator G of the multi-temporal image generation network content , and get the intermediate result image I0∈R H×W×3 ;

[0074] The remote sensing image A' with missing information and the first time series image group {B1, B2, ..., B N} Input content generator G content The initial feature extractor in performs feature extraction, and then the extracted features are spliced ​​in pairs in the spectral channel dimension. The spliced ​​features are input into the feature fusion module for feature fusion, and then the fused features are spliced ​​in the spectral channel dimension and input into the intermediate result output module to obtain the intermediate result image I0.

[0075] (2) The intermediate result image I0 and the foreground mask M f After splicing in the spectral channel dimension, the foreground code z of the intermediate result image I0 is obtained by inputting the domain code sub-extractor E1. f_0 ∈R H×W×L ;

[0076] (3) The information-missing remote sensing image A' and the background mask M b The spectral channel dimension is stitched, and then the I0 foreground code z of the intermediate result image is obtained. f_0 The splicing result is input into the style generator G style In the process, after the primary feature extractor, feature fusion module and final result output module, the restored image C is obtained.

[0077] S32 obtains the restored image C and the remote sensing data A' with missing information, the first time series image group {B1, B2, . . . , B N}Merge based on the channel dimension to obtain the merged fake data concat(C,(A',B1,B2,......,B N ), and simultaneously the remote sensing sample A with complete information and the remote sensing data A' with missing information, the first time series image group {B1, B2, ..., B N}Merge based on the channel dimension to get the merged true data concat(A,(A',B1,B2,......,B N )), input the above two combined data into the content discriminator D content , get the discriminant value D of the combined false data content (concat(C,(A',B1,B2,......,B N )) and the discriminant value D of the combined true data content (concat(A,(A',B1,B2,...,B N ));

[0078] The remote sensing data A' with missing information and the background mask M b The background code z obtained by the domain code sub-extractor E2 b , and the generated repair image C and foreground mask M f The foreground code z obtained by the domain code sub-extractor E4 f_1 ∈R H×W×L Merge based on channel dimension to get the merged fake data concat(C,(A',M b ,M f )); At the same time, the remote sensing data A' with missing information and the background mask M b The background code z obtained by the domain code sub-extractor E2 b ∈R H×W×L , with complete remote sensing sample A and foreground mask M f The foreground code z obtained by the domain code sub-extractor E3 f ∈R H×W×LMerge based on channel dimension to get the merged true data concat(A,(A',M b ,M f )); Input the above two combined data into the style discriminator D style , get the discriminant value D of the combined false data style (concat(C,(A',M b ,M f )) and the discriminant value D of the combined true data style (concat(A,(A',M b ,M f )).

[0079] S33 calculates a comprehensive loss function of the multi-temporal image generation network, and updates network parameters of the multi-temporal image generation network through the comprehensive loss function;

[0080] The loss function includes a first loss function L1, a second loss function L2, and a third loss function L3.

[0081] The first loss function L1 is the reconstruction loss, which is obtained by calculating the image difference between the repaired image C obtained by the multi-temporal image generation network and the remote sensing image A with complete information. The specific calculation method is as follows:

[0082] L1=E||I0-A||1+E||G(Z)-A||1

[0083] Among them, E||I0-A||1 represents the difference between the intermediate result image I0 and the remote sensing image A with complete information; E||G(Z)-A||1 represents the difference between the restored image C and the remote sensing image A with complete information; G(Z)=C,Z=[A',B1,B2,......,B N ,Mb,Mf].

[0084] Manhattan distance is used to calculate the pixel difference between the intermediate result image I0 and the remote sensing image A with complete information, and the pixel difference between the restored image C and the remote sensing image A with complete information to obtain the reconstruction loss. The calculation formula is as follows:

[0085]

[0086]

[0087] n represents the spectral channel dimension of the image (1≤n≤3), i represents the number of rows of image pixels (1≤i≤H), j represents the number of columns of image pixels (1≤j≤W), Represents the pixel point in the i-th row and j-th column of the n-th channel of image A'. Represents the pixel point in the i-th row and j-th column of the n-th channel of image C', Represents the pixel at the i-th row and j-th column of the n-th channel of image I0'.

[0088] The second loss function L2 is the triplet loss, which is obtained by calculating the similarity of the domain encoding between the intermediate result image I0, the repaired image C and the remote sensing image A with complete information. The specific process is as follows:

[0089] Intermediate result image I0 and foreground mask M f After splicing in the spectral channel dimension, the domain code sub-extractor E1 is input to obtain the foreground code z of the intermediate result image f_0 ; Remote sensing image A' with missing information and background mask M b After splicing in the spectral channel dimension, the background code z of the remote sensing image A' with missing information is obtained by inputting the domain code sub-extractor E2. b ; Remote sensing image A with complete information and foreground mask M f After splicing in the spectral channel dimension, the domain code sub-extractor E3 is input to obtain the foreground code z of the remote sensing image A with complete information. f ∈R H×W×L ; Repair image C and foreground mask M f After splicing in the spectral channel dimension, the domain code sub-extractor E4 is input to obtain the foreground code z of the repaired image C. f_1 .

[0090] Repair the foreground code z of image C f_1 The background code z of the remote sensing image A' with missing information should be b Similarly, the foreground code z of the intermediate result image I0 f_0 Different; repair the foreground code z of image C f_1 Should be the foreground code z of the remote sensing image A with complete information f Similarly, the foreground code z of the intermediate result image I0 f_0 The specific calculation method is as follows:

[0091] L2=L fb +L ff

[0092] L fb =L(z b ,z f_0 ,z f_1 =max(d(z b ,z f_1 )-d(z f_0 ,z f_1 )+m,0)

[0093] L ff =L(z f ,z f_0 ,zf_1 =max(d(z f ,z f_1 )-d(z f ,z f_0 )+m,0)

[0094] Among them, L fb represents the similarity loss of the domain encoding of the image background area, L ff represents the similarity loss of the domain encoding of the foreground area of ​​the image, d(·) represents the Euclidean distance, and m is a boundary. By minimizing L fb , so that d(z b ,z f_1 ) is at least greater than d(z f_0 ,z f_1 ) is smaller than a margin m. By minimizing L ff , so that d(z f ,z f_1 ) is at least greater than d(z f ,z f_0 ) is smaller than a boundary m.

[0095] The third loss function L3 is calculated by calculating the discriminant value D(concat(G(Z),(A',B1,B2,......,B N ))) and the discriminant value D(concat(A,(A',B1,B2,......,B N The difference of ))) is:

[0096] L3=L Dcontent +L Dstyle

[0097] L Dcontent =E[log(D content (Z,A))]+E[log(1-D content (Z,G(Z)))]

[0098] L Dstyle =E[max(0,1-D style (A))]+E[max(0,1+D style (C))]

[0099] Among them, L Dcontent Content discriminator loss function, L Dstyle represents the style discriminator loss function; G(Z)=C,Z=[A',B1,B2,......,B N ,Mb,Mf].

[0100] The comprehensive loss function is calculated based on the first loss function, the second loss function and the third loss function:

[0101] L=αL1+βL2+γL3

[0102] Among them, L1 is the first loss function, L2 is the second loss function, L3 is the third loss function, α, β, and γ represent the weight parameters of the first loss function, the second loss function, and the third loss function, respectively.

[0103] S34 obtains a new training sample group, and repeats steps S3-1 to S3-3 to perform iterative training until the comprehensive loss function converges, thereby obtaining a trained multi-temporal image generation network.

[0104] S4, acquiring the remote sensing image to be restored and the time series images of the same area as the remote sensing image to be restored, and generating a remote sensing image data set to be restored;

[0105] The remote sensing image to be restored is a missing image, including any one or more combinations of a remote sensing image with shadows, a remote sensing image covered by clouds, and a remote sensing image with dead pixels;

[0106] The image group to be restored includes remote sensing images to be restored, time-series images, background masks, and foreground masks.

[0107] The method for obtaining the image group to be repaired is as follows:

[0108] (1) Obtain a remote sensing image X0 to be restored;

[0109] (2) Perform image retrieval in the image library according to the missing area in the remote sensing image to be restored, and obtain a second time series image group {X1, X2, ..., X3, ..., X4, ..., X5, ..., X6, ..., X7, ..., X8, ..., X9, ..., X10, ..., X11, ..., X12, ..., X13, ..., X14, ..., X15, ..., X16, ..., X17, ..., n}; The time range of the image retrieval is: based on the time point of the remote sensing image to be restored, no more than 15 days before and after; the image closest to the remote sensing image to be restored is preferred as the time series image;

[0110] (3) Generate background mask M based on the remote sensing image to be restored b and the corresponding foreground mask M f ;

[0111] (4) The above image data are combined into a remote sensing image data set to be restored {X0, X1, X2, ..., X n ,M b ,M f}.

[0112] S5. Input the image group to be restored into the trained multi-temporal image generation network to obtain the restored remote sensing image.

[0113] The specific process is as follows:

[0114] (1) The image group to be restored is input into the encoder of the trained multi-temporal image generation network, and the feature extractor E1 extracts background features and style features of the image to be restored; the feature extractor E2 extracts content features of each temporal image respectively;

[0115] (2) The extracted background features, style features, and content features are fused and reconstructed through an encoder to obtain the restored image Y.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image restoration method based on time-series images, characterized in that: The following steps are involved: S1: Acquire a training sample set, where the training sample set includes multiple training sample groups in different regions; Each training sample group includes a remote sensing image A with complete information, a remote sensing image A' with missing information, and a first time series image set {B1, B2, ..., B N A foreground mask M f and a background mask M b ; S2: construct a multi-temporal image generation network, wherein the multi-temporal image generation network includes a domain code extractor E, a multi-temporal generator G and a discriminator D; S3: training the multi-temporal image generation network using the acquired training sample set to obtain a trained multi-temporal image generation network; S4: Acquire the remote sensing image to be restored and the time series images of the same area as the remote sensing image to be restored, and generate a remote sensing image data group to be restored; S5: input the image group to be restored into the trained multi-temporal image generation network to obtain the restored remote sensing image; In step S2, the domain code extractor E includes domain code sub-extractors E1, E2, E3, and E4; each domain code sub-extractor includes multiple local convolutional layer structures; the multi-phase generator G includes a content generator G content and style generator G style ; The discriminator D includes a content discriminator D content and style discriminator D style ; The specific process of training the multi-temporal image generation network in step S3 is as follows: S31 The remote sensing image A' with missing information in the training sample group and the first time series image set {B1, B2, . . . , B N }、Foreground mask M f and background mask M b Input the multi-temporal image generation network to obtain the repaired image C; S32 compares the obtained restored image C with the remote sensing image A' with missing information, the first time series image set {B1, B2, . . . , B N } to obtain the merged false data, and at the same time, the remote sensing image A with complete information and the remote sensing image A' with missing information, the first time series image set {B1, B2, ..., B N } to obtain the combined true data, and input the combined false data and true data into the discriminator D respectively to obtain the discriminant value of the combined false data and the discriminant value of the combined true data; S33 calculates a comprehensive loss function of the multi-temporal image generation network, and updates parameters of the multi-temporal image generation network through the comprehensive loss function; S34 obtains a new training sample group, and repeats steps S31 to S33 to perform iterative training until the comprehensive loss function converges, thereby obtaining a trained multi-temporal image generation network.

2. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: The content generator G content and style generator G style Both include an initial feature extractor, a feature fusion module, and an intermediate result output module; The content discriminator D content is the PatchGAN network structure; the style discriminator D style It is the PixelGAN network structure.

3. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: The specific steps of obtaining the repaired image C in step S31 are: The remote sensing image A' with missing information in the training sample group and the first time series image set {B1, B2, ..., B N }, input the content generator G of the multi-temporal image generation network content , and obtain the intermediate result image I0; The intermediate result image I0 and the foreground mask M f After splicing in the spectral channel dimension, it is input into the domain code sub-extractor E1 to obtain the foreground code of the intermediate result image I0; The remote sensing image A' with missing information and the background mask M b The foreground encoding of the intermediate result image I0 is concatenated in the channel dimension and then input into the style generator G style The restored image C is obtained.

4. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: The process of inputting the merged false data and true data into the discriminator D to obtain the discrimination values ​​of the true data and the false data in step S32 includes image style discrimination and image content discrimination; wherein the image style discrimination discriminates the background coding of the remote sensing image A' with missing information, the foreground coding of the remote sensing image A with complete information, and the foreground coding input of the restored image C; and the image content discrimination discriminates the image input in the merged false data and the true data; The background encoding of the information-missing remote sensing image A' is composed of the information-missing remote sensing data A' and the background mask M b Obtained by domain code sub-extractor E2; The foreground encoding of the complete remote sensing image A consists of the complete remote sensing sample A and the foreground mask M f Obtained by the domain code sub-extractor E3; The foreground encoding of the repaired image C is composed of the repaired image C and the foreground mask M f Obtained by domain code sub-extractor E4.

5. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: The loss function in step S33 includes a first loss function L1, a second loss function L2, and a third loss function L3, wherein: The first loss function L1 is the reconstruction loss, which is obtained by calculating the image difference between the restored image C obtained by the multi-temporal image generation network and the remote sensing image A with complete information; The second loss function L2 is the triplet loss, which is obtained by calculating the similarity of the domain encoding between the intermediate image I0, the restored image C and the real information complete image A; The third loss function L3 is obtained by calculating the difference between the discriminant value of the merged false data and the discriminant value of the merged true data.

6. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: In step S1, the first time-series image set {B1, B2, . . . , B N } is a set of images of the remote sensing image A with complete information in the same area and at different time phases; the remote sensing image A' with missing information is a simulated remote sensing image with missing information of the remote sensing image A with complete information; the foreground mask M f By the background mask M b Complementary generation, where M f =1-M b .

7. The remote sensing image restoration method based on time-series images according to claim 1, characterized in that: The image group to be restored in step S4 includes the remote sensing image to be restored, the second time series image set, the background mask and the foreground mask; the remote sensing image to be restored is any one or more combinations of remote sensing images with shadows, remote sensing images covered by clouds and remote sensing images with dead pixels.

Images