Optical remote sensing image cloud and fog removing method and system based on generative adversarial network

Abstract

The present application relates to a remote sensing image cloud and fog removing method and system based on a generative adversarial network, comprising the steps of: obtaining a plurality of historical optical remote sensing images and their corresponding SAR images, making a sample library, the sample library comprising a cloud mask sample library and a cloud removing sample library; using the cloud mask sample library to train an FCN model to build a cloud mask generation model; using the cloud removing sample library to train a pix2pix model to build a cloud removing model; obtaining real-time clouded optical remote sensing images and their corresponding SAR images, inputting the cloud mask generation model and the cloud removing model, and obtaining optical remote sensing images with the cloud and fog removed. The present application establishes a cloud mask sample library and a cloud removing sample library, builds a cloud mask generation model and a cloud removing model respectively, realizes optical remote sensing image ground object restoration and reconstruction under SAR image assistance, and can meet the cloud and fog removing requirements of optical remote sensing images, especially under thick cloud conditions.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method and system for dehazing remote sensing images based on generative adversarial networks. Background Technology

[0002] Optical remote sensing images have advantages such as high resolution, multispectral density, and ease of processing, but they are susceptible to interference from weather conditions such as clouds, fog, and rain, which can affect image quality and performance. Dehazing of optical remote sensing images is an important area in remote sensing image processing. Over the past few decades, researchers have proposed various methods to address this problem, including methods based on statistical estimation, methods based on physical models, methods based on multi-temporal data, and methods based on deep learning.

[0003] Statistical estimation-based methods: Establish a mathematical model based on gray value statistics and use the correlation between gray values ​​of adjacent pixels to predict the gray value of cloud and fog pixels. However, this method is mainly applicable to thin cloud processing. When there are large areas of thick clouds covering the optical remote sensing image, this type of method cannot be used.

[0004] Physical model-based methods: These methods establish mathematical models based on physical processes such as atmospheric optics and imaging principles, and remove cloud and fog information through image analysis and parameter inversion. However, these methods often require a large amount of meteorological data, involve a large amount of computation, depend on specific scenarios, and have low accuracy.

[0005] Multi-temporal data-based methods utilize multiple simultaneous optical remote sensing image sequences to construct coverage models at different time points. They employ techniques such as interpolation or variation methods to remove cloud and fog information. However, these methods require sufficient multi-temporal data, and sample selection and classification are costly.

[0006] Deep learning-based methods typically use nearby optical remote sensing images of both cloudy and cloudless areas for training to build a conversion model. However, these methods often have low accuracy in extracting features from thick clouds.

[0007] Therefore, a method is needed to effectively remove thin and thick clouds from optical remote sensing images. Summary of the Invention

[0008] The purpose of this invention is to effectively remove thin and thick clouds from optical remote sensing images, and to provide a method and system for dehazing remote sensing images based on generative adversarial networks.

[0009] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0010] A method for removing clouds and fog from remote sensing images based on generative adversarial networks includes the following steps:

[0011] Step 1: Acquire several historical optical remote sensing images and their corresponding SAR images, and create a sample library, which includes a cloud mask sample library and a cloud removal sample library.

[0012] Step 2: Train the FCN model using a cloud mask sample library to build a cloud mask generation model;

[0013] Step 3: Train the pix2pix model using a cloud-free sample library to build the cloud-free model;

[0014] Step 4: Acquire real-time cloud-covered optical remote sensing images and their corresponding SAR images, input the cloud mask generation model and the cloud removal model, and obtain cloud-removed optical remote sensing images.

[0015] Furthermore, step 1 specifically includes the following steps:

[0016] Acquire several optical remote sensing images, and select optical remote sensing images of six bands: 4, 5, 6, 7, 8, and 12; acquire the corresponding SAR images of the optical remote sensing images, and select the polarization mode of the SAR images as VV and VH.

[0017] Preprocessing of optical remote sensing images and SAR images includes geometric correction, radiometric calibration, projection transformation, spatial registration, and brightness equalization.

[0018] Optical remote sensing images were cropped to a size of 512*512 using a sliding window with an overlap of 5, resulting in optical remote sensing image samples and corresponding SAR image samples for different time phases and different land cover types.

[0019] The cropped optical remote sensing image samples are divided into cloud-covered optical images and cloudless optical images. The clouds and fog in the cloud-covered optical images are manually vectorized, and the vector data is binarized to generate cloud mask labels.

[0020] The cloud-covered optical images and cloud mask tags constitute a cloud mask sample library, and the cloudless optical images and SAR images constitute a de-clouding sample library.

[0021] Furthermore, step 2 specifically includes the following steps:

[0022] The cloud mask sample library is used as the training set for the FCN model. The initial learning rate of the FCN model is set to 0.001 and decreases as the training process progresses. The loss function is cross-entropy and the optimizer is SGD. The FCN model is iteratively trained until the loss converges, thus obtaining the cloud mask generation model.

[0023] Furthermore, step 3 specifically includes the following steps:

[0024] The cloud-free sample library is used as the training set for the pix2pix model, which includes a generator network and an adversarial network. The generator network and the adversarial network are trained alternately until Nash equilibrium is reached, thus obtaining the cloud-free model.

[0025] The total loss function Loss of the pix2pix model is:

[0026] Loss=λ1LcGan+λ2L1+λ3L p

[0027] LcGan=E{log{D[I REAL ,G(I SAR )]}}+E{log{1-D[I REAL ,G(I SAR )]}}

[0028] L1 = E[I REAL -G(I SAR )]

[0029]

[0030] Where Loss is the total loss function of the pix2pix model; LcGAN is the adversarial loss function; L1 is the conditional loss function; and L... p I represents the perceptual loss function; λ represents the weights of each loss function; REAL For real images, G(I) SAR ) represents the image generated by the generator; G represents the generator, D represents the discriminator; E represents the expectation operation; F i Let N represent the feature representation of the image at the i-th layer in the pix2pix model, where N represents the number of feature layers.

[0031] Furthermore, step 4 specifically includes the following steps:

[0032] Acquire real-time cloud-covered optical remote sensing images and their corresponding SAR images, automatically register the optical remote sensing images and their corresponding SAR images, and obtain overlapping areas with the same resolution and the same geographic reference through resampling, projection and overlapping area cropping.

[0033] The overlapping region is input into the cloud mask generation model, and the cloud mask generation model generates a cloud mask;

[0034] The automatically registered cloud-covered optical remote sensing image, the corresponding SAR image, and the cloud mask output by the cloud mask generation model are all input into the cloud removal model, and the cloud removal model generates the optical remote sensing image corresponding to the SAR image.

[0035] The optical remote sensing image generated by the cloud removal model within the cloud mask area is replaced with the corresponding area of ​​the cloud-covered optical remote sensing image to complete the cloud removal process of the cloud-covered optical remote sensing image, resulting in a cloud-removed optical remote sensing image.

[0036] A remote sensing image dehazing system based on generative adversarial networks includes:

[0037] The sample library creation module is used to acquire several historical optical remote sensing images and their corresponding SAR images to create a sample library, which includes a cloud mask sample library and a cloud removal sample library.

[0038] The model building module is used to train the FCN model using a cloud mask sample library to build a cloud mask generation model; and to train the pix2pix model using a de-clouded sample library to construct a de-clouded model.

[0039] The automatic dehazing module is used to acquire real-time cloud-covered optical remote sensing images and their corresponding SAR images. By inputting the cloud mask generation model and the dehazing model, the cloud-removed optical remote sensing images are obtained.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention establishes a cloud mask sample library and a declouding sample library, and constructs cloud mask generation models and declouding models respectively, to achieve ground feature restoration and reconstruction of optical remote sensing images with SAR image assistance, thereby realizing declouding and fog removal processing to make them closer to the target image. It can meet the declouding and fog removal requirements of optical remote sensing images, especially under thick cloud conditions. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating the model construction process for an embodiment of the present invention;

[0044] Figure 2 This is a flowchart illustrating the process of removing clouds and fog using a model in an embodiment of the present invention;

[0045] Figure 3 This is a block diagram of the system modules of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0047] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.

[0048] Example:

[0049] This invention is achieved through the following technical solution: a method for removing clouds and fog from remote sensing images based on generative adversarial networks, comprising the following steps:

[0050] Please see Figure 1 The flowchart for constructing the model for this method is as follows: Step 1, acquire several historical optical remote sensing images and their corresponding SAR images, and create a sample library, which includes a cloud mask sample library and a cloud removal sample library.

[0051] Acquire several optical remote sensing images, selecting those in bands 4, 5, 6, 7, 8, and 12. Acquire the corresponding SAR images, selecting VV and VH polarization modes for the SAR images. Perform geometric correction, radiometric calibration, projection transformation, spatial registration, and brightness equalization preprocessing on both the optical and SAR images.

[0052] Optical remote sensing images were cropped to a size of 512*512 using a sliding window with an overlap of 5, resulting in optical remote sensing image samples and corresponding SAR image samples for different time phases and land cover types. The cropped optical remote sensing image samples were divided into cloud-covered and cloudless optical images. Clouds and fog in the cloud-covered optical images were manually vectorized, and the vector data was binarized to generate cloud mask labels.

[0053] The cloud-covered optical images and cloud mask tags constitute a cloud mask sample library, and the cloudless optical images and SAR images constitute a de-clouding sample library.

[0054] It should be noted that since SAR images (synthetic aperture radar images) can penetrate clouds and are not affected by clouds and fog, when dividing optical images with clouds and optical images without clouds, SAR images are automatically classified as optical images without clouds. Therefore, SAR images only exist in the cloudless sample library, while SAR images do not exist in the cloud mask sample library.

[0055] Step 2: Train the FCN model using a cloud mask sample library to build a cloud mask generation model.

[0056] The cloud mask sample library is used as the training set for the FCN model. The initial learning rate of the FCN model is set to 0.001 and decreases as the training process progresses. The loss function is cross-entropy and the optimizer is SGD. The FCN model is iteratively trained until the loss converges, thus obtaining the cloud mask generation model.

[0057] The cross-entropy loss function of the FCN model is calculated as follows:

[0058]

[0059] Where i represents the i-th clouded optical image, j represents the j-th clouded optical image, N represents the total number of clouded optical images in the cloud mask sample library; α is the regularization parameter, μ and υ are preset constants, γ is the scaling factor; k represents the k-th cloud mask label, K represents the total number of cloud mask label types; φ1 and φ2 are preset weights; ω i Let ω be the weight of the cloud mask label corresponding to the i-th cloudy optical image. j Let be the weight of the cloud mask label corresponding to the j-th cloudy optical image;

[0060] Where, x i The cloud mask label represents the i-th optical image with clouds.

[0061] Where, ω i,k The weight of the cloud mask label of the i-th cloudy optical image belongs to the k-th class.

[0062] Step 3: Train the pix2pix model using a cloud-free sample library to build the cloud-free model.

[0063] The de-clouded sample library is used as the training set for the pix2pix model, which includes a generator network and an adversarial network. The generator network and the adversarial network are trained alternately until Nash equilibrium is reached, thus obtaining the de-clouded model.

[0064] To address issues such as blurred boundaries and image distortion that may occur when deep learning networks extract features from optical remote sensing images, this scheme introduces a perceptual loss function into the generative network. The total loss function of the pix2pix model is:

[0065] Loss=λ1LcGan+λ2L1+λ3L p

[0066] LcGan=E{log{D[I REAL ,G(I SAR )]}}+E{log{1-D[I REAL ,G(I SAR )]}}

[0067] L1 = E[I REAL -G(I SAR )]

[0068]

[0069] Where Loss is the total loss function of the pix2pix model; LcGAN is the adversarial loss function; L1 is the conditional loss function; and L... p I represents the perceptual loss function; λ represents the weights of each loss function; REAL For real images, G(I) SAR G represents the image generated by the generator; D represents the discriminator; E represents the expectation operation, which calculates the loss value of the discriminator D and updates its parameters using real data and generated fake data in each iteration update; F i Let N represent the feature representation of the image at the i-th layer in the pix2pix model, where N represents the number of feature layers.

[0070] Please see Figure 2 The flowchart below shows the process of obtaining cloud-removed optical remote sensing images using the constructed model in this invention. Step 4: Obtain real-time cloud-covered optical remote sensing images and their corresponding SAR images. Input the cloud mask generation model and the cloud removal model to obtain cloud-removed optical remote sensing images.

[0071] Real-time cloud-covered optical remote sensing images and their corresponding SAR images are acquired. The optical remote sensing images and their corresponding SAR images are automatically registered. Through resampling, projection, and overlapping region cropping, overlapping regions with the same resolution and geographic reference are obtained. The overlapping regions are input into the cloud mask generation model, which generates a cloud mask.

[0072] The automatically registered cloud-covered optical remote sensing image, the corresponding SAR image, and the cloud mask output by the cloud mask generation model are all input into the declouding model. The declouding model generates an optical remote sensing image corresponding to the SAR image. The optical remote sensing image generated by the declouding model within the cloud mask area is then used to replace the corresponding area of ​​the cloud-covered optical remote sensing image, completing the cloud and fog removal process of the cloud-covered optical remote sensing image and obtaining a cloud-removed optical remote sensing image.

[0073] The replacement process is as follows:

[0074] M result =M REAL ⊙M CLOUD ^+M FAKE ⊙M CLOUD

[0075] Among them, M result To remove clouds and fog from the optical remote sensing image matrix, M REAL M represents a cloud-covered optical remote sensing image matrix. CLOUD M is a Boolean matrix representing the cloud mask, where 1 represents a cloud and 0 represents a cloudless area; CLOUD ^ represents the non-cloudy region matrix obtained by bitwise inversion, M FAKE The optical remote sensing image matrix generated for the cloud removal model.

[0076] Please see Figure 3 The present invention also proposes a remote sensing image dehazing system based on generative adversarial networks, comprising:

[0077] The sample library creation module is used to acquire several historical optical remote sensing images and their corresponding SAR images to create a sample library, which includes a cloud mask sample library and a cloud removal sample library.

[0078] The model building module is used to train the FCN model using a cloud mask sample library to build a cloud mask generation model; and to train the pix2pix model using a de-clouded sample library to construct a de-clouded model.

[0079] The automatic dehazing module is used to acquire real-time cloud-covered optical remote sensing images and their corresponding SAR images. By inputting the cloud mask generation model and the dehazing model, the cloud-removed optical remote sensing images are obtained.

[0080] In detail, the sample library creation module acquires several optical remote sensing images, selecting optical remote sensing images in six bands: 4, 5, 6, 7, 8, and 12. It then acquires the corresponding SAR images, selecting VV and VH polarization modes for the SAR images. Preprocessing is performed on the optical and SAR images, including geometric correction, radiometric calibration, projection transformation, spatial registration, and brightness equalization. A sliding window is used to crop the optical remote sensing images to a size of 512*512, with an overlap of 5, resulting in optical remote sensing image samples and corresponding SAR image samples for different time phases and land cover types. The cropped optical remote sensing image samples are divided into cloud-covered and cloudless optical images. Clouds and fog in the cloud-covered optical images are manually vectorized, and the vector data is binarized to generate cloud mask labels. The cloud-covered optical images and cloud mask labels constitute a cloud mask sample library, while the cloudless optical images and SAR images constitute a de-clouding sample library.

[0081] The model building module includes a cloud mask generation model building unit and a cloud removal model building unit. The cloud mask generation model building unit is used to train the FCN model using a cloud mask sample library to build a cloud mask generation model. The cloud removal model building unit uses a cloud removal sample library to train the pix2pix model to construct a cloud removal model.

[0082] The cloud mask generation model building unit uses the cloud mask sample library as the training set of the FCN model, sets the initial learning rate of the FCN model to 0.001 and decreases it as the training process progresses, selects the loss function cross-entropy and the optimizer sgd, and iteratively trains the FCN model until the loss converges, thus obtaining the cloud mask generation model.

[0083] The cross-entropy loss function of the FCN model is calculated as follows:

[0084]

[0085] Where i represents the i-th clouded optical image, j represents the j-th clouded optical image, N represents the total number of clouded optical images in the cloud mask sample library; α is the regularization parameter, μ and υ are preset constants, γ is the scaling factor; k represents the k-th cloud mask label, K represents the total number of cloud mask label types; φ1 and φ2 are preset weights; ω i Let ω be the weight of the cloud mask label corresponding to the i-th cloudy optical image. j Let be the weight of the cloud mask label corresponding to the j-th cloudy optical image;

[0086] Where, x i The cloud mask label represents the i-th optical image with clouds.

[0087] Where, ω i,k The weight of the cloud mask label of the i-th cloudy optical image belongs to the k-th class.

[0088] The cloud removal model construction unit uses the cloud removal sample library as the training set for the pix2pix model and trains the pix2pix model, which includes a generator network and an adversarial network. The generator network and the adversarial network are trained alternately until Nash equilibrium is reached, thus obtaining the cloud removal model.

[0089] The total loss function Loss of the pix2pix model is:

[0090] Loss=λ1LcGan+λ2L1+λ3L p

[0091] LcGan=E{log{D[I REAL ,G(I SAR )]}}+E{log{1-D[I REAL ,G(I SAR )]}}

[0092] L1 = E[I REAL -G(I SAR )]

[0093]

[0094] Where Loss is the total loss function of the pix2pix model; LcGAN is the adversarial loss function; L1 is the conditional loss function; and L... p I represents the perceptual loss function; λ represents the weights of each loss function; REAL For real images, G(I) SAR G represents the image generated by the generator; D represents the discriminator; E represents the expectation operation, which calculates the loss value of the discriminator D and updates its parameters using real data and generated fake data in each iteration update; F i Let N represent the feature representation of the image at the i-th layer in the pix2pix model, where N represents the number of feature layers.

[0095] The automatic dehazing module acquires real-time cloud-covered optical remote sensing images and their corresponding SAR images. It automatically registers the optical remote sensing images and the corresponding SAR images, and through resampling, projection, and overlapping region cropping, obtains overlapping regions with the same resolution and geographic reference. The overlapping regions are input into the cloud mask generation model, which generates a cloud mask. The automatically registered cloud-covered optical remote sensing image, the corresponding SAR image, and the cloud mask output by the cloud mask generation model are then input into the dehazing model, which generates the optical remote sensing image corresponding to the SAR image. The optical remote sensing image generated by the dehazing model within the cloud mask area is then used to replace the corresponding area of ​​the cloud-covered optical remote sensing image, completing the cloud removal process and obtaining a dehazed optical remote sensing image.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for removing cloud and fog from remote sensing images based on a generative adversarial network, characterized in that: The method comprises the following steps: Step 1, obtaining a plurality of historical optical remote sensing images and corresponding SAR images, and making a sample library, wherein the sample library comprises a cloud mask sample library and a cloud removal sample library; Step 2, training an FCN model using the cloud mask sample library to construct a cloud mask generation model; The step 2 specifically comprises the following steps: The cloud mask sample library is used as the training set of the FCN model, the initial learning rate of the FCN model is set to 0.001 and is decreased during the training process, the loss function cross-entropy is selected, the optimizer sgd is selected, the FCN model is iteratively trained until the loss converges, and the cloud mask generation model is obtained; Step 3, training a pix2pix model using the cloud removal sample library to construct a cloud removal model; The step 3 specifically comprises the following steps: The cloud removal sample library is used as the training set of the pix2pix model, the pix2pix model is trained, the pix2pix model comprises a generator and a discriminator, the generator and the discriminator are alternately trained until a Nash equilibrium is reached, and the cloud removal model is obtained; The total loss function Loss of the pix2pix model is: wherein, Loss is the total loss function of the pix2pix model; LcGan is the adversarial loss function, L1 is the conditional loss function, L p is the perceptual loss function; is the weight of each loss function; I REAL is the real image, G(I SAR ) is the image generated by the generator; G represents the generator, D represents the discriminator; E represents the expected value operation; F i is the feature representation of the image in the i-th layer of the pix2pix model, and N represents the number of feature layers; Step 4, obtaining real-time optical remote sensing images with clouds and corresponding SAR images, inputting the cloud mask generation model and the cloud removal model, and obtaining optical remote sensing images with clouds removed; The step 4 specifically comprises the following steps: Real-time optical remote sensing images with clouds and corresponding SAR images are obtained, the optical remote sensing images and the corresponding SAR images are automatically registered, the same resolution and the same geographical reference of the overlapping areas are obtained through resampling, projection and overlapping area cropping; The overlapping areas are input into the cloud mask generation model, and the cloud mask generation model generates a cloud mask; The automatically registered optical remote sensing images with clouds, the corresponding SAR images and the cloud mask generated by the cloud mask generation model are collectively input into the cloud removal model, and the cloud removal model generates an optical remote sensing image corresponding to the SAR image; The optical remote sensing image generated by the cloud removal model within the cloud mask range is replaced into the corresponding area of the optical remote sensing image with clouds, the cloud and fog removal processing of the optical remote sensing image with clouds is completed, and an optical remote sensing image with clouds removed is obtained; the replacement process is as follows: where M result is the cloud-free optical remote sensing image matrix, M REAL is the cloudy optical remote sensing image matrix, M CLOUD is the cloud mask Boolean matrix, 1 for cloudy and 0 for cloud-free; is the non-cloudy region matrix obtained by bitwise NOT, M FAKE is the cloud-free optical remote sensing image matrix generated by the cloud removal model.

2. The remote sensing image de-clouding method based on a generative adversarial network according to claim 1, characterized in that: The step 1 specifically comprises the following steps: A plurality of optical remote sensing images are obtained, and optical remote sensing images of 4, 5, 6, 7, 8 and 12 bands are selected; SAR images corresponding to the optical remote sensing images are obtained, and the polarization modes of the SAR images are selected as VV and VH; The optical remote sensing images and the SAR images are preprocessed through geometric correction, radiation calibration, projection change, spatial registration and brightness equalization; The optical remote sensing images are cropped according to the size of 512*512 by using a sliding window, the overlapping degree of the sliding window is 5, and optical remote sensing image samples and corresponding SAR image samples of different time phases and different ground object types are obtained; The cropped optical remote sensing image samples are divided into optical images with clouds and optical images without clouds, the cloud and fog in the optical images with clouds are manually vector-drawn, and the drawn vector data is binarized to generate a cloud mask label; The cloud optical image and the cloud mask label constitute a cloud mask sample library, and the cloud-free optical image and the SAR image constitute a cloud-free sample library.

3. A remote sensing image cloud and fog removal system based on a generative adversarial network, characterized in that: The method comprises the following steps: A sample library manufacturing module is configured to obtain a plurality of historical optical remote sensing images and corresponding SAR images, and manufacture a sample library, which comprises a cloud mask sample library and a cloud-free sample library; A model construction module is configured to train an FCN model using the cloud mask sample library to construct a cloud mask generation model, and train a pix2pix model using the cloud-free sample library to construct a cloud-free model; The cloud mask sample library is used as the training set of the FCN model, the initial learning rate of the FCN model is set to 0.001 and is decreased during the training process, the loss function cross-entropy is selected, the optimizer sgd is selected, the FCN model is iteratively trained until the loss converges, and the cloud mask generation model is obtained; The cloud-free sample library is used as the training set of the pix2pix model, the pix2pix model is trained, the pix2pix model comprises a generator and a discriminator, the generator and the discriminator are alternately trained until a Nash equilibrium is reached, and the cloud-free model is obtained; The total loss function Loss of the pix2pix model is as follows: An automatic cloud and fog removal module is configured to obtain real-time clouded optical remote sensing images and corresponding SAR images, input the cloud mask generation model and the cloud-free model, and obtain cloud and fog removed optical remote sensing images; wherein, Loss is the total loss function of the pix2pix model; LcGan is the adversarial loss function, L1 is the conditional loss function, L p is the perceptual loss function; is the weight of each loss function; I REAL is the real image, G(I SAR ) is the image generated by the generator; G represents the generator, D represents the discriminator; E represents the expected value operation; F i is the feature representation of the image in the i-th layer in the pix2pix model, and N represents the number of feature layers. Real-time clouded optical remote sensing images and corresponding SAR images are obtained, the optical remote sensing images and the corresponding SAR images are automatically registered, the same resolution and the same geographical reference of the overlapping area are obtained through resampling, projection and overlapping area cropping; The overlapping area is input into the cloud mask generation model, and the cloud mask generation model generates a cloud mask; The clouded optical remote sensing images after automatic registration, the corresponding SAR images, and the cloud mask generated by the cloud mask generation model are collectively input into the cloud-free model, and the cloud-free model generates SAR image corresponding optical remote sensing images; The optical remote sensing images generated by the cloud-free model within the cloud mask range are replaced into the corresponding area of the clouded optical remote sensing images, cloud and fog removal processing of the clouded optical remote sensing images is completed, and cloud and fog removed optical remote sensing images are obtained; the replacement process is as follows: ​ where M result is the cloud-free optical remote sensing image matrix, M REAL is the cloudy optical remote sensing image matrix, M CLOUD is the cloud mask Boolean matrix, 1 for cloudy, 0 for cloud-free; is the non-cloudy region matrix obtained by bitwise NOT, M FAKE is the cloud-free optical remote sensing image matrix generated by the cloud removal model.

Images