Progressive repair framework for cloud removal from optical remote sensing images and sar images

Abstract

The present application relates to a kind of optical remote sensing image and SAR image fusion's progressive repair framework cloud removal method, belong to remote sensing image cloud removal field, the progressive repair framework is specifically divided into two stages of rough, refinement, and embedding coherent semantic attention layer CSA in the second stage refinement repair network, to retain context structure, more effectively predict missing part by modeling the semantic correlation between the features of missing area, in addition, in order to improve the structural similarity and global consistency of image, the present application also constructs loss function when distinguishing cloudy area and cloudless area, and the boundary of cloudy area is constrained, so that cloud removal image structure is clearer.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image cloud removal, and relates to a progressive repair framework cloud removal method for fusing optical remote sensing images and SAR images. BACKGROUND

[0002] Satellite images have been widely used in the fields of remote sensing, computer vision, environmental science and meteorology, such as observing the ground surface by means of satellite images in order to monitor the natural environment, improve traffic and respond to disasters in a timely manner. According to a study by ISCCP, clouds cover about 67% of the area of the earth's surface. Therefore, how to remove the clouds in the optical remote sensing images is an important problem for the application and development of remote sensing images. The traditional remote sensing data reconstruction methods can be divided into four categories: spatial method, spectral method, temporal method and hybrid method.

[0003] The spatial-based method uses cloud-free areas to recover the image, but if the cloud coverage area is large, it is usually not ideal. The spectral-based method uses the spectral characteristics of the optical sensor image to obtain additional data for reconstructing the missing information, but they can also be contaminated by clouds. The temporal-based method estimates the missing information by integrating cloud-free corresponding images at different times, but this method is limited to the frequent use of cloud-free images. The hybrid method combines the first three methods to remove the clouds. For example, Chen et al. found the most suitable pixels for the damaged area using a spatio-temporal Markov random field. Benabdelkader et al. improved the context reconstruction process using spatial spectral information.

[0004] With the development of synthetic aperture radar (SAR), the cloud interference in SAR images has been eliminated due to its high penetration of clouds. Therefore, SAR can obtain real feature information under the cloud-shielded area. However, due to the differences in imaging mechanisms between SAR and optical remote sensors, the further development of SAR-based cloud removal methods has been hindered.

[0005] At present, the in-depth study of deep learning has opened up a new field for optical image cloud removal technology. Its rapid development has provided powerful feature extraction capability for the application of SAR images. In order to generate simulated images similar to real images, Meraner et al. fused SAR images and optical images into a deep residual neural network. Gao et al. adopted a two-step method, first converting SAR images into simulated optical images, and then fusing simulated optical images, SAR images and cloudy images into a generative adversarial network to reconstruct the damaged area. The above experiments have verified the effectiveness of the fusion of SAR and optical remote sensing images.

[0006] However, the above prior art generally has the following problems:

[0007] (1) Due to the discontinuity of local pixels, the existing method usually produces image content with blurred texture and distorted structure. From the perspective of semantic level, the local pixel discontinuity is mainly because these methods ignore the semantic relevance and feature continuity of the missing area.

[0008] (2) Unlike the task of reconstructing a smooth image in the image restoration task, cloud removal of remote sensing images needs to focus on restoring the real information of ground objects. While thick clouds make the information of ground objects under the clouds almost completely missing, leading to the fact that the recovery results of the existing cloud removal methods using SAR and optical remote sensing image fusion are generally not clear in the boundary information of ground objects in thick cloud areas.

[0009] (3) Most of the existing methods do not adequately consider the differences between optical remote sensing images and SAR images, resulting in poor fusion effect. SUMMARY

[0010] Therefore, the purpose of the present application is to provide a progressive repair framework cloud removal method for optical remote sensing image and SAR image fusion, which enhances the extraction and utilization of global information so as to keep the color, texture and structure information of the reconstructed area and cloud-free area globally consistent.

[0011] To achieve the above purpose, the present application provides the following technical scheme:

[0012] A progressive repair framework cloud removal method for optical remote sensing image and SAR image fusion, comprising the following steps:

[0013] S1: Obtain an optical remote sensing image dataset and divide it into a training set, a validation set and a test set; the dataset includes Sentinel-1 SAR images, Sentinel-2 cloudy optical images and Sentinel-2 near-time cloud-free optical images;

[0014] S2: Batch input of Sentinel-2 cloudy optical images and Sentinel-1 SAR images into the progressive repair framework, training of the missing area based on simple dilated convolution combined with L1 reconstruction loss to obtain a rough prediction I p ; extraction and fusion of global and local information of SAR images and RGB images to obtain a refined repaired image I r ; calculation of the loss function of the model according to the Sentinel-2 cloud-free image and the cloud removal image I r of the refined network repair; evaluation of the precision of the cloud removal model and adjustment of the network model parameters according to the change of the model precision; completion of model convergence after multiple adjustments and training;

[0015] S3: Based on the model trained in step S2, batch input of cloud area validation sample dataset to generate cloud-free optical images and complete the cloud removal task.

[0016] Further, the optical remote sensing image dataset in step S1 is SEN12MS-CR; wherein the VV and VH polarization images of the SAR image are cut into values [-25, 0] and [-32.5, 0] respectively and scaled to the range [0, 1]; the bands of all optical images are truncated to the value [0, 10,000] and then normalized to the range [0, 1]; only the VH polarization SAR data and the RGB band optical images are used as input during training.

[0017] Further, the progressive repair framework in step S2 is specifically divided into two stages of rough and refinement;

[0018] The rough repair stage adopts a general encoder-decoder network structure, and the network includes 11 layers, the first 5 layers form a contraction path, and the 6th to 10th layers form an expansion path; in the contraction path, the size of the jth layer of all three networks is n1xn2x(24x2 j-1 ), j = 1, 2, 3, 4, 5, the convolution block performs multi-channel two-dimensional convolution with a step of 1 or 2; the expansion path has a symmetrical structure with the transpose convolution block of the generator G and the discriminator D, and the number of channels in each layer is doubled; the 11th layer is a standard output layer with convolution operation and sigmoid activation function; the Conv Block is composed of multi-channel 2-D convolution and rectified linear unit ReLU, and the transposed Conv Block is composed of transposed 2-D convolution, batch normalization BN and ReLU, and the BN operation is performed before the activation function in this step;

[0019] The refinement repair stage in step S2 includes a residual branch and a reconstruction branch; the residual branch copies and retains the input Sentinel-2 cloudy image, waiting to be merged with the output of the reconstruction branch; the reconstruction branch has three steps: preprocessing, encoding-decoding and image recovery; in the preprocessing, the shallow features of the multi-source images are extracted and mapped to the shape specified in the encoding stage; the encoding-decoding part is a U-shaped structure, the shallow features obtained after preprocessing first enter the encoding stage to obtain high-dimensional features; in the encoding stage, the high-dimensional features are decoded to obtain a reconstructed feature map with the same shape as the input; in addition, a coherent semantic attention layer CSA is embedded in the fourth layer of the encoder; the structure of the decoder is symmetrical to the encoder without the CSA layer, and all 4x4 convolutions are deconvolutions; in the image recovery stage, a mapping is established to ensure that the output of the reconstruction branch is consistent with the form of the RGB image, and finally the reconstructed image is merged with the RGB input image; the coherent semantic attention layer CSA is used to build the correlation between the deep features of the missing areas.

[0020] Further, the calculation steps of the coherent semantic attention layer CSA are as follows:

[0021] Input the feature map set Fin ;

[0022] Extract known regions The image patch is reshaped into a convolutional filter and applied to the missing region M;

[0023] The maximum cross-correlation value of the records was calculated. And the closest matching neural patch

[0024]

[0025] Use the most similar context m i initialization

[0026] Calculate the similarity between two adjacent generated patches

[0027] Get m i Attention diagram A i :

[0028]

[0029]

[0030] Pay attention to diagrams A1 and A. n Combined, we obtain the attention matrix;

[0031] Reuse of known regions The feature map F is obtained by deconvolution. out ;

[0032] Output the reconstructed feature map set F out .

[0033] Furthermore, in the refinement stage of the progressive restoration framework, different activation functions are used during feature extraction, resulting in different value ranges for each layer of the feature map. Consequently, the number of channels C in the obtained depth reconstruction feature map differs from the number of channels in the target image. The transformation process is accomplished through Decloud-IR, which accepts the feature map with channel C and expands the output to any dimension according to the requirements of different tasks. The Decloud-IR module consists of a squeeze excitation module SE, a 3×3 convolution, and a sigmoid activation function. The SE module learns the focal points of different channels. The 3×3 convolution transforms the feature map into the same shape as the target image and introduces the inductive bias of the convolution operation into the Transformer-based module. Finally, the sigmoid function converts the value of the feature map into the [0,1] interval and outputs the result, thereby obtaining the cloud-removed image.

[0034] Further, step S2 specifically comprises the following steps: using a learning perceptual image block similarity LPIPS to calculate the reconstruction loss, LPIPS maps the predicted image and the real image to the same dimension in the same way, and gives priority to the perceptual similarity between them, and the definition of LPIPS is described as follows:

[0035]

[0036] Wherein, the lower the value of LPIPS, the more similar the two images are, and vice versa; the specific content of the formula includes: extracting a feature stack from the L layer and performing unit-normalization in the channel dimension; using a vector to scale the number of activation channels, calculating the L2 distance; and finally averaging in space and summing in channel.

[0037] For any position in the feature map M, the feature space is set to the corresponding layer of CSA in the target layer and the decoder respectively to calculate the L2 distance, and the consistency loss L is defined c as follows:

[0038]

[0039] Wherein, Φ n is the activation map of the selected layer in the data set, I in is the input, I p is the first-stage rough prediction result, I r is the final output result; CSA(·) represents the feature after using the CSA layer, CSA d (·) is the corresponding feature in the decoder, using I in as the input of the refinement network of the predicted final result I p . r

[0040] On the other hand, the application provides an optical remote sensing image and SAR image fusion progressive repair framework cloud removal device, comprising: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the optical remote sensing image and SAR image fusion progressive repair framework cloud removal method as described above.

[0041] In another aspect, a computer readable storage medium is provided, and computer program instructions are stored on the computer readable storage medium, and the computer program instructions are executed by a processor to realize the steps of the optical remote sensing image and SAR image fusion progressive repair framework cloud removal method as described above.

[0042] The application has the following beneficial effects:

[0043] ​1) An incremental inpainting framework for fusing optical remote sensing images and SAR images is proposed, which is specifically divided into two stages of rough and refinement. In the first stage, the Sentinel-1 SAR image and the Sentinel-2 image with clouds are input into the rough network to output the rough prediction I p . The network is composed of 4 convolutional groups with skip connections to connect the features from each layer of the encoder and the corresponding decoder layer, and is trained explicitly using the L1 reconstruction loss. In the second stage, the rough prediction output I p and the Sentinel-1 SAR image are input into the refinement network to extract and fuse the global and local information of the SAR image and the RGB image, so as to maintain the global consistency of the color, texture and structure information of the reconstructed area and the cloud-free area.

[0044] 2) In the refinement stage of the incremental inpainting framework, the coherent semantic attention layer CSA is embedded to preserve the context structure and more effectively predict the missing part by modeling the semantic correlation between the features of the missing area;

[0045] 3) In order to improve the structural similarity, visual interpretability and global consistency of the image, the loss function is constructed to distinguish the cloud area and the cloud-free area, and the boundary of the cloud area is constrained, so that the structure of the cloud-removed image is clearer.

[0046] Other advantages, objects and features of the present application will be in part apparent and in part pointed out hereinafter in the specification, and it is to be understood that certain changes can be made in the application by those skilled in the art without departing from the scope of the present application. The objects and other advantages of the present application can be realized and obtained by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings as follows, wherein:

[0048] Figure 1 is the overall flowchart of the present application;

[0049] Figure 2 is the incremental inpainting framework structure diagram of the present application;

[0050] Figure 3 is the rough inpainting network structure diagram of the present application;

[0051] Figure 4 is the refinement inpainting network structure diagram of the present application;

[0052] Figure 5 is the decloud-IR structure diagram of the refinement inpainting of the present application. DETAILED DESCRIPTION

[0053] Following, the advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the specification. The present application can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0054] Among them, the drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation on the present application; in order to better illustrate the embodiments of the present application, some components in the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings can be omitted.

[0055] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the positional relationship described in the drawings is only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0056] As Figure 1 shown, the present application proposes a gradual repair framework for cloud removal of optical remote sensing image and SAR image fusion, which specifically includes the following steps:

[0057] Step 1: Collect the required data set and pre-process. The data used in this paper comes from the large-scale data set SEN12MS-CR. This data set covers all continents, weather and seasonal geographical and weather conditions, a total of 169 non-overlapping regions. The image size of each region is about 5200x4000 pixels, and the data is sliced according to the size of 256x256 and 50% spatial overlap. A total of 122,218 triple samples are generated. Each sample includes a corrected and georeferenced Sentinel 1 SAR image, a cloud-free Sentinel 2 image and a cloud-covered Sentinel 2 image, wherein the acquisition time of the cloud-free and cloud-covered images is close.

[0058] The VV and VH polarization images of SAR images are cut into values [-25, 0] and [-32.5, 0] respectively and scaled to the range [0, 1]. The bands of all optical images are truncated to values [0, 10,000] and then normalized to the range [0, 1]. Only the VH polarization SAR data and the RGB band optical images are used as input during training. According to the ratio of 85:10:5, all data are divided into training set, validation set and test set, 29207, 3436 and 1718 respectively. The data set is divided in a random way.

[0059] S2: Batch the Sentinel-2 cloudy optical images and Sentinel-1 SAR images into the progressive repair framework, train the missing area based on a simple dilated convolution combined with an L1 reconstruction loss to obtain a rough prediction I p ; Further extract and fuse the global and local information of the SAR image and the RGB image to obtain the refined repaired image I r ; Calculate the loss function of the model according to the Sentinel-2 cloud-free image and the cloud-removed image I r of the refined network, and evaluate the precision of the cloud removal model. According to the change of the model precision, adjust the network model parameters, and complete the model convergence after multiple adjustments and training.

[0060] As shown in Figure 2 , the progressive repair framework is specifically divided into rough and refined repair networks.

[0061] As shown in Figure 3 , the rough repair network adopts a general encoder-decoder network structure, and the network includes 11 layers. The first five layers form a contraction path, and the sixth to tenth layers form an expansion path. In the contraction path, the size of the jth layer of all three networks is n1x n2x (24x2 j-1 ), j = 1, 2, 3, 4, 5, and the convolution block (black arrow) performs multi-channel two-dimensional convolution with a step of 1 or 2. The expansion path (from the sixth layer to the tenth layer) has a symmetrical structure with the transpose convolution block of the generator G and the discriminator D, but due to the copy and connection operation, the number of channels of the U-Net in each layer is doubled. The 11th layer is a standard output layer with convolution operation and sigmoid activation function. The Conv Block is composed of multi-channel 2-D convolution and rectified linear unit (ReLU), and the transposed Conv Block is composed of transposed 2-D convolution, batch normalization (BN) and ReLU. The present application follows and performs the BN operation before the activation function.

[0062] The refined repair network module includes two key branches: a residual branch and a reconstruction branch.

[0063] The residual branch replicates and retains the input Sentinel-2 cloud image, waiting to be merged with the output of the reconstruction branch. The reconstruction branch has three steps: pre-processing, encoding-decoding, and image restoration. In the first step, shallow features of the multi-source image are extracted and mapped to the shape specified in the encoding stage. The encoding-decoding part is a U-shaped structure, and the shallow features obtained after pre-processing first enter the encoding stage to obtain high-dimensional features. In the encoding stage, the high-dimensional features are decoded to obtain the reconstructed feature map with the same shape as the input. It is worth noting that the coherent semantic attention layer (CSA) is embedded in the fourth layer of the encoder. The structure of the decoder is symmetrical to the encoder without the CSA layer, and all 4x4 convolutions are deconvolutions. In the image restoration stage, a mapping is established to ensure that the output of the reconstruction branch is consistent with the form of the RGB image, and finally the reconstructed image is merged with the RGB input image.

[0064] wherein the coherent semantic attention layer (CSA) is used to build the correlation between the deep features of the missing area, and regardless of whether the missing area is irregular or centered, the CSA can achieve good repair effect. The specific steps of the CSA layer are shown in Table 1:

[0065] Table 1

[0066]

[0067] As shown in the accompanying Figure 4 , during the feature extraction process, the value range of each layer of the feature map is different due to the use of different activation functions, and the number of channels (C) of the obtained deep reconstructed feature map is also different from the number of channels of the target image. In order to normalize the value range of the deep feature map and convert the dimension to the same dimension as the target optical image, Decloud-IR is designed to complete the transformation process. Decloud-IR accepts a feature map with C channels, and according to the requirements of different tasks, the output can be expanded to any dimension, thereby improving the output flexibility of the entire network and the scalability of the task.

[0068] As Figure 5 shown, the Decloud-IR module is composed of a squeeze-and-excitation (SE) module, a 3x3 convolution, and a sigmoid activation function. The SE module can learn the focus points of different channels and has been widely used in image denoising, image super-resolution, and other low-level vision tasks. The 3x3 convolution converts the feature map to the same shape as the target image and introduces the inductive bias of the convolution operation into the refinement repair module. Finally, the sigmoid function converts the value of the feature map to the [0, 1] interval and output. Thus, the cloud-removed image is obtained.

[0069] Since the cloud image, the cloud-free optical image and the SAR image are not acquired simultaneously when using the dataset created by real images, it cannot be ensured that the ground object information is completely consistent, and the color and brightness information of the two images will inevitably be affected due to the thick cloud existing between the cloud image and the cloud-free optical image. Technically, the ground real cloud-free image corresponding to the cloud image cannot be obtained. Therefore, the present application uses the learned perceptual image patch similarity (LPIPS) to calculate the reconstruction loss. LPIPS can map the predicted image and the real image to the same dimension in the same way and prioritize the perceptual similarity between them. The definition of LPIPS is described as follows:

[0070]

[0071] Wherein, the lower the value of LPIPS, the more similar the two images are, and vice versa, the difference is large. The specific content of the formula includes: extracting the feature stack from the L layer and performing unit-normalization in the channel dimension. The vector is used to scale the number of activation channels, and finally the L2 distance is calculated. Finally, the space is averaged and the sum is summed in the channel.

[0072] In addition, for any position in the feature map M, the present application sets the feature space to the corresponding layer of CSA in the target layer and the decoder respectively to calculate the L2 distance. In order to match the shape of the feature map, the present application defines the consistency loss L c as:

[0073]

[0074] Wherein, Φ n is the activation map of the selected layer in the dataset, I in is the input, I p is the first-stage rough prediction result, I r is the final output result. CSA(·) represents the feature after using the CSA layer, CSA d (·) is the corresponding feature in the decoder, using I in as the condition, I p is used as the input of the refinement network to predict the final result I r .

[0075] In an embodiment of the present application, the present application further includes a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the above-mentioned any optical remote sensing image and SAR image fusion progressive repair framework cloud removal method.

[0076] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by computer program related hardware. The foregoing computer program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes various storage medium capable of storing program codes, such as ROM, RAM, magnetic disc or optical disc.

[0077] An optical remote sensing image and SAR image fusion progressive repair framework cloud removal device, comprising a processor and a memory; the memory is used for storing a computer program; the processor is connected with the memory and is used for executing the computer program stored in the memory, so that the optical remote sensing image and SAR image fusion progressive repair framework cloud removal device executes any above-mentioned optical remote sensing image and SAR image fusion progressive repair framework cloud removal method.

[0078] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A progressive inpainting framework cloud removal method for fusing optical remote sensing images with SAR images, characterized in that: Comprising the following steps: S1: Obtain optical remote sensing image dataset and divide it into training set, validation set and test set; the dataset includes Sentinel-1 SAR image, Sentinel-2 cloudy optical image and Sentinel-2 near-time cloud-free optical image; S2: batch the Sentinel 2 clouded optical image and the Sentinel 1 SAR image into the progressive repair framework, train the missing area based on a simple dilated convolution combined with The reconstruction loss trains the missing area to obtain a rough prediction ; extract and fuse the global and local information of the SAR image and the RGB image to obtain the refined repaired image ; according to the Sentinel 2 cloud-free image and the cloud-removed image refined by the network The loss function of the calculation model is calculated and the cloud removal model precision is evaluated, the network model parameters are adjusted according to the model precision change, and the model convergence is completed after multiple adjustments and training; the progressive repair framework in step S2 is specifically divided into two stages of rough and refinement; wherein the rough repair stage adopts a general encoder-decoder network structure, the network includes 11 layers, the first 5 layers form a contraction path, and the 6th to 10th layers form an expansion path; in the contraction path, the size of the jth layer of all three networks is , j = 1, 2, 3, 4, 5, the convolution block performs multi-channel two-dimensional convolution with a step of 1 or 2; the expansion path has a symmetrical structure with the transpose convolution blocks of the generator G and the discriminator D, the number of channels in each layer is doubled; the 11th layer is a standard output layer with convolution operation and sigmoid activation function; the Conv Block is composed of multi-channel 2-D convolution and rectified linear unit ReLU , the transposed Conv Block is composed of transposed 2-D convolution, batch normalization BN and ReLU , in this step, the BN operation is followed and executed before the activation function; The refinement repair stage in step S2 includes a residual branch and a reconstruction branch; the residual branch copies and retains the input Sentinel-2 cloudy image, and waits to be merged with the output of the reconstruction branch; the reconstruction branch has three steps: preprocessing, encoding-decoding and image recovery; in the preprocessing, shallow features of multi-source images are extracted and mapped to the shape specified in the encoding stage; the encoding-decoding part is a U-shaped structure, the shallow features obtained after preprocessing first enter the encoding stage to obtain high-dimensional features; in the encoding stage, the high-dimensional features are decoded to obtain a reconstructed feature map with the same shape as the input; in addition, a coherent semantic attention layer (CSA) is embedded in the fourth layer of the encoder; the structure of the decoder is symmetrical to the encoder without the CSA layer, and all 4x4 convolutions are deconvolutions; in the image recovery stage, a mapping is established to ensure that the output of the reconstruction branch is consistent with the form of the RGB image, and finally the reconstructed image is merged with the RGB input image; the coherent semantic attention layer (CSA) is used to build the correlation between the deep features of the missing areas; The calculation steps of the coherent semantic attention layer (CSA) are as follows: inputting a feature map set of a current batch ; extracting the image patch in the known region and reshaping it into a convolution filter, applied to the missing region M； The maximum cross-correlation value is calculated and the closest matching neural patch : Using the most similar context Initialization ; Computing the similarity between two adjacent generated patches Attention map of interest : , = 0 Attention is paid to the figure With The attention matrix is obtained in combination Reuse known areas As deconvolution to get feature maps set ; output reconstructed feature map set ; S3: Based on the model trained in step S2, input the cloud area validation sample dataset in batches to generate cloud-free optical images and complete the cloud removal task.

2. The progressive inpainting framework for cloud removal method of fusing optical remote sensing images with SAR images according to claim 1, characterized in that: The optical remote sensing image dataset in step S1 is SEN12MS-CR; the VV and VH polarization images of the SAR image are cut into values [-25, 0] and [-32.5, 0] respectively and scaled to the range [0, 1]; the bands of all optical images are truncated to values [0, 10,000] and then normalized to the range [0, 1]; only VH polarization SAR data and RGB band optical images are used as input during training.

3. The progressive inpainting framework for cloud removal of optical remote sensing image and SAR image fusion method according to claim 1, characterized in that: In the feature extraction process in the progressive repair framework refinement repair stage, the value range of each layer of the feature map is different using different activation functions, and the obtained depth reconstruction feature map channel number C is also different from the channel number of the target image; the transformation process is completed by Decloud-IR, Decloud-IR accepts the feature map of channel C, and outputs expansion to any dimension according to the requirements of different tasks; the Decloud-IR module is composed of a squeezing excitation module SE, a 3*3 convolution and an sigmoid activation function; the SE module learns the focus point of different channels; the 3*3 convolution converts the feature map into the same shape as the target image, and introduces the inductive bias of the convolution operation into the Transformer-based module; finally, sigmoid the function converts the value of the feature map to the interval [0, 1] and the output, so as to obtain the cloud removal image.

4. The progressive repair framework for cloud removal of optical remote sensing image and SAR image fusion method according to claim 1, characterized in that: Step S2 specifically comprises the following steps: using the learning perceptual image block similarity LPIPS to calculate the reconstruction loss, LPIPS In the same way, the predicted image and the real image are mapped to the same dimension, and the perceptual similarity between them is prioritized, as described by the following formula LPIPS Definition: Wherein, the lower the value of LPIPS represents the more similar the two images, and vice versa; the specific content of the formula includes: extracting the feature stack from the L layer and unit-normalizing in the channel dimension; using the vector to scale the number of activation channels, calculating the L2 distance; finally, averaging in space and summing in channel. For any position in the feature map M, the feature space is set to the respective layer of the CSA in the CSA target layer and the decoder to compute the L2 distance, defining the consistency loss is: wherein is the activation map of the selected layer in the data set, is the input, is the first stage coarse prediction result, is the final output result; denotes the features after using the CSA layer, is the corresponding feature in the decoder, using as the conditioning to the refinement network that predicts the final result .

5. A progressive repair framework cloud removal apparatus for fusing optical remote sensing images with SAR images, characterized in that, Comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the progressive repair framework cloud removal method for fusing optical remote sensing images and SAR images according to any one of claims 1-4.

6. A computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the steps of the progressive repair framework cloud removal method for fusing optical remote sensing images and SAR images according to any one of claims 1-4.

Images