Remote sensing sar-optical image fusion cloud removal method based on generative adversarial network

Abstract

The application discloses a remote sensing SAR-optical image fusion cloud removing method based on a generative adversarial network, and belongs to the field of remote sensing optical image processing.The application aims at the problem that the existing remote sensing optical image cloud removing method is difficult to remove and restore thick cloud regions.The application comprises the following steps: introducing an attention module and a conversion module into a generative adversarial network based on the generative adversarial network and fusing SAR image information; inputting a multi-cloud image into the attention module to generate an attention map; inputting the multi-cloud image and a false optical image into a generator to restore a multi-cloud region; fusing the multi-cloud image and the output of the generator into a cloud-free image according to the attention map; inputting a true value image and the generated cloud-free image into a discriminator for judgment of truth and falsehood; introducing an attention loss and a loss function of the generator and the discriminator, and updating the parameters of the model by using a gradient descent algorithm.The method can effectively remove thin cloud and thick cloud regions in an optical image.

Description

TECHNICAL FIELD

[0001] The present application relates to a remote sensing SAR-optical image fusion cloud removal method based on a generative adversarial network, and belongs to the field of remote sensing optical image processing in remote sensing digital image processing. BACKGROUND

[0002] Remote sensing technology plays an important role in earth observation tasks. Since remote sensing optical images are easily disturbed by clouds and fog, the ground target is blocked, thereby affecting the interpretation of remote sensing images, and therefore cloud removal of remote sensing optical images has become an important issue in the field of remote sensing image processing.

[0003] The thickness of the clouds in the remote sensing optical image is not the same, and the ground target is completely blocked in the thick cloud area, and the gray value of the ground target is contaminated in the thin cloud area; in addition to the clouds, the shadow area caused by the thick clouds in the image will also affect the interpretation of the ground target. Therefore, cloud removal of remote sensing optical images is needed to restore the target disturbed by the clouds or cloud shadows.

[0004] In recent years, many scholars have devoted to improving the effect of cloud removal algorithm, and the proposed cloud removal methods are mainly divided into four categories: spatial-based method, spectral-based method, multi-temporal-based method and hybrid method. The spatial-based method uses information from cloud-free areas to restore missing data, which can effectively fill small gaps in the image; however, it is difficult to restore images with large cloud-polluted areas. The spectral-based method uses multi-spectral data to restore missing data, which can obtain satisfactory visual performance and can handle thin clouds, but when multiple or all spectral bands are contaminated, the image cannot be effectively restored. The multi-temporal-based method uses auxiliary information provided by data obtained during the similar period to restore cloud-polluted areas. However, when the ground target changes significantly during data collection, the multi-temporal-based method is no longer applicable. The hybrid method tries to combine the advantages of spatial-based, spectral-based and multi-temporal-based methods for cloud removal, but for richer target scenes, it cannot obtain good cloud removal effect at present. SUMMARY

[0005] In view of the problem that the existing remote sensing optical image cloud removal method is difficult to remove and restore thick cloud areas, the present application provides a remote sensing SAR-optical image fusion cloud removal method based on a generative adversarial network.

[0006] The remote sensing SAR-optical image fusion cloud removal method based on a generative adversarial network of the present application comprises,

[0007] Step one: collecting paired cloud optical images I c , SAR images I s and selected scene true optical images I t as training sample images;

[0008] Step two: a remote sensing optical image cloud removal model is established, and a training sample image is used for training to obtain a trained remote sensing optical image cloud removal model;

[0009] The remote sensing optical image cloud removal model comprises a conversion module, an attention module, a generator and a discriminator;

[0010] SAR image I s The converted module obtains a false optical image I f ; the false optical image I f is concatenated with the cloudy optical image I c by channel, and then is input into the generator, and the cloudy area of the cloudy optical image I c is recovered by the generator to obtain a generator-generated optical image; the cloudy optical image I c is input into the attention module to generate an attention map I M ; the attention map I M is element-wise multiplied with the generator-generated optical image, and the result is inverted and element-wise multiplied with the cloudy optical image I c , and the two multiplication results are added to obtain a recovered cloud-free image I g ;

[0011] The SAR image I s , the recovered cloud-free image I g and the selected scene real optical image I t are concatenated and input into the discriminator for true or false judgment of the image;

[0012] In the model training process, a conversion module loss function, an attention loss function, a generator loss function and a discriminator loss function are introduced, and the parameters of the remote sensing optical image cloud removal model are updated and optimized by a gradient descent algorithm; when a preset condition is met, a trained remote sensing optical image cloud removal model is obtained;

[0013] Step three: a pair of cloudy optical image I c and corresponding SAR image I s of a target scene are input into the trained remote sensing optical image cloud removal model, and a cloud-removed optical image of the cloudy optical image I c is output by the generator.

[0014] According to the remote sensing SAR-optical image fusion cloud removal method based on the generative adversarial network, the pair of cloudy optical image I c and SAR image I s in step one have the same size and resolution;

[0015] The conversion module, the attention module and the generator are realized by the same U-net network.

[0016] According to the remote sensing SAR-optical image fusion cloud removing method based on the generative adversarial network, the generator comprises 8 down-sampling layers and 8 up-sampling layers, each down-sampling layer comprises a convolution layer with a 4*4 convolution kernel, an activation layer and an instance normalization layer; the first 7 up-sampling layers in the 8 up-sampling layers in sequence comprise an activation layer, a deconvolution layer with a 4*4 convolution kernel and an instance normalization layer, and the eighth up-sampling layer comprises a convolution layer and an activation layer.

[0017] According to the remote sensing SAR-optical image fusion cloud removing method based on the generative adversarial network, in the 16 sampling layers of the generator, the i-th layer of the down-sampling layer and the 9-i-th layer of the up-sampling layer are connected by skip connection; i=1, 2, 3,..., 8.

[0018] According to the remote sensing SAR-optical image fusion cloud removing method based on the generative adversarial network, the first four network layers of the discriminator each comprise 5 convolution layers with a 4*4 kernel, 4 activation layers and 4 instance normalization layers; and the last network layer comprises a convolution layer.

[0019] According to the remote sensing SAR-optical image fusion cloud removing method based on the generative adversarial network, in step two, the attention loss function is represented as L A , which is defined as the L1 norm of the attention map I M :

[0020]

[0021] In the formula, A(I c ) = I M , A represents an attention module; N represents the total number of cloudy optical images I c in the training sample image, represents the n-th cloudy optical image I c ; and ||·||1 represents the L1 norm;

[0022] The attention loss function L A is used to encourage the attention module to focus on the cloudy area.

[0023] According to the remote sensing SAR-optical image fusion cloud removing method based on the generative adversarial network, in step two, the discriminator loss function is represented as L D :

[0024]

[0025] In the formula, E represents the expectation of the result calculated by taking all I ts as input, I represents the decision result of the decision unit on the input image. ts For SAR image I s With the selected scene real optical image I t Cascaded results To represent all I gs The expected value is calculated using the input result, I. gs For SAR image I s Compared with the restored cloudless image I g Cascaded results;

[0026] Among them, the restored cloudless image I g The method to obtain it is as follows:

[0027] I g =I M ⊙G(I c ,I f )+(1-I M )⊙I c ,

[0028] In the formula G(I) c ,I f ) represents the optical image generated by the generator, and ⊙ represents element-wise dot product.

[0029] According to the cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks of the present invention, in step two, the generator loss function is expressed as L. G :

[0030] L G =L adv +λ1L1(I t ,I g )+λ2L SSIM (I t ,I g ),

[0031] In the formula L adv To counteract the loss, λ1 is the weight of the L1 norm loss function, L SSIM Let λ be the structural similarity loss function, and λ2 be the loss function for the restored cloudless image I. g and real optical images of selected scenes I t The weights of the structural similarity loss function; the values ​​of λ1 and λ2 range from 0 to 10;

[0032]

[0033]

[0034] In the formula I represents the nth restored cloudless image. g , I represents the nth real optical image of the selected scene. t ;

[0035]

[0036] In the formula, SSIM represents structural similarity.

[0037] According to the cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks of the present invention, the loss function of the conversion module is expressed as L. T :

[0038] I T =L1(I f ,I t )+λ3L SSIM (I f ,I t ),

[0039] In the formula, λ3 represents the value for the pseudo-optical image I. f and real optical images of selected scenes I t The weights of the structural similarity loss function range from 0 to 10.

[0040] In the cloud removal method of remote sensing SAR-optical image fusion based on generative adversarial networks according to the present invention, in step two, the cloud removal model of remote sensing optical image is trained using the PyTorch deep learning framework.

[0041] The beneficial effects of the present invention are as follows: The method of the present invention addresses the problem that current methods are only applicable to targets with single-feature cloud cover for cloud removal, especially the inability to effectively restore thick cloud areas, and can effectively improve the cloud removal effect of remote sensing images.

[0042] This invention's method is based on a generative adversarial network (GAN) and fuses SAR image information, introducing an attention module and a transformation module into the GAN. A cloudy image is input to the attention module to generate an attention map; then, the cloudy image and a fake optical image are input to the generator to reconstruct the cloudy region. Based on the attention map, the cloudy image and the generator's output are fused into a cloudless image. Subsequently, the ground truth image and the generated cloudless image are concatenated with the SAR image respectively and input to a discriminator, which determines whether the two types of images are "real" or "fake." An attention loss and loss functions for the generator and discriminator are introduced, and a gradient descent algorithm is used to update the model's parameters. The attention module in this invention can effectively detect thin clouds, thick clouds, and other regions in optical images.

[0043] The method can effectively remove thick clouds, thin clouds, cloud shadows and the like, and can effectively remove thick clouds, thin clouds, cloud shadows and the like for target images of various forms of cloud and fog blockage. Through experimental verification, compared with other SAR image-based cloud removal methods (SAR-opt-GAN, Simulation-Fusion GAN and the like), the method can obtain a cloud-free image with higher quality. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the remote sensing SAR-optical image fusion cloud removal method based on a generative adversarial network according to the present application;

[0045] Figure 2 is a network structure diagram for obtaining the cloud-free image I g ; in the diagram, Transposed Conv+InstanceNorm+ReLU represents a convolution layer+an instance normalization layer+an ReLU activation function, Transposed Conv+Tanh represents a transposed convolution layer+an Tanh activation function, Conv+InstanceNorm+LeakyReLU represents a convolution layer+an instance normalization layer+an ReLU activation function, Addition by pixel represents pixel-by-pixel addition, Multiply by pixel represents pixel-by-pixel multiplication, and Negate and multiply by pixel represents taking negation and then pixel-by-pixel multiplication;

[0046] Figure 3 is a decision diagram of the decision maker; in the diagram, Conv represents a convolution layer;

[0047] Figure 4 is a comparison diagram of simulation experimental results of the method according to the present application;

[0048] Figure 5 is a comparison diagram of true cloud experimental results of the method according to the present application;

[0049] Figure 6 is an ablation experimental result diagram. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0051] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0053] Specific Implementation Method 1: Combination Figures 1 to 3 As shown, this invention provides a cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks, including:

[0054] Step 1: Acquire paired cloud-covered optical images of the selected scene. c SAR image I s and real optical images of selected scenes I t As training sample images;

[0055] Step 2: Establish a cloud removal model for remote sensing optical images and train it using training sample images to obtain the trained cloud removal model for remote sensing optical images;

[0056] The cloud removal model for remote sensing optical images includes a transformation module, an attention module, a generator, and a decision maker;

[0057] SAR image I s The pseudo optical image I is obtained through the conversion module. f ; to use fake optical images I f With cloud optical image I c The data is cascaded through channels and then fed into the generator, which recovers the cloud-covered optical image I. c In cloud-covered areas, an optical image is generated by the generator; cloud-covered optical image I c Attention map I is generated by the attention module M Attention Map I M The optical image generated by the generator is multiplied element-wise, and then inverted and multiplied with the cloud-covered optical image I. c Perform element-wise dot products and add the two dot product results to obtain the restored cloudless image I. g ;

[0058] SAR image I s Compared with the restored cloudless image I g and real optical images of selected scenes I t The images are then cascaded and input into a decision unit to determine whether they are real or fake.

[0059] During model training, loss functions for the transformation module, attention, generator, and decision makers are introduced. The parameters of the remote sensing optical image cloud removal model are optimized and updated using the gradient descent algorithm. After the preset conditions are met, such as when the loss no longer decreases and becomes basically stable, the trained remote sensing optical image cloud removal model is obtained.

[0060] Step three: input the paired cloudy optical image I c and the corresponding SAR image I s into the trained remote sensing optical image cloud removal model to generate the cloud-removed optical image. c Step four: output the cloud-removed optical image.

[0061] The embodiment is based on a generative adversarial network and fuses SAR image information to remove clouds from remote sensing optical images, wherein the paired cloudy optical image I c and the SAR image I s are images collected for the same region. During model training, an attention loss function and loss functions of the generator and the discriminator are established according to the output results of each module; then, according to the established loss functions, the gradient descent method is used to optimize the parameters of the deep learning model; after training, a trained cloud removal model is obtained, and the model is saved; during testing, the paired optical image and the SAR image are input into the trained cloud removal model to generate a cloud-removed result.

[0062] In the embodiment, the preset condition for ending the model training can be 300 iteration cycles, and each iteration cycle completes a round of training of all training sample images.

[0063] Further, the paired cloudy optical image I c and the SAR image I s in step one have the same size and resolution.

[0064] As an example, the conversion module, the attention module, and the generator are implemented using the same U-net network.

[0065] As shown in Figure 2 , the generator in the embodiment includes 8 down-sampling layers and 8 up-sampling layers, each down-sampling layer includes a convolution layer with a 4x4 convolution kernel, an activation layer, and an instance normalization layer; the first 7 up-sampling layers in the 8 up-sampling layers include an activation layer, a deconvolution layer with a 4x4 convolution kernel, and an instance normalization layer in sequence, and the 8th up-sampling layer includes a convolution layer and an activation layer.

[0066] In order to fuse low-level and high-level features and reduce information loss, in the 16 sampling layers of the generator, a skip connection is performed between the i-th down-sampling layer and the 9-i-th up-sampling layer; i = 1, 2, 3, …, 8.

[0067] Further, as shown in Figure 3 , the first four network layers of the discriminator in the embodiment each include 5 convolution layers with a 4x4 kernel, 4 activation layers, and 4 instance normalization layers; and the last network layer includes a convolution layer.

[0068] Furthermore, in step two of this embodiment, the attention loss function is expressed as L A Attention loss function L A To encourage the attention network to focus on cloudy areas rather than the entire image, an attention graph I is defined. M L1 norm:

[0069]

[0070] In the formula A(I c ) = I M A represents the attention module; N represents the optical image I containing clouds in the training sample images. c The total number, I represents the nth optical image with clouds. c ||·||1 represents the L1 norm;

[0071] Attention loss function L A This is used to encourage the attention module to focus on cloudy areas.

[0072] In step two, the decision loss function is expressed as L D In this implementation, the adversarial loss from LSGAN is used:

[0073]

[0074] In the formula This indicates that for all I ts The expected value is calculated using the input result. I represents the decision result of the decision unit on the input image. ts For SAR image I s With the selected scene real optical image I t Cascaded results To represent all I gs The expected value is calculated using the input result, I. gs For SAR image I s Compared with the restored cloudless image I g Cascaded results;

[0075] Among them, the restored cloudless image I g The method to obtain it is as follows:

[0076] I g =I M ⊙G(I c ,I f )+(1-I M )⊙I c ,

[0077] In the formula G(I) c ,I f) represents the generator generated optical image obtained by the generator, and represents element-wise point multiplication.

[0078] In this embodiment, I M is used as prior information, and I c is the cloud-free area, and the output of the generator is used to recover the multi-cloud area.

[0079] In step two, the generator loss function is represented as L G :

[0080] L G = L adv + λ1L1(I t ,I g ) + λ2L SSIM (I t ,I g ),

[0081] wherein L adv is the adversarial loss, λ1 is the weight of the L1 norm loss function, L SSIM is the structural similarity loss function, λ2 is the weight of the structural similarity loss function for the recovered cloud-free image I g and the selected scene real optical image I t , and the two weights can be adjusted according to the generated image during training, and in general cases, the value range is within 0-10;

[0082]

[0083]

[0084] wherein I represents the nth recovered cloud-free image I g , represents the nth selected scene real optical image I t ;

[0085]

[0086] wherein SSIM represents the structural similarity.

[0087] The conversion module loss function is represented as L T :

[0088] I T = L1(I f ,I t ) + λ3L SSIM (I f ,I t ),

[0089] wherein λ3 is the weight of the structural similarity loss function for the false optical image If and selected scene real optical image I t The weight of the structural similarity loss function is adjusted according to the condition of the generated image during training, and the value is generally within 0-1.

[0090] In this embodiment, a loss function is designed for training the cloud removal model. After the model training is completed, the paired SAR image and optical image are input into the model, and the cloud is removed through the conversion module, attention module and generator. The method has good cloud removal effect on cloud images of different thicknesses in different environments.

[0091] As an example, in step two, the remote sensing optical image cloud removal model is trained using the pytorch deep learning framework. Pytorch has an automatic differentiation mechanism, and gradient descent can be used to update the model parameters to obtain the trained model, complete the training phase, and save the trained model. During testing, the paired optical image and SAR image are input into the model to generate cloud-free images. Specific embodiments:

[0093] The effectiveness of the method is illustrated by experiments as follows:

[0094] 1. Experimental data:

[0095] To verify the proposed remote sensing optical image cloud removal model, real and simulated data experiments for cloud removal are performed. In the real data experiment, the SEN12MS-CR dataset is used, which contains 122,218 pairs of corresponding Sentinel-1 dual-polarization SAR data, Sentinel-2 multispectral images and Sentinel-2 multispectral images covered by cloud layers. The Sentinel-2 multispectral image includes 13 bands, and in this embodiment, three bands of R, G and B are used.

[0096] In the simulation experiment, 1342 pairs of cloud-free images and SAR images are randomly selected from SEN12MS-CR, and 1342 simulated cloudy images are generated using these cloud-free images. Among them, 1000 pairs of data are used to train the model, and the remaining 342 pairs of data are used to test the performance of the model. In the real cloud experiment, 1436 pairs of cloudy images, cloud-free images and SAR images are randomly selected from the SEN12MS-CR dataset. 936 pairs of data are selected to train the model and 500 pairs of data are selected to test the model.

[0097] 2. Experimental results and analysis:

[0098] To verify the superiority of the cloud removal method proposed in the present application, comparisons are made with other advanced cloud removal methods, including a pix2pix model, a SAR-opt-GAN model, a Simulation-Fusion GAN model and a GLF-CR model. All the models are retrained on the SEN12MS-CR dataset.

[0099] In Figure 4 , the cloud removal effects of the various methods in the simulation experiment are shown, Figure 4 column (a) shows the SAR images, column (b) shows the attention maps, column (c) shows the cloudy images, columns (d) to (h) correspond to the results obtained by the pix2pix model, the SAR-opt-GAN model, the Simulation-Fusion GAN model, the GLF-CR model and the model of the present application, and column (i) shows the ground truth images.

[0100] It can be seen from Figure 4 that the attention module can accurately detect the cloudy areas. In the attention map, the values of the cloudy areas are large, and the values of the non-cloudy areas are small. Since the pix2pix model does not use the SAR image as auxiliary information, it cannot recover the information of the thick cloud area. Compared with the pix2pix model, the SAR-opt-GAN model can recover the information of part of the thick cloud area, but the recovery performance is limited due to the inaccurate recovery of many areas. The Simulation-Fusion GAN model improves the performance by recovering the information of the thick cloud area. However, some areas become blurred and important image details are lost. Similarly, the GLF-CR model recovers very blurred areas, and the details of the thick cloud coverage area cannot be well recovered. Compared with the other methods, the model proposed in the present application can reconstruct the areas blocked by thick clouds and most accurately recover the details.

[0101] The cloud removal results are evaluated by three indicators, root mean square error (RMSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as follows:

[0102]

[0103] In the formula, MSE is the mean square error, h is the number of columns of the image pixels, l is the number of rows of the image pixels, I(p, q) is the pth row qth column pixel of the image I, and K(p, q) is the pth row qth column pixel of the image K.

[0104] The root mean square error (RMSE) is the square root of the MSE.

[0105]

[0106] In the formula, MAX IFor the maximum value of the pixels that can occur in an image I, for example, the maximum value of a pixel in an 8-bit image is 255.

[0107]

[0108] where x is the image x, y is the image y, μ x and μ y are the mean of x, y respectively, σ xy denotes the covariance of x and y, σ x and σ y are the standard deviation of the images x and y respectively, c1 and c2 are constants respectively.

[0109] After calculating the above indicators, the average of all test images is obtained, and the experimental results of the simulation experiment are shown in Table 1. The cloud removal results obtained by the model of the application are better than those of other models in the three indicators. Through observation of the visual effect of cloud removal and quantitative index evaluation, the results show that the model of the application has good cloud removal effect.

[0110] Table 1 Comparison of cloud removal effect evaluation indicators of different models in simulation experiment

[0111]

[0112] Figure 5 shows the cloud removal effect of each method in the real cloud experiment, Figure 5 column (a) shows the SAR image, column (b) shows the attention map, column (c) shows the cloudy image, columns (d) to (h) correspond to the results obtained by the pix2pix model, the SAR-opt-GAN model, the Simulation-Fusion GAN model, the GLF-CR model and the model of the application, and column (i) shows the ground truth image.

[0113] The results show that the results are basically consistent with the results of the simulation experiment, and the attention map can extract the distribution of the real cloud. The results provided by the model of the application are better than those of other methods in vision, and the ground objects blocked by thick clouds can be restored. In addition, the details of the land cover in the cloud-free area are most accurately preserved in the result image of the method of the application.

[0114] For the real cloud experiment, three indicators of peak signal-to-noise ratio (PSNR), root mean square error (RMSE) and structural similarity (SSIM) are also used to evaluate the cloud removal results. The average of all test images is obtained, and the experimental results of the real cloud experiment are shown in Table 2. The cloud removal results obtained by the model of the application are better than those of other models in the three indicators.

[0115] In the real cloud experiment, through observation of the visual effect of cloud removal and quantitative index evaluation, the results show that the model of the application has good cloud removal effect.

[0116] Table 2 Comparison of cloud removal effect evaluation indexes of different models in true cloud experiment

[0117]

[0118] In order to verify the role of each module in the method of the present application, an ablation experiment is performed, the attention module and the conversion module are removed respectively, and the experimental effect is evaluated, and the experimental results are as shown in Figure 6

[0119] Figure 6 In the table, the areas boxed in the lower part of the picture in the first row and the third column, the upper part of the picture in the first row and the third column, the lower part of the picture in the second row and the second column, and the upper part of the picture in the second row and the third column represent areas that cannot be accurately restored, and other boxes in all pictures represent well-restored areas. As can be easily observed from Figure 6 it can be easily observed that, in the absence of the attention module or the conversion module, the cloud removal result is severely damaged. When the conversion module is removed, the SAR image is directly input into the generator. Due to the existence of speckle noise in the SAR image, the restoration result of the cloudy area is severely affected. Therefore, this shows that the conversion module can effectively eliminate noise, thereby ensuring the restoration performance. When the attention module is removed, it can be observed that the cloud-free area in some images is not well preserved, because the attention module ensures that the cloud-free area remains unchanged during the reconstruction process.

[0120] Table 3 shows the quantitative comparison of the proposed model with and without the attention module and the conversion module, which shows that the introduction of the attention module and the conversion module helps to improve the quality of cloud removal.

[0121] Table 3 Ablation experiment results

[0122]

[0123] In summary, the method of the present application can adapt to more rich scenes to realize optical image cloud removal.

[0124] Although the present application is described herein with reference to particular embodiments, it should be understood that these examples are merely illustrative of the principles and applications of the present application. It should therefore be understood that numerous modifications can be made to the illustrative embodiments and that other arrangements can be devised without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that the features described in connection with one embodiment can be used in conjunction with other embodiments described herein. It should also be understood that features described in connection with separate embodiments can be used in other described embodiments.​

Claims

1. A cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks, characterized in that... include, Step 1: Acquire paired cloud-covered optical images of the selected scene. SAR images and real optical images of selected scenes As training sample images; Step 2: Establish a cloud removal model for remote sensing optical images and train it using training sample images to obtain the trained cloud removal model for remote sensing optical images; The cloud removal model for remote sensing optical images includes a transformation module, an attention module, a generator, and a decision maker; SAR images The conversion module obtains a fake optical image. ; fake optical images With cloud optical images The images are cascaded through channels and then fed into a generator to reconstruct the cloud-covered optical image. In cloud-covered areas, an optical image is generated by the generator; cloud-covered optical image. Attention map generated by the attention module Attention map Perform element-wise multiplication with the optical image generated by the generator, then invert the result and combine it with the cloud-covered optical image. Perform element-wise dot products and add the two dot product results to obtain the restored cloudless image. ; SAR images respectively Compared with the restored cloudless image and real optical images of selected scenes The images are then cascaded and input into a decision unit to determine whether they are real or fake. During model training, loss functions for the transformation module, attention, generator, and decision maker are introduced. The parameters of the remote sensing optical image cloud removal model are optimized and updated using the gradient descent algorithm. After the preset conditions are met, the trained remote sensing optical image cloud removal model is obtained. Step 3: Combine paired cloud-covered optical images of the target scene. and the corresponding SAR image The cloud removal model is input into the remote sensing optical image training process, and the generator outputs a cloud-covered optical image. Optical images after cloud removal; In step two, the attention loss function is expressed as: Defined as attention map of Norm: ， In the formula = A represents the attention module; N represents the optical image containing clouds in the training sample images. The total number, This represents the nth optical image with clouds. ; express Norm; Attention loss function Used to encourage the attention module to focus on cloudy areas; In step two, the decision loss function is expressed as: : ， In the formula Indicates all The expected value is calculated from the input result. This indicates the decision result of the decision-maker on the input image. SAR image With real optical images of the selected scene Cascaded results To indicate for all The expected value is calculated from the input result. SAR image Compared with the restored cloudless image Cascaded results; Among them, the restored cloudless image The method to obtain it is as follows: ， In the formula This indicates that the generator produces an optical image. This represents element-wise dot product; In step two, the generator loss function is expressed as follows: : ， In the formula To combat the losses, for The weights of the norm loss function The structural similarity loss function is... For the restored cloudless image and real optical images of selected scenes The weights of the structural similarity loss function; and The value range is 0-10; ， ， In the formula This represents the nth restored cloudless image. , Represents the nth real optical image of the selected scene. ; ， In the formula Indicates structural similarity; The loss function of the conversion module is expressed as: : ， In the formula To target fake optical images and real optical images of selected scenes The weights of the structural similarity loss function range from 0 to 10.

2. The cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks according to claim 1, characterized in that, Paired cloud-covered optical images in step one and SAR images They have the same size and resolution; The transformation module, attention module, and generator are implemented using the same U-net network.

3. The cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks according to claim 2, characterized in that, The generator includes 8 downsampling layers and 8 upsampling layers. Each downsampling layer includes a convolutional layer with a 4×4 kernel, an activation layer, and an instance normalization layer. The first 7 upsampling layers in sequence include an activation layer, a deconvolutional layer with a 4×4 kernel, and an instance normalization layer. The 8th upsampling layer includes a convolutional layer and an activation layer.

4. The cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks according to claim 3, characterized in that, In the generator's 16 sampling layers, there is a skip connection between the i-th downsampling layer and the (9-i)-th upsampling layer; i=1,2,3,...

8.

5. The cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks according to claim 4, characterized in that, The first four network layers of the decision maker each consist of five convolutional layers with 4×4 kernels, four activation layers, and four instance normalization layers; the last network layer consists of a convolutional layer.

6. The cloud removal method for remote sensing SAR-optical image fusion based on generative adversarial networks according to claim 5, characterized in that, In step two, the cloud removal model for remote sensing optical images is trained using the PyTorch deep learning framework.

Images