Image cloud removing method and system based on optical remote sensing image and SAR image

Abstract

The application provides an image cloud removing method and system based on optical remote sensing images and SAR images, and comprises the following steps: designing a heterogeneous feature cooperative convolution module, considering the influence of different degrees of cloud and shadow, and effectively improving the precision of optical SAR image fusion cloud removing results by using a cloud-cloud shadow self-attention module in a spatial domain, solving the problems of blurred results generated by a traditional convolutional neural network in the optical SAR image fusion cloud removing task, and distorted results of cloud center position recovery; in the high-dimensional feature part of the network, a Transformer module is introduced to learn global features and global information, and a solution is provided for generating high-fidelity optical images in the optical SAR image fusion cloud removing.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and in particular to an image declouding method and system based on optical remote sensing images and SAR images. Background Technology

[0002] With the continuous development of modern remote sensing technology, the amount of remote sensing image data that can be obtained has increased rapidly. Remote sensing imaging technology, because it can periodically obtain real-time information about objects on the ground, has become a focus of attention and extensive research. However, due to the influence of weather and climate factors, especially the influence of clouds in the atmosphere, optical remote sensing images obtained by remote sensing satellites are often interfered with by clouds due to their weak penetration ability. This generally manifests as cloud obscuring of ground objects and interference from cloud shadow areas on the representation of ground features, thus affecting the processing and use of remote sensing images. Therefore, removing clouds and cloud shadow areas from remote sensing images to restore obscured information has become a hot research area in remote sensing image processing.

[0003] Synthetic Aperture Radar (SAR) possesses various advantages such as all-weather, all-time coverage, and cloud penetration. However, SAR images suffer from problems like speckle noise, making them difficult to interpret. Fusion of optical and SAR image data can achieve information complementarity between imaging from different types of sensors, better supporting downstream tasks of remote sensing imagery. Current research on cloud removal from remote sensing images using SAR imagery (hereinafter referred to as optical SAR image fusion cloud removal) mainly falls into two categories: traditional fusion methods based on mechanistic models and methods based on deep learning.

[0004] Traditional image fusion methods include spatial domain-based fusion and transform domain-based fusion. Spatial domain-based fusion is a simple and fast method that directly operates on pixels to obtain the fusion result. Common methods include HIS, Brovey, PCA, GS, and weighted average. Transform domain methods, as one of the most popular algorithms in image fusion, have also developed rapidly in recent years. They mainly include multi-scale transforms and model-based methods. Multi-scale transforms include pyramid transform, wavelet transform, ridge transform, shear wave transform, curvelet transform, NSCT, and NSST, while model-based methods include Pulse Coupled Network (PCNN) and sparse representation. Spatial domain methods are simple, intuitive, and fast, but the simple superposition operation fusion rules cannot guarantee fusion accuracy. For transform domain algorithms, due to computational cost limitations and the difficulty of algorithm implementation, these algorithms are difficult to implement practically. In recent years, research on cloud removal in optical SAR image fusion has gradually shifted towards deep learning.

[0005] For cloud removal in optical SAR image fusion, convolutional neural networks (CNNs) and generative adversarial networks (GANs) are currently the most widely used deep learning methods. To overcome the difficulties of manual feature extraction and the challenge of colorizing SAR images in traditional methods, Chen Keming et al. used CNNs to achieve automatic fusion of optical and SAR remote sensing images; Zhang et al. trained a 13-layer network based on AlexNet to extract image features, solving the problem of manual feature extraction and achieving better intelligent image fusion. Schmitt et al. used an autoencoder and a hybrid density network to learn Sentinel-1 images from registered Sentinel-2 image sets for colorization, and then performed image interpretation. Meraner et al. concatenated SAR images with input optical images and then used a deep residual neural network to predict cloud-free optical images of the target.

[0006] To further improve the quality of image fusion, generative adversarial networks (GANs) have been widely applied to optical SAR fusion and cloud removal tasks. The GAN proposed by Goodfellow et al. is one of the most popular networks in the field of deep learning. Grohnfeldt et al. proposed a SAR-OPT-CGAN conditional GAN ​​architecture specifically for handling the fusion of SAR and multispectral images. The emergence of GANs has brought opportunities for optical and SAR image fusion, and numerous studies have demonstrated the feasibility of information fusion through image translation and showcased the potential of deep learning methods in image fusion. However, GANs still face many bottlenecks, as they cannot effectively transform specific ground features and texture information.

[0007] In recent years, the emergence of the Transformer architecture has provided a feasible supplement to the local perceptual characteristics of CNN convolution. Fang et al. designed a cloud removal method based on global-local fusion (GLF-CR), which introduces a self-attention mechanism to establish global dependencies. It performed well in the cloud removal task of fusion of SAR and multispectral images. The success of GLF-CR proves the importance of global information in remote sensing cloud removal tasks. However, the actual cloud removal effect of the above methods still has key problems such as low result sharpness, serious loss of ground feature details and textures, and distortion of cloud center recovery results. Summary of the Invention

[0008] This invention provides an image declouding method and system based on optical remote sensing images and SAR images, which solves the defects of existing declouding methods for remote sensing fused images, such as poor anti-interference effect and low processing accuracy.

[0009] In a first aspect, the present invention provides an image cloud removal method based on optical remote sensing images and SAR images, comprising:

[0010] Acquire SAR image sample data and multispectral remote sensing image sample data, and perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouded dataset.

[0011] A neural network for declouding optical SAR images is constructed by deep co-convolutional fusion of remote sensing images. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The optical SAR image deep co-convolutional fusion remote sensing image declouding neural network is trained using the training dataset to obtain the optical SAR image declouding model.

[0012] The multispectral remote sensing image to be processed and the corresponding SAR image at the same location are input into the optical SAR image declouding model, and the image declouding result is output. The image declouding result includes several declouding result slices.

[0013] The cloud-removed result slices are aggregated according to the original slice index to obtain the complete image cloud-removed result. The stitching edges of the complete image cloud-removed result are then filtered to obtain the final image cloud-removed result.

[0014] According to the present invention, an image cloud removal method based on optical remote sensing imagery and SAR imagery is provided, which acquires SAR image sample data and multispectral remote sensing image sample data, and performs dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion cloud-removed dataset, including:

[0015] Two adjacent temporal images of the multispectral remote sensing image sample data are collected in the same area. The two adjacent temporal images include a first temporal image with a cloud content ratio lower than a first preset cloud content ratio and a second temporal image with a cloud content ratio higher than the preset cloud content ratio but lower than a second preset cloud content ratio. The first temporal image is used to create a supervision label, and the second temporal image is used to create training data.

[0016] Atmospheric correction is performed on the two adjacent temporal images to obtain a preprocessed remote sensing image of a preset level. Each band of the preprocessed remote sensing image of the preset level is upsampled according to the highest resolution band to obtain an optical preprocessed image.

[0017] The SAR image sample data is sequentially subjected to thermal noise removal, orbit correction, radiometric calibration, speckle filtering, terrain correction, and decibel reduction to obtain a SAR preprocessed image. The SAR preprocessed image is located in the same position as the optical preprocessed image and has a preset temporal difference range.

[0018] The SAR preprocessed image is reprojected according to the corresponding optical preprocessed image, and the SAR preprocessed image and the optical preprocessed image are mapped to the same coordinate space. The SAR preprocessed image is cropped according to the optical preprocessed image so that the position range of the SAR preprocessed image and the optical preprocessed image are consistent. The cropped SAR preprocessed image and the optical preprocessed image are cropped into pixel blocks of a preset pixel size to obtain the initial optical SAR image fusion cloud removal dataset.

[0019] The initial optical SAR image fusion de-cloud dataset is constructed by cropping each band of the optical image to a preset band range and redistributing it to a preset distribution interval. The vertical-vertical polarization (VV) channel and vertical-horizontal polarization (VH) channel of the initial optical SAR image fusion de-cloud dataset are then normalized to the preset distribution interval using maximum and minimum values.

[0020] According to the present invention, an image cloud removal method based on optical remote sensing imagery and SAR imagery is provided, which constructs a deep co-convolutional fusion neural network for optical SAR imagery and remote sensing imagery cloud removal, including:

[0021] The network structure is defined as consisting of a cloud-to-cloud shadow attention module, a feature encoding module, a Transformer module, and a feature decoding module.

[0022] The optical SAR image fusion de-clouding dataset is input into the cloud-cloud shadow attention module. The cloud-cloud shadow attention module generates weight maps of cloud areas and cloud shadow areas. A preset activation function is used to map the weights in the weight maps of cloud areas and cloud shadow areas to between 0 and 1.

[0023] The feature encoding module includes four feature encoding layers. The weight maps of the cloud area and the cloud shadow area are input into the feature encoding module, and the encoded SAR image and the feature map of the cloudy optical image are output.

[0024] The Transformer module includes 8 Transformer coding layers, each Transformer coding layer includes 8 attention heads, and each Transformer coding layer consists of a self-attention module and a fully connected feedforward network layer;

[0025] The feature decoding module includes four feature decoding layers.

[0026] According to the present invention, an image declouding method based on optical remote sensing imagery and SAR imagery replaces the Conv2d convolutional module in the optical SAR imagery deep co-convolutional fusion remote sensing image declouding neural network with a co-convolutional module, wherein the co-convolutional module comprises:

[0027]

[0028] Wherein, cld′ is the output feature map, cld and sar are the input feature maps of the cloudy remote sensing image and SAR image respectively, M is the input attention map, W is the convolution kernel, b is the bias of the convolution operation, sum(1) represents the sum of the elements of a matrix with all elements equal to 1 and the same size as the convolution kernel, and sum(M) represents the sum of the elements of a matrix with all elements equal to 1 and the same size as the convolution kernel.

[0029] According to the present invention, an image declouding method based on optical remote sensing imagery and SAR imagery is provided. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The training dataset is used to train the optical SAR imagery deep co-convolutional fusion remote sensing image declouding neural network to obtain an optical SAR image declouding model, comprising:

[0030] The optical SAR image deep co-convolutional fusion remote sensing image declouding neural network was trained using SAR images, clouded remote sensing images, and cloudless remote sensing images from the training dataset, and the network parameters were initialized.

[0031] The combined loss function is constructed by using the L1 Loss loss function with the same weights and the LPIPS loss function that learns perceptual image patch similarity:

[0032]

[0033] Wherein, L1Loss represents the pixel-level difference, and LPIPSLoss represents the L2 level loss between the model output and the supervision label in high-dimensional space;

[0034]

[0035] Where X(i,j,k) represents the pixel value of the model output at position (i,j) in channel k, Y(i,j,k) represents the pixel value of the supervision label at position (i,j) in channel k, and H, W, and C represent the height, width, and number of channels of the image slice, respectively.

[0036]

[0037] Where y and y0 represent the model output and the ground truth, respectively; H and W represent the height and width of the image; and h and w represent the index of the pixel's position in the image.

[0038] The Adam optimizer is used to reduce the gradient during backpropagation, and the network parameters are adjusted until the model converges, thus obtaining the cloud removal model for the optical SAR image.

[0039] The image cloud removal method based on optical remote sensing images and SAR images provided by the present invention further includes:

[0040] The accuracy of the prediction results of the optical SAR image declouding model is evaluated using multiple preset evaluation indicators.

[0041] Secondly, the present invention also provides an image cloud removal system based on optical remote sensing images and SAR images, comprising:

[0042] The data processing module is used to acquire SAR image sample data and multispectral remote sensing image sample data, and to perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouded dataset.

[0043] The training module is used to construct a neural network for declouding optical SAR images through deep co-convolutional fusion. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The training dataset is used to train the optical SAR image deep co-convolutional fusion remote sensing image declouding neural network to obtain an optical SAR image declouding model.

[0044] The slicing module is used to input the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and output the image declouding result, which includes several declouding result slices.

[0045] The processing module is used to aggregate the several declouded result slices according to the original slice index to obtain a complete image declouded result, and to filter the stitching edges of the complete image declouded result to obtain the final image declouded result.

[0046] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image declouding method based on optical remote sensing images and SAR images as described above.

[0047] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image declouding method based on optical remote sensing images and SAR images as described above.

[0048] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image declouding method based on optical remote sensing images and SAR images as described above.

[0049] The cloud removal method and system based on optical remote sensing images and SAR images provided by this invention, by designing a heterogeneous feature collaborative convolution module, simultaneously considers the influence of different degrees of cloud and fog cover and cloud shadows, and by adopting a spatial domain cloud-cloud shadow self-attention module, effectively improves the accuracy of the cloud removal results of optical SAR image fusion, and solves the problems of blurry results and distorted cloud center position recovery results generated by traditional convolutional neural networks in optical SAR image fusion cloud removal tasks; in the high-dimensional feature part of the network, a Transformer module is introduced to learn global features and global information, providing a solution for generating high-fidelity optical images through optical SAR image fusion cloud removal. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is one of the flowcharts illustrating the image cloud removal method based on optical remote sensing images and SAR images provided by the present invention;

[0052] Figure 2 This is the second flowchart illustrating the image cloud removal method based on optical remote sensing images and SAR images provided by the present invention.

[0053] Figure 3 This is a network structure diagram of the optical SAR image collaborative convolutional neural network provided by the present invention;

[0054] Figure 4 This is a structural diagram of the cloud-cloud shadow attention module provided by the present invention;

[0055] Figure 5 This is a structural diagram of the Enocder / Decoder module provided by the present invention;

[0056] Figure 6 This is a structural diagram of the TransformerEncoder module with global feature enhancement provided by the present invention;

[0057] Figure 7 This is a schematic diagram of the image declouding system based on optical remote sensing images and SAR images provided by the present invention;

[0058] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] Figure 1 This is one of the flowcharts illustrating the image cloud removal method based on optical remote sensing images and SAR images provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0061] Step 100: Obtain SAR image sample data and multispectral remote sensing image sample data, and perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset.

[0062] Step 200: Construct a neural network for declouding optical SAR images through deep co-convolutional fusion. Divide the optical SAR image fusion declouding dataset into a training dataset and a test dataset. Use the training dataset to train the neural network for declouding optical SAR images through deep co-convolutional fusion to obtain the optical SAR image declouding model.

[0063] Step 300: Input the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and output the image declouding result, which includes several declouding result slices.

[0064] Step 400: Aggregate the several declouded result slices according to the original slice index to obtain the complete image declouded result, and filter the stitching edges of the complete image declouded result to obtain the final image declouded result.

[0065] Specifically, the cloud removal method based on collaborative convolutional global feature intelligent learning of optical remote sensing images and SAR images proposed in this embodiment of the invention includes:

[0066] For SAR and multispectral remote sensing image data, data preprocessing, coordinate transformation, and image cropping are performed to obtain the corresponding initial optical SAR image fusion decloud dataset. The image pairs in the initial optical SAR image fusion decloud dataset are preprocessed before model input. For optical images, band cropping and redistribution are performed to adapt to subsequent model training.

[0067] Then, a deep co-convolutional fusion remote sensing image declouding neural network for optical SAR images was constructed. This neural network consists of four modules. The obtained dataset was divided into training dataset and test dataset. The constructed optical SAR deep co-convolutional fusion remote sensing image declouding neural network was trained using the corresponding SAR images, clouded remote sensing images and cloudless remote sensing images in each training dataset. The network parameters were initialized, and L1 Loss combined with Learned Perceptual Image Patch Similarity (LPIPS) Loss was used. The two losses were linearly combined with a weight of 1:1 as the loss function for model training. The training used SAR images and corresponding clouded optical images as inputs, and corresponding cloudless optical images as supervision labels for training, to obtain the optical SAR image declouding model.

[0068] Furthermore, for a multispectral remote sensing image without supervised labels (cloud-free optical image), preprocessing (mainly atmospheric correction) is performed first. Then, the image is cropped to an N*N size according to the cropping size of the preprocessed dataset (no longer needing to overlap). SAR images at the same location are obtained and preprocessed and cropped in the same way to match the model input. The pre-trained model is used for prediction to obtain the cloud removal result corresponding to the image. The result consists of a set of several N*N pixel slices. Using the obtained cloud removal result slices, the slices are aggregated according to the index before the original slices to obtain the complete cloud removal result of the image. Finally, the stitching edges of the aggregated complete image are filtered to eliminate the image information differences at the slice stitching edges, resulting in the final cloud removal result.

[0069] This invention designs a heterogeneous feature collaborative convolution module that considers the effects of varying degrees of cloud cover and cloud shadows. By employing a spatial domain cloud-cloud shadow self-attention module, it effectively improves the accuracy of declouding results in optical SAR image fusion. This solves the problems of blurred results and distorted cloud center location recovery results generated by traditional convolutional neural networks in optical SAR image fusion declouding tasks. In the high-dimensional feature part of the network, a Transformer module is introduced to learn global features and global information, providing a solution for generating high-fidelity optical images through optical SAR image fusion declouding.

[0070] In one embodiment, SAR image sample data and multispectral remote sensing image sample data are acquired, and dataset processing is performed on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion de-clouding dataset, including:

[0071] Two adjacent temporal images of the multispectral remote sensing image sample data are collected in the same area. The two adjacent temporal images include a first temporal image with a cloud content ratio lower than a first preset cloud content ratio and a second temporal image with a cloud content ratio higher than the preset cloud content ratio but lower than a second preset cloud content ratio. The first temporal image is used to create a supervision label, and the second temporal image is used to create training data.

[0072] Atmospheric correction is performed on the two adjacent temporal images to obtain a preprocessed remote sensing image of a preset level. Each band of the preprocessed remote sensing image of the preset level is upsampled according to the highest resolution band to obtain an optical preprocessed image.

[0073] The SAR image sample data is sequentially subjected to thermal noise removal, orbit correction, radiometric calibration, speckle filtering, terrain correction, and decibel reduction to obtain a SAR preprocessed image. The SAR preprocessed image is located in the same position as the optical preprocessed image and has a preset temporal difference range.

[0074] The SAR preprocessed image is reprojected according to the corresponding optical preprocessed image, and the SAR preprocessed image and the optical preprocessed image are mapped to the same coordinate space. The SAR preprocessed image is cropped according to the optical preprocessed image so that the position range of the SAR preprocessed image and the optical preprocessed image are consistent. The cropped SAR preprocessed image and the optical preprocessed image are cropped into pixel blocks of a preset pixel size to obtain the initial optical SAR image fusion cloud removal dataset.

[0075] The initial optical SAR image fusion de-cloud dataset is constructed by cropping each band of the optical image to a preset band range and redistributing it to a preset distribution interval. The vertical-vertical polarization (VV) channel and vertical-horizontal polarization (VH) channel of the initial optical SAR image fusion de-cloud dataset are then normalized to the preset distribution interval using maximum and minimum values.

[0076] Specifically, such as Figure 2 As shown, in this embodiment of the invention, when acquiring basic sample data, Sentinel-1 SAR data and Sentinel-2 optical image data are obtained from the European Space Agency's official website. Atmospheric bridging and cropping preprocessing are performed on the optical images, and the SAR images are preprocessed, reprojected, and cropped according to the aforementioned steps. This results in a dataset consisting of multiple corresponding SAR images, cloud-covered optical images, and cloudless optical images.

[0077] For processing multispectral remote sensing imagery, two adjacent images from the same region are acquired. One image has a cloud cover of less than 3% for subsequent supervision label creation, while the other image has a cloud cover between 3% and 50% for model training data. For both images, atmospheric correction is first performed to obtain L2A-level preprocessed remote sensing images. Then, for each band of the multispectral image, upsampling is performed on the band with the highest resolution (generally the visible light band). Finally, a consistent resolution, atmospherically corrected multispectral remote sensing image is obtained. This step requires selecting multiple regions with different climates and seasons to obtain richer training data.

[0078] For SAR image processing, the following preprocessing operations are performed in sequence: thermal noise removal, orbit correction, radiometric calibration, speckle filtering, terrain correction, and decibel reduction to obtain preprocessed SAR image products. It should be noted that the acquired and processed SAR images must be located in the same location as the optical images so that the model can be trained under supervision. Furthermore, the SAR images must be close in temporal phase to the corresponding optical images, with the temporal phase difference limited to 10 days.

[0079] The preprocessed optical and SAR images are fused. First, the SAR image is reprojected onto the corresponding optical image to map it to the same coordinate space. Then, the SAR image is cropped according to the outer contour of the optical image to obtain a SAR image with a location range completely consistent with the optical image. Finally, both the optical and SAR images are cropped into several 512*512 pixel image blocks. During the cropping process, overlap between adjacent image blocks is allowed, for example, allowing a 50% overlap area to increase the sample size. This step ultimately yields several complete pairs of cloud-free fused optical and SAR image data. Each pair contains three image blocks: a SAR image (containing both VV and VH features), a cloud-covered remote sensing image, and a cloudless remote sensing image. All three are located in the same geographical area, were captured at similar times, and are all 512*512 pixel in size.

[0080] Finally, the obtained image pairs are preprocessed before model input. For optical images, each band is cropped to the range of [0,10000] and redistributed to the range of [0,1] for subsequent model training. For the VV and VH channels of SAR images, the maximum and minimum value normalization method is used to redistribute them to [0,1].

[0081] In one embodiment, constructing a depth-coordinated convolutional fusion remote sensing image declouding neural network for optical SAR images includes:

[0082] The network structure is defined as consisting of a cloud-to-cloud shadow attention module, a feature encoding module, a Transformer module, and a feature decoding module.

[0083] The optical SAR image fusion de-clouding dataset is input into the cloud-cloud shadow attention module. The cloud-cloud shadow attention module generates weight maps of cloud areas and cloud shadow areas. A preset activation function is used to map the weights in the weight maps of cloud areas and cloud shadow areas to between 0 and 1.

[0084] The feature encoding module includes four feature encoding layers. The weight maps of the cloud area and the cloud shadow area are input into the feature encoding module, and the encoded SAR image and the feature map of the cloudy optical image are output.

[0085] The Transformer module includes 8 Transformer coding layers, each Transformer coding layer includes 8 attention heads, and each Transformer coding layer consists of a self-attention module and a fully connected feedforward network layer;

[0086] The feature decoding module includes four feature decoding layers.

[0087] Specifically, the optical SAR image depth-cooperative convolutional fusion remote sensing image cloud removal neural network constructed in the embodiments of the present invention is as follows: Figure 3 As shown, the network structure includes four modules:

[0088] The first module is the cloud-cloud shadow attention module, such as... Figure 4 As shown, the model first obtains a weight map of the cloud region and the cloud shadow region through a cloud shadow attention module, and uses tanh as the activation function to map the weights between 0 and 1. The subsequent co-convolution module is implemented based on this weight map. To enhance the network's learning of the cloud shadow weights, residual connections based on cloud shadow attention are used at the model's input and output positions as a prompt, making the network's learning of the cloud shadow weights more stable (e.g., ...). Figure 3 (As shown). Both the Encoder and decoder modules require simultaneous input of optical image features, SAR features, and cloud shadow features;

[0089] The second module is the feature coding module, which consists of four feature coding layers and outputs feature maps of the encoded SAR image and the cloud-covered optical image.

[0090] The third intermediate layer module is the Transformer module, which contains eight Transformer EncoderLayers. Each Transformer Encoder Layer has eight attention heads. The structure of the Transformer EncoderLayer is as follows: Figure 6As shown, a TransformerEncoderLayer consists of a self-attention module and a fully connected feedforward network (FFN) layer. In the model, the high-dimensional features obtained by the encoder are used as the input to the transformer. Since the Transformer structure lacks position sensitivity, position embedding is performed before the high-dimensional features from the encoder are input to the transformer to introduce positional information. After the self-attention module and the FFN layer, residual connections are used to ensure the stability of the gradient during model training. At the same time, the residual connections ensure that the positional information embedded in the position embedding can be passed to deeper layers of the network.

[0091] The fourth module is the feature decoding module, which consists of four feature decoding layers. The structures of the encoding and decoding layers are similar, such as... Figure 5 As shown, the encoding layer does not have an UpSample upsampling submodule.

[0092] For the encoding and decoding modules in the model, the input features are SAR, optical image features, or their representations during the encoding / decoding process. The input also includes an attention map, which is either the output of the cloud-cloud shadow module or its result after max pooling, depending on the number of encoding layers. The encoding / decoding module primarily performs a co-convolution operation on the SAR and optical image features based on the attention map. The co-convolution sub-module receives one SAR image feature, one optical image feature, and an attention map of corresponding size as input. Within the co-convolution module, the SAR and optical features are co-fused, with the result being the output of the optical feature. Additionally, a convolution transformation is performed on the SAR feature, resulting in its output. Max pooling is then applied to the attention map to obtain its output.

[0093] Co-convolutional modules are used to replace traditional Conv2d convolutional modules in image processing. In remote sensing image declouding tasks, cloud-covered remote sensing images are used as input, and cloudless remote sensing images are used as labels for learning. However, when traditional convolutional layers are used to remove thick clouds from remote sensing images, a large number of pixel values ​​are close to 255, resulting in a significant waste of computation. Furthermore, convolutional kernels cannot distinguish between cloud-covered and cloud-free areas during computation, and are not sensitive to the information difference between the two. Co-convolution solves these two problems by using attention maps to distinguish between cloudless and cloud-covered areas and adaptively learning the attention size for different degrees of cloud cover. It performs well in thin cloud removal, thick cloud removal, and cloud shadow removal scenarios.

[0094] A single co-convolution operation can be expressed by the following formula:

[0095]

[0096] Wherein, cld′ is the output feature map, cld and sar are the input feature maps of the cloudy remote sensing image and SAR image respectively, M is the input attention map, W is the convolution kernel, b is the bias of the convolution operation, sum(1) represents the sum of the elements of a matrix with all elements equal to 1 and the same size as the convolution kernel, and sum(M) represents the sum of the elements of a matrix with all elements equal to 1 and the same size as the convolution kernel.

[0097] The co-convolutional input consists of a set of features from cloud-covered remote sensing images and SAR images, along with an attention map. The transformation process for cloud-covered remote sensing image features has already been described. For SAR image features, convolution processing is also required to transform them into another feature space. This ensures that the attention mechanism of the attention map can coordinate and fuse the features from both optical and SAR images. For the attention map (c), it's crucial to ensure that the change in spatial resolution after the co-convolutional layer matches the changes in spatial resolution of the cloud-covered remote sensing and SAR image features. This is achieved by performing a max-pooling operation on the attention map. It's important to note that the attention map always has only one channel; a broadcast operation is required during multiplication with optical or SAR image features.

[0098] 1) This invention introduces a spatial domain-based cloud-cloud shadow self-attention module, enabling the model to exhibit good generalization ability in a series of tasks such as thin cloud and fog removal, cirrus cloud removal, thick cloud removal, and cloud shadow removal. It also introduces a collaborative convolution module for optical and SAR images, which adaptively learns features from both SAR and optical images, balancing the penetration of SAR image information into clouds and fog with the accurate representation of ground feature details in optical images. Compared to traditional convolution modules, this module avoids distortion in the cloud removal effect in the central cloud region. The convolution results have different expressions for different levels of cloud coverage, meaning the convolution is dynamic for different cloud coverage scenarios. Furthermore, a Transformer structure is introduced in the high-dimensional feature part of the model encoding, enhancing the model's ability to learn global image information and improving its generalization performance.

[0099] In one embodiment, the optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The training dataset is used to train the optical SAR image depth-co-convolutional fusion remote sensing image declouding neural network to obtain an optical SAR image declouding model, including:

[0100] In this embodiment of the invention, the obtained dataset is divided into a training dataset and a test dataset. The optical SAR deep co-convolutional fusion remote sensing image declouding neural network constructed in step 3 is trained using the corresponding SAR images, clouded remote sensing images, and cloudless remote sensing images in each training dataset. The network parameters are initialized, and L1 Loss combined with LPIPS Loss is used. The two losses are linearly combined with a 1:1 weight as the loss function for model training. The training uses SAR images and corresponding clouded optical images as inputs, and corresponding cloudless optical images as supervision labels for training, to obtain a model that converges to the optical SAR image fusion declouding training of the created dataset.

[0101] Specifically, the loss training function used includes an L1 Loss part and an LPIPS Loss part, and the calculation formula is as follows:

[0102]

[0103] Wherein, L1Loss represents the pixel-level difference, and LPIPSLoss represents the L2 level loss between the model output and the supervision label in high-dimensional space;

[0104]

[0105] Where X(i,j,k) represents the pixel value of the model output at position (i,j) in channel k, Y(i,j,k) represents the pixel value of the supervision label at position (i,j) in channel k, and H, W, and C represent the height, width, and number of channels of the image slice, respectively.

[0106]

[0107] Where y and y0 represent the model output and the ground truth, respectively; H and W represent the height and width of the image; and h and w represent the index of the pixel's position in the image.

[0108] For each training process, after calculating the loss function, the Adam optimizer is used to reduce the gradient for backpropagation, and the parameters are continuously adjusted until the model converges.

[0109] In one embodiment, multiple preset evaluation indicators are used to evaluate the accuracy of the prediction results of the optical SAR image declouding model.

[0110] Specifically, this embodiment of the invention also evaluates the accuracy of the trained and converged model. It predicts on the prediction set and compares the predictions with cloudless multispectral remote sensing images in the test dataset. Mean Absolute Error (MAE), Structural Similarity (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Spectral Angle Measure (SAM) are calculated as evaluation metrics, as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] SSIM = l(X,Y)*C(X,Y)*s(X,Y)

[0118]

[0119] in, The model prediction result is represented by x, which represents the supervision label corresponding to the output. For the accuracy evaluation metric SSIM, this metric measures the similarity between two images in terms of brightness, contrast, and image structure. C1, C2, and C3 are constants introduced to avoid division by zero. μσ represents the mean and variance of the images, respectively. X and σ Y This represents the covariance between image X and image Y.

[0120] The image cloud removal system based on optical remote sensing images and SAR images provided by the present invention is described below. The image cloud removal system based on optical remote sensing images and SAR images described below can be referred to in correspondence with the image cloud removal method based on optical remote sensing images and SAR images described above.

[0121] Figure 7 This is a schematic diagram of the image cloud removal system based on optical remote sensing images and SAR images provided in an embodiment of the present invention, as shown below. Figure 7 As shown, it includes: a data processing module 71, a training module 72, a slicing module 73, and a processing module 74, wherein:

[0122] The data processing module 71 is used to acquire SAR image sample data and multispectral remote sensing image sample data, and to process the SAR image sample data and multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset. The training module 72 is used to construct an optical SAR image deep co-convolutional fusion remote sensing image declouding neural network, divide the optical SAR image fusion declouding dataset into a training dataset and a test dataset, and use the training dataset to train the optical SAR image deep co-convolutional fusion remote sensing image declouding neural network to obtain an optical SAR image declouding model. The slicing module 73 is used to input the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and output the image declouding result, which includes several declouding result slices. The processing module 74 is used to aggregate the several declouding result slices according to the original slice index to obtain a complete image declouding result, and to filter the stitching edges of the complete image declouding result to obtain the final image declouding result.

[0123] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logic instructions in the memory 830 to execute an image declouding method based on optical remote sensing images and SAR images. This method includes: acquiring SAR image sample data and multispectral remote sensing image sample data; performing dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset; constructing an optical SAR image deep co-convolutional fusion remote sensing image declouding neural network; dividing the optical SAR image fusion declouding dataset into a training dataset and a test dataset; training the optical SAR image deep co-convolutional fusion remote sensing image declouding neural network using the training dataset to obtain an optical SAR image declouding model; inputting the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and outputting an image declouding result, the image declouding result including several declouding result slices; aggregating the several declouding result slices according to the original slice index to obtain a complete image declouding result; filtering the stitching edges of the complete image declouding result to obtain the final image declouding result.

[0124] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image declouding methods based on optical remote sensing images and SAR images provided by the above methods. These methods include: acquiring SAR image sample data and multispectral remote sensing image sample data; performing dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset; and constructing an optical SAR image deep co-convolutional fusion remote sensing image declouding neural network. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The optical SAR image deep co-convolutional fusion remote sensing image declouding neural network is trained using the training dataset to obtain an optical SAR image declouding model. The multispectral remote sensing image to be processed and the corresponding SAR image at the same location are input into the optical SAR image declouding model, and the image declouding result is output. The image declouding result includes several declouding result slices. The several declouding result slices are aggregated according to the original slice index to obtain a complete image declouding result. The stitching edges of the complete image declouding result are filtered to obtain the final image declouding result.

[0126] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image declouding method based on optical remote sensing images and SAR images provided by the above methods. The method includes: acquiring SAR image sample data and multispectral remote sensing image sample data; performing dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset; constructing an optical SAR image deep co-convolutional fusion remote sensing image declouding neural network; dividing the optical SAR image fusion declouding dataset into a training dataset and a test dataset; training the optical SAR image deep co-convolutional fusion remote sensing image declouding neural network using the training dataset to obtain an optical SAR image declouding model; inputting the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and outputting an image declouding result, the image declouding result including several declouding result slices; aggregating the several declouding result slices according to the original slice index to obtain a complete image declouding result; filtering the stitching edges of the complete image declouding result to obtain the final image declouding result.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud removal method for images based on optical remote sensing images and SAR images, characterized in that, include: Acquire synthetic aperture radar (SAR) image sample data and multispectral remote sensing image sample data, and perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouding dataset. A neural network for declouding optical SAR images is constructed by deep co-convolutional fusion of remote sensing images. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The optical SAR image deep co-convolutional fusion remote sensing image declouding neural network is trained using the training dataset to obtain the optical SAR image declouding model. The multispectral remote sensing image to be processed and the corresponding SAR image at the same location are input into the optical SAR image declouding model, and the image declouding result is output. The image declouding result includes several declouding result slices. The cloud removal result slices are aggregated according to the original slice index to obtain the complete image cloud removal result. The stitching edges of the complete image cloud removal result are filtered to obtain the final image cloud removal result. The construction of a cloud removal neural network for optical SAR image depth-coordinated convolutional fusion remote sensing imagery includes: The network structure is defined as consisting of a cloud-to-cloud shadow attention module, a feature encoding module, a Transformer module, and a feature decoding module. The optical SAR image fusion de-clouding dataset is input into the cloud-cloud shadow attention module. The cloud-cloud shadow attention module generates weight maps of cloud areas and cloud shadow areas. A preset activation function is used to map the weights in the weight maps of cloud areas and cloud shadow areas to between 0 and 1. The feature encoding module includes four feature encoding layers. The weight maps of the cloud area and the cloud shadow area are input into the feature encoding module, and the encoded SAR image and the feature map of the cloudy optical image are output. The Transformer module includes 8 Transformer coding layers, each Transformer coding layer includes 8 attention heads, and each Transformer coding layer consists of a self-attention module and a fully connected feedforward network layer; The feature decoding module includes four feature decoding layers; The Conv2d convolutional module in the optical SAR image depth co-convolutional fusion remote sensing image declouding neural network is replaced with a co-convolutional module, which includes: in, To output the feature map, and denoted as feature maps of the input cloud-covered remote sensing image and SAR image, respectively; M is the input attention map; W is the convolution kernel; and b is the bias of the convolution operation. This represents the sum of the elements of a matrix of the same size as the convolution kernel, where all elements are 1. This represents the sum of the elements of a matrix whose elements are all 1 and have the same size as the convolution kernel.

2. The image cloud removal method based on optical remote sensing imagery and SAR imagery according to claim 1, characterized in that, Acquire SAR image sample data and multispectral remote sensing image sample data, and perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouded dataset, including: Two adjacent temporal images of the multispectral remote sensing image sample data are collected in the same area. The two adjacent temporal images include a first temporal image with a cloud content ratio lower than a first preset cloud content ratio and a second temporal image with a cloud content ratio higher than the preset cloud content ratio but lower than a second preset cloud content ratio. The first temporal image is used to create a supervision label, and the second temporal image is used to create training data. Atmospheric correction is performed on the two adjacent temporal images to obtain a preprocessed remote sensing image of a preset level. Each band of the preprocessed remote sensing image of the preset level is upsampled according to the highest resolution band to obtain an optical preprocessed image. The SAR image sample data is sequentially subjected to thermal noise removal, orbit correction, radiometric calibration, speckle filtering, terrain correction, and decibel reduction to obtain a SAR preprocessed image. The SAR preprocessed image is located in the same position as the optical preprocessed image and has a preset temporal difference range. The SAR preprocessed image is reprojected according to the corresponding optical preprocessed image, and the SAR preprocessed image and the optical preprocessed image are mapped to the same coordinate space. The SAR preprocessed image is cropped according to the optical preprocessed image so that the position range of the SAR preprocessed image and the optical preprocessed image are consistent. The cropped SAR preprocessed image and the optical preprocessed image are cropped into pixel blocks of a preset pixel size to obtain the initial optical SAR image fusion cloud removal dataset. The initial optical SAR image fusion de-cloud dataset is constructed by cropping each band of the optical image to a preset band range and redistributing it to a preset distribution interval. The vertical-vertical polarization (VV) channel and vertical-horizontal polarization (VH) channel of the initial optical SAR image fusion de-cloud dataset are then normalized to the preset distribution interval using maximum and minimum values.

3. The image cloud removal method based on optical remote sensing imagery and SAR imagery according to claim 1, characterized in that, The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The training dataset is used to train the optical SAR image depth-co-convolutional fusion remote sensing image declouding neural network to obtain the optical SAR image declouding model, including: The optical SAR image deep co-convolutional fusion remote sensing image declouding neural network was trained using SAR images, clouded remote sensing images, and cloudless remote sensing images from the training dataset, and the network parameters were initialized. The combined loss function is constructed by using the L1 Loss loss function with the same weights and the LPIPS loss function that learns perceptual image patch similarity: in, Represents differences at the pixel level. The L2 level loss of the model output and the supervision label in high-dimensional space; in, This represents the position of the model output in channel k. The pixel value at position ) The supervisory label represents the channel. Location ( The pixel value at position ) These represent the height, width, and number of channels of the image slice, respectively. Where y and y0 represent the model output and the true value, respectively. Represents the height and width of the image. , An index representing the position of a pixel in the image; The Adam optimizer is used to reduce the gradient during backpropagation, and the network parameters are adjusted until the model converges, thus obtaining the cloud removal model for the optical SAR image.

4. The image cloud removal method based on optical remote sensing imagery and SAR imagery according to claim 1, characterized in that, Also includes: The accuracy of the prediction results of the optical SAR image declouding model is evaluated using multiple preset evaluation indicators.

5. An image cloud removal system based on optical remote sensing imagery and SAR imagery, based on the image cloud removal method based on optical remote sensing imagery and SAR imagery as described in any one of claims 1 to 4, characterized in that, include: The data processing module is used to acquire SAR image sample data and multispectral remote sensing image sample data, and to perform dataset processing on the SAR image sample data and the multispectral remote sensing image sample data to obtain an optical SAR image fusion declouded dataset. The training module is used to construct a neural network for declouding optical SAR images through deep co-convolutional fusion. The optical SAR image fusion declouding dataset is divided into a training dataset and a test dataset. The training dataset is used to train the optical SAR image deep co-convolutional fusion remote sensing image declouding neural network to obtain an optical SAR image declouding model. The slicing module is used to input the multispectral remote sensing image to be processed and the corresponding SAR image at the same location into the optical SAR image declouding model, and output the image declouding result, which includes several declouding result slices. The processing module is used to aggregate the several declouded result slices according to the original slice index to obtain a complete image declouded result, and to filter the stitching edges of the complete image declouded result to obtain the final image declouded result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image declouding method based on optical remote sensing images and SAR images as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image declouding method based on optical remote sensing images and SAR images as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image declouding method based on optical remote sensing images and SAR images as described in any one of claims 1 to 4.

Images