A multimodal sequence fusion cloud removal method and system robust to temporal surface changes

Abstract

The present invention provides a multimodal sequence fusion cloud removal method and system that is robust to temporal surface changes. The principle of the present invention is: using a deep convolutional neural network to extract multi-temporal SAR features and perform distance measurement on them to obtain temporal change relationships, and applying the change information in a spatial attention manner to the optical-SAR comprehensive features extracted using the deep convolutional neural network, so that the network can more specifically select valuable reference data for completing cloud-occluded images. The method of the present invention can better solve the problem of decreased reconstruction performance of the multimodal sequence fusion cloud removal algorithm when using sequence data containing surface changes. The proposed method can use SAR data to capture changes in ground objects in multi-temporal reference images, thereby being able to more specifically select valuable reference data for completing cloud-occluded images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, relates to a multimodal sequence fusion cloud removal algorithm, and specifically relates to a multimodal sequence fusion cloud removal method and system that is robust to temporal surface changes. Background Art

[0002] Optical remote sensing data is often obscured by cloud cover and shadows, which, to a certain extent, limits its widespread application and effectiveness. To overcome this challenge, cloud removal algorithms have emerged. As a key technology in remote sensing image processing, their primary goal is to accurately identify and remove cloud obscurations and shadows from remote sensing images, thereby restoring a clearer image of the Earth's surface. Therefore, in-depth research and improvement of cloud removal algorithms is a key issue in improving remote sensing data processing capabilities.

[0003] Advances in remote sensing technology have greatly enriched the types of data available. These data, in the form of time series, encompass a variety of observation methods, including optical and radar. Using these regularly collected multi-source remote sensing data as auxiliary data can effectively improve cloud removal performance. On the one hand, due to the movement and variability of clouds, data acquired at other times may provide clear surface information in areas obscured by clouds. On the other hand, the penetrating power of synthetic aperture radar (SAR) enables it to capture surface information beneath cloud cover. In recent years, deep learning cloud removal algorithms leveraging multimodal sequence data have made considerable progress. However, in the real world, the surface changes due to factors such as seasonal changes, urbanization, and natural disturbances, resulting in potential inconsistencies between data collected at different times. Existing cloud removal methods for multitemporal and multimodal remote sensing imagery primarily rely on optical data and struggle to properly address the degradation in reconstruction performance caused by surface changes. Summary of the Invention

[0004] The existing technical issues can be addressed by more effectively utilizing multi-temporal SAR imagery. Multi-temporal SAR imagery can help determine which areas have undergone significant changes and which areas have remained stable, allowing for more targeted selection of unchanged data for image correction. Based on this idea, the present invention proposes a multimodal sequence fusion cloud removal algorithm that is robust to temporal surface changes. This algorithm aims to fully utilize the useful information in multimodal sequence remote sensing imagery to improve cloud removal performance.

[0005] The technical solution of the present invention is a multimodal sequence fusion cloud removal method that is robust to temporal surface changes, comprising the following specific steps:

[0006] Step 1: splice the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(Iopt , I sar ), where I opt is a multi-phase optical image, I sar is a multi-temporal SAR image, E is an encoder composed of a convolutional neural network;

[0007] Step 2: Input the SAR image sequence into the convolutional neural network to extract the multi-temporal SAR features F sar =E sar (I sar ), where E sar is a SAR encoder composed of a convolutional neural network;

[0008] Step 3, F sar Input change attention module CA selects a specific time point in the input time series as the benchmark and calculates the similarity between the SAR features of other time points and the SAR features of the benchmark time point, which is recorded as Attn c ;

[0009] Step 4: F a and Attn c After splicing and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion t , and F a Multi-temporal fusion is performed in a weighted sum manner;

[0010] Step 5: Input the fused features into the decoder D to obtain the predicted cloud-free image I pred .

[0011] Furthermore, the encoder E consists of a 1×1 convolutional layer and an MBConv module.

[0012] Furthermore, the SAR encoder E sar It consists of a 1×1 convolutional layer and an MBConv module.

[0013] Furthermore, the mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors at the spatial coordinates (i, j) of any other time point t and the reference time point b.

[0014] Further, Attn t The specific calculation process is as follows: the concatenated feature cat(F a , Attn c) Through the shallow convolutional network, the spatial cloud cover information and the ground feature change information are integrated to obtain the feature F b , then F b Downsampling is performed in the spatial dimension to reduce the computational complexity, and F c ; From the perspective of tensor, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the time series length, C is the number of feature channels, H and W are the length and width of the feature, that is, F c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is calculated through the self-attention mechanism to obtain the attention weight Attn t_d , and finally Attn t_d Upsample to the original spatial scale to get Attn t .

[0015] Furthermore, the decoder D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

[0016] Furthermore, the encoder E and SAR encoder E are trained by back propagation algorithm. sar , change attention module CA, self-attention module LTAE, and decoder D are trained.

[0017] Furthermore, during the training process, the loss function L is calculated as (I pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I GT The Euclidean distance is used to measure the accuracy of the predicted image.

[0018] Furthermore, the cloud removal effect is evaluated using peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM) and mean absolute error (MAE).

[0019] The present invention also provides a multimodal sequence fusion cloud removal system that is robust to temporal surface changes, comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a multimodal sequence fusion cloud removal method that is robust to temporal surface changes as described in the above scheme.

[0020] The principle of the present invention is to use a deep convolutional neural network to extract multi-phase SAR features and perform distance measurement on them to obtain the temporal change relationship, and then apply the change information to the optical-SAR comprehensive features extracted by the deep convolutional neural network in the form of spatial attention, so that the network can more specifically select valuable reference data for completing cloud-occluded images.

[0021] The purpose of this invention is to provide a multimodal sequence fusion cloud removal method that is robust to temporal surface changes. This method can better address the degradation in reconstruction performance that occurs when multimodal sequence fusion cloud removal algorithms use sequence data containing surface changes. The proposed method can use SAR data to capture surface feature changes in multi-temporal reference imagery, thereby enabling more targeted selection of valuable reference data for cloud-obstructed image completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram of the deep convolutional neural network model used in this invention. DETAILED DESCRIPTION

[0023] The present invention provides a multimodal sequence fusion cloud removal algorithm that is robust to temporal surface changes. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, an embodiment of the present invention provides a multi-modal sequence fusion cloud removal method that is robust to temporal surface changes, comprising the following steps:

[0025] Step 1: Concatenate the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(I opt , I sar ), where E consists of a 1×1 convolutional layer and an MBConv module. The MBConv module is an efficient and lightweight variant of CNN. It was first proposed in MobileNetV2 to improve the computational efficiency and model performance of the network. In the task of declouding multi-temporal and multi-modal remote sensing imagery, it is crucial to preserve image details. To achieve this goal, the network usually avoids downsampling the feature map during image reconstruction to maintain high-resolution feature representation. However, this often leads to a significant increase in the number of model parameters and computational complexity, especially when processing large-scale remote sensing data. The introduction of the MBConv module is precisely to effectively reduce the number of model parameters and computational complexity without sacrificing the details of the feature map.

[0026] Step 2: Input the SAR image sequence into the convolutional neural network to extract multi-temporal SAR features Fsar =E sar (I sar ), where E sar It also consists of a 1×1 convolutional layer and an MBConv module.

[0027] Step 3: F sar Input the change attention module CA to calculate the multi-phase change attention map Attn c The calculation process is as follows: select the first time point in the input time series as the benchmark, regard the SAR feature map as a set of feature vectors, and calculate the cosine similarity of the SAR features of other time points and the SAR features of the benchmark time point pixel by pixel, that is, Attn c .

[0028] The mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors at the spatial coordinates (i, j) of any other time point t and the reference time point b.

[0029] Step 4: F a and Attn c After splicing and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion t , and F a Multi-temporal fusion is performed in a weighted sum manner. Multi-temporal fusion aims to selectively aggregate the input multi-temporal features into single-temporal features according to the cloud cover, and finally decode them into a cloud-free image. a The addition of can make LTAE more inclined to select the area where the ground objects in the reference image have not changed when performing multi-temporal fusion, thereby obtaining better reconstruction results. The LTAE module is a self-attention mechanism module proposed by scholars and optimized for time series data. This structure consists of a fully connected layer and a self-attention layer. Compared with the multi-head self-attention module in the classic Transformer encoder, the fully connected layer corresponding to the query vector and the value vector is omitted. This design enables LTAE to improve computational efficiency and reduce the number of model parameters. Attn t The specific calculation process is as follows: the concatenated feature cat(F a , Attn c ) Through the shallow convolutional network, the spatial cloud cover information and the ground feature change information are integrated to obtain the feature F b Then F bDownsampling is performed in the spatial dimension to reduce the computational complexity, and F c From the perspective of tensors, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the time series length, C is the number of feature channels, and H and W are the length and width of the feature. c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is calculated through the self-attention mechanism to obtain the attention weight Attn t_d Finally, Attn t_d Upsample to the original spatial scale to get Attn t .

[0030] Step 5: Input the fused features into the decoder D composed of a convolutional neural network to obtain the predicted cloud-free image I pred D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

[0031] Step 6: Use the back propagation algorithm to calculate E and E sar , LTAE, D and other modules are trained using the Adam optimizer. The initial learning rate is set to 0.001, the decay coefficient is set to 0.8, and a total of 20 rounds of training are performed.

[0032] The training process calculates the loss function L = (I pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I GT The Euclidean distance is used to measure the accuracy of the predicted image.

[0033] The experiments used the SEN12MS-CR-TS dataset, a multimodal and multitemporal dataset designed specifically for cloud removal from optical satellite imagery. The dataset derives data from two key missions under the European Space Agency (ESA) Copernicus program: Sentinel-1 and Sentinel-2. The Sentinel-1 satellite carries a C-band synthetic aperture radar (SAR), capable of providing cloud-penetrating surface imaging, while the Sentinel-2 satellite provides high-resolution multispectral imagery. To evaluate the cloud removal performance, this method uses the following analytical metrics: peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM), and mean absolute error (MAE). Experimental results on the SEN12MS-CR-TS dataset are shown in Table 1. Analysis of the experimental results demonstrates that this method effectively utilizes multitemporal SAR imagery to more specifically select unchanged data for image correction, thereby improving cloud removal performance.

[0034] Table 1 Experimental results on the SEN12MS-CR-TS dataset

[0035]

[0036] On the other hand, an embodiment of the present invention also provides a multimodal sequence fusion cloud removal system that is robust to temporal surface changes, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a multimodal sequence fusion cloud removal method that is robust to temporal surface changes as described in the above scheme.

[0037] The present invention provides a multimodal sequence fusion cloud removal method and system that is robust to temporal surface changes. This method utilizes SAR data to capture changes in ground objects in multi-temporal reference images, enabling more targeted selection of valuable reference data for completing cloud-obstructed images. The network simultaneously inputs multi-temporal optical image data and SAR image data. The network includes two feature extractors, one for extracting the combined features of the optical-SAR image and the other for extracting SAR image features. In the network's forward pass, the network measures distances between SAR features at different time points to obtain ground object changes in the multi-temporal reference images. This feature is then applied as spatial attention to the combined optical-SAR features, which are then fed into a self-attention module (LTAE) for weighted fusion. Finally, the fused features are fed into a convolutional neural network to obtain a predicted cloud-free image. The multimodal sequence fusion cloud removal method, robust to temporal surface changes, provided by the present invention, effectively utilizes multi-temporal SAR imagery to more targetedly select unchanged data for image correction, thereby improving the cloud removal effect.

[0038] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A multimodal sequence fusion cloud removal method that is robust to temporal surface changes, characterized by: The following steps are involved: Step 1: splice the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(I opt ,I sar ), where I opt is a multi-phase optical image, I sar is a multi-temporal SAR image, E is an encoder composed of a convolutional neural network; Step 2: Input the SAR image sequence into the convolutional neural network to extract the multi-temporal SAR features F sar =E sar (I sar ), where E sar is a SAR encoder composed of a convolutional neural network; Step 3, F sar Input change attention module CA selects a specific time point in the input time series as the benchmark and calculates the similarity between the SAR features of other time points and the SAR features of the benchmark time point, which is recorded as Attn c ; Step 4: F a and Attn c After splicing and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion t , and F a Multi-temporal fusion is performed in a weighted sum manner; Step 5: Input the fused features into the decoder D to obtain the predicted cloud-free image I pred .

2. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The encoder E consists of a 1×1 convolutional layer and an MBConv module.

3. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: SAR Encoder E sar It consists of a 1×1 convolutional layer and an MBConv module.

4. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors of any other time point t and the reference time point b at the spatial coordinates (i, j).

5. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: Attn t The specific calculation process is as follows: the concatenated feature cat(F a ,Attn c ) Through the shallow convolutional network, the spatial cloud cover information and the ground feature change information are integrated to obtain the feature F b , then F b Downsampling is performed in the spatial dimension to reduce the computational complexity, and F c ; From the perspective of tensor, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the time series length, C is the number of feature channels, H and W are the length and width of the feature, that is, F c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is calculated through the self-attention mechanism to obtain the attention weight Attn t_d , and finally Attn t_d Upsample to the original spatial scale to get Attn t .

6. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The decoder D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

7. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: Through the back propagation algorithm, the encoder E and SAR encoder E sar , change attention module CA, self-attention module LTAE, and decoder D are trained.

8. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 7, characterized in that: Calculate the loss function L during training pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I Gt The Euclidean distance is used to measure the accuracy of the predicted image.

9. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: It also includes the evaluation of cloud removal effect using peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM) and mean absolute error (MAE).

10. A multi-modal sequence fusion cloud removal system that is robust to temporal surface changes, characterized by: The invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a multi-modal sequence fusion cloud removal method that is robust to temporal surface changes as claimed in any one of claims 1 to 9.

Images