Multi-modal sequence fusion cloud removal method and system for time sequence surface change robustness

Abstract

The invention provides a multi-modal sequence fusion cloud removal method and system for time sequence surface change robustness. According to the principle of the method, a deep convolutional neural network is used for extracting multi-temporal SAR features, distance measurement is carried out on the multi-temporal SAR features to obtain a time sequence change relation, and change information acts on optical-SAR comprehensive features extracted by using the deep convolutional neural network in a space attention mode. And the network can select valuable reference data in a more targeted manner for complementing the cloud occlusion image. According to the method, the problem that the reconstruction performance is reduced when sequence data containing earth surface changes are used in a multi-modal sequence fusion cloud removal algorithm can be better solved. According to the method provided by the invention, the surface feature change in the multi-temporal reference image can be captured by using the SAR data, so that valuable reference data can be selected in a more targeted manner for complementing the cloud occlusion image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a multimodal sequence fusion cloud removal algorithm, and specifically to a multimodal sequence fusion cloud removal method and system that is robust to temporal surface changes. Background Art

[0002] Optical remote sensing data is often interfered by clouds and shadows, which to some extent limits the breadth and effectiveness of its application. In order to overcome this problem, cloud removal algorithms have emerged. As a key technology in the field of remote sensing image processing, its main goal is to accurately identify and remove cloud occlusion and shadow areas in remote sensing images, thereby restoring clearer surface images. Therefore, in-depth research and improvement of cloud removal algorithms is a key issue in improving remote sensing data processing capabilities.

[0003] The progress of remote sensing technology has greatly enriched the types of data that can be obtained. These data exist in the form of time series, covering a variety of observation methods such as optics and radar. Using these regularly collected multi-source remote sensing data as auxiliary data can effectively improve the declouding performance: on the one hand, due to the movement and change of clouds, data acquired at other time points may provide clear surface information for areas obscured by clouds; on the other hand, the penetration ability of synthetic aperture radar (SAR) enables it to capture surface information under cloud cover. In recent years, deep learning declouding algorithms using multimodal sequence data have made some progress. However, in the real world, the surface will change due to factors such as seasonal changes, urbanization and natural disturbances, resulting in certain inconsistencies between data collected at different times. Existing multi-temporal and multi-modal remote sensing image declouding methods mainly rely on optical data, which is difficult to properly handle the degradation of reconstruction performance caused by surface changes. Summary of the invention

[0004] In view of the problems in the prior art, this problem is expected to be solved by more effectively utilizing multi-temporal SAR images. Multi-temporal SAR images can help determine which areas have undergone significant changes and which areas remain stable, so that unchanged data can be selected more specifically for image correction. Based on this idea, the present invention proposes a multi-modal sequence fusion cloud removal algorithm that is robust to temporal surface changes, aiming to make full use of the useful information in multi-modal sequence remote sensing images and improve the de-clouding performance.

[0005] The technical solution of the present invention is a multi-modal sequence fusion cloud removal method that is robust to temporal surface changes, comprising the following specific steps:

[0006] Step 1: splice the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(Iopt , I sar ), where I opt is a multi-phase optical image, I sar is a multi-temporal SAR image, E is an encoder composed of a convolutional neural network;

[0007] Step 2: Input the SAR image sequence into the convolutional neural network to extract multi-temporal SAR features F sar =E sar (I sar ), where E sar is a SAR encoder composed of a convolutional neural network;

[0008] Step 3: F sar The input change attention module CA selects a specific time point in the input time series as the benchmark and calculates the similarity between the SAR features of other time points and the SAR features of the benchmark time point, denoted as Attn c ;

[0009] Step 4: F a and Attn c After concatenation and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion t , and for F a Multi-temporal fusion is performed in a weighted sum manner;

[0010] Step 5: Input the fused features into the decoder D to obtain the predicted cloud-free image I pred .

[0011] Furthermore, the encoder E consists of a 1×1 convolutional layer and an MBConv module.

[0012] Furthermore, the SAR encoder E sar It consists of a 1×1 convolutional layer and an MBConv module.

[0013] Furthermore, the mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors at the spatial coordinates (i, j) of any other time point t and the reference time point b.

[0014] Further, Attn t The specific calculation process is as follows: the concatenated feature cat(F a , Attn c) Through the shallow convolutional network, the spatial cloud coverage information and the ground object change information are integrated to obtain the feature F b , and then F b Downsampling is performed in the spatial dimension to reduce the computational complexity, and F is obtained. c ; From the perspective of tensors, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the time series length, C is the number of feature channels, H and W are the length and width of the feature, that is, F c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is used to calculate the attention weight Attn through the self-attention mechanism t_d , and finally Attn t_d Upsample to the original spatial scale to get Attn t .

[0015] Furthermore, the decoder D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

[0016] Furthermore, the encoder E and SAR encoder E are trained by back propagation algorithm. sar , change attention module CA, self-attention module LTAE, and decoder D are trained.

[0017] Furthermore, during the training process, the loss function L is calculated as (I pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I GT The Euclidean distance is used to measure the accuracy of the predicted image.

[0018] Furthermore, it also includes evaluating the cloud removal effect using peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM) and mean absolute error (MAE).

[0019] The present invention also provides a multimodal sequence fusion cloud removal system that is robust to temporal surface changes, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a multimodal sequence fusion cloud removal method that is robust to temporal surface changes as described in the above scheme.

[0020] The principle of the present invention is to use a deep convolutional neural network to extract multi-phase SAR features and perform distance measurement on them to obtain the temporal change relationship, and to apply the change information in the form of spatial attention to the optical-SAR comprehensive features extracted using the deep convolutional neural network, so that the network can more specifically select valuable reference data for completing cloud-occluded images.

[0021] The purpose of the present invention is to provide a multimodal sequence fusion cloud removal method that is robust to temporal surface changes, which can better solve the problem of decreased reconstruction performance of the multimodal sequence fusion cloud removal algorithm when using sequence data containing surface changes. The proposed method can use SAR data to capture the changes of objects in multi-temporal reference images, so that valuable reference data can be selected more specifically to complete cloud-occluded images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram of the deep convolutional neural network model used in the present invention. DETAILED DESCRIPTION

[0023] The present invention provides a multi-modal sequence fusion cloud removal algorithm that is robust to temporal surface changes. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, an embodiment of the present invention provides a multi-modal sequence fusion cloud removal method that is robust to temporal surface changes, comprising the following steps:

[0025] Step 1: Concatenate the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(I opt , I sar ), where E consists of a 1×1 convolutional layer and an MBConv module. The MBConv module is an efficient and lightweight variant of CNN. It was first proposed in MobileNetV2 to improve the computational efficiency and model performance of the network. In the multi-temporal and multi-modal remote sensing image declouding task, it is crucial to preserve image details. To achieve this goal, the network usually avoids downsampling the feature map during image reconstruction to maintain high-resolution feature representation. However, this often leads to a significant increase in the number of model parameters and the amount of computation, especially when processing large-scale remote sensing data. The introduction of the MBConv module is precisely to effectively reduce the number of model parameters and the amount of computation without sacrificing the details of the feature map.

[0026] Step 2: Input the SAR image sequence into the convolutional neural network to extract multi-temporal SAR features Fsar =E sar (I sar ), where E sar It also consists of a 1×1 convolutional layer and an MBConv module.

[0027] Step 3: F sar Input the change attention module CA and calculate the multi-phase change attention map Attn c The calculation process is as follows: select the first time point in the input time series as the benchmark, regard the SAR feature map as a set of feature vectors, and calculate the cosine similarity of the SAR features of other time points and the SAR features of the benchmark time point pixel by pixel, that is, Attn c .

[0028] The mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors at the spatial coordinates (i, j) of any other time point t and the reference time point b.

[0029] Step 4: F a and Attn c After concatenation and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion t , and for F a Multi-temporal fusion is performed in a weighted sum manner. Multi-temporal fusion aims to selectively aggregate the input multi-temporal features into single-temporal features according to the cloud cover, and finally decode them into a cloud-free image. a The addition of can make LTAE more inclined to select the area where the ground objects in the reference image have not changed when performing multi-temporal fusion, thereby obtaining better reconstruction results. The LTAE module is a self-attention mechanism module proposed by scholars and optimized for time series data. This structure consists of a fully connected layer and a self-attention layer. Compared with the multi-head self-attention module in the classic Transformer encoder, the fully connected layer corresponding to the query vector and the value vector is omitted. This design enables LTAE to improve computational efficiency and reduce the number of model parameters. Attn t The specific calculation process is as follows: the concatenated feature cat(F a , Attn c ) Through the shallow convolutional network, the spatial cloud coverage information and the ground object change information are integrated to obtain the feature F b Then, F bDownsampling is performed in the spatial dimension to reduce the computational complexity, and F is obtained. c From the perspective of tensors, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the length of the time series, C is the number of feature channels, and H and W are the length and width of the feature. c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is used to calculate the attention weight Attn through the self-attention mechanism t_d Finally, Attn t_d Upsample to the original spatial scale to get Attn t .

[0030] Step 5: Input the fused features into the decoder D composed of a convolutional neural network to obtain the predicted cloud-free image I pred D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

[0031] Step 6: Use the back propagation algorithm to derive E and E sar , LTAE, D and other modules are trained using the Adam optimizer. The initial learning rate is set to 0.001, the decay coefficient is set to 0.8, and a total of 20 rounds of training are performed.

[0032] The training process calculates the loss function L = (I pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I GT The Euclidean distance is used to measure the accuracy of the predicted image.

[0033] The experiment uses the SEN12MS-CR-TS dataset, which is a multimodal and multi-temporal dataset designed for optical satellite image cloud removal tasks. The data of the dataset comes from two key missions under the Copernicus program of the European Space Agency (ESA): Sentinel-1 and Sentinel-2. The Sentinel-1 satellite carries a C-band synthetic aperture radar (SAR) that can provide surface imaging that penetrates clouds, while the Sentinel-2 satellite provides high-resolution multispectral images. For the evaluation of the declouding effect, the analysis indicators used in this method are: peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM) and mean absolute error (MAE). The experimental results on the SEN12MS-CR-TS dataset are shown in Table 1. The analysis of the experimental results shows that this method can effectively use multi-temporal SAR images to more specifically select unchanged data for image correction, thereby improving the declouding effect.

[0034] Table 1 Experimental results on the SEN12MS-CR-TS dataset

[0035]

[0036] On the other hand, an embodiment of the present invention also provides a multimodal sequence fusion cloud removal system that is robust to temporal surface changes, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a multimodal sequence fusion cloud removal method that is robust to temporal surface changes as described in the above scheme.

[0037] The present invention provides a multi-modal sequence fusion cloud removal method and system that is robust to temporal surface changes. The method uses SAR data to capture the changes of objects in multi-temporal reference images, so that valuable reference data can be more targeted to complete cloud-occluded images. The network simultaneously inputs multi-temporal optical image data and SAR image data. The network includes two feature extractors, which are used to extract the comprehensive features of optical-SAR images and SAR image features respectively. In the forward process of the network, the changes of objects in the multi-temporal reference images are obtained by measuring the distance between the SAR features at different time points, and the optical-SAR comprehensive features are acted on as spatial attention, and then the self-attention module LTAE is input for weighted fusion, and finally the fused features are input into the convolutional neural network to obtain the predicted cloud-free image. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes provided by the present invention can effectively use multi-temporal SAR images to more targetedly select unchanged data for image correction, thereby improving the cloud removal effect.

[0038] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A multi-modal sequence fusion cloud removal method that is robust to temporal surface changes, characterized in that: The following steps are involved: Step 1: splice the optical and SAR image sequences and input them into the convolutional neural network to extract the multi-temporal optical-SAR image comprehensive features F a =E(I opt ,I sar ), where I opt is a multi-phase optical image, I sar is a multi-temporal SAR image, and E is an encoder composed of a convolutional neural network; Step 2: Input the SAR image sequence into the convolutional neural network to extract multi-temporal SAR features F sar =E sar (I sar ), where E sar is a SAR encoder composed of a convolutional neural network; Step 3: F sar The input change attention module CA selects a specific time point in the input time series as the benchmark and calculates the similarity between the SAR features of other time points and the SAR features of the benchmark time point, denoted as Attn c ; Step 4: F a and Attn c After concatenation and convolution fusion in the channel dimension, the self-attention module LTAE is input to calculate the attention weight Attn for multi-temporal fusion. t , and for F a Multi-temporal fusion is performed in a weighted sum manner; Step 5: Input the fused features into the decoder D to obtain the predicted cloud-free image I pred .

2. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The encoder E consists of a 1×1 convolutional layer and an MBConv module.

3. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: SAR Encoder E sar It consists of a 1×1 convolutional layer and an MBConv module.

4. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The mathematical expression of similarity calculation is: in, For Attn c The value at position (i, j) and time t, and Respectively represent the feature vectors at the spatial coordinates (i, j) of any other time point t and the reference time point b.

5. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: Attn t The specific calculation process is as follows: the concatenated feature cat(F a ,Attn c ) Through the shallow convolutional network, the spatial cloud coverage information and the ground object change information are integrated to obtain the feature F b , and then F b Downsampling is performed in the spatial dimension to reduce the computational complexity, and F is obtained. c ; From the perspective of tensors, F c From the shape [B, T, C, H, W] to [B*H*W, T, C], where B is the batch size, T is the length of the time series, C is the number of feature channels, H and W are the length and width of the feature, that is, F c As a vector set with a sequence length of T and a feature vector size of C, the input LTAE is used to calculate the attention weight Attn through the self-attention mechanism t_d , and finally Attn t_d Upsample to the original spatial scale to get Attn t .

6. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: The decoder D consists of five serially connected MBConv modules, a 1×1 convolutional layer, and a Sigmoid function.

7. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: Through the back propagation algorithm, the encoder E and SAR encoder E sar , change attention module CA, self-attention module LTAE, and decoder D are trained.

8. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 7, characterized in that: The loss function L is calculated during the training process. pred -I GT ) 2 , the loss function is calculated by predicting the image I pred With the target image I Gt The Euclidean distance is used to measure the accuracy of the predicted image.

9. The multi-modal sequence fusion cloud removal method that is robust to temporal surface changes according to claim 1, characterized in that: It also includes the evaluation of cloud removal effect using peak signal-to-noise ratio (PSNR), spectral angle mapping (SAM), unit dimensionless structural similarity index (SSIM) and mean absolute error (MAE).

10. A multi-modal sequence fusion cloud removal system that is robust to temporal surface changes, characterized by: The invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a multi-modal sequence fusion cloud removal method that is robust to temporal surface changes as described in any one of claims 1 to 9.

Images