Remote sensing optical time-series image reconstruction method, device, terminal and storage medium

Abstract

The present invention provides a remote sensing optical time-series image reconstruction method, device, terminal and storage medium, which belong to the field of remote sensing technology. The method includes: obtaining remote sensing optical time-series images, SAR time-series images and their respective shooting date sequences; obtaining a first SAR time-series image and a noisy cloud optical time-series image; inputting the noisy cloud optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud optical time-series image and the noise diffusion step number t into a target sampler, iteratively calling the trained diffusion model t times, obtaining a reconstructed cloud-free optical image sequence, and the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence constitute a reconstructed remote sensing optical time-series image. The present invention introduces SAR time-series images and shooting date sequences as prior information, combines the high-quality image generation capability of the multimodal diffusion model, and guides the reconstruction of cloud images, which can effectively improve the reliability of remote sensing optical time-series image reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a remote sensing optical time-series image reconstruction method, device, terminal and storage medium. Background Art

[0002] Satellite long-term observations are essential for monitoring changes and trends in the surface environment. However, cloud cover causes frequent data gaps, and the quality of optical image time series is seriously affected. To solve this problem, it is necessary to reconstruct cloud images and restore cloud-free images.

[0003] There are many technologies available, among which remote sensing image restoration methods based on deep learning are a common type. Among these methods, restoration technology based on a single image is a mainstream solution. This technology mainly uses the prior information contained in cloud-free optical images or SAR images (synthetic aperture radar images) that are adjacent in time to restore the missing areas caused by cloud cover. However, the restoration method based on a single image has certain limitations. Since it only relies on the information of a single image and lacks information on temporal continuity and change trends, it is difficult to accurately capture the dynamic changes of the ground at different time points, resulting in low reliability of remote sensing optical time series image reconstruction.

[0004] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a remote sensing optical time-series image reconstruction method, device, terminal and storage medium are provided, aiming to solve the problem of low accuracy of remote sensing optical time-series image reconstruction in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a remote sensing optical time-series image reconstruction method, the method comprising:

[0008] Acquire remote sensing optical time-series images and their corresponding shooting date sequences of the target study area within the target time period, as well as SAR time-series images and their corresponding shooting date sequences, wherein the remote sensing optical time-series images include cloud-covered optical time-series images and cloud-free optical time-series images;

[0009] The remote sensing optical time-series image is paired with the SAR time-series image based on the shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, and Gaussian noise is added to the cloud area in the remote sensing optical time-series image to obtain a noisy cloud optical time-series image, wherein the noisy cloud optical time-series image is composed of a plurality of noisy cloud images;

[0010] The noisy cloud-containing optical time-series images, the first SAR time-series images, the shooting date sequence of the cloud-containing optical time-series images, and a preset noise diffusion step number t are input into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, and a reconstructed remote sensing optical time-series image is composed of the cloud-free optical time-series images and the reconstructed cloud-free optical image sequence, and the reconstructed cloud-free image sequence is composed of multiple reconstructed cloud-free images.

[0011] In one implementation, pairing the remote sensing optical time-series image with the SAR time-series image based on shooting dates to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image includes:

[0012] Taking the shooting date sequence of the remote sensing optical time-series image as a reference, the SAR time-series image with the closest shooting date in the shooting date sequence is selected for pairing, so as to obtain a first SAR time-series image paired with the remote sensing optical time-series image.

[0013] In one embodiment, adding Gaussian noise to the cloud region in the remote sensing optical time-series image to obtain the noisy cloud optical time-series image includes:

[0014] Performing cloud detection on each image in the remote sensing optical time-series images to obtain a number of cloud regions;

[0015] Gaussian noise is added to each of the cloud regions to obtain a noisy cloud optical time-series image.

[0016] In one implementation, the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t are input into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, including:

[0017] Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, calling a trained diffusion model through the target sampler, processing using the diffusion model, and outputting a reconstructed image sequence of the t-1th diffusion step number, wherein the reconstructed image sequence is a reconstructed optical image sequence;

[0018] The reconstructed image sequence output by the diffusion model each time and the shooting dates of the first SAR time series image and the cloud-containing optical time series image are iteratively input into the trained diffusion model for processing by using a target sampler until the number of iterations reaches t times, thereby obtaining a reconstructed cloud-free optical image sequence of the 0th diffusion step.

[0019] In one embodiment, the diffusion model includes a coding layer, a plurality of basic reconstruction modules, a linear layer and a decoding layer; the noisy cloud optical time series image, the first SAR time series image, the shooting date sequence of the cloud optical time series image, and a preset noise diffusion step number t are input into a target sampler, the trained diffusion model is called by the target sampler, and the diffusion model is used for processing to output a reconstructed image sequence of the t-1th diffusion step number, including:

[0020] Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into the target sampler;

[0021] The target sampler is used to transmit the noisy and cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and the preset noise diffusion step number t to the coding layer, and the optical time-series image code, the SAR time-series image code, the date code and the noise diffusion step code are obtained through processing by the coding layer;

[0022] After the optical time-series image code and the SAR time-series image code are added to the date code respectively, the codes are inputted into a first basic reconstruction module together with the noise diffusion step code, and fusion features are obtained after processing. The basic reconstruction module is used to perform feature fusion on the input codes;

[0023] Iteratively taking the fusion feature output by the last basic reconstruction module as the updated optical time-series image code, and inputting it together with the SAR time-series image code, the date code and the noise diffusion step code into the next basic reconstruction module for processing, until the last basic reconstruction module outputs the final fusion feature;

[0024] The final fusion feature output by the last basic reconstruction module is input into the linear layer, and the intermediate feature is obtained by processing the linear layer;

[0025] The intermediate features are input to the decoding layer, and are decoded by the decoding layer to obtain a reconstructed image sequence of the t-1th diffusion step.

[0026] In one embodiment, the basic reconstruction module includes a multimodal cross attention module, a time attention module, a spatial attention module and a feedforward neural network; the optical time series image code and the SAR time series image code are respectively added to the date code, and then input together with the noise diffusion step code into the first basic reconstruction module, and the fusion features are obtained after processing, including:

[0027] After adding the optical time sequence image code and the SAR time sequence image code to the date code respectively, the codes are input into the multimodal cross attention module together with the noise diffusion step code, and the multimodal cross attention module uses the optical time sequence image code as a query feature and the SAR time sequence image code as a key feature and a value feature for fusion, and outputs a first basic fusion feature, where the first basic fusion feature contains the cloud prior information provided by the SAR time sequence image;

[0028] Inputting the first basic fusion feature into the time attention module, and optimizing the first basic fusion feature in the time dimension by the time attention module to obtain a second basic fusion feature;

[0029] Inputting the second basic fusion feature into the spatial attention module, and optimizing the second basic fusion feature in the spatial dimension by the spatial attention module to obtain a third basic fusion feature;

[0030] The third basic fusion feature is input into a feedforward neural network, and the feedforward neural network is used to optimize the potential relationship in the third basic fusion feature to obtain a fusion feature.

[0031] In one embodiment, the step of training the diffusion model includes:

[0032] Acquire a training set, wherein the training set includes several groups of training data, each group of training data includes: SAR time series training images, cloud-free optical time series training images, and cloud-containing simulated optical time series training images generated by simulating cloud conditions based on the cloud-free optical time series training images, and their corresponding shooting date sequences;

[0033] Initialize the diffusion model to be trained;

[0034] Iterative training is performed according to preset rounds. In each round of training, all of the cloud-simulated optical time-series training images, the SAR time-series training images, the shooting date sequence corresponding to the cloud-simulated optical time-series training images, and a preset number of training diffusion steps are used as inputs, and all of the cloud-free optical time-series training images are used as true values. Forward diffusion and reverse denoising are performed in sequence to train the diffusion model to be trained, and back propagation is performed after calculating the loss function of each round to update the model parameters;

[0035] When the loss function converges to a preset value, the training is terminated to obtain a trained diffusion model.

[0036] In a second aspect, an embodiment of the present invention provides a remote sensing optical time-series image reconstruction device, the device comprising:

[0037] A data acquisition module is used to acquire remote sensing optical time-series images and their corresponding shooting date sequences of the target research area within the target time period, as well as SAR time-series images and their corresponding shooting date sequences, wherein the remote sensing optical time-series images include cloud-covered optical time-series images and cloud-free optical time-series images;

[0038] an image processing module, configured to pair the remote sensing optical time-series image with the SAR time-series image based on a shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, add Gaussian noise to a cloud region in the remote sensing optical time-series image to obtain a noisy cloud-containing optical time-series image, wherein the noisy cloud-containing optical time-series image is composed of a plurality of noisy cloud-containing images;

[0039] A time series image reconstruction module is used to input the noisy cloud-containing optical time series image, the first SAR time series image, the shooting date sequence of the cloud-containing optical time series image, and a preset noise diffusion step number t into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, and the reconstructed remote sensing optical time series image is composed of the cloud-free optical time series image and the reconstructed cloud-free optical image sequence, and the reconstructed cloud-free image sequence is composed of multiple reconstructed cloud-free images.

[0040] In a third aspect, an embodiment of the present invention further provides a terminal, comprising: a memory, a processor, and a remote sensing optical time-series image reconstruction program stored in the memory and executable on the processor, wherein the remote sensing optical time-series image reconstruction program implements the steps of the remote sensing optical time-series image reconstruction method as described above when executed by the processor.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a remote sensing optical time-series image reconstruction program, and the remote sensing optical time-series image reconstruction program can be executed to implement the steps of the remote sensing optical time-series image reconstruction method as described above.

[0042] Beneficial effects of the present invention: The present invention obtains remote sensing optical time-series images, SAR time-series images and their respective shooting date sequences; obtains the first SAR time-series image and the noisy cloud optical time-series image; inputs the noisy cloud optical time-series image, the first SAR time-series image, the cloud optical time-series image's shooting date sequence and the noise diffusion step number t into the target sampler, and iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence, and the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence constitute a reconstructed remote sensing optical time-series image. The present invention introduces SAR time-series images and shooting date sequences as prior information, combines the high-quality image generation capability of the multimodal diffusion model, and guides the reconstruction of the cloud image, which can effectively improve the reliability of remote sensing optical time-series image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a preferred embodiment of the remote sensing optical time-series image reconstruction method in the present invention.

[0044] Figure 2 It is a schematic diagram of the diffusion model processing flow in the present invention.

[0045] Figure 3 It is a schematic diagram of forward diffusion and reverse denoising in the present invention.

[0046] Figure 4 It is a schematic diagram of the comparison between the method of the present invention and the existing method on the test set.

[0047] Figure 5 It is a schematic diagram of remote sensing optical time-series images with clouds under real conditions in the present invention.

[0048] Figure 6 It is a schematic diagram of the reconstructed remote sensing optical time-series image in the present invention.

[0049] Figure 7 It is a structural schematic diagram of a preferred embodiment of the remote sensing optical time-series image reconstruction device in the present invention.

[0050] Figure 8 It is a terminal principle block diagram of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0052] Satellite long-term observations are essential for monitoring changes and trends in the surface environment. However, cloud cover causes frequent data gaps, and the quality of optical image time series is seriously affected. To solve this problem, it is necessary to reconstruct cloud images and restore cloud-free images.

[0053] There are many technologies available, among which remote sensing image restoration methods based on deep learning are a common type. Among these methods, restoration technology based on a single image is a mainstream solution. This technology mainly uses the prior information contained in temporally adjacent cloud-free optical images or SAR images (synthetic aperture radar images) to restore the missing areas caused by cloud cover. However, the restoration method based on a single image has certain limitations. Since it only relies on the information of a single image and lacks information on temporal continuity and change trends, it is difficult to accurately capture the dynamic changes of the ground at different time points, resulting in low accuracy in the reconstruction of remote sensing optical time series images.

[0054] In view of the above-mentioned defects of the prior art, the present invention provides a remote sensing optical time-series image reconstruction method, device, terminal and storage medium, which belongs to the field of remote sensing technology. The method includes: obtaining remote sensing optical time-series images and their corresponding shooting date sequences, as well as SAR time-series images and their corresponding shooting date sequences; obtaining a first SAR time-series image and a noisy cloud optical time-series image; the noisy cloud optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud optical time-series image, and the preset noise diffusion step number t target sampler are iteratively called t times by the target sampler to obtain a reconstructed cloud-free optical image sequence with the 0th diffusion step number, and the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence constitute a reconstructed remote sensing optical time-series image. The present invention introduces SAR time-series images and shooting date sequences as prior information, combines the high-quality image generation capability of the multimodal diffusion model, guides the reconstruction of cloud images, and can effectively improve the reliability of remote sensing optical time-series image reconstruction.

[0055] See also Figure 1 The remote sensing optical time-series image reconstruction method described in the embodiment of the present invention comprises the following steps:

[0056] Step S100, acquiring remote sensing optical time-series images and their corresponding shooting date sequences, as well as SAR time-series images and their corresponding shooting date sequences of the target study area within the target time period, wherein the remote sensing optical time-series images include cloud optical time-series images and cloud-free optical time-series images.

[0057] Specifically, for the target research area, the optical remote sensing sensor and the synthetic aperture radar sensor are used to monitor together. The optical remote sensing sensor can take remote sensing optical time-series images at regular intervals, and the synthetic aperture radar sensor can also take SAR time-series images at regular intervals. In the present invention, the main focus is on the presence of clouds in the target time period. At this time, the remote sensing optical time-series images include cloud optical time-series images and cloudless optical time-series images. Among them, the cloud optical time-series images are composed of cloud images, and the cloudless optical time-series images are composed of cloudless images. The cloud images are images containing clouds, and the cloudless images are images that do not contain clouds. It can be understood that the purpose of the present invention is to reconstruct the cloud area in the remote sensing optical time-series images and process them into cloudless images. The cloudless images in the remote sensing optical time-series images themselves remain unchanged, that is, they do not need to be reconstructed.

[0058] See also Figure 1 The remote sensing optical time-series image reconstruction method described in the embodiment of the present invention comprises the following steps:

[0059] Step S200: Pair the remote sensing optical time series image with the SAR time series image based on the shooting date to obtain a first SAR time series image corresponding to the remote sensing optical time series image; add Gaussian noise to the cloud area in the remote sensing optical time series image to obtain a noisy cloud optical time series image, wherein the noisy cloud optical time series image is composed of a plurality of noisy cloud images.

[0060] Specifically, the remote sensing optical time series image is paired with the SAR time series image based on the shooting date to obtain the remote sensing optical time series image and the first SAR time series image with equal sequence lengths, and then Gaussian noise is added to the cloud area of ​​the remote sensing optical time series image to meet the input requirements of the diffusion model.

[0061] In one implementation, the remote sensing optical time-series image is paired with the SAR time-series image based on a shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, including:

[0062] Taking the shooting date of the remote sensing optical time-series image as a reference, the SAR time-series image with the closest shooting date is selected for pairing, so as to obtain a first SAR time-series image paired with the remote sensing optical time-series image.

[0063] Specifically, a date nearest neighbor matching algorithm is used to match the remote sensing optical time series image and the SAR time series image to obtain a first SAR time series image paired with the remote sensing optical time series image.

[0064] In one implementation, adding Gaussian noise to the cloud region in the remote sensing optical time-series image to obtain the noisy cloud optical time-series image includes:

[0065] Performing cloud detection on each image in the remote sensing optical time-series images to obtain a number of cloud regions;

[0066] Gaussian noise is added to each of the cloud regions to obtain a noisy cloud optical time-series image.

[0067] Specifically, cloud detection is performed on each image in the optical time-series images with clouds using the CloudSEN12 cloud detection method.

[0068] See also Figure 1 The remote sensing optical time-series image reconstruction method described in the embodiment of the present invention further includes the following steps:

[0069] Step S300: input the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, and the reconstructed remote sensing optical time-series image is composed of the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence, and the reconstructed cloud-free image sequence is composed of a plurality of reconstructed cloud-free images.

[0070] Specifically, the noisy cloud optical time series image is the object that needs to be reconstructed, and it needs to be processed and reconstructed into a cloud-free image. The shooting date sequence of the cloud optical time series image and the first SAR time series image are used as the prior information for image reconstruction, and the noise diffusion step number t is the reflection of the noise level of the current step in the diffusion model. Figure 2 As shown, the shooting date sequence of the noisy cloud optical time series image, the first SAR time series image, the cloud optical time series image, and the preset noise diffusion step number t are input into the target sampler, and the target sampler iteratively calls the trained diffusion model t times. Through iterative calls, a high-quality cloud-free image sequence can be gradually restored.

[0071] In the prior art, there are many methods for recovering missing data caused by cloud occlusion. In the early days, statistical models were used for evaluation, such as using linear interpolation to obtain linear coefficients by statistics of known pixels in the time series, and then calculating missing values ​​through linear models. In addition, seasonal change models can be established using harmonic functions, logarithmic functions, and polynomial functions to recover missing data. However, these methods are usually based on strong assumptions and are not suitable for rapid surface changes. Some other statistical methods use similar pixels for reconstruction. For example, the Neighborhood Similar Pixel Interpolator (NSPI) method uses a weighted linear model to achieve reconstruction with the help of similar pixels in spatial and temporal dimensions; the Cascade Temporal and Spatial Reconstruction Model (CTS) method uses a progressive missing value filling scheme to ensure that similar pixels are close to the target pixels in space. However, when the number of similar pixels is insufficient, the effect of such methods will be unstable.

[0072] In recent years, remote sensing image restoration methods based on deep learning have developed rapidly. Deep learning can extract complex spatiotemporal and deep features from a large amount of data, and these features are crucial for time series reconstruction of remote sensing images. Depending on the input conditions and the reconstruction object, the existing methods mainly include single image-based methods and image sequence-based methods.

[0073] Single-image based methods use prior information from temporally adjacent cloud-free optical images or synthetic aperture radar (SAR) images to restore missing areas. For example, the temporal information embedding network (TIIN) method introduces a temporal branch to inject compensatory information from temporally adjacent optical images. Due to the lack of time series information, such methods usually perform poorly in highly dynamic and heterogeneous areas. SAR can penetrate clouds and provide structural information of the surface beneath the clouds. The introduction of SAR can make the reconstruction results more reliable. The parallel generative adversarial network (Parallel-GAN) method uses the strategy of SAR-optical image conversion for reconstruction. Taking synthetic aperture radar images as input, a pixel-level mapping from synthetic aperture radar images to optical images is learned through a generative adversarial network (GAN). However, due to the significant differences between optical and SAR sensors, it is challenging to directly learn this mapping, and the learned pixel-level mapping is not stable for complex land cover types. In addition, since it only relies on the information of a single image and lacks temporal continuity and change trend information, it is difficult to accurately capture the dynamic changes of the ground at different time points, resulting in low accuracy of remote sensing optical time series image reconstruction.

[0074] The image sequence-based method considers the correlation of time and space in the reconstruction process, ensuring the spatial continuity of the ground objects while fully exploring the temporal variation law. The rich information contained in the image sequence improves the reliability of the reconstruction. The Uncertainty quantification for cloud removal in optical satellite time series (UnCRtainTS) method is a method that maps multiple cloud images into a single cloud-free optical image, but the input sequence only contains three images and cannot directly reconstruct the complete image sequence. The U-Net temporal imputation lightweight image sequence encoder (U-TILISE) method uses a lightweight convolutional neural network (CNN) and temporal attention to extract the spatiotemporal features of the optical image time series for reconstruction, achieving better performance than the commonly used linear interpolation method. In addition, the U-TILISE method also attempts to introduce SAR time series for construction, but experimental results show that stacking optical images and SAR images as network input has not been proven to significantly improve the effect.

[0075] In order to improve the reliability and versatility of optical image time series reconstruction, the present invention proposes a remote sensing optical image time series reconstruction method based on a multimodal sequence diffusion model (i.e., RESTORE-DiT). The present invention applies the DiT (DiffusionTransformer) generation framework to remote sensing optical image time series reconstruction, and uses multimodal prior information to guide reliable reconstruction. Specifically, in addition to using the spatiotemporal information of the optical time series itself, SAR time series images are used as the main prior conditions, and the periodic changes of ground objects and the differences in observation time intervals are considered in the reconstruction in combination with date information. In this way, high-quality, dense and cloudless optical image sequences can be effectively generated, thereby better serving surface observation tasks such as vegetation phenology extraction, crop classification, urban planning, and natural disaster assessment.

[0076] In one implementation, the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t are input into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, including:

[0077] Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, calling a trained diffusion model through the target sampler, processing using the diffusion model, and outputting a reconstructed image sequence of the t-1th diffusion step number, wherein the reconstructed image sequence is a reconstructed optical image sequence;

[0078] The reconstructed image sequence output by the diffusion model each time and the shooting dates of the first SAR time series image and the cloud-containing optical time series image are iteratively input into the trained diffusion model for processing by using a target sampler until the number of iterations reaches t times, thereby obtaining a reconstructed cloud-free optical image sequence of the 0th diffusion step.

[0079] Specifically, by iteratively calling the model, each iteration can reduce a portion of the noise, so that the image can be closer to a real cloud-free optical image after multiple iterations.

[0080] In one implementation, the diffusion model includes a coding layer, a plurality of basic reconstruction modules, a linear layer, and a decoding layer; the noisy cloud optical time series image, the first SAR time series image, the shooting date sequence of the cloud optical time series image, and the preset noise diffusion step number t are input into a target sampler, the trained diffusion model is called by the target sampler, and the diffusion model is used for processing to output a reconstructed image sequence of the t-1th diffusion step number, including:

[0081] Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into the target sampler;

[0082] The target sampler is used to transmit the noisy and cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and the preset noise diffusion step number t to the coding layer, and the optical time-series image code, the SAR time-series image code, the date code and the noise diffusion step code are obtained through processing by the coding layer;

[0083] After adding the optical time-series image code and the SAR time-series image code to the date code respectively, the codes are input into a first basic reconstruction module together with the noise diffusion step code, and fusion features are obtained after processing;

[0084] Iteratively taking the fusion feature output by the last basic reconstruction module as the updated optical time-series image code, and inputting it together with the SAR time-series image code, the date code and the noise diffusion step code into the next basic reconstruction module for processing, until the last basic reconstruction module outputs the fusion feature;

[0085] Inputting the fused features output by the last basic reconstruction module into the linear layer, and obtaining the intermediate features through processing by the linear layer;

[0086] The intermediate features are input to the decoding layer, and are decoded by the decoding layer to obtain a reconstructed image sequence with t-1 diffusion steps.

[0087] Specifically, in the coding layer of the diffusion model, the noisy cloud optical time-series image and the first SAR time-series image are spatially divided into multiple N×N patches, and then linearly transformed to obtain the optical time-series image coding: and SAR time series image coding For the shooting date sequence of cloud optical time series images, convert each date into the format of the day of the year. For example, January 3 is the 3rd day. In this way, the date data is converted into a simpler, continuous numerical representation, which is convenient for subsequent mathematical processing. Then use the sine function to process the shooting date sequence in the format of the day of the year to obtain the date code For the noise diffusion step number encoding F t The noise diffusion steps are also obtained by using the sine function.

[0088] After the code is obtained, it is first input into the first basic reconstruction module, and then the output of the first basic reconstruction module is used as the updated optical time series image code, and is input into the next basic reconstruction module together with the SAR time series image code, date code and noise diffusion step code for processing. This operation is iterated until the last basic reconstruction module outputs the final fusion feature. Then it is linearly changed through the linear layer to obtain the intermediate feature, and then the intermediate feature is decoded by the decoding layer to obtain the reconstructed image sequence under t-1 diffusion steps.

[0089] The above process is to call the trained diffusion model once, and obtain the reconstructed image sequence under t-1 diffusion steps from the noisy cloud optical time series image with t diffusion steps. At this time, the reconstructed image sequence under t-1 diffusion steps still contains noise. Therefore, it is necessary to use the target sampler to iteratively call the trained diffusion model t times to restore high-quality cloud-free optical time series images.

[0090] In one implementation, the number of the basic reconstruction modules is four.

[0091] In one implementation, the basic reconstruction module includes a multimodal cross attention module, a temporal attention module, a spatial attention module and a feedforward neural network; the optical time series image code and the SAR time series image code are respectively added to the date code, and then input together with the noise diffusion step code into the first basic reconstruction module, and the fusion features are obtained after processing, including:

[0092] After adding the optical time sequence image code and the SAR time sequence image code to the date code respectively, the codes are input into the multimodal cross attention module together with the noise diffusion step code, and the multimodal cross attention module uses the optical time sequence image code as a query feature and the SAR time sequence image code as a key feature and a value feature for fusion, and outputs a first basic fusion feature, where the first basic fusion feature contains the cloud prior information provided by the SAR time sequence image;

[0093] Inputting the first basic fusion feature into the time attention module, and optimizing the first basic fusion feature in the time dimension by the time attention module to obtain a second basic fusion feature;

[0094] Inputting the second basic fusion feature into the spatial attention module, and optimizing the second basic fusion feature in the spatial dimension by the spatial attention module to obtain a third basic fusion feature;

[0095] The third basic fusion feature is input into a feedforward neural network, and the feedforward neural network is used to optimize the potential relationship in the third basic fusion feature to obtain a fusion feature.

[0096] Specifically, the multimodal cross-attention module aims to utilize the spatiotemporal characteristics of SAR time-series images to guide the reconstruction process of noisy optical time-series images. The multimodal cross-attention module consists of an adaptive layer normalization layer, a cross-temporal attention layer, and a cross-spatial attention layer. The temporal attention module consists of an adaptive layer normalization layer and a temporal self-attention layer, and the spatial attention module consists of an adaptive layer normalization layer and a spatial self-attention layer. The feedforward neural network consists of a multi-layer perceptron (MLP). The first basic fusion feature output by the multimodal cross-attention module can be expressed as By focusing on the optimization of specific dimensions or relationships through different modules, a more comprehensive fusion feature can be obtained in the end, providing high-quality input for subsequent image reconstruction.

[0097] In one implementation, the step of training the diffusion model includes:

[0098] A training set is obtained, wherein the training set includes several groups of training data, each group of training data includes:

[0099] SAR time series training images, cloud-free optical time series training images, and cloud-simulated optical time series training images generated by simulating cloud conditions based on the cloud-free optical time series training images, and their corresponding shooting date sequences, the cloud-simulated optical time series training images and the SAR time series training images are in a paired relationship;

[0100] Initialize the diffusion model to be trained;

[0101] Iterative training is performed according to preset rounds. In each round of training, all of the cloud-simulated optical time-series training images, the SAR time-series training images, the shooting date sequence corresponding to the cloud-simulated optical time-series training images, and a preset number of training diffusion steps are used as inputs, and all of the cloud-free optical time-series training images are used as true values. Forward diffusion and reverse denoising are performed in sequence to train the diffusion model to be trained, and back propagation is performed after calculating the loss function of each round to update the model parameters;

[0102] When the loss function converges to a preset value, the training is terminated to obtain a trained diffusion model.

[0103] Specifically, before obtaining the training set, a training set is first constructed. A multimodal time-series remote sensing image dataset is obtained, which covers multiple sets of remote sensing image data. Each set of remote sensing image data includes an initial optical time-series image, an initial SAR time-series image, and respective shooting date sequences taken for the same area within the same time range. The multimodal time-series remote sensing image dataset can be a PASTIS-R public dataset. The PASTIS-R public dataset contains image sequences of two modes, SAR and optical, collected from Sentinel-1 and sentinel-2 satellites, respectively. The image size is 128×128, and a total of 2433 sets of sequences are included, each sequence containing 38 to 61 images. The initial optical time-series image contains a cloud optical image set and a cloud-free optical image set. Cloud detection is performed on the initial optical time-series image to distinguish between a cloud optical image set and a cloud-free optical image set. It can be understood that all cloud optical images are cloud images, and all cloud-free optical images are cloud-free images. The cloud detection process will also obtain a cloud mask corresponding to each cloud image. All cloud masks of each set of remote sensing image data are randomly sampled to obtain the simulated cloud mask corresponding to each set of remote sensing image data. The simulated cloud mask is added to each cloud-free image of each set of remote sensing image data, and Gaussian noise is added to the simulated cloud mask to obtain a number of cloud-simulated images, i.e., images simulating cloud conditions. All cloud-simulated images form a cloud-simulated optical image set. The shooting date sequence corresponding to the cloud-free optical image set is used as the shooting date sequence corresponding to the cloud-simulated optical time-series training image. The SAR image closest to the shooting date of the cloud-simulated image is selected from the initial SAR time-series images of the same set of remote sensing image data to form a SAR time-series image set. By performing the above operations on each set of remote sensing image data, multiple sets of SAR time-series image sets, cloud-simulated optical image sets, and cloud-optical image sets with corresponding relationships can be obtained. Then, all the SAR time series image sets and the corresponding cloud-simulated optical image sets and cloud-optical image sets are selected according to a preset ratio to obtain several groups of training data, each group of training data includes SAR time series training images, cloud-free optical time series training images, and cloud-simulated optical time series training images generated by simulating cloud conditions on the basis of cloud-free optical time series training images, as well as their corresponding shooting date sequences.

[0104] In one implementation, the preset ratio is 80%, and 80% of the data is used as a training set, and the remaining 20% ​​of the data is used as a test set.

[0105] In each iterative training, there are two processes, forward diffusion and reverse denoising, such as Figure 3 In the process of forward diffusion, noise is gradually added to the cloud-free optical time series training image according to the following formula: ,in, Indicates that at a given xt-1 Under the condition of t The probability distribution of , N(.) represents the normal distribution, is a predefined variance index, t represents the number of diffusion steps, I represents the identity matrix, and x t-1 represents the cloud-free optical time series training image after adding t-1 step noise, x t represents the cloud-free optical temporal training image after adding t steps of noise. The noise of each step is added to the cloud mask area. 0 represents a cloud-free optical time-series image, which contains multiple cloud-free images. For each cloud-free image, forward diffusion and reverse denoising are iterated T times. When T is large enough, x 0 Will eventually become a pure Gaussian noise image x T T can be set to 1000 times. In the reverse denoising process, the diffusion model accurately predicts the mean and variance of the Gaussian distribution by learning the input noise image and the number of diffusion steps, thereby realizing the restoration from the noisy image to a relatively clearer image at the previous moment.

[0106] The relevant formula for the reverse denoising process is as follows: , where Indicates that at a given x t Under the condition of t-1 The probability distribution of θ is the parameter of the diffusion model. According to the diffusion model, x t and the mean of the normal distribution predicted by the diffusion step number t, is the diffusion model according to X t and the variance of the Gaussian distribution predicted by t.

[0107] The formula of the loss function is , where E represents expectation, is the prediction result of the model, and c represents the conditional prior of reconstruction. After calculating the loss function, the Adam optimizer is used for back propagation to update the model parameters.

[0108] After the diffusion model is trained, the diffusion step number t is preset, and the trained diffusion model is iteratively called t times using the target sampler to restore high-quality cloud-free optical time-series images.

[0109] In the classical diffusion model, forward diffusion adds T steps of noise, and backward denoising also requires T steps of denoising, that is, the model needs to do T inferences, which is very inefficient. In the present invention, the target sampler can be a DPM-sovler sampler. In this case, the diffusion step number t is set to 1. The DPM-sovler sampler only needs one inference to recover the reconstructed cloud-free image sequence from the noisy cloud image sequence under T diffusion steps. It and the original cloud-free optical time-series image together constitute the reconstructed remote sensing optical time-series image, which aims to more effectively acquire and restore the remote sensing optical time-series image information covered by clouds, and improve the efficiency and accuracy of data processing and analysis.

[0110] The diffusion model of the present invention is a sequence-to-sequence model, the input is a sequence of cloud optical images with cloud masks, and the output is a high-quality cloud-free optical time-series image. Compared with the single image reconstruction algorithm, this method has higher reconstruction efficiency and can efficiently utilize the cloud-free pixel information in the sequence. In addition, the present invention makes full use of multimodal priors for optical time-series reconstruction. The SAR image sequence of the corresponding date provides an accurate cloud-under-the-cloud prior for each optical image. Compared with the existing time-series reconstruction methods, it can more accurately reconstruct high-dynamic scenes such as vegetation, and the reliability of the reconstruction results is greatly improved. In addition, the input data conditions of the present invention are loose, support time series with irregular time intervals, and at the same time, have a high tolerance for cloud detection errors. Compared with the existing time-series reconstruction methods, it can better suppress the negative impact of cloud detection errors on reconstruction, and the stability of the reconstruction results is stronger.

[0111] In order to verify the practicality of this example, it is compared with other optical image reconstruction methods, including the Deep Sentinel-2 Cloud Removal method (DSen2-CR), the linear interpolation method (Linear), the U-Net temporal imputation lightweight image sequence encoder (U-TILISE), and the U-Net temporal imputation lightweight image sequence encoder with SAR (U-TILISE-SAR). The experiment uses the PyTorch deep learning framework to build the model, and trains and tests on the PASTIS-R dataset. The quantitative performance of different methods on the test set is shown in the figure below. Figure 4 As shown. Figure 4It can be seen that this example has achieved the best performance in terms of MAE (mean absolute error), RMSE (root mean square error), SAM (spectral angle mapping), PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index), indicating that the optical time series image reconstructed by this example is closest to the real image and is significantly better than other comparison methods in terms of reconstruction reliability.

[0112] Figure 5 This is a schematic diagram of remote sensing optical time-series images with clouds in real situations. It can be seen that there are clouds blocking the view at certain moments. Figure 6 It is a schematic diagram of a reconstructed remote sensing optical time-series image. From the reconstruction results, it can be seen that the area blocked by clouds and shadows is completely restored and naturally connected with the surrounding objects. The reconstructed time-series image conforms to the law of vegetation change, which verifies the effectiveness of the method of the present invention in practical applications.

[0113] In summary, the present invention obtains remote sensing optical time-series images, SAR time-series images and their respective shooting date sequences; obtains the first SAR time-series image and the noisy cloud optical time-series image; inputs the noisy cloud optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud optical time-series image and the noise diffusion step number t into the target sampler, and iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence, and the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence constitute a reconstructed remote sensing optical time-series image. The present invention introduces SAR time-series images and shooting date sequences as prior information, combines the high-quality image generation capability of the multimodal diffusion model, and guides the reconstruction of the cloud image, which can effectively improve the reliability of remote sensing optical time-series image reconstruction.

[0114] In one embodiment, if Figure 7 As shown, based on the above remote sensing optical time-series image reconstruction method, the present invention also provides a remote sensing optical time-series image reconstruction device, including:

[0115] The data acquisition module 100 is used to acquire remote sensing optical time-series images and their corresponding shooting date sequences of the target research area within the target time period, and SAR time-series images and their corresponding shooting date sequences, wherein the remote sensing optical time-series images include cloud-covered optical time-series images and cloud-free optical time-series images;

[0116] The image processing module 200 is used to pair the remote sensing optical time-series image with the SAR time-series image based on the shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, add Gaussian noise to the cloud area in the remote sensing optical time-series image to obtain a noisy cloud optical time-series image, wherein the noisy cloud optical time-series image is composed of a plurality of noisy cloud images;

[0117] The time series image reconstruction module 300 is used to input the noisy cloud-containing optical time series image, the first SAR time series image, the shooting date sequence of the cloud-containing optical time series image, and a preset noise diffusion step number t into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, and the reconstructed remote sensing optical time series image is composed of the cloud-free optical time series image and the reconstructed cloud-free optical image sequence, and the reconstructed cloud-free image sequence is composed of multiple reconstructed cloud-free images.

[0118] In one embodiment, the device further comprises:

[0119] The first pairing unit is used to select the SAR time-series image with the closest shooting date in the shooting date sequence of the remote sensing optical time-series image to pair with it, so as to obtain the first SAR time-series image paired with the remote sensing optical time-series image.

[0120] In one embodiment, the device further comprises:

[0121] A first cloud detection unit is used to perform cloud detection on each image in the remote sensing optical time-series image to obtain a number of cloud areas;

[0122] The first noise adding unit is used to add Gaussian noise to each of the cloud regions to obtain a noisy cloud optical time-series image.

[0123] In one embodiment, the device further comprises:

[0124] a single processing unit, configured to input the noisy and cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, call the trained diffusion model through the target sampler, perform processing using the diffusion model, and output a reconstructed image sequence of the t-1th diffusion step number, wherein the reconstructed image sequence is a reconstructed optical image sequence;

[0125] The first iteration unit is used to iteratively input the reconstructed image sequence output by the diffusion model each time and the shooting dates of the first SAR time series image and the cloud-containing optical time series image into the trained diffusion model for processing by using a target sampler until the number of iterations reaches t times, thereby obtaining a reconstructed cloud-free optical image sequence of the 0th diffusion step.

[0126] In one embodiment, the diffusion model includes a coding layer, a plurality of basic reconstruction modules, a linear layer and a decoding layer; the device also includes:

[0127] a data input unit, configured to input the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into the target sampler;

[0128] an encoding unit, configured to transmit the noisy and cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t to the encoding layer, and obtain the optical time-series image code, the SAR time-series image code, the date code, and the noise diffusion step code through processing by the encoding layer;

[0129] A first processing unit is used for adding the optical time-series image code and the SAR time-series image code to the date code respectively, and inputting the code and the noise diffusion step code together into a first basic reconstruction module to obtain a fusion feature after processing;

[0130] A second iterative unit is used to iteratively use the fusion feature output by the previous basic reconstruction module as the updated optical time series image code, and input it together with the SAR time series image code, the date code and the noise diffusion step code to the next basic reconstruction module for processing until the last basic reconstruction module outputs the fusion feature;

[0131] A linear transformation unit, used for inputting the fusion features output by the last basic reconstruction module into the linear layer, and obtaining intermediate features through processing by the linear layer;

[0132] A decoding unit is used to input the intermediate features into the decoding layer, and perform decoding processing through the decoding layer to obtain a reconstructed cloud-free image sequence under t-1 diffusion steps.

[0133] In one embodiment, the basic reconstruction module includes a multimodal cross attention module, a temporal attention module, a spatial attention module and a feedforward neural network; the device also includes:

[0134] A first fusion unit is used for adding the optical time sequence image code and the SAR time sequence image code to the date code respectively, and inputting the code and the noise diffusion step code together into the multimodal cross attention module, and the multimodal cross attention module uses the optical time sequence image code as a query feature and the SAR time sequence image code as a key feature and a value feature for fusion, and outputs a first basic fusion feature, wherein the first basic fusion feature includes the cloud prior information provided by the SAR time sequence image;

[0135] A second fusion unit is used to input the first basic fusion feature into the time attention module, and the time attention module optimizes the first basic fusion feature in the time dimension to obtain a second basic fusion feature;

[0136] A third fusion unit is used to input the second basic fusion feature into the spatial attention module, and the spatial attention module optimizes the second basic fusion feature in the spatial dimension to obtain a third basic fusion feature;

[0137] The final fusion unit is used to input the third basic fusion feature into a feedforward neural network, and optimize the potential relationship in the third basic fusion feature through the feedforward neural network to obtain a fusion feature.

[0138] In one embodiment, the device further comprises:

[0139] A training set acquisition unit is used to acquire a training set, wherein the training set includes a plurality of groups of training data, each group of training data includes: SAR time series training images, cloud-free optical time series training images, and cloud-containing simulated optical time series training images generated by simulating cloud conditions based on the cloud-free optical time series training images, and their corresponding shooting date sequences;

[0140] An initialization unit, used to initialize the diffusion model to be trained;

[0141] a parameter updating unit, configured to take the cloud-simulated optical time-series training image, the SAR time-series training image, the shooting date sequence of the cloud-simulated optical time-series training image, and a preset number of training diffusion steps as input, and take the cloud-free optical time-series training image as a true value, iteratively train the training diffusion model, and in each iterative training, sequentially perform forward diffusion and reverse denoising, and perform back propagation to update model parameters after calculating the loss function;

[0142] The training ending unit is used to end the training when the loss function converges to a preset value to obtain a trained diffusion model.

[0143] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 8As shown. The terminal includes a processor, a memory, a network interface and a display screen connected via a device bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device and a remote sensing optical time-series image reconstruction program. The internal memory provides an environment for the operation of the operating device and the remote sensing optical time-series image reconstruction program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the remote sensing optical time-series image reconstruction program is executed by the processor, the steps of any one of the above-mentioned remote sensing optical time-series image reconstruction methods are implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0144] Those skilled in the art will understand that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, a terminal is provided, which includes a memory, a processor, and a remote sensing optical time-series image reconstruction program stored in the memory and executable on the processor. When the remote sensing optical time-series image reconstruction program is executed by the processor, the steps of any one of the remote sensing optical time-series image reconstruction methods provided in the embodiments of the present invention are implemented.

[0146] An embodiment of the present invention further provides a computer-readable storage medium, on which a remote sensing optical time-series image reconstruction program is stored. When the remote sensing optical time-series image reconstruction program is executed by a processor, the steps of any remote sensing optical time-series image reconstruction method provided by an embodiment of the present invention are implemented.

[0147] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0148] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0149] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A remote sensing optical time-series image reconstruction method, characterized in that: The method comprises: Acquire remote sensing optical time-series images and their corresponding shooting date sequences of the target study area within the target time period, as well as SAR time-series images and their corresponding shooting date sequences, wherein the remote sensing optical time-series images include cloud-covered optical time-series images and cloud-free optical time-series images; The remote sensing optical time-series image is paired with the SAR time-series image based on the shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, and Gaussian noise is added to the cloud area in the remote sensing optical time-series image to obtain a noisy cloud optical time-series image, wherein the noisy cloud optical time-series image is composed of a plurality of noisy cloud images; Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, the target sampler iteratively calls the trained diffusion model t times, and obtains a reconstructed cloud-free optical image sequence of the 0th diffusion step number, the cloud-free optical time-series image and the reconstructed cloud-free optical image sequence constitute a reconstructed remote sensing optical time-series image, and the reconstructed cloud-free optical image sequence consists of a plurality of reconstructed cloud-free images; The noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t are input into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, including: Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, calling a trained diffusion model through the target sampler, processing using the diffusion model, and outputting a reconstructed image sequence of the t-1th diffusion step number, wherein the reconstructed image sequence is a reconstructed optical image sequence; The reconstructed image sequence output by the diffusion model each time and the shooting dates of the first SAR time series image and the cloud-containing optical time series image are iteratively input into the trained diffusion model for processing by using a target sampler until the number of iterations reaches t times, thereby obtaining a reconstructed cloud-free optical image sequence of the 0th diffusion step.

2. The remote sensing optical time-series image reconstruction method according to claim 1, characterized in that: Pairing the remote sensing optical time-series image with the SAR time-series image based on shooting dates to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, including: Taking the shooting date sequence of the remote sensing optical time-series image as a reference, the SAR time-series image with the closest shooting date in the shooting date sequence is selected for pairing, so as to obtain a first SAR time-series image paired with the remote sensing optical time-series image.

3. The remote sensing optical time-series image reconstruction method according to claim 1, characterized in that: Adding Gaussian noise to the cloud area in the remote sensing optical time-series image to obtain a noisy cloud optical time-series image, including: Performing cloud detection on each image in the remote sensing optical time-series images to obtain a number of cloud regions; Gaussian noise is added to each of the cloud regions to obtain a noisy cloud optical time-series image.

4. The remote sensing optical time-series image reconstruction method according to claim 1, characterized in that: The diffusion model includes a coding layer, a plurality of basic reconstruction modules, a linear layer and a decoding layer; the shooting date sequence of the noisy cloud optical time series image, the first SAR time series image, the cloud optical time series image, and a preset noise diffusion step number t are input into a target sampler, the trained diffusion model is called by the target sampler, and the diffusion model is used for processing to output a reconstructed image sequence of the t-1th diffusion step number, including: Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into the target sampler; The target sampler is used to transmit the noisy and cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and the preset noise diffusion step number t to the coding layer, and the optical time-series image code, the SAR time-series image code, the date code and the noise diffusion step code are obtained through processing by the coding layer; After the optical time-series image code and the SAR time-series image code are added to the date code respectively, the codes are inputted into a first basic reconstruction module together with the noise diffusion step code, and fusion features are obtained after processing. The basic reconstruction module is used to perform feature fusion on the input codes; Iteratively taking the fusion feature output by the last basic reconstruction module as the updated optical time-series image code, and inputting it together with the SAR time-series image code, the date code and the noise diffusion step code into the next basic reconstruction module for processing, until the last basic reconstruction module outputs the final fusion feature; The final fusion feature output by the last basic reconstruction module is input into the linear layer, and the intermediate feature is obtained by processing the linear layer; The intermediate features are input to the decoding layer, and are decoded by the decoding layer to obtain a reconstructed image sequence of the t-1th diffusion step.

5. The remote sensing optical time-series image reconstruction method according to claim 4, characterized in that: The basic reconstruction module includes a multimodal cross attention module, a time attention module, a spatial attention module and a feedforward neural network; the optical time series image code and the SAR time series image code are respectively added to the date code, and then input together with the noise diffusion step code into the first basic reconstruction module, and the fusion features are obtained after processing, including: After adding the optical time sequence image code and the SAR time sequence image code to the date code respectively, the codes are input into the multimodal cross attention module together with the noise diffusion step code, and the multimodal cross attention module uses the optical time sequence image code as a query feature and the SAR time sequence image code as a key feature and a value feature for fusion, and outputs a first basic fusion feature, where the first basic fusion feature contains the cloud prior information provided by the SAR time sequence image; Inputting the first basic fusion feature into the time attention module, and optimizing the first basic fusion feature in the time dimension by the time attention module to obtain a second basic fusion feature; Inputting the second basic fusion feature into the spatial attention module, and optimizing the second basic fusion feature in the spatial dimension by the spatial attention module to obtain a third basic fusion feature; The third basic fusion feature is input into a feedforward neural network, and the feedforward neural network is used to optimize the potential relationship in the third basic fusion feature to obtain a fusion feature.

6. The remote sensing optical time-series image reconstruction method according to claim 1, characterized in that: The training steps of the diffusion model include: Acquire a training set, wherein the training set includes several groups of training data, each group of training data includes: SAR time series training images, cloud-free optical time series training images, and cloud-containing simulated optical time series training images generated by simulating cloud conditions based on the cloud-free optical time series training images, and their corresponding shooting date sequences; Initialize the diffusion model to be trained; Iterative training is performed according to preset rounds. In each round of training, all of the cloud-simulated optical time-series training images, the SAR time-series training images, the shooting date sequence corresponding to the cloud-simulated optical time-series training images, and a preset number of training diffusion steps are used as inputs, and all of the cloud-free optical time-series training images are used as true values. Forward diffusion and reverse denoising are performed in sequence to train the diffusion model to be trained, and back propagation is performed after calculating the loss function of each round to update the model parameters; When the loss function converges to a preset value, the training is terminated to obtain a trained diffusion model.

7. A remote sensing optical time-series image reconstruction device, characterized in that: include: A data acquisition module is used to acquire remote sensing optical time-series images and their corresponding shooting date sequences of the target research area within the target time period, as well as SAR time-series images and their corresponding shooting date sequences, wherein the remote sensing optical time-series images include cloud-covered optical time-series images and cloud-free optical time-series images; an image processing module, configured to pair the remote sensing optical time-series image with the SAR time-series image based on a shooting date to obtain a first SAR time-series image corresponding to the remote sensing optical time-series image, add Gaussian noise to a cloud region in the remote sensing optical time-series image to obtain a noisy cloud-containing optical time-series image, wherein the noisy cloud-containing optical time-series image is composed of a plurality of noisy cloud-containing images; a time series image reconstruction module, for inputting the noisy cloud-containing optical time series image, the first SAR time series image, the shooting date sequence of the cloud-containing optical time series image, and a preset noise diffusion step number t into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence of the 0th diffusion step number, and the cloud-free optical time series image and the reconstructed cloud-free optical image sequence form a reconstructed remote sensing optical time series image, and the reconstructed cloud-free optical image sequence is composed of a plurality of reconstructed cloud-free images; The noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t are input into a target sampler, and the target sampler iteratively calls the trained diffusion model t times to obtain a reconstructed cloud-free optical image sequence with a 0th diffusion step number, including: Inputting the noisy cloud-containing optical time-series image, the first SAR time-series image, the shooting date sequence of the cloud-containing optical time-series image, and a preset noise diffusion step number t into a target sampler, calling a trained diffusion model through the target sampler, processing using the diffusion model, and outputting a reconstructed image sequence of the t-1th diffusion step number, wherein the reconstructed image sequence is a reconstructed optical image sequence; The reconstructed image sequence output by the diffusion model each time and the shooting dates of the first SAR time series image and the cloud-containing optical time series image are iteratively input into the trained diffusion model for processing by using a target sampler until the number of iterations reaches t times, thereby obtaining a reconstructed cloud-free optical image sequence of the 0th diffusion step.

8. A terminal, characterized in that: The terminal includes a memory, a processor, and a remote sensing optical time-series image reconstruction program stored in the memory and executable on the processor. When the remote sensing optical time-series image reconstruction program is executed by the processor, the steps of the remote sensing optical time-series image reconstruction method as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a remote sensing optical time-series image reconstruction program, and when the remote sensing optical time-series image reconstruction program is executed by the processor, the steps of the remote sensing optical time-series image reconstruction method according to any one of claims 1 to 6 are implemented.

Images