Optical image reconstruction method and device based on conditional diffusion model

Abstract

This invention discloses an optical image reconstruction method and apparatus based on a conditional diffusion model. The method includes: extracting observations at pixel locations from optical and radar image data to obtain optical and radar time series; inputting the time series as training data into the model, the training process including adding noise to the optical time series through a forward diffusion process to obtain a missing observation sequence; extracting temporal and physical features of the time series through a multimodal denoising network, and using the reconstructed missing observation sequence as the training target; inputting the optical and radar time series to be reconstructed into the multimodal denoising network, and outputting the reconstructed optical time series. By employing the above technical solution, extracting time series at image pixel locations and using the reconstructed time series as the model output target overcomes the limitations of overall image generation and effectively restores the detailed features of optical images; the temporal and physical features are decoupled, avoiding mutual interference between radar image data and optical image data.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing image processing and long-term series analysis technology, and in particular to an optical image reconstruction method and apparatus based on a conditional diffusion model. Background Technology

[0002] Currently, optical satellite time series reconstruction mainly relies on traditional interpolation methods and single-modal deep learning techniques. However, when faced with long-term data gaps in high cloud-covered areas, the reconstruction accuracy and reliability of these methods significantly decrease, making it difficult to meet the high-precision application requirements such as vegetation dynamic monitoring.

[0003] To overcome this limitation, multimodal fusion schemes for fusing auxiliary data sources such as satellite optical imagery and synthetic aperture radar (SAR) have begun to be studied.

[0004] However, effectively fusing optical and SAR images—two heterogeneous data sources—has become a major challenge. Existing technologies, whether pre- or post-fusion, struggle to fully and effectively capture the complex spatiotemporal dependencies between the two modalities, often leading to spectral distortion or temporal deviations in the reconstruction results. Furthermore, the inherent speckle noise in SAR image data, and the differences in physical interpretation and representation between SAR and optical images, further complicate cross-modal feature alignment. Summary of the Invention

[0005] Purpose of the invention: This invention provides an optical image reconstruction method based on a conditional diffusion model, aiming to solve the technical problem of low accuracy of physical features and temporal features in reconstructed optical images in the prior art.

[0006] Technical Solution: This invention provides an optical image reconstruction method based on a conditional diffusion model, comprising: acquiring optical image data through an optical satellite, acquiring corresponding radar image data through synthetic aperture radar, and aligning the radar image data with the optical image data in time steps; at multiple time steps, using pixels as independent processing objects, extracting spectral band observations of the optical image data at the image pixel positions and backscattering coefficient observations of the radar image data at the image pixel positions to obtain optical time series and radar time series; inputting the optical time series and radar time series as training data into a conditional diffusion model; the conditional diffusion model includes a forward diffusion process and a multimodal denoising network, wherein the training process includes: masking the elements in the optical time series through the forward diffusion process to define the missing parts of the simulated observations, adding noise to the masked parts according to the selected denoising iteration steps and the corresponding noise iteration intensity change table, and taking the noisy part of the sequence as the missing observation sequence; extracting the temporal and physical features within the optical time series through the multimodal denoising network, and extracting... The study focuses on the interaction of optical and radar time series, extracting temporal and physical features. A multimodal denoising network is trained using these extracted features and an auxiliary feature set. The training process involves using the multimodal denoising network to iteratively remove noise and restore the original spectral band observations from the missing observation sequence based on the number of denoising iterations and the noise intensity corresponding to each iteration. The training objective of the multimodal denoising network is to reduce the loss between the noise added during forward diffusion and the noise predicted by the network. The auxiliary feature set includes an optical auxiliary sequence, a table of denoising iteration steps, and a table of noise iteration intensity changes. The optical auxiliary sequence is a sequence of spectral band predictions generated within the time period corresponding to the missing observation sequence using effective observation sequences without masks in the optical time series. The actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set are input into the trained multimodal denoising network to obtain the reconstructed actual optical time series. The reconstructed optical image data is then obtained by reshaping the actual optical time series at multiple image pixel locations.

[0007] Specifically, the radar image data is resampled and interpolated over time to align the radar image data with the optical image data in time steps.

[0008] Specifically, the masking of elements in the optical time series includes: masking observations in the same spectral band at all time steps, and / or masking observations in one or more spectral bands at the same time step.

[0009] Specifically, for the optical time series of the mask portion, noise is gradually added at each iteration step according to the selected number of denoising iteration steps. The noise intensity of each iteration step is determined according to the noise iteration intensity change table; the added noise intensity is positively correlated with the number of iteration steps.

[0010] Specifically, through the intra-module interaction module in the multimodal denoising network, for the same optical time series, the temporal characteristics of the changes of the observation values ​​of the same spectral band with time step, as well as the physical characteristics of the relationship between the observation values ​​of multiple spectral bands at a single time step, are extracted.

[0011] Specifically, through the inter-module interaction module in the multimodal denoising network, the temporal and physical features of the optical time series extracted by the intra-module interaction module are used as queries, and the radar time series is used as the key and value. By matching the corresponding key and value through the query, the temporal features of the temporal correlation between the optical time series and the radar time series, as well as the physical features of the physical correlation between the spectral band observation values ​​of the optical time series at each time step and the backscattering coefficient observation values ​​of the radar time series, are extracted.

[0012] Specifically, the multimodal denoising network is built based on the Transformer network structure and adopts a multi-layer residual structure. Each residual layer includes one or more intra-module interaction modules and inter-module interaction modules. The intra-module interaction modules and the inter-module interaction modules include dimensionality decomposition attention heads.

[0013] Specifically, the generation process of the optically assisted sequence includes: if the time step corresponding to the mask part is within the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask part is generated by interpolation based on the effective observation sequence; if the time step corresponding to the mask part is outside the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask part is generated by extrapolation of the first and last observation values ​​based on the effective observation sequence.

[0014] Specifically, noise is added to the mask part according to the different denoising iteration steps and the corresponding noise iteration intensity change table selected multiple times to obtain multiple missing observation sequences. Multiple corresponding reconstructed sequences are obtained by denoising and restoration. The mean of multiple reconstructed sequences is used as the target expected value for reconstructing the missing observation sequence.

[0015] This invention also provides an optical image reconstruction device based on a conditional diffusion model, comprising: a preprocessing unit, a time series extraction unit, a training unit, and a reconstruction unit, wherein: the preprocessing unit is used to acquire optical image data through optical satellites and corresponding radar image data through synthetic aperture radar, and to align the radar image data with the optical image data in time steps; the time series extraction unit is used to extract spectral band observations of the optical image data at the image pixel positions and backscattering coefficient observations of the radar image data at the image pixel positions at multiple time steps, with pixels as independent processing objects, to obtain optical time series and radar time series; the training unit is used to input the optical time series and radar time series as training data into the conditional diffusion model; the conditional diffusion model includes a forward diffusion process and a multimodal denoising network, wherein the training process includes: masking the elements in the optical time series through the forward diffusion process to define the missing parts of the simulated observations, adding noise to the masked parts according to the selected number of denoising iteration steps and the corresponding noise iteration intensity change table, and taking the noisy part of the sequence as the missing observation sequence; and using the multimodal denoising network... The network extracts temporal and physical features from the optical time series, and extracts temporal and physical features from the interaction between the optical and radar time series. A multimodal denoising network is trained using the extracted temporal and physical features, along with an auxiliary feature set. The training process involves using the multimodal denoising network to remove noise step-by-step according to the number of denoising iterations and the noise intensity corresponding to each iteration step, restoring the original spectral band observations in the missing observation sequence. The training objective of the multimodal denoising network is to reduce the loss between the noise added during the forward diffusion process and the noise predicted by the multimodal denoising network. The auxiliary features... The feature set includes an optical auxiliary sequence, a denoising iteration step count, and a noise iteration intensity variation table. The optical auxiliary sequence is a sequence of spectral band prediction values ​​generated in the time period corresponding to the missing observation sequence using an effective observation sequence without a mask in the optical time series. The reconstruction unit is used to input the actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set into the trained multimodal denoising network to obtain the reconstructed actual optical time series. The reconstructed optical image data is obtained by reshaping the actual optical time series at multiple image pixel positions.

[0016] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: It extracts the observation values ​​of optical image data and radar image data at the image pixel positions as time series, and uses the reconstructed time series as the output target of the model, breaking through the limitations of the original "overall image generation", avoiding the negative impact of unrelated parts on the generation of local features in the image, and can more effectively restore the detailed features of optical images; it decouples the independent and interactive temporal and physical features of optical time series and radar time series, avoiding radar image data from contaminating optical image data or radar image data from being masked by optical image data, so that the model can effectively utilize the temporal and physical features of radar image data to restore optical image data, and improves the accuracy of physical features and temporal features of the reconstructed optical image. Attached Figure Description

[0017] Figure 1 A schematic diagram of the internal process of the conditional diffusion model provided by the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0019] In this embodiment of the invention, optical image data is acquired by optical satellites, and corresponding radar image data is acquired by Synthetic Aperture Radar (SAR).

[0020] In practice, multi-temporal optical image data can be multispectral data acquired through the Sentinel-2 satellite, including blue, green, red, and near-infrared spectral bands, with a spatial resolution of 10m and a revisit period of 5 days.

[0021] In practice, multi-temporal radar imagery data can be dual-polarization synthetic aperture radar data acquired through the Sentinel-1 satellite, using a C-band sensor, containing VV polarization and VH polarization data, with a spatial resolution of 5m×20m and a revisit period of 12 days.

[0022] In this embodiment of the invention, radar image data is aligned with optical image data in terms of time steps.

[0023] In practice, a timestep usually refers to the corresponding time information of an element in a sequence, such as the date of generation of an element, which is the timestep of that element.

[0024] In this embodiment of the invention, the radar image data is resampled and interpolated over time to align the radar image data with the optical image data in time steps.

[0025] In practice, the SAR data is resampled from its original resolution to 10 meters to match the resolution of the Sentinel-2 optical data. The resampling method can use bilinear interpolation to ensure that the two types of data are accurately aligned in space.

[0026] In practice, optical image data and SAR image data can be time-registered to have the same time series length, cover the same geographical area, and span a period of one year or longer.

[0027] In practice, due to the different revisit cycles of optical satellites and SAR satellites, the original data acquisition dates of the two are naturally mismatched. Therefore, the acquisition date of the optical data is used as the reference time step, and the SAR image data is processed by linear interpolation to complete the SAR data on the acquisition date of the optical data (the SAR estimate of the missing time step is calculated based on the ratio of adjacent actual SAR observations and time intervals). This makes the SAR data and the optical data have completely overlapping time steps and the same length, forming multimodal time series data that can be directly paired.

[0028] In practice, optical and radar image data can be preprocessed. Optical data preprocessing includes: acquiring Sentinel-2Level-2A surface reflectance products with cloud cover below 10% from the Copernicus Data Space Ecosystem; selecting four spectral bands with a spatial resolution of 10m: blue light (B2, center wavelength 490nm), green light (B3, center wavelength 560nm), red light (B4, center wavelength 665nm), and near-infrared (B8, center wavelength 832nm); and applying the SCL (Scene Classification Layer) cloud mask product to automatically identify and remove invalid pixels such as clouds, cloud shadows, and cloud edges, retaining only high-quality observations under clear-sky conditions. The SAR data was preprocessed as follows: Sentinel-1A GRD (Ground Range Detected) products from the same region and time period as the optical data were acquired, with the working mode being interferometric wide swath (IW), using VV and VH dual polarization data, and the orbit being ascending; orbit correction was performed using a precise orbit file, thermal noise was removed, radiometric calibration was performed to convert pixel values ​​into backscattering coefficients, and terrain correction was performed using the SRTM digital elevation model to eliminate terrain distortion.

[0029] In this embodiment of the invention, at multiple time steps, spectral band observation values ​​of optical image data at image pixel locations are extracted to obtain an optical time series; and backscattering coefficient observation values ​​of radar image data at image pixel locations are extracted to obtain a radar time series.

[0030] In practice, pixels are treated as independent processing objects. Time series are extracted from optical data at the pixel level. Each pixel contains observations of multiple spectral bands in the time series, forming an optical time series X = (x1, x2, x3, ..., x...). L )∈R D×L Where D represents the number of spectral bands, L represents the length of the time series, and the elements in the series, such as x1, are spectral band observations, specifically one or more of the four bands (blue B2, green B3, red B4, and near-infrared B8). The time series is extracted from the radar data at the pixel level, with each pixel containing backscattering coefficient observations, forming the radar time series S = (s1, s2, s3…s…). L )∈R F×L In this context, F represents the two polarization channels VV and VH, and L also represents the time series length. The elements in the series, such as s1, are the backscattering coefficient observations, specifically the observations obtained through one or both of the two polarization channels VV and VH.

[0031] In practical implementation, the observed values ​​at the pixel positions of optical and radar image data are extracted as a time series, and the reconstructed time series is used as the output target of the model. This overcomes the limitations of the original "overall image generation" and avoids the negative impact of unrelated parts on the generation of local features in the image, enabling more effective recovery of the detailed features of optical images. Furthermore, using the spectral bands in the optical time series as independent channels and feature dependencies for recovering the reconstructed sequence is a significant improvement of this invention. This avoids spectral physical contradictions, such as the fact that some NDVI values ​​obtained by the original "overall image generation" exceed a reasonable range.

[0032] See Figure 1 This is a schematic diagram of the internal process of the conditional diffusion model provided by the present invention.

[0033] In this embodiment of the invention, optical time series and radar time series are used as training data input into the conditional diffusion model; the conditional diffusion model includes a forward diffusion process and a multimodal denoising network.

[0034] In practical implementation, the conditional diffusion model is a key model in this invention, mainly comprising a forward diffusion process and a multimodal denoising network. The forward diffusion process and the backward denoising process are two Markov chain processes. The forward diffusion process destroys the data by gradually adding Gaussian noise to the optical time series, while the backward denoising process gradually recovers and reconstructs the original noisy data through a neural network.

[0035] In this embodiment of the invention, a forward diffusion process is used to mask the elements in the optical time series to define the missing parts of the simulated observations. Noise is added to the masked parts according to the selected denoising iteration steps and the corresponding noise iteration intensity change table. The part of the sequence with noise is taken as the missing observation sequence. In specific implementation, during the training phase, "missing observation" samples similar to real cloud pollution are artificially constructed. That is, "missing targets to be imputed" (corresponding to the masked part) and "conditional observations as reference" (corresponding to the unmasked part) are separated from the complete optical time series through the mask. These missing targets to be imputed are then used to simulate cloud pollution missing scenarios of different degrees and modes by adding noise. Training the multimodal denoising network to predict noise requires learning based on "optical time series with missing observation sequences + SAR time series". The forward diffusion process defines the missing parts of the simulated observations through the mask. Gaussian noise is gradually added to the "missing targets" by adding noise to the masked parts. Subsequently, under the guidance of "conditional observations (the part of the optical time series without noise) + SAR time series", the model predicts these added noises and restores the elements in the original optical time series. Without diverse "missing observation" samples constructed based on the masking scheme, the model lacks training data that matches the real-world scenario, making it unable to learn noise patterns under different missing observation modes. Consequently, it cannot achieve accurate noise prediction and subsequent reverse denoising and reconstruction. The masking scheme provides training samples adapted to the real-world scenario for the model to predict noise.

[0036] In this embodiment of the invention, for the optical time series of the mask portion, noise is gradually added at each iteration step according to the selected number of denoising iteration steps, and the noise intensity of each iteration step is determined according to the noise iteration intensity change table; the added noise intensity is positively correlated with the number of iteration steps.

[0037] In practical implementation, the number of denoising iteration steps refers to the number of (denoising) iteration steps. It can be determined randomly or through other suitable methods. It is the number of iterations that control the addition and removal of noise in the conditional diffusion model. It is a calculation step within the model, and its function is to make the addition and removal of noise gradual rather than a one-time completion, so as to simulate the reasonable process of real data from clear to blurry and then from blurry to clear. The reverse process starts from iteration step K (pure noise) and ends at iteration step 1. Each iteration step predicts and removes the noise added in the current iteration step based on auxiliary features such as effective optical observations, radar data, and optical auxiliary sequences. That is, first remove a large amount of noise, transforming pure noise into blurry values ​​with a small number of features, and then gradually remove a small amount of noise, refining the blurry values ​​into a reconstruction result close to the original clear values. Finally, through K iteration steps, the recovery from pure noise to the reconstructed missing region is completed, which can significantly improve the effect of noise prediction and removal.

[0038] In practice, the noise iteration intensity variation table can be predefined. It is a set of values ​​that corresponds one-to-one with each denoising iteration step. Its purpose is to provide a clear noise intensity standard for the noise added in each iteration step, ensuring that the noise gradually accumulates from weak to strong during the forward diffusion process, rather than being added randomly. Each entry in the table corresponds to an iteration step. For example, if the number of iteration steps is set to 50, the table contains 50 values. The larger the iteration step, the larger the corresponding value. For example, the value corresponding to iteration step 1 is smaller and the noise is weaker, while the value corresponding to iteration step 50 is the largest and the noise is strongest. The values ​​in the intensity variation table are determined before model training through linear scheduling or cosine scheduling, etc. Once determined, they remain fixed during training as a uniform basis for noise addition.

[0039] In this embodiment of the invention, observations in the same spectral band at all time steps are selected for masking, or / and observations in one or more spectral bands at the same time step are selected for masking.

[0040] In practical implementation, one or more spectral bands at the same time step are selected for masking. The spectral mask targets different spectral bands within the same time step, randomly or according to certain rules selecting some bands for masking (e.g., masking the near-infrared and short-wave infrared bands). This invalidates the observations in these bands (e.g., setting them to 0 or using special markings), while retaining valid data in other bands (e.g., red, green, and blue bands). This is used to simulate scenarios where "specific spectral bands are missing due to sensor malfunction, atmospheric interference, etc." (e.g., near-infrared band data is invalidated due to water vapor absorption).

[0041] In practice, the same spectral band is selected for masking at all time steps. The time mask is for the observations of the same spectral band at different time steps. The time mask can randomly select some time steps to mask (such as masking day 5 and day 15), that is, to invalidate all or part of the band observations at these time steps, while retaining the valid data of other time steps (such as day 1 and day 10).

[0042] In practice, spectral masking and temporal masking can be performed on spectral bands individually (e.g., only the near-infrared band is masked, and this band is invalidated at all time steps); time steps can be masked individually (e.g., only day 5 is masked, and all or some bands at this time step are invalidated); or the two types of masking can be combined (e.g., masking the red band of day 5 and the near-infrared band of all time steps).

[0043] In practice, a scheme using spectral and temporal masks is adopted. Instead of masking all spectral bands at a specific time step, combinations of time and spectral bands are independently selected for masking, resulting in richer training samples. The mask ratio can vary from 20% to 80%, effectively simulating data loss caused by different degrees of cloud contamination and improving the model's adaptability and generalization ability to various loss patterns. During training, different mask patterns are randomly generated for each batch of data to ensure that the model can learn to handle various cloud contamination and loss patterns that may occur in reality.

[0044] In this embodiment of the invention, the process of generating the optical auxiliary sequence is as follows: if the time step corresponding to the mask part is within the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask part is generated by interpolation based on the effective observation sequence; if the time step corresponding to the mask part is outside the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask part is generated by extrapolation of the first and last observation values ​​based on the effective observation sequence.

[0045] In practice, the optical time series obtained by linear interpolation (or extrapolation of the first and last observations) can be introduced as auxiliary boundary information. This can alleviate the problem of spectral band discontinuity between the effective observation sequence segment and the missing observation sequence segment in the non-autoregressive interpolation model, and guide the multimodal denoising network to predict noise and interpolate to reconstruct the missing observation sequence.

[0046] In practice, the effective observation period refers to the time interval covered by effective optical observation data (cloud-free optical data) within the effective observation sequence. Specifically, it is the range between the minimum and maximum time points (from the earliest to the latest time point) across all time steps corresponding to the effective optical observation data. Points earlier than the minimum or later than the maximum time point are outside the effective observation period. For example, for satellite data from January 1st to December 31st, 2023, only March 1st, May 1st, and August 1st were cloud-free (effective observations), so the effective observation period is from March 1st to August 1st, 2023.

[0047] In practice, the target time step for interpolating missing values ​​must be outside the effective observation range, i.e., earlier than the time step corresponding to the first observation or later than the time step corresponding to the last observation. If the target time step is before the time of the first observation (minimum time point), the first observation (the observation at the minimum time point) is used directly as the interpolation result for that target time step; if the target time step is after the time of the last observation (maximum time point), the last observation (the observation at the maximum time point) is used directly as the interpolation result for that target time step.

[0048] In practical implementation, "spectral jumps" are prone to occur at the adjacent boundaries between the effective observation segment and the interpolation segment (missing observation segment). Essentially, this is because the diffusion model is initialized from pure noise, lacking initial constraints on the overall trend of the time series. In this invention, linear interpolation (within the effective observation range) and extrapolation of the first and last observations (outside the effective observation range) are performed based on the observed values ​​to generate a "rough but physically consistent time series benchmark," which conforms to the prior generation logic. The linear interpolation results are combined with the temporal and physical characteristics of the optical time series to form a composite prior, guiding the diffusion process to optimize details along the physical trend. The physical prior and the temporal prior work together to solve the boundary inconsistency problem of non-autoregressive models.

[0049] In practical implementation, the optical auxiliary sequence, along with temporal embedding, feature embedding, and observation mask, can be transformed into an auxiliary feature set through a feature projection layer. This set is then fused with cross-modal features from optics and radar, and input into a multimodal denoising network. Temporal embedding and feature embedding are pre-processed auxiliary information input to the multimodal denoising network. Their function is to enable the network to recognize temporal and feature attributes. Temporal embedding converts time step information (the observation date of the optical data) into a vector form that the network can process, allowing the network to clearly identify which time step the currently processed data corresponds to, providing basic attribute annotations for subsequent learning of temporal associations. Feature embedding converts spectral band attributes (such as blue, green, and red bands in optical data) into vector form, allowing the network to clearly identify which spectral band the currently processed data corresponds to, providing basic attribute annotations for subsequent learning of feature associations. The observation mask is a binary label indicating whether an observation is valid or missing; it is a binary vector or matrix composed of 0s and 1s. Its core function is to distinguish between valid and missing observation sequences in the optical data, rather than the observation values ​​themselves. Guided by temporal embedding, feature embedding, and observation masking, multimodal denoising networks can learn temporal and physical features more fully and effectively, and effectively predict noise and reconstruct missing observation sequences.

[0050] In this embodiment of the invention, a multimodal denoising network is used to extract temporal and physical features within the optical time series, as well as temporal and physical features of the interaction between the optical and radar time series. The multimodal denoising network is trained using the extracted temporal and physical features and an auxiliary feature set. The training process involves using the multimodal denoising network to remove noise step by step according to the number of denoising iterations and the noise intensity corresponding to each iteration step to restore the original spectral band observation values ​​in the missing observation sequence. The loss between reducing (minimizing) the noise added during the forward diffusion process and the noise predicted by the multimodal denoising network is taken as the training objective of the multimodal denoising network, thereby optimizing the model parameters of the multimodal denoising network.

[0051] In this embodiment of the invention, the multimodal denoising network is built based on the Transformer network structure and adopts a multi-layer residual structure. Each residual layer includes one or more intra-module interaction modules and inter-module interaction modules. The intra-module interaction modules and the inter-module interaction modules include dimensionality decomposition attention heads.

[0052] In this embodiment of the invention, through the intra-module interaction module in the multimodal denoising network, for the same optical time series, the temporal features of the changes of the observation values ​​of the same spectral band with time step, as well as the physical features of the relationship between the observation values ​​of multiple spectral bands at a single time step, are extracted.

[0053] In this embodiment of the invention, through the inter-module interaction module in the multimodal denoising network, the temporal and physical features of the optical time series extracted by the intra-module interaction module are used as queries, and the radar time series is used as the key and value. By matching the corresponding key and value through the query, the temporal features of the temporal correlation between the optical time series and the radar time series, as well as the physical features of the physical correlation between the spectral band observation values ​​of the optical time series at each time step and the backscattering coefficient observation values ​​of the radar time series, are extracted.

[0054] In practical implementation, the intra-module interaction module processes optical time series data. Its input source is the optical time series. The intra-module interaction module learns the internal relationships of the optical time series through the temporal self-attention layer and the feature self-attention layer. The temporal self-attention layer independently processes each optical single variable (single spectral band) time series (such as the time series of optical blue light, green light, etc.), and the feature self-attention layer independently processes each time step of the optical time series (such as multispectral band data of a certain date). These processing objects all come from the preprocessed optical satellite time series, including valid observation sequences and noisy missing observation sequences that serve as reconstruction targets, directly reflecting the input of the optical time series.

[0055] In practice, the interactive module within the module first performs time self-attention for optical time series, and then performs feature self-attention. This follows a progressive logic of first modeling the time series dynamics of a single spectral band, and then integrating the collaborative relationships of multiple bands within the same time step, in order to gradually strengthen the inherent consistency of the optical data itself.

[0056] In practical implementation, the core function of performing temporal self-attention first is to decompose the optical time series along the feature dimension, independently modeling the dynamic changes of each single spectral band (such as the near-infrared band) over time (such as the upward trend of near-infrared reflectance during the vegetation growth cycle). This first captures the temporal dependencies of a single spectral band, laying the foundation for the temporal dimension features of the optical data and avoiding the loss of key temporal variation patterns of single bands during subsequent multi-band integration. The core function of performing feature self-attention second is to decompose the data along the time dimension after acquiring the single-band temporal features through temporal self-attention. For each time step (such as a certain observation date), it models the collaborative relationships between multiple bands (such as the NDVI ratio constraint between the red band and the near-infrared band). Based on the already stable single-band temporal features, it further strengthens the spectral consistency of multiple bands within the same time step, ensuring that the optical data, while having a reasonable temporal dynamic, also meets the physical logic of the spectral dimension, such as the inter-band correlation patterns of vegetation spectra.

[0057] In practical implementation, the inter-module interaction module integrates optical and radar time series, with its input explicitly including both modal time series. The inter-module interaction module employs a time-cross attention layer and a feature-cross attention layer. The time-cross attention layer uses feature codes from the optical time series as queries and feature codes from the radar time series as keys and values. Similarly, the feature-cross attention layer uses feature codes from the optical time series as queries and feature codes from the radar time series as keys and values. Through this "optical-SAR" query-key-value interaction, the fusion of the two modal data is achieved.

[0058] In practical implementation, the intermodal interaction module first models the temporal correlation between single-spectral bands and single-polarization SAR by splitting along the feature dimension through temporal cross-attention, such as the synchronous increase of near-infrared reflectance and VH-polarized backscattering during the corn jointing stage. Then, it models the physical correlation between multispectral bands and multi-polarization SAR within the same time step by splitting along the time dimension through feature cross-attention, such as the matching of vegetation color reflected by optical RGB bands and canopy roughness reflected by SAR. The approach of first using temporal cross-attention and then feature cross-attention follows a progressive logic of first establishing cross-modal temporal correlations and then deepening the matching of physical features within the same time step, achieving a precise and reasonable fusion of optical and SAR modes.

[0059] In practical implementation, the core function of performing temporal cross-attention first is to decompose along the feature dimension, modeling the temporal correlation between a single spectral band of optical (such as the near-infrared band) and a single polarization of SAR (such as VH polarization). This establishes cross-modal temporal consistency at the single feature and single polarization level, ensuring that the two modes match in their temporal trends, thus building a temporal cross-modal bridge for subsequent multi-feature fusion. The core function of performing feature cross-attention second is to decompose along the time dimension after establishing the cross-modal temporal correlation through temporal cross-attention. For each time step, it models the physical correlation between optical multispectral bands and SAR multipolarization. For example, the vegetation color reflected by the optical RGB band matches the canopy roughness reflected by the SAR. Based on the stable cross-modal temporal relationship, the physical features of multispectral bands and multipolar SAR are further fused within the same time step, ensuring that cross-modal fusion is not only temporally synchronized but also conforms to the physical properties of surface targets.

[0060] Existing multimodal fusion methods rely on spatiotemporal separation and cross-attention. They first match the local structures of optics and SAR in the spatial dimension, and then associate the dynamics of different frames in the temporal dimension. Essentially, they are modal fusion dominated by spatial similarity. This mechanism has two limitations. On the one hand, it does not decouple intramodal dependencies (such as correlations between spectral bands and between SAR polarization channels) and intermodal dependencies (such as correlations between near-infrared spectral bands and SAR VH polarization). On the other hand, it does not distinguish between dynamic correlations in the temporal dimension and physical correlations in the feature dimension. Due to these two reasons, SAR noise can easily contaminate optical spectral band features, or optical spectral band features can mask SAR structural information.

[0061] Existing cross-attention schemes directly mix intra-modal and inter-modal information. In contrast, the solution provided in this invention incorporates a dimensionality decomposition attention mechanism, achieving four-dimensional decoupling (intra-modal / inter-modal, temporal / physical features) for multimodal fusion through a two-stage interaction. Furthermore, it first learns intra-modal features and then dynamically associates inter-modal features. This decouples the independent and interactive temporal and physical features of optical and radar time series, preventing mutual interference between radar and optical image data. This allows the model to effectively utilize the temporal and physical features of radar image data to reconstruct optical image data, improving the accuracy of both physical and temporal features in the reconstructed optical image.

[0062] In this embodiment of the invention, noise is added to the mask part according to the different denoising iteration steps and the corresponding noise iteration intensity change table selected multiple times to obtain multiple missing observation sequences. Multiple corresponding reconstructed sequences are obtained by denoising and restoration. The mean of multiple reconstructed sequences is used as the target expected value for reconstructing the missing observation sequences.

[0063] In practice, multiple reconstruction sequences can be output through multiple samplings. The mean of the multiple reconstruction sequences can be calculated as the expected value of the reconstruction target and output as the final reconstruction result. The standard deviation and confidence interval of the reconstruction sequence distribution can be calculated to quantify the uncertainty of the reconstruction result, and 50% and 90% confidence intervals can be provided to provide reliability indicators for risk assessment and decision-making in downstream applications.

[0064] In practical implementation, multiple sampling refers to sampling the inverse denoising process of the multimodal denoising network. The inverse denoising process starts with random noise following a normal distribution. Based on the effective observation sequence, radar time series, and linear interpolation auxiliary information, it iteratively denoises through noise prediction to recover and reconstruct the target (missing observation sequence). However, due to the randomness in the inverse denoising process (such as differences in the initial random noise and random sampling of the Gaussian distribution during iteration), each complete iteration from random noise to the final reconstructed target will generate a different reconstruction result. Multiple sampling means repeating this inverse denoising process to obtain multiple independent reconstruction samples, based on which the mean, standard deviation, and confidence interval can be calculated.

[0065] In this embodiment of the invention, the actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set are input into a trained multimodal denoising network to obtain the reconstructed actual optical time series (spectral band time series of pixel positions). The reconstructed optical image data is obtained by reshaping (recombining the time series of multiple pixel positions) using the reconstructed actual optical time series at multiple image pixel positions.

[0066] In practice, the incomplete optical time series to be reconstructed is processed according to the same preprocessing process as the training data to ensure that the input data format is consistent with that during training. The SAR time series corresponding to the spatiotemporal location is extracted as conditional information. After undergoing the same preprocessing process as the training data, it is spatiotemporally aligned with the optical data to provide cross-modal auxiliary information for the conditional diffusion model.

[0067] In practical implementation, the system outputs complete and highly spectrally faithful optical image data along with its uncertainty range. Specifically, this includes: the output reconstruction results possess high spectral fidelity, with the reconstructed spectral reflectance values ​​maintaining a high degree of consistency with the actual observed values ​​in spectral characteristics, avoiding the spectral distortion problems that may arise with traditional methods; the reconstruction results maintain temporal continuity, accurately capturing key time points of vegetation phenological changes, such as the beginning of the green leaf period, peak period, and senescence period; the system outputs the uncertainty range of the time series for each reconstructed pixel, including the standard deviation, 50% confidence interval, and 90% confidence interval, providing users with a quantitative assessment of the reliability of the reconstruction results; furthermore, the system outputs... The reconstructed results are applicable to downstream remote sensing applications such as vegetation dynamics monitoring and land surface change analysis. Specifically, in vegetation dynamics monitoring, the reconstructed complete time series can support the continuous calculation of vegetation indices (such as NDVI and EVI), providing a reliable data foundation for crop growth monitoring and forest health assessment. In land surface change analysis, the high-quality optical image data reconstruction results can support remote sensing applications such as land use change detection, ecosystem monitoring, and disaster impact assessment. In agricultural applications, the reconstructed optical time series can support precision agriculture applications such as crop type identification, growth stage monitoring, and yield prediction, and is particularly suitable for agricultural areas prone to fog and cloud cover.

[0068] This invention also provides an optical image reconstruction device based on a conditional diffusion model, comprising: a preprocessing unit, a time series extraction unit, a training unit, and a reconstruction unit, wherein: the preprocessing unit is used to acquire optical image data through optical satellites and corresponding radar image data through synthetic aperture radar, and to align the radar image data with the optical image data in time steps; the time series extraction unit is used to extract spectral band observations of the optical image data at the image pixel positions and backscattering coefficient observations of the radar image data at the image pixel positions at multiple time steps, with pixels as independent processing objects, to obtain optical time series and radar time series; the training unit is used to input the optical time series and radar time series as training data into the conditional diffusion model; the conditional diffusion model includes a forward diffusion process and a multimodal denoising network, wherein the training process includes: masking the elements in the optical time series through the forward diffusion process to define the missing parts of the simulated observations, adding noise to the masked parts according to the selected number of denoising iteration steps and the corresponding noise iteration intensity change table, and taking the noisy part of the sequence as the missing observation sequence; and using the multimodal denoising network... The network extracts temporal and physical features from the optical time series, and extracts temporal and physical features from the interaction between the optical and radar time series. A multimodal denoising network is trained using the extracted temporal and physical features, along with an auxiliary feature set. The training process involves using the multimodal denoising network to remove noise step-by-step according to the number of denoising iterations and the noise intensity corresponding to each iteration step, restoring the original spectral band observations in the missing observation sequence. The training objective of the multimodal denoising network is to reduce the loss between the noise added during the forward diffusion process and the noise predicted by the multimodal denoising network. The auxiliary features... The feature set includes an optical auxiliary sequence, a denoising iteration step count, and a noise iteration intensity variation table. The optical auxiliary sequence is a sequence of spectral band prediction values ​​generated in the time period corresponding to the missing observation sequence using an effective observation sequence without a mask in the optical time series. The reconstruction unit is used to input the actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set into the trained multimodal denoising network to obtain the reconstructed actual optical time series. The reconstructed optical image data is obtained by reshaping the actual optical time series at multiple image pixel positions.

[0069] In specific implementations, the optical image reconstruction apparatus based on the conditional diffusion model provided by the present invention may refer to the optical image reconstruction method based on the conditional diffusion model provided by the present invention for the methods, steps or functions executed by the execution unit.

Claims

1. An optical image reconstruction method based on a conditional diffusion model, characterized in that, include: Optical image data is acquired through optical satellites, and corresponding radar image data is acquired through synthetic aperture radar. The radar image data is then aligned with the optical image data in time step. At multiple time steps, pixels are treated as independent processing objects. Spectral band observations of optical image data at the image pixel locations and backscattering coefficient observations of radar image data at the image pixel locations are extracted to obtain optical time series and radar time series. Optical time series and radar time series are used as training data input into the conditional diffusion model; The conditional diffusion model includes a forward diffusion process and a multimodal denoising network, wherein the training process includes: A forward diffusion process is used to mask elements in the optical time series to define missing parts of the simulated observations. Noise is added to the masked parts according to the selected denoising iteration steps and the corresponding noise iteration intensity change table. The noisy parts of the sequence are then considered as missing observation sequences. A multimodal denoising network is used to extract temporal and physical features within the optical time series, as well as temporal and physical features of the interaction between the optical and radar time series. The multimodal denoising network is trained using these two types of temporal and physical features, along with auxiliary feature sets. The training process involves using the multimodal denoising network to perform denoising... The original spectral band observations in the missing observation sequence are obtained by iterating step by step, using the number of iterations and the noise intensity corresponding to each iteration. The training objective of the multimodal denoising network is to reduce the loss between the noise added by the forward diffusion process and the noise predicted by the multimodal denoising network. The auxiliary feature set includes an optical auxiliary sequence, the number of denoising iterations, and a table of noise iteration intensity changes. The optical auxiliary sequence is a sequence of spectral band predictions generated using effective observation sequences without masks in the optical time series, and the sequence of spectral band predictions is in the same time period as the missing observation sequence. The actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set are input into the trained multimodal denoising network to obtain the reconstructed actual optical time series. The reconstructed optical image data is obtained by merging the reconstructed actual optical time series at multiple image pixel locations.

2. The optical image reconstruction method based on the conditional diffusion model according to claim 1, characterized in that, The alignment of radar image data with optical image data at time steps includes: Temporal resampling and interpolation are performed on the radar image data to align the radar image data with the optical image data in time steps.

3. The optical image reconstruction method based on the conditional diffusion model according to claim 1, characterized in that, The masking of elements in the optical time series includes: Masking can be performed by selecting observations in the same spectral band across all time steps, or / and by selecting observations in one or more spectral bands at the same time step.

4. The optical image reconstruction method based on the conditional diffusion model according to claim 1, characterized in that, The addition of noise to the mask portion according to the selected denoising iteration step number and the corresponding noise iteration intensity change table includes: For the optical time series of the mask portion, noise is gradually added at each iteration step according to the selected number of denoising iteration steps. The noise intensity of each iteration step is determined according to the noise iteration intensity change table. The added noise intensity is positively correlated with the number of iteration steps.

5. The optical image reconstruction method based on the conditional diffusion model according to claim 4, characterized in that, The extraction of temporal and physical features within the optical time series includes: Through the intra-module interaction module in the multimodal denoising network, for the same optical time series, the temporal characteristics of the changes of the observation values ​​of the same spectral band with time step, as well as the physical characteristics of the relationship between the observation values ​​of multiple spectral bands at a single time step, are extracted.

6. The optical image reconstruction method based on the conditional diffusion model according to claim 5, characterized in that, The extraction of temporal and physical features of the interaction between optical and radar time series includes: Through the inter-module interaction module in the multimodal denoising network, the temporal and physical features of the optical time series extracted by the intra-module interaction module are used as queries, and the radar time series is used as the key and value. By matching the corresponding key and value through the query, the temporal features of the temporal correlation between the optical time series and the radar time series, as well as the physical features of the physical correlation between the spectral band observations of the optical time series and the backscattering coefficient observations of the radar time series at each time step, are extracted.

7. The optical image reconstruction method based on the conditional diffusion model according to claim 6, characterized in that, The multimodal denoising network is built on the Transformer network structure and adopts a multi-layer residual structure. Each residual layer includes one or more intra-module interaction modules and inter-module interaction modules. The intra-module interaction modules and the inter-module interaction modules include multi-head self-attention for dimension splitting.

8. The optical image reconstruction method based on the conditional diffusion model according to claim 4, characterized in that, The process of generating the optically assisted sequence includes: If the time step corresponding to the mask portion is within the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask portion is generated by interpolation based on the effective observation sequence. If the time step corresponding to the mask portion is outside the effective observation time period of the effective observation sequence, the spectral band prediction value of the time step of the mask portion is generated by extrapolation of the first and last observation values ​​based on the effective observation sequence.

9. The optical image reconstruction method based on the conditional diffusion model according to claim 4, characterized in that, The training process of the multimodal denoising network also includes: Noise is added to the mask part according to the different denoising iteration steps and the corresponding noise iteration intensity change table selected multiple times to obtain multiple missing observation sequences. Multiple corresponding reconstructed sequences are obtained by denoising and restoration. The mean of multiple reconstructed sequences is used as the target expected value for reconstructing the missing observation sequence.

10. An optical image reconstruction device based on a conditional diffusion model, characterized in that, include: The unit comprises a preprocessing unit, a time series extraction unit, a training unit, and a reconstruction unit, wherein: The preprocessing unit is used to acquire optical image data through optical satellites, acquire corresponding radar image data through synthetic aperture radar, and align the radar image data with the optical image data in time steps. The time series extraction unit is used to extract the spectral band observation values ​​of optical image data at the image pixel positions and the backscattering coefficient observation values ​​of radar image data at the image pixel positions at multiple time steps, taking pixels as independent processing objects, to obtain optical time series and radar time series. The training unit is used to input optical time series and radar time series as training data into the conditional diffusion model; the conditional diffusion model includes a forward diffusion process and a multimodal denoising network, wherein the training process includes: A forward diffusion process is used to mask elements in the optical time series to define missing parts of the simulated observations. Noise is added to the masked parts according to the selected denoising iteration steps and the corresponding noise iteration intensity change table. The noisy parts of the sequence are then considered as missing observation sequences. A multimodal denoising network is used to extract temporal and physical features within the optical time series, as well as temporal and physical features of the interaction between the optical and radar time series. The multimodal denoising network is trained using these two types of temporal and physical features, along with auxiliary feature sets. The training process involves using the multimodal denoising network to perform denoising... The original spectral band observations in the missing observation sequence are obtained by iterating step by step, using the number of iterations and the noise intensity corresponding to each iteration. The training objective of the multimodal denoising network is to reduce the loss between the noise added by the forward diffusion process and the noise predicted by the multimodal denoising network. The auxiliary feature set includes an optical auxiliary sequence, the number of denoising iterations, and a table of noise iteration intensity changes. The optical auxiliary sequence is a sequence of spectral band predictions generated using effective observation sequences without masks in the optical time series, and the sequence of spectral band predictions is in the same time period as the missing observation sequence. The reconstruction unit is used to input the actual optical time series to be reconstructed, the corresponding actual radar time series, and the corresponding auxiliary feature set into a trained multimodal denoising network to obtain the reconstructed actual optical time series. The reconstructed optical image data is obtained by merging the reconstructed actual optical time series at multiple image pixel positions.

Images