Mask-aware temporal sequence generation adversarial network and vegetation index time series reconstruction method

Abstract

The present application relates to the technical field of remote sensing data processing and artificial intelligence, and discloses a mask perception time sequence generation adversarial network and a vegetation index time sequence reconstruction method, comprising: a mask perception time position coding module, which is used for modeling time intervals and observation reliability; a time-gated deep hybrid module, which is used for modeling long-time dependence; a multi-scale feature modeling module, comprising constructing a multi-scale feature extraction branch; extracting spatial features at different scales; and fusing multi-scale features. The present application solves the defects that the existing model cannot fill the irregular data gaps in the vegetation index remote sensing image and the long-time observation missing.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing data processing and artificial intelligence, specifically to a mask-sensing temporal generative adversarial network and a method for reconstructing vegetation index time series data, particularly suitable for forest phenological monitoring and analysis. This invention is applied in remote sensing data processing systems to process vegetation index time series data acquired by remote sensing sensors, thereby improving data quality and the accuracy of subsequent analysis. Background Technology

[0002] In recent years, remote sensing technology has developed rapidly, acquiring large-scale, multi-temporal surface observation data. Among these, MODIS (Moderate Resolution Imaging Spectroradiometer) provides long-term, continuous time-series data of vegetation indices. Enhanced vegetation indices (EVIs), due to their high sensitivity to changes in vegetation growth, are widely used in forest phenology monitoring, ecological environment assessment, and climate change analysis. However, in actual remote sensing observations, due to factors such as cloud pollution, atmospheric interference, snow cover, and changes in sensor observation conditions, irregular and prolonged data gaps are common in MODIS EVI time series. These gaps not only reduce the continuity of the time-series data but also introduce significant errors in subsequent phenological parameter extraction, affecting the reliability of the analysis results.

[0003] Existing technology 1:

[0004] In the field of vegetation index time series reconstruction, early studies mainly employed time series modeling methods, such as Savitzky-Golay filtering, spline interpolation, and harmonic analysis. These methods utilize the periodic characteristics of vegetation growth to smooth or fit the time series, thereby filling in missing data. Roerink et al. (2000) proposed a HANTS filtering method based on harmonic analysis to fill gaps. In addition, some methods utilize redundant information in the time dimension to achieve data reconstruction through weighted regression, low-rank matrix recovery, and tensor completion. Zeng et al. (2013) developed a multi-time weighted linear regression (WLR) method, which combines multi-time reference information and non-reference regularization algorithms to fill gaps. Chuet et al. (2021) designed an adaptive spatiotemporal tensor completeness method to fully utilize multi-dimensional spatiotemporal information to reconstruct long-term NDVI time series in cloud-prone areas.

[0005] Disadvantages of existing technology 1:

[0006] The above methods usually rely on the assumption of stationarity or periodicity of time series. When there are long-term continuous or irregular missing data, they are prone to over-smoothing or oscillation, making it difficult to accurately recover the key changes in vegetation growth, especially when the phenological transition phase is poor.

[0007] Existing technology 2:

[0008] To overcome the limitations of purely temporal methods, some studies have incorporated spatial information, reconstructing data through neighborhood pixel similarity or image patch matching. Examples include Neighborhood Similarity Pixel Interpolation (NSPI), image patch-based spatiotemporal reconstruction methods, and cloud removal models, which utilize spatial redundancy information to compensate for missing data. In recent years, with the development of deep learning technology, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs) have been applied to remote sensing data reconstruction tasks, improving reconstruction accuracy by learning complex spatiotemporal feature distributions. For instance, Pathak et al. (2016) combined an autoencoder with a GAN to generate and optimize missing textures. Chen et al. (2019) proposed a network composed of a CNN for content, texture, and spectrum generation to remove thick clouds from ZY-3 satellite images. Li et al. (2020) proposed a Convolutional-Mapping-Deconvolutional Network that integrates synthetic aperture radar (SAR) and low-resolution heterogeneous images to remove high-resolution images with thick clouds. Zhao et al. (2022) proposed a multi-model GAN ​​to counteract consistency loss and achieved a significant improvement in reconstruction accuracy.

[0009] Disadvantages of existing technology 2:

[0010] Existing time series reconstruction methods rely on stationarity or periodicity assumptions, which makes it difficult to accurately recover the dynamic changes of vegetation indices when there are long-term continuous or irregular missing data.

[0011] Existing methods do not differentiate and model the reliability of observation data, treating valid observations and low-quality observations the same, which can easily introduce noise and thus affect the accuracy of reconstruction results.

[0012] Existing methods are insufficient in modeling long-term dependencies, making it difficult to recover key phenological change information across long time intervals;

[0013] Existing spatial information-based methods rely on the assumption of regional homogeneity, which leads to unstable reconstruction results in areas with strong spatial heterogeneity, such as forests. Summary of the Invention

[0014] To address the shortcomings of existing technologies in handling irregular, long-term missing data, modeling observation reliability, and modeling long-term dependencies, this invention proposes a mask-aware temporal generative adversarial network and a vegetation index time series reconstruction method.

[0015] The technical solution provided by this invention is as follows:

[0016] The mask-aware temporal generative adversarial network includes: a mask-aware temporal location encoding module, a temporally gated deep mixing module, and a multi-scale feature modeling module.

[0017] (1) Mask-aware temporal location encoding module, used to model time intervals and observation reliability; calculate the time distance between each time step and the most recent valid observation; construct a confidence weight function based on time distance; fuse the confidence weight with the temporal location encoding; embed the fused encoding into the input features;

[0018] (2) Temporally gated deep mixing module, used to model long-term dependencies; the temporally gated deep mixing module includes a linear mapping layer, a one-dimensional convolutional layer and a gated activation function; the input features are decomposed into gated signals and content signals; the content signals are subjected to one-dimensional temporal convolution to extract temporal features; the gated signals are activated to generate gated weights and the temporal features are modulated using the gated weights to achieve information selection and fusion;

[0019] (3) Multi-scale feature modeling module, including constructing a multi-scale feature extraction branch; extracting spatial features at different scales; and fusing multi-scale features.

[0020] Furthermore, the mask-aware temporal position encoding module includes the following steps:

[0021] S1: Obtain MODIS Enhanced Vegetation Index (EVI) time series data and its quality control data;

[0022] S2: Construct an observation mask based on the quality control data, where valid observations are marked as 1 and missing or low-quality observations are marked as 0.

[0023] S3: Calculate the time interval between each time step in the time series and the most recent valid observation;

[0024] S4: Construct a time confidence weight function based on the time interval and weight the time position code;

[0025] S5: The weighted temporal location encoding is fused with the EVI input features to achieve joint modeling of observation reliability and temporal information. The designed new loss functions include: reconstruction loss based on observation mask, temporal consistency loss, and adversarial loss.

[0026] Furthermore, the time-gated deep mixing module includes the following steps:

[0027] S1: Decompose the input time features into a gated signal and a content signal through a linear mapping;

[0028] S2: Perform a one-dimensional convolution operation on the content signal along the time dimension to extract local temporal features;

[0029] S3: Perform nonlinear activation processing on the gating signal to obtain the gating weight;

[0030] S4: Use the gating weights to perform weighted fusion of content signals to achieve long-term dependency modeling.

[0031] Furthermore, the multi-scale feature modeling module includes:

[0032] The input data is encoded by a multi-scale feature extraction branch, and features at different scales are fused to improve the ability to express spatial heterogeneity.

[0033] The vegetation index time series reconstruction method uses the aforementioned network; the method includes the following steps: 1) data preprocessing and sample construction; 2) EVI time series reconstruction based on generative adversarial networks; 3) phenological parameter extraction based on the reconstruction results;

[0034] First, MODIS EVI time series data and its quality control data are acquired. Low-quality observations are masked to construct input data. Specifically, the quality control information is used to identify cloud contamination, snow cover, and anomalous observations. A binary observation mask is constructed, in which valid observations are marked as 1 and missing observations are marked as 0. The EVI data and the observation mask are used together as model input.

[0035] This invention obtains phenological parameters from time-series images of reconstructed vegetation indices after training.

[0036] The technical solution of the present invention has the following beneficial effects:

[0037] 1) This invention improves the reconstruction accuracy of vegetation index time series, and therefore has certain commercial value.

[0038] 2) This invention reconstructs missing data based on a generative model, so that the reconstruction results maintain numerical consistency while having good structural continuity, which helps to improve the reliability of subsequent phenological parameter extraction and can be well applied to industrial needs.

[0039] 3) This invention can be applied to the field of remote sensing vegetation monitoring, including forest phenology analysis, ecological environment assessment and climate change research, and has certain application value. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of the generative adversarial network structure described in this invention;

[0042] Figure 3 This is a schematic diagram of the mask sensing time position encoding module of the present invention;

[0043] Figure 4 This is a schematic diagram of the time-gated deep mixing module structure of the present invention;

[0044] Figure 5 This is a comparison chart of the EVI reconstruction effects of the method of the present invention and existing methods.

[0045] Figure 6 This is one of the schematic diagrams showing the phenological parameters extracted based on the reconstructed EVI time series;

[0046] Figure 7 The second illustration shows the results of phenological parameters extracted based on the reconstructed EVI time series.

[0047] Figure 8 The third illustration shows the results of phenological parameters extracted based on the reconstructed EVI time series;

[0048] Figure 9 The fourth illustration shows the results of phenological parameters extracted based on the reconstructed EVI time series;

[0049] Figure 10 The fifth illustration shows the results of phenological parameters extracted based on the reconstructed EVI time series;

[0050] Figure 11 This is the sixth illustration of the phenological parameter results extracted based on the reconstructed EVI time series. Detailed Implementation

[0051] The specific embodiments of the present invention are described in detail below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the following specific embodiments. Various modifications or equivalent substitutions can be made to the present invention without departing from the spirit and scope of the claims, and all such modifications and substitutions should fall within the scope of protection of the present invention.

[0052] This invention utilizes the existing deep learning framework TensorFlow and its corresponding programming libraries, primarily including NumPy, Pandas, and Tensor. TensorFlow mainly employs deep learning models, including linear and convolutional modules.

[0053] The specific implementation principle of the solution is as follows:

[0054] The vegetation index time series reconstruction method proposed in this invention transforms gap filling into an image generation task for vegetation index time series reconstruction. The overall process is as follows: Figure 1 As shown.

[0055] This method mainly includes the following three stages: 1) data preprocessing and sample construction; 2) EVI time series reconstruction based on generative adversarial networks; 3) phenological parameter extraction based on reconstruction results.

[0056] First, MODIS EVI time series data and its quality control data are acquired. Low-quality observations are masked to construct the input data. Specifically: the quality control information is used to identify cloud contamination, snow cover, and anomalous observations; a binary observation mask is constructed, where valid observations are marked as 1 and missing observations are marked as 0; the EVI data and the observation mask are used together as the model input.

[0057] The Mask-Aware Temporal Generative Adversarial Network (MTMGAN) of this invention has the following structure: Figure 2 As shown. This network mainly consists of a generator and a discriminator. Unlike existing GAN models that do not consider observation reliability, this invention introduces a time-interval-based weight encoding mechanism to achieve differentiated modeling of different observation qualities, specifically including:

[0058] (1) Mask-aware temporal position encoding module (MTPE), such as Figure 3 As shown, this invention proposes a mask-aware temporal location encoding module for modeling time intervals and observation reliability. The specific implementation is as follows: calculating the temporal distance between each time step and the most recent valid observation; constructing a confidence weight function based on the temporal distance; fusing the confidence weights with the temporal location encoding; and embedding the fused encoding into the input features. Through this method, the model can distinguish between reliable observations and interpolated data, improving time series modeling capabilities.

[0059] (2) Time-gated deep mixing module (TGDM), such as Figure 4 As shown, this invention proposes a time-gated deep mixing module for modeling long-term dependencies. The time-gated deep mixing module includes a linear mapping layer, a one-dimensional convolutional layer, and a gated activation function. Specifically, the input features are decomposed into a gate signal and a content signal; the content signal is subjected to a one-dimensional temporal convolution to extract temporal features; the gate signal is activated to generate gate weights, which are then used to modulate the temporal features, achieving information selection and fusion. This module can effectively model long-term sequence dependencies without introducing a highly complex attention mechanism.

[0060] (3) Multi-scale feature modeling module: To adapt to the spatial heterogeneity of forest areas, this invention introduces a multi-scale feature modeling structure. Specifically, it includes constructing a multi-scale feature extraction branch; extracting spatial features at different scales; and fusing multi-scale features to improve the model's adaptability to complex spatial structures.

[0061] In the embodiments, the method of the present invention was validated on the MODIS EVI dataset. Compared with existing methods such as SRGAN, the mean absolute error (MAE) was reduced by approximately 0.7%, and the structural similarity (SSIM) was improved by approximately 3.4%, verifying the effectiveness of the method in terms of time series reconstruction accuracy. The method of the present invention can be deployed on servers or embedded systems to achieve automated remote sensing data processing. The reconstruction results of the present invention are visualized as follows: Figure 5 As shown. The results of obtaining phenological parameters in this embodiment are as follows. Figures 6 to 11 As shown.

Claims

1. A mask-aware temporal generative adversarial network, characterized by: It includes a mask-aware temporal position encoding module, a temporally gated depth fusion module, and a multi-scale feature modeling module; (1) Mask-aware temporal location encoding module, used to model time intervals and observation reliability; calculate the time distance between each time step and the most recent valid observation; construct a confidence weight function based on time distance; fuse the confidence weight with the temporal location encoding; embed the fused encoding into the input features; (2) Temporally gated deep mixing module, used to model long-term dependencies; the temporally gated deep mixing module includes a linear mapping layer, a one-dimensional convolutional layer and a gated activation function; it decomposes the input features into a gated signal and a content signal; it performs one-dimensional temporal convolution on the content signal to extract temporal features; The gating signal is activated to generate gating weights. The gating weights are then used to modulate the time characteristics to achieve information selection and fusion. (3) Multi-scale feature modeling module, including the construction of a multi-scale feature extraction branch; Extracting spatial features at different scales; Multi-scale features are fused.

2. The mask-aware temporal generative adversarial network according to claim 1, characterized in that: The mask-sensing time-position encoding module, regarding the modeling time interval and observation reliability, includes the following steps: S1: Obtain MODIS Enhanced Vegetation Index (EVI) time series data and its quality control data; S2: Construct an observation mask based on the quality control data, where valid observations are marked as 1 and missing observations are marked as 0; S3: Calculate the time interval between each time step and the most recent valid observation; S4: Construct a time confidence weight function based on the time interval and weight the time position code; S5: The weighted temporal location code is fused with the EVI input features to achieve joint modeling of observation reliability and temporal information.

3. The mask-aware temporal generative adversarial network according to claim 1, characterized in that: The time-gated deep blending module, for modeling long-term dependencies, includes the following steps: S1: Decompose the input time features into a gated signal and a content signal through a linear mapping; S2: Perform a one-dimensional convolution operation on the content signal along the time dimension to extract local temporal features; S3: Perform nonlinear activation processing on the gating signal to obtain the gating weight; S4: Use the gating weights to perform weighted fusion of the content signals to obtain the time modeling results.

4. A method for reconstructing vegetation index time series, characterized in that, The method employs the mask-aware temporal generative adversarial network as described in any one of claims 1 to 3; the method includes the following steps: 1) data preprocessing and sample construction; 2) EVI time series reconstruction based on the generative adversarial network; 3) phenological parameter extraction based on the reconstruction results; First, MODIS EVI time series data and its quality control data are acquired. Low-quality observations are masked to construct input data. Specifically, the quality control information is used to identify cloud contamination, snow cover, and anomalous observations. A binary observation mask is constructed, in which valid observations are marked as 1 and missing observations are marked as 0. The EVI data and the observation mask are used together as model input.

Images