Method and system for obtaining complete sequence of ground observation evapotranspiration based on multi-source data and machine learning

Abstract

The present application relates to a kind of based on multi-source data and machine learning ground observation evapotranspiration complete sequence acquisition method and system.The steps of the method include: preparing high-quality evapotranspiration data and time-continuous reference variable;Training machine learning model, construct the nonlinear relationship of evapotranspiration and reference variable, and evaluate the performance of machine learning model;Using the machine learning model trained completely to interpolate the evapotranspiration of missing time, produce time-continuous evapotranspiration data.The present application effectively solves the problem of how to obtain more real day-scale data, can provide high-quality continuous true value for ground verification;At the same time, it solves the problem of how to obtain long-time complete sequence data, can provide powerful data support for climate change analysis.

Description

Technical Field

[0001] This invention relates to the field of ground flux observation data processing technology, specifically to a method and system for obtaining complete sequences of ground-based evapotranspiration based on multi-source data and machine learning. Background Technology

[0002] Evapotranspiration, as a key link in the water cycle and energy balance, directly reflects the interaction and feedback between hydrological processes on the land surface and the atmosphere, and is of great significance for water resource management and watershed hydrological cycle research. In recent years, researchers have developed an increasing number of remote sensing evapotranspiration models and products. These models and products must be verified using ground-based measured data before they can be used for eco-hydrological process analysis and global climate change monitoring. Eddy covariance technology, as an effective means of obtaining true values ​​of ground fluxes, has widely applied its latent heat flux (the energy form of evapotranspiration) to related work on verifying the accuracy of evapotranspiration remote sensing. However, after a series of data quality control processes, a large amount of low-quality eddy covariance data is unusable. Simultaneously, due to various factors such as power supply and sensor failures, system maintenance and calibration, and unfavorable micrometeorological conditions, a large amount of ground-based eddy covariance data is missing. This leads to: 1) missing hourly-scale data, resulting in inaccurate daily-scale evapotranspiration acquisition and affecting the reliability of verification data; 2) missing daily-scale data, resulting in incomplete long-term observation data and affecting data analysis based on observations from actual ground stations. Therefore, it is necessary to perform reasonable interpolation of the missing data to obtain the complete sequence of ground-observed evapotranspiration.

[0003] Among existing data imputation techniques, traditional techniques and machine learning techniques are the two main types. Traditional techniques (including lookup tables, marginal distribution sampling, mean diurnal variation, and nonlinear regression) have been around for a long time and are relatively mature. They are effective for imputing small amounts of missing data, but they cannot handle long intervals of missing data (7 consecutive days or more), which limits their application. Machine learning algorithms have received widespread attention due to their excellent time series modeling capabilities, and they often outperform traditional methods in flux imputation. The core of this technique is to imput missing evapotranspiration by establishing a relationship with reference variables (such as light, temperature, humidity, and pressure). However, machine learning techniques have significant problems in obtaining complete evapotranspiration sequences. These problems mainly manifest in the following ways: existing techniques typically use ground-based meteorological observations to obtain reference variables, but these reference variable datasets, like eddy current observations, are subject to external interference, resulting in significant data gaps. We call the problem of missing data in both eddy current observations and reference data the "double discontinuity problem," which prevents us from obtaining complete observation sequences. Therefore, developing a robust method for obtaining complete evapotranspiration sequences from ground observations is crucial. Summary of the Invention

[0004] To overcome the aforementioned problems, this invention provides a method and system for acquiring a complete ground-based observational evapotranspiration sequence based on multi-source data and machine learning. This method uses high-resolution atmospheric driving data and satellite remote sensing data as reference variables to ensure temporal continuity; it uses machine learning algorithms to establish a nonlinear relationship between evapotranspiration and the reference variables, thereby constructing a robust machine learning model and evaluating its performance; it uses the constructed model to predict evapotranspiration at missing times, and merges this prediction with evapotranspiration data at times with those not missing to form a continuous, complete ground-based observational evapotranspiration sequence, thus completing this invention.

[0005] Specifically, the technical solution adopted by this invention to solve its technical problem is as follows:

[0006] A method for obtaining complete evapotranspiration sequences from ground observations based on multi-source data and machine learning includes the following steps:

[0007] Prepare high-quality evapotranspiration data and time-continuous reference variables;

[0008] Train a machine learning model, construct a non-linear relationship between evapotranspiration and a reference variable, and evaluate the performance of the machine learning model;

[0009] The missing evapotranspiration data is imputed using a well-trained machine learning model to generate temporally continuous evapotranspiration data.

[0010] Furthermore, the high-quality evapotranspiration data is obtained using the following method:

[0011] Evapotranspiration is obtained from eddy current observations (i.e. latent heat flux). Low-quality data is removed by quality control, and missing data is also filtered out to obtain high-quality evapotranspiration at the time when no data is missing, denoted as ET.

[0012] Furthermore, the reference variables include meteorological reference variables and vegetation reference variables, denoted as M respectively. RD and V RD The meteorological reference variables were selected according to the Penman-Monteith formula. High-resolution atmospheric driving data and remote sensing satellite data were chosen as the data sources for these reference variables. The Savitzky-Golay filtering method was used to smooth the time series of remote sensing satellite data and process it into a continuous format. The nearest neighbor pixel method was used to extract the complete point-scale time series of these reference variables from the above-mentioned meteorological and remote sensing gridded data (Savitzky-Golay filtered remote sensing and meteorological data), and its temporal resolution was processed to be the same as that of evapotranspiration observation. Here, grid refers to rasterized remote sensing and meteorological data, and point scale refers to a specific grid cell at the location of the station.

[0013] Furthermore, the evapotranspiration data at the time points without missing values ​​and the reference variable data are merged and randomly divided into two parts to form the training set and the test set.

[0014] Furthermore, for the machine learning model, a machine learning algorithm is selected and parameters are adjusted to ensure optimal computational results and efficiency. The model is trained using a training set. Finally, the nonlinear relationship established by the machine learning model is expressed by the following formula:

[0015] ET = f ML (M RD V RD )

[0016] Among them, f ML This represents a nonlinear equation established by a machine learning algorithm between evapotranspiration and a reference variable.

[0017] Furthermore, after training is completed, the performance of the constructed machine learning model is evaluated on the test set using commonly used model evaluation metrics.

[0018] Furthermore, using a fully trained machine learning model and a dataset of reference variables for missing evapotranspiration times, the evapotranspiration at the missing times is predicted and merged with ground observations at the non-missing times to form a continuous, complete sequence of ground-observed evapotranspiration.

[0019] A system for acquiring complete evapotranspiration sequences from ground observations based on multi-source data and machine learning, comprising:

[0020] The data preparation module is used to prepare high-quality evapotranspiration data and time-continuous reference variables;

[0021] The model building and evaluation module is used to train machine learning models, construct the nonlinear relationship between evapotranspiration and reference variables, and evaluate the performance of machine learning models.

[0022] The continuous evapotranspiration data generation module is used to interpolate evapotranspiration at missing moments using a well-trained machine learning model, generating time-continuous evapotranspiration data.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] (1) According to the method for obtaining a complete evapotranspiration sequence based on multi-source data and machine learning provided by the present invention, starting from the data source, atmospheric driving data is applied to the interpolation of flux data as one of the data sources of reference variables. It can easily and conveniently obtain time-continuous reference variables, effectively solve the "double discontinuity problem" of missing ground meteorological observations, get rid of the constraints of ground meteorological observations, and realize the acquisition of a complete evapotranspiration sequence at eddy observation stations that do not have supporting meteorological observations.

[0025] (2) According to the method for obtaining the complete sequence of ground observation evapotranspiration based on multi-source data and machine learning provided by the present invention, starting from the selection of variables, the reference variables are selected based on the classic evapotranspiration calculation formula, which has certain theoretical support; at the same time, compared with the technique that only uses meteorological data as reference variables, this method can easily introduce more vegetation reference variables such as NDVI and EVI, making the nonlinear relationship between evapotranspiration and reference variables more stable, and greatly improving the accuracy and stability of the results.

[0026] (3) According to the method for obtaining complete evapotranspiration sequence based on multi-source data and machine learning provided by the present invention, starting from the interpolation method, compared with the traditional technology, the interpolation method of this method is more efficient, more operable and more reliable.

[0027] Ultimately, this method effectively solves the problem of how to obtain more realistic daily-scale data, providing high-quality continuous true values ​​for ground validation; at the same time, it solves the problem of how to obtain long-term complete sequence data, providing strong data support for climate change analysis. Attached Figure Description

[0028] Figure 1 A flowchart illustrating the technical process of this invention is shown.

[0029] Figure 2 The graph shows the overall performance evaluation of the machine learning model constructed in this invention.

[0030] Figure 3 The figure shows a time series plot of the data imputation effect of the present invention for different data missing lengths. Detailed Implementation

[0031] The present invention will be further described in detail below through preferred embodiments and examples. Through these exemplary descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0033] It should be noted that since evapotranspiration observed from ground eddy currents is recorded in the form of latent heat flux, latent heat flux is used instead of evapotranspiration in the specific implementation and examples.

[0034] The method for obtaining complete evapotranspiration sequences from ground observations based on multi-source data and machine learning provided by this invention, such as... Figure 1As shown, the method includes the following steps:

[0035] Preferably, in step 1, high-quality latent heat flux data and time-continuous reference variables are prepared, including the following sub-steps;

[0036] Sub-step 1: The latent heat flux data is a quality-controlled dataset published by the site. The latent heat flux is measured by an eddy covariance instrument set up at the site. It is usually accompanied by a quality label level. Low-quality data is removed according to the quality label, and missing values ​​are then filtered out to obtain the latent heat flux at 30-minute intervals with no missing values, denoted as LE.

[0037] Sub-step 2 involves using meteorological and vegetation reference variables. High-resolution atmospheric driving data and remote sensing satellite data are selected as the data sources for the reference variables. The meteorological reference variables are selected based on the Penman-Monteith equation (i.e., the PM equation), including temperature T. a The variables used for data collection included wind speed (W), vapor pressure differential (VPD), radiation radiation (RAD), and atmospheric pressure (AP); vegetation reference variables included normalized difference vegetation index (NDVI) and vegetation enhancement index (EVI). The Savitzky-Golay filtering method was used to smooth the time series of remote sensing satellite data and render it continuous. The nearest neighbor pixel method was used to extract the complete time series of the observation station locations from the aforementioned remote sensing and meteorological data, and its temporal resolution was adjusted to be the same as that of evapotranspiration observations.

[0038] In a preferred embodiment, when selecting meteorological reference variables in sub-step 2, the variables that can be provided by atmospheric driving data should be flexibly selected, covering multiple elements such as light, temperature, wind, pressure, water, and vegetation.

[0039] Sub-step 3 involves merging the latent heat flux and reference variable data at time points without missing values ​​to form a training set and a test set. Theoretically, the ratio of the training set to the test set should be higher than 3:2.

[0040] Preferably, in step 2, the training set from step 1 is used to train a machine learning model, construct a nonlinear relationship between latent heat flux and reference variables, and evaluate the model performance.

[0041] In step 2, the machine learning algorithms include, but are not limited to, Artificial Neural Networks (ANN), Random Forests (RF), and Deep Neural Networks (DNN). The parameters of each machine learning algorithm are adjusted according to its own characteristics to ensure optimal computational results and efficiency. The model is trained using the training set from step 1. Finally, the nonlinear relationship established by the machine learning model is expressed by the following formula:

[0042] LE = f ML (T a,W,VPD,RAD,AP,NDVI or EVI,…)

[0043] Among them, f ML This represents a nonlinear equation established by a machine learning algorithm between latent heat flux and a reference variable.

[0044] After training, the performance of the constructed model is evaluated on the test set using model evaluation metrics, including but not limited to root mean square error (RMSE), normalized root mean square error (NRMSE), mean absolute error (MAE), mean bias (Bias), and coefficient of determination (R²). 2 wait.

[0045] Preferably, in step 3, the latent heat flux at the missing time is predicted using a fully trained model and a reference variable dataset of the missing latent heat flux time, and then merged with the ground observations at the non-missing time to form a continuous complete sequence of ground observation latent heat flux.

[0046] It should be understood that the contents not described in detail in the description of the method of the present invention are all common parameters or operating methods that are easy for those skilled in the art to think of, or conventional conditions in the art, and therefore detailed descriptions of them can be omitted.

[0047] The present invention will be further described below with reference to specific embodiments.

[0048] For three stations in the flux observation network of the Heihe River Basin in my country, a method for obtaining complete latent heat flux sequences based on multi-source data and machine learning was adopted to obtain complete latent heat flux sequences from ground observations. The method includes the following steps:

[0049] Step 1: Select three stations, Arou, Daman and Sidaoqiao, from the Heihe River Basin flux observation network to obtain latent heat flux observation data from 2013 to 2021 and the corresponding spatiotemporally continuous reference variables.

[0050] In sub-step 1, latent heat flux was measured by eddy covariance instruments installed at the three stations. The eddy covariance instruments were installed at heights of 3.5m, 4.5m, and 8m, respectively, with a data acquisition frequency of 10Hz. After a series of processing steps including WPL correction, coordinate axis rotation, and outlier removal, a quality rating was added, and the data was finally processed into a data product with a time resolution of 30 minutes. The product quality rating was divided into two levels: level three and level nine. For level three, data with a rating of 3 were removed; for level nine, data with quality ratings of 7-9 were removed, as they were considered unsuitable for quantitative research. Missing values ​​marked as -6999 were also removed. The final latent heat flux dataset with no missing values ​​was obtained, denoted as LE.

[0051] Sub-step 2 involves selecting high-resolution atmospheric driving data and MODIS NDVI data from the Heihe River Basin as data sources for reference variables. The meteorological reference variables are selected based on the Penman-Monteith formula and the variables available from the atmospheric driving data, including temperature T. a Wind speed (W), specific humidity (Q), downwave shortwave radiation (DSR), downwave longwave radiation (DLR), and atmospheric pressure (AP) were used as the reference variable for vegetation, with the normalized difference vegetation index (NDVI) as the reference variable. The Savitzky-Golay filter was used to smooth the MODIS NDVI data time series and render it continuous. The nearest neighbor method was used to extract the complete point-scale time series of these variables from the gridded data, and the time resolution was adjusted to 30 minutes to match the latent heat flux observations.

[0052] Sub-step 3 involves randomly dividing the reference variable and the latent heat flux data at time points without missing values ​​into two parts, forming a training set and a test set. The training set accounts for 70%, and the test set accounts for 30%.

[0053] In step 2, the Random Forest (RF) algorithm is used to model the machine learning model. Triple-fold cross-validation is used to automatically select the optimal combination of hyperparameters, ensuring the best computational results and efficiency. The model is then trained using the training set from step 1. Finally, the nonlinear relationship established by the machine learning model is expressed by the following formula:

[0054] LE = f RF (T a ,W,Q,DLR,DSR,AP,NDVI)

[0055] Among them, f RF This represents a nonlinear equation established by the random forest algorithm between latent heat flux and reference variables.

[0056] After training, the performance of the constructed model is evaluated on the test set using model evaluation metrics, including root mean square error (RMSE), normalized root mean square error (NRMSE), mean absolute error (MAE), mean bias (Bias), and coefficient of determination (R²). 2 The model's performance at the three sites is as follows: Figure 2 As shown, the data imputation effect of this technique for different data missing lengths is as follows: Figure 3 As shown. According to Figure 2 , Figure 3 It can be seen that this technology has a good ability to recover complete sequences on different underlying surfaces such as grassland, crops and shrubs and with different data missing lengths.

[0057] Step 3: Using the fully trained model and the complete reference variable dataset, predict the latent heat flux at the missing time points, and merge it with the ground observations at the non-missing time points to form a continuous complete sequence of ground observation latent heat fluxes.

[0058] Another embodiment of the present invention provides a system for obtaining complete sequences of ground-based evapotranspiration based on multi-source data and machine learning, comprising:

[0059] The data preparation module is used to prepare high-quality evapotranspiration data and time-continuous reference variables;

[0060] The model building and evaluation module is used to train machine learning models, construct the nonlinear relationship between evapotranspiration and reference variables, and evaluate the performance of machine learning models.

[0061] The continuous evapotranspiration data generation module is used to interpolate evapotranspiration at missing moments using a well-trained machine learning model, generating time-continuous evapotranspiration data.

[0062] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0063] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0064] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0065] The present invention has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present invention. Those skilled in the art should understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the present invention, and all such modifications and improvements fall within the scope of the present invention.

Claims

1. A method for obtaining a complete sequence of ground-based evapotranspiration based on multi-source data and machine learning, characterized in that, Includes the following steps: Prepare high-quality evapotranspiration data and time-continuous reference variables; Train a machine learning model, construct a non-linear relationship between evapotranspiration and a reference variable, and evaluate the performance of the machine learning model; The missing evapotranspiration data is imputed using a fully trained machine learning model to produce continuous evapotranspiration data over time. The nonlinear relationship established by the machine learning model is expressed by the following formula: AND=f ML (M RD ,V RD ) Among them, f ML M represents the nonlinear equation established by the machine learning algorithm between evapotranspiration and the reference variable. RD V represents the meteorological reference variable. RD This represents the vegetation reference variable.

2. The method according to claim 1, characterized in that, The evapotranspiration data was obtained from eddy current observations. Low-quality data was removed through quality control, and missing data was also filtered out to obtain high-quality evapotranspiration data at the time when no data was missing.

3. The method according to claim 1, characterized in that, The reference variables include meteorological reference variables and vegetation reference variables. The meteorological reference variables are selected according to the Penman-Monteith formula. High-resolution atmospheric driving and remote sensing satellite data are selected as the data sources for the reference variables. The Savitzky-Golay filtering method is used to smooth the time series of remote sensing satellite data and process it into a continuous form. The nearest neighbor pixel method is used to extract the complete time series of the reference variables at the point scale from the filtered remote sensing data and meteorological data, and its time resolution is processed to be the same as that of evapotranspiration observation.

4. The method according to claim 1, characterized in that, The evapotranspiration data at the time points without missing values ​​and the reference variable data are combined and randomly divided into two parts to form the training set and the test set.

5. The method according to claim 1, characterized in that, The machine learning model is trained using a training set, and a machine learning algorithm is selected and its parameters are adjusted to ensure optimal computational results and efficiency.

6. The method according to claim 1, characterized in that, After training, the performance of the machine learning model is evaluated on the test set using model evaluation metrics.

7. The method according to claim 1, characterized in that, The process of using a fully trained machine learning model to interpolate evapotranspiration at missing moments to produce continuous evapotranspiration data includes: using a fully trained machine learning model and a reference variable dataset of missing evapotranspiration moments to predict evapotranspiration at missing moments, and merging this prediction with ground observations at moments without missing moments to form a continuous, complete sequence of ground-observed evapotranspiration.

8. A system for acquiring complete sequences of ground-based evapotranspiration based on multi-source data and machine learning, characterized in that, include: The data preparation module is used to prepare high-quality evapotranspiration data and time-continuous reference variables; The model building and evaluation module is used to train machine learning models, construct the nonlinear relationship between evapotranspiration and reference variables, and evaluate the performance of machine learning models. The continuous evapotranspiration data generation module is used to interpolate the evapotranspiration at missing moments using a well-trained machine learning model, generating time-continuous evapotranspiration data. The nonlinear relationship established by the machine learning model is expressed by the following formula: AND=f ML (M RD ,V RD ) Among them, f ML M represents the nonlinear equation established by the machine learning algorithm between evapotranspiration and the reference variable. RD V represents the meteorological reference variable. RD This represents the vegetation reference variable.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images