Multi-layer high-resolution soil moisture generation method based on ensemble learning

Abstract

The invention discloses a multi-layer high-resolution soil moisture generation method based on ensemble learning, which comprises the following steps: step 1, data collection and preprocessing: collecting multi-source remote sensing data of a research area and ground actual measurement soil moisture content site data of different depths, and respectively preprocessing the collected data; 2, data space-time reconstruction: discriminating the space-time continuity of the preprocessed multi-source remote sensing data, and performing data space-time reconstruction on space-time discontinuous data; step 3, constructing and training a multi-layer soil moisture downscaling model: constructing and training the multi-layer soil moisture downscaling model in the research area by using an ensemble learning algorithm; and step 4, generating a multi-layer high-resolution soil moisture result. According to the method, a multi-layer high-resolution soil moisture result can be obtained, the efficiency and precision of regional soil moisture monitoring are greatly improved, the method has wide business popularization and application value, and technical support is provided for precise irrigation of crops.

Description

Technical Field

[0001] The invention belongs to the technical field of soil moisture remote sensing, and in particular relates to a multi-layer high-resolution soil moisture generation method based on ensemble learning. Background Art

[0002] Soil moisture is one of the characteristic elements of regional water cycle, agricultural irrigation management and climate change. It also plays an important role in hydrology, meteorology, agriculture and other disciplines. Soil moisture (SM) is the main source of water absorbed by surface vegetation, which is crucial to the growth and development of crops. Therefore, accurate simulation and prediction of soil moisture is of great significance to crop yield and food security. With the development of remote sensing technology, soil moisture can be obtained from different active and passive microwave sensors, such as Advanced Microwave Scanning Radiometer-Earth Observing System (AMSR-E), Advanced Scatterometer (ASCAT), AMSR-2, SMOS (Soil Moisture and Ocean Salinity Mission), SMAP (Soil Moisture Active Passive Mission) and ESA-CCI (The European Space Agency's Climate Change Initiative). The temporal and spatial resolution of the above soil moisture remote sensing products is mostly 10 to 40 km, which cannot meet the accuracy requirements of existing regional agricultural drought monitoring.

[0003] In order to improve the spatiotemporal resolution of soil moisture data, some scholars have proposed several downscaling methods, mainly including polynomial regression fitting method, DISPATCH method and machine learning method. Among them, the polynomial regression fitting method is simple to model and does not require measured soil moisture data. At present, this method has been widely used in the downscaling research of remote sensing soil moisture products. Compared with the polynomial fitting method, the DISPATCH method has a better theoretical and physical basis because the scale change depends on partial derivatives, Taylor series expansion and projection technology. In addition, this method studies the soil evaporation process and is classified as a physical method. However, both the polynomial regression fitting method and the DISPATCH method rely more on the accuracy of the original soil moisture remote sensing product. When the original remote sensing product is not highly correlated with the ground measured soil moisture station value or has a certain deviation, the downscaling result is often poor.

[0004] In recent years, many studies have used machine learning methods to achieve soil moisture downscaling. However, a single machine learning algorithm may have the problem of overfitting, and ensemble learning is an integrated model that integrates multiple machine learning models. Through a certain fusion strategy, it can often obtain significantly better generalization performance than a single model. Since the ensemble learning model has higher accuracy and model robustness, it can not only achieve complementary advantages between models, but also reduce the dependence on the data required for training.

[0005] Most of the existing soil moisture downscaling methods use coarse-resolution passive microwave soil moisture products, vegetation cover and surface temperature information, and other surface parameters obtained from optical / thermal sensors to obtain high-resolution soil moisture results. Although the advantages of multi-source remote sensing products are combined, the generated high-resolution soil moisture results rely heavily on the accuracy of the original soil moisture remote sensing products, and rarely or only use soil moisture station data for result correction, resulting in low accuracy of the generated products. In addition, the existing technology only generates soil moisture results for the surface layer (0-10 cm). Studies have shown that under drought stress conditions, summer maize mainly absorbs soil water at 0-20, 20-40, and 40-60 cm during the jointing-tasseling period; mainly absorbs soil water at 0-20 cm during the tasseling-filling period; and mainly absorbs soil water at 0-20 cm during the filling-maturity period. In the vertical direction, the root growth of different soybean varieties can break through the soil layer of 100 cm, and the soil layer of 40-60 cm has the greatest correlation with soybean yield. Therefore, the soil moisture results generated by the existing technology cannot well reflect the moisture conditions in the deep soil.

[0006] Based on this, the present invention proposes a multi-layer high-resolution soil moisture generation method based on ensemble learning, which combines the advantages of multi-source remote sensing products and different types of data from measured soil moisture sites to generate multi-layer high-resolution soil moisture results, thereby solving the problem that the existing high-resolution soil moisture simulation results are of low accuracy and can only generate surface high-resolution soil moisture results. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-layer high-resolution soil moisture generation method based on ensemble learning to solve the above technical problems.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention discloses a multi-layer high-resolution soil moisture generation method based on ensemble learning, the method comprising the following steps:

[0010] Step 1, data collection and preprocessing: collect multi-source remote sensing data and ground-measured soil moisture site data of different depths in the study area for many years, and preprocess the collected multi-source remote sensing data and ground-measured soil moisture site data respectively to obtain preprocessed multi-source remote sensing data and ground-measured soil moisture site data;

[0011] The multi-source remote sensing data includes a soil moisture dataset, a vegetation index dataset, a surface temperature dataset, a precipitation dataset, a digital elevation dataset, a slope dataset and a soil texture dataset; the soil moisture dataset includes a SMAP SM dataset and an ERA5-LAND SM dataset of different layer depths; the vegetation index dataset includes a MODIS NDVI / EVI dataset and an ERA5-LAND LAI dataset; the surface temperature dataset includes a MODIS LST dataset and an ERA5-LAND LST dataset; the layer depth of the soil moisture dataset corresponds to the depth of the ground soil moisture station data;

[0012] Step 2, data spatiotemporal reconstruction: the spatiotemporal continuity of the preprocessed multi-source remote sensing data is judged, the spatiotemporal continuous data and the spatiotemporal discontinuous data are distinguished, and the spatiotemporal discontinuous data are reconstructed to obtain spatiotemporal continuous data; the spatiotemporal continuous data obtained by data reconstruction are merged with the distinguished spatiotemporal continuous data to obtain all spatiotemporal continuous multi-source remote sensing data;

[0013] Step 3, construct a multi-layer soil moisture downscaling model and train it: match the spatial longitude and latitude of all spatiotemporally continuous multi-source remote sensing data with the ground-measured soil moisture sites, obtain the remote sensing pixel values ​​of the points where the ground-measured soil moisture sites are located, use the matched remote sensing pixel values ​​as input data, and use the preprocessed ground-measured soil moisture site data as output data. The input data and output data constitute the model training data set. Use the integrated learning algorithm to construct a multi-layer soil moisture downscaling model in the study area, and use the model training data set to train the model until the model training is completed;

[0014] Step 4: Generate multi-layer high-resolution soil moisture results: All spatiotemporally continuous multi-source remote sensing data are used as input data and input into the trained multi-layer soil moisture downscaling model to generate multi-layer high-resolution soil moisture results, that is, generate multi-layer soil moisture results with a spatial resolution of 1 km, and verify the generated results with the measured soil moisture station values ​​on the spatiotemporal scale.

[0015] Furthermore, the step 1 of collecting multi-source remote sensing data and ground-measured soil moisture site data of the study area for many years specifically refers to collecting data of the study area for more than five years.

[0016] Furthermore, the collected multi-source remote sensing data and ground-measured soil moisture site data described in step 1 are preprocessed separately to obtain the preprocessed multi-source remote sensing data and ground-measured soil moisture site data: after projecting, resampling and cropping the multi-source remote sensing data, multi-source remote sensing data with consistent projection coordinate system, spatial range and resolution are obtained; after data quality control of the ground-measured soil moisture site data, a ground-measured soil moisture site data set with a complete time series and no abnormal values ​​is obtained.

[0017] Furthermore, the SMAP SM dataset in step 1 is a microwave dataset with a temporal resolution of 1 day and a spatial resolution of 9 km; the MODIS LST dataset and the MODIS NDVI / EVI dataset are both optical remote sensing datasets, the MODIS LST dataset has a temporal resolution of 1 day and a spatial resolution of 1 km, and the MODIS NDVI / EVI dataset has a temporal resolution of 16 days and a spatial resolution of 1 km; the ERA5-LAND SM dataset, the ERA5-LAND LST dataset and the ERA5-LAND LAI dataset are reanalysis datasets, all with a temporal resolution of 1 day and a spatial resolution of 9 km; the precipitation dataset has a temporal resolution of 1 day and a spatial resolution of 5 km; the digital elevation dataset, the slope dataset and the soil texture dataset are all static data with a spatial resolution of 1 km.

[0018] Furthermore, the spatiotemporal discontinuous data distinguished in step 2 include MODIS NDVI / EVI dataset and MODIS LST dataset;

[0019] The specific process of performing spatiotemporal reconstruction on the spatiotemporal discontinuous data to obtain spatiotemporal continuous data is as follows:

[0020] For the MODIS NDVI / EVI dataset, the Savitzky-Golay filtering method is used to interpolate the dataset with a time resolution of 16 days into a dataset with a time resolution of 1 day;

[0021] For the MODIS LST dataset, the multiple regression method is used to fill the missing data in the MODIS LST dataset to generate a spatiotemporally continuous surface temperature dataset in the study area, that is, a seamless daily surface temperature dataset with a spatial resolution of 1 km. The specific formula is:

[0022] LST value =a×LST ERA5_value +b×NDVI 1d_value +c×DEM value +d (1)

[0023] LSTnon_value =a×LST ERA5_non_value +b×NDVI 1d_non_value +c×DEM non_value +d (2)

[0024] Where: LST value and LST non_value They are the areas with values ​​and areas without values ​​in MODIS LST; ERA5_value and LST ERA5_non_value They are ERA5-LAND LST with and without values ​​in the corresponding areas of MODIS LST; NDVI 1d_value and NDVI 1d_non_value They are the 1-day NDVI data with and without values ​​in the corresponding area of ​​MODIS LST; DEM value and DEM non_value They are the digital elevation data with and without values ​​in the corresponding area of ​​MODIS LST; a, b, c, d are the parameters of the formula;

[0025] Finally, LST value and LST non_value Splicing to generate seamless daily land surface temperature dataset LST seamless .

[0026] Furthermore, the multi-layer soil moisture downscaling model in step 3 is:

[0027]

[0028] Where: f is the ensemble learning algorithm; SMAP SM For SMAP SM data; ERA5 SM ERA5-LAND SM data; ERA5 LAI ERA5-LAND LAI data; LST seamless For seamless daily surface temperature data; NDVI 1d 、EVI 1d They are the reconstructed NDVI and EVI data respectively; Precipitation is precipitation data; Soil texture is soil texture data; DEM is digital elevation data; Slope is slope data.

[0029] Furthermore, the specific process of using the model training data set to train the model until the model training is completed in step 3 is: the model training data set is divided into a training set and a test set in a ratio of 8:2, and the model is trained using the training set. During the model training process, the optuna and GridSearchCV methods are used to adjust the hyperparameters of the ensemble learning algorithm. After finding the optimal parameter combination of each hyperparameter in the training set, the optimal parameter combination is verified in the test set, and evaluated using evaluation indicators. When the evaluation indicator results tend to be stable and reach their respective optimal values, the model training is completed.

[0030] Furthermore, the evaluation indicators include mean absolute error MAE, determination coefficient R 2 The smaller the mean absolute error MAE and the root mean square error RMSE, the greater the determination coefficient R 2 The larger the value, the smaller the deviation between the predicted result and the measured result, that is, the better the predicted result of the optimal parameter combination, the mean absolute error MAE and the determination coefficient R 2 The calculation formulas for the root mean square error and RMSE are:

[0031]

[0032] In the above formula: is the predicted value of soil moisture; i is the true value; is the average value; i is the index value of the data sequence; n is the total length of the data sequence.

[0033] Furthermore, the generation of multiple layers of high-resolution soil moisture results in step 4 includes high-resolution soil moisture results at depths of 10, 20, and 40 cm;

[0034] Using SMAP SM data 0-5cm and ERA5-LAND SM data 0-7cm as model input data and the ground soil moisture measured at a depth of 10cm as model output data, high-resolution soil moisture results at a depth of 10cm are generated; using SMAP SM data 0-100cm and ERA5-LAND SM data 7-28cm as model input data and the ground soil moisture measured at a depth of 20cm are used as model output data, high-resolution soil moisture results at a depth of 20cm are generated; using SMAP SM data 0-100cm and ERA5-LAND SM data 28-100cm as model input data and the ground soil moisture measured at a depth of 40cm are used as model output data, high-resolution soil moisture results at a depth of 40cm are generated.

[0035] The beneficial effect of the present invention is that the method of the present invention uses an integrated learning algorithm to construct a multi-layer soil moisture downscaling model and train the model, and finally generates multi-layer high-resolution soil moisture results based on the trained model. The method of the present invention solves the problems of inconsistent resolution and large data quality differences in multi-source heterogeneous data; by fusing multi-source remote sensing data and ground-measured soil moisture site data, combined with an integrated learning algorithm, multi-layer high-resolution soil moisture results can be obtained. Compared with traditional downscaling methods, the efficiency and accuracy of regional soil moisture monitoring are greatly improved, and it has broad business promotion and application value, which is conducive to improving the accuracy of regional drought monitoring and early warning, and provides technical support for precision irrigation of crops.

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a principle flow chart of the multi-layer high-resolution soil moisture generation method of the present invention;

[0038] Figure 2 It is an elevation map of the target area of ​​the present invention;

[0039] Figure 3 This is a distribution map of soil moisture sites in the target area of ​​the present invention;

[0040] Figure 4 A schematic diagram of the MODIS NDVI / EVI reconstruction results provided by the present invention;

[0041] Figure 5 A schematic diagram of the MODIS LST vacant area filling results provided by the present invention;

[0042] Figure 6 It is a schematic diagram of the parameter optimization results of the multi-layer soil moisture downscaling model of the present invention;

[0043] Figure 7 It is a schematic diagram of the time scale verification result of the multi-layer soil moisture downscaling model of the present invention;

[0044] Figure 8 Schematic diagram of the verification results of the downscaling results generated by the traditional downscaling method and the measured sites;

[0045] Fig. 9 It is a schematic diagram of the comparison and verification results of the multi-layer soil moisture downscaling results of the present invention and the actual measurement site;

[0046] Fig.10 This is a before and after comparison diagram of soil moisture in the target area provided by the present invention. DETAILED DESCRIPTION

[0047] The present invention discloses a multi-layer high-resolution soil moisture generation method based on ensemble learning, such as Figure 1 As shown, the method comprises the following steps:

[0048] Step 1. Data collection and preprocessing: Collect and organize multi-source remote sensing data of the study area for many years (generally more than 5 years) and ground-measured soil moisture station data at different depths. The multi-source remote sensing data include soil moisture dataset (SoilMoisture, including SMAP SM datasets and ERA5-LAND SM datasets at different depths), vegetation index dataset (Vegetation, including MODIS NDVI / EVI datasets and ERA5-LAND LAI datasets), land surface temperature dataset (LandSurface Temperature, including MODIS LST datasets and ERA5-LAND LST datasets), precipitation dataset (Precipitation), digital elevation dataset (DEM), slope dataset (Slope) and soil texture dataset (Soiltexture).

[0049] The layer depth of the soil moisture data set corresponds to the depth of the ground measured soil moisture site data. Generally, the depth of the remote sensing soil moisture product close to the layer depth of the ground measured soil moisture site is selected. In the present invention, the SMAP SM data 0-5cm and the ERA5-LAND SM data 0-7cm correspond to the ground measured soil moisture site data at a depth of 10cm; the SMAP SM data 0-100cm and the ERA5-LAND SM data 7-28cm correspond to the ground measured soil moisture site data at a depth of 20cm; the SMAP SM data 0-100cm and the ERA5-LAND SM data 28-100cm correspond to the ground measured soil moisture site data at a depth of 40cm.

[0050] The collected multi-source remote sensing data and ground-measured soil moisture station data are preprocessed separately. The collected multi-source remote sensing data are generally in NC or TIF format. After preprocessing such as projection, resampling and clipping, multi-source remote sensing data with consistent projection coordinate system, spatial range and resolution are obtained. The projection format is WGS84 projection. After data quality control of the ground-measured soil moisture station data, a ground-measured soil moisture station data set with complete time series and no abnormal values ​​is obtained.

[0051] Among them, the SMAP SM dataset is a microwave dataset with a temporal resolution of 1 day and a spatial resolution of 9 km; the MODIS LST dataset and the MODIS NDVI / EVI dataset are both optical remote sensing datasets, the MODIS LST dataset has a temporal resolution of 1 day and a spatial resolution of 1 km, and the MODIS NDVI / EVI dataset has a temporal resolution of 16 days and a spatial resolution of 1 km; the ERA5-LAND SM dataset, the ERA5-LAND LST dataset, and the ERA5-LAND LAI dataset are reanalysis datasets with a temporal resolution of 1 day and a spatial resolution of 9 km; the precipitation dataset has a temporal resolution of 1 day and a spatial resolution of 5 km; the digital elevation dataset, the slope dataset, and the soil texture dataset are all static data with a spatial resolution of 1 km. Generally, the HEG tool (HDF-EOS To GeoTIFF Conversion Tool) is used to batch convert SMAP soil moisture data to WGS84 projection; the MRT (MODIS Reprojection Tool) batch processing software is used to convert the sinusoidal projection of MODIS products to WGS84 projection. The data were regridded at a resolution of 1 km according to the MODIS grid system to achieve a uniform spatial resolution.

[0052] Step 2, data spatiotemporal reconstruction: the spatiotemporal continuity of the preprocessed multi-source remote sensing data is determined, and the spatiotemporal continuous data and the spatiotemporal discontinuous data are distinguished, and the spatiotemporal discontinuous data are reconstructed to obtain spatiotemporal continuous data; the spatiotemporal continuous data obtained by data reconstruction are merged with the distinguished spatiotemporal continuous data to obtain all spatiotemporal continuous multi-source remote sensing data.

[0053] The spatiotemporal discontinuous data distinguished include MODIS NDVI / EVI datasets and MODIS LST datasets. Although the NDVI and EVI datasets provided by MODIS products have a high spatial resolution (1km), the data time interval is 16d, which cannot meet the requirements of generating daily soil moisture results. Therefore, the data needs to be reconstructed to generate daily high-resolution data. Existing studies have found that due to the small amplitude of canopy changes within a certain period of time, the vegetation index curve should be a continuous and smooth curve. Therefore, based on the MODIS NDVI / EVI dataset obtained in step 1, the Savitzky-Golay filtering method is used to interpolate the dataset with a time resolution of 16d into a dataset with a time resolution of 1d.

[0054] Specifically, the 16-day original NDVI / EVI data sequence is fitted using the Savitzky-Golay filtering method, which is implemented by calling the savgol_filter() function in Python. The function has two important parameters, one is the length of the sliding window (window_length), and the other is the order of the polynomial (K). Among them, window_length must be a positive odd number, and the smaller the value, the closer the curve is to the true curve; the larger the value, the better the smoothing effect; the K value must be less than the length of window_length. The larger the K value, the closer the curve is to the true curve; the smaller the K value, the better the curve smoothing. Considering that the time interval of NDVI / EVI is 16 days, the window_length length is selected as 7 and the K value is 4 for filtering.

[0055] The LST dataset provided by MODIS products has a temporal resolution of 1d and a spatial resolution of 1km. However, due to cloud cover, the MODIS LST dataset usually has a large proportion of missing areas, and although the ERA5-LAND LST dataset has no missing parts, its spatial resolution is relatively coarse (9km) and cannot meet the input data resolution requirements. Therefore, two original surface temperature datasets with different spatial resolutions are used: the MODIS LST dataset and the ERA5-LAND LST dataset, digital elevation data, and the MODIS NDVI / EVI dataset with a temporal resolution of 1d obtained after interpolation processing. The multiple regression method is used to fill in the missing data in the MODIS LST data to generate a spatiotemporally continuous surface temperature dataset in the study area, that is, a seamless daily surface temperature dataset with a spatial resolution of 1km. The specific formula is:

[0056] LST value =a×LST ERA5_value +b×NDVI 1d_value +c×DEM value +d (1)

[0057] LST non_value =a×LST ERA5_non_value +b×NDVI 1d_non_value +c×DEM non_value +d (2)

[0058] Where: LST value and LST non_value They are the areas with values ​​and areas without values ​​in MODIS LST; ERA5_value and LST ERA5_non_value They are ERA5-LAND LST with and without values ​​in the corresponding areas of MODIS LST; NDVI 1d_value and NDVI1d_non_value They are the 1-day NDVI data with and without values ​​in the corresponding area of ​​MODIS LST; DEM value and DEM non_value They are the digital elevation data with and without values ​​in the corresponding area of ​​MODIS LST; a, b, c, d are the parameters of the formula;

[0059] Finally, LST value and LST non_value Splicing to generate seamless daily land surface temperature dataset LST seamless .

[0060] Step 3, construct a multi-layer soil moisture downscaling model and train it: Use an ensemble learning algorithm to construct a multi-layer soil moisture downscaling model. There are three commonly used fusion strategies for ensemble learning: Bagging, Boosting and Stacking. In the specific application process of this technology, the method in the optimal ensemble learning strategy can be selected in combination with the data type and computer performance for application. Since the training process of Stacking is more complicated than that of Bagging and Boosting, an additional meta-learner needs to be trained, which increases the computational burden. Therefore, the present invention preferably selects the typical random forest (RF) algorithm in Bagging to construct the multi-layer soil moisture downscaling model in the present invention.

[0061] First, all spatiotemporally continuous multi-source remote sensing data are matched with the ground-measured soil moisture sites in spatial latitude and longitude to obtain the remote sensing pixel values ​​of the ground-measured soil moisture sites. The matched remote sensing pixel values ​​are used as input data, and the preprocessed ground-measured soil moisture site data are used as output data. The input data and output data constitute the model training data set. The model training data set is divided into a training set and a test set according to the ratio (generally 8:2). The multi-layer soil moisture downscaling model in the study area is constructed using an integrated learning algorithm, and the model is trained using the training set.

[0062] The constructed multi-layer soil moisture downscaling model is expressed as:

[0063]

[0064] Where: f is the ensemble learning algorithm; SMAP SM For SMAP SM data; ERA5 SM ERA5-LAND SM data; ERA5 LAI ERA5-LAND LAI data; LST seamless For seamless daily surface temperature data; NDVI 1d 、EVI 1dThey are the reconstructed NDVI and EVI data respectively; Precipitation is precipitation data; Soil texture is soil texture data; DEM is digital elevation data; Slope is slope data.

[0065] Specifically, the integrated learning algorithm modeling process is implemented by loading scikit-learn in the Python language environment.

[0066] During the model training process, optuna and GridSearchCV methods are used to adjust the hyperparameters of the ensemble learning algorithm. For the RF algorithm, the hyperparameters that need to be adjusted are the number of decision trees (n_estimators), the depth of the decision tree (max_depth), the maximum number of features selected when establishing the decision tree (max_features), and the minimum number of samples required for segmentation (min_sample_split). After using optuna and GridSearchCV methods to find the optimal (i.e., minimum error) parameter combination of various hyperparameters in the training set, the optimal parameter combination is verified in the test set, and an evaluation index system is constructed for evaluation. When the evaluation index results tend to be stable and reach their respective optimal values, the model training is considered to be completed.

[0067] The mean absolute error (MAE), determination coefficient (R 2 ) and root mean square error (RMSE) are used as evaluation indicators. The smaller the mean absolute error (MAE) and root mean square error (RMSE), the higher the determination coefficient (R 2 ) is larger, the smaller the deviation between the predicted result and the measured result is, that is, the better the prediction result of the optimal parameter combination is. 2 ) and the root mean square error (RMSE) are calculated as follows:

[0068]

[0069] In the above formula: is the predicted value of soil moisture; i is the true value; is the average value; i is the index value of the data sequence; n is the total length of the data sequence.

[0070] Step 4: Generate multi-layer high-resolution soil moisture results: After model training, only the pixel values ​​within the 1km grid where the soil moisture station is located are obtained. For the other grid points without soil moisture stations, all the above processed spatiotemporal continuous multi-source remote sensing data (including SMAP SM, ERA5-LAND SM, ERA5-LAND LAI, LSTseamless, NDVI, etc.) are required. 1d 、EVI1d , Precipitation, Soil texture, DEM, Slope data) as input data, input to the multi-layer soil moisture downscaling model after training in step 3, and finally generate soil moisture results for all 1km grid points in the area of ​​10, 20 and 40cm, that is, soil moisture results with a spatial resolution of 1km, and verify the generated results with the measured soil moisture station values ​​in time and space scales, where the time scale is directly compared with the long sequence values ​​of the measured soil moisture station, and the spatial scale is compared using the spatial distribution of the evaluation index. At this time, the soil moisture results of the 1km grid point are 10 times higher in resolution than the original 9km SMAP SM and ERA5-LAND SM, so the generated 1km soil moisture results are collectively referred to as high-resolution soil moisture results.

[0071] Since the existing ground soil moisture measurement stations have 10, 20 and 40 cm depth data, the multi-layer high-resolution soil moisture results generated by the present invention are 10, 20 and 40 cm. In the process of generating high-resolution soil moisture results, remote sensing soil moisture data has a greater impact on the final result. Therefore, when generating multi-layer high-resolution soil moisture results, remote sensing soil moisture data of different layer depths are selected as model input data. Using the surface data of SMAP SM data at 0-5cm and the surface data of ERA5-LAND SM data at 0-7cm as the model input data, and the ground soil moisture measured at a depth of 10cm as the model output data, the high-resolution soil moisture results at a depth of 10cm are finally generated; using the SMAP SM data at 0-100cm and the ERA5-LANDSM data at 7-28cm as the model input data, and the ground soil moisture measured at a depth of 20cm as the model output data, the high-resolution soil moisture results at a depth of 20cm are finally generated; using the SMAP SM data at 0-100cm and the ERA5-LANDSM data at 28-100cm as the model input data, and the ground soil moisture measured at a depth of 40cm as the model output data, the high-resolution soil moisture results at a depth of 40cm are finally generated. Since other remote sensing data (such as leaf area index, surface temperature data, etc.) do not have multi-layer results, when generating multi-layer high-resolution soil moisture, only the remote sensing soil moisture data of different depths (including SMAP SM and ERA5-LAND SM) and other remote sensing data (ERA5-LAND LAI, LST seamless , Precipitation, Soil texture, DEM, Slope data) are input into the trained multi-layer soil moisture downscaling model, and finally multi-layer high-resolution (1km) soil moisture results of 10, 20 and 40cm are generated.

[0072] Embodiment 1

[0073] This example uses Jilin Province as the research area. Figure 2 As shown; Based on the SMAP microwave surface (0-5cm) soil moisture data and the ERA5-LAND soil moisture data surface (0-7cm) soil moisture dataset from 2017 to 2022, taking the random forest (RF) algorithm as an example, a multi-layer soil moisture downscaling model is constructed to generate high-resolution soil moisture results for the surface layer of 10cm.

[0074] This embodiment discloses a multi-layer high-resolution soil moisture generation method based on ensemble learning, comprising the following steps:

[0075] Step 1. Data collection and preprocessing: This embodiment collects multi-source remote sensing data and ground-measured soil moisture site data in the study area from January 1, 2017 to December 31, 2022, and preprocesses the collected multi-source remote sensing data and ground-measured soil moisture site data respectively.

[0076] The multi-source remote sensing data includes soil moisture dataset (Soil Moisture), vegetation index dataset (Vegetation), land surface temperature dataset (Land Surface Temperature), precipitation dataset (Precipitation), digital elevation dataset (DEM), slope dataset (Slope) and soil texture dataset (SoilTexture). This embodiment collects the SMAP SM dataset with a time resolution of 1d and a spatial resolution of 9km and the ERA5-LAND SM dataset with a time resolution of 1d and a spatial resolution of 9km as soil moisture datasets; collects the MODIS NDVI / EVI dataset with a time resolution of 16d and a spatial resolution of 1km and the ERA5-LAND LAI dataset with a time resolution of 1d and a spatial resolution of 9km as vegetation index datasets; collects the MODIS LST dataset with a time resolution of 1d and a spatial resolution of 1km and the ERA5-LAND The LST dataset is used as the surface temperature dataset; a digital elevation dataset with a spatial resolution of 1 km is also collected, and the slope dataset of the area of ​​1 km is calculated through ArcGIS10.8; a precipitation dataset with a temporal resolution of 1d and a spatial resolution of 5 km is collected; a soil texture dataset with a spatial resolution of 1 km is collected, and the soil texture is divided into several types: sand, silt, and clay; as shown in Table 1. After preprocessing such as projection, resampling, and clipping, the collected multi-source remote sensing data is obtained with consistent projection coordinate system, spatial range, and resolution.

[0077] Table 1

[0078]

[0079] The ground-measured soil moisture station data collected in this embodiment include 8 o'clock data of 305 soil moisture stations in the study area from 2017 to 2022 as the daily value results of the day. After data quality control, 303 stations meet the quality requirements. The spatial distribution of soil moisture stations is as follows: Figure 3 shown.

[0080] Step 2, data spatiotemporal reconstruction: the spatiotemporal continuity of the preprocessed multi-source remote sensing data is determined, and the spatiotemporal continuous data and the spatiotemporal discontinuous data are distinguished, and the spatiotemporal discontinuous data are reconstructed to obtain spatiotemporal continuous data; the spatiotemporal continuous data obtained by data reconstruction are merged with the distinguished spatiotemporal continuous data to obtain all spatiotemporal continuous multi-source remote sensing data.

[0081] The identified spatiotemporal discontinuous data include MODIS NDVI / EVI dataset and MODIS LST dataset. Taking advantage of the fact that the vegetation index has a small change in the canopy within a certain period of time and the vegetation index curve should be a continuous and smooth curve, the Savitzky-Golay filtering method is used to reconstruct the MODIS NDVI / EVI time series 16d result into a 1d result.

[0082] The specific reconstruction process is: implemented by calling the savgol_filter() function in Python. This function has two important parameters, one is the length of the sliding window (window_length), and the other is the order of the polynomial (K). Among them, window_length must be a positive odd number, and the smaller the value, the closer the curve is to the true curve; the larger the value, the better the smoothing effect; the K value must be less than the length of window_length. The larger the K value, the closer the curve is to the true curve; the smaller the K value, the better the curve smoothing. Considering that the time interval of NDVI / EVI is 16 days, select window_length length of 7 and K value of 4 for filtering. The interpolated daily NDVI data results are as follows: Figure 4 As shown, it can be seen that the NDVI spatial results vary little, with only slight differences in the plains in the northwest.

[0083] This embodiment combines the 1km resolution MODIS LST daily dataset and the 9km resolution ERA5-LAND LST dataset to fill the gap in the MODIS LST dataset. The specific formula is:

[0084] LST value =a×LSTERA5_value +b×NDVI 1d_value +c×DEM value +d

[0085] LST non_value =a×LST ERA5_non_value +b×NDVI 1d_non_value +c×DEM non_value +d

[0086] Where: LST value and LST non_value They are the areas with values ​​and areas without values ​​in MODIS LST; ERA5_value and LST ERA5_non_value They are ERA5-LAND LST with and without values ​​in the corresponding areas of MODIS LST; NDVI 1d_value and NDVI 1d_non_value They are the 1d NDVI data with and without values ​​in the corresponding area of ​​MODIS LST; DEM value and DEM non_value are the digital elevation data with and without values ​​in the MODIS LST corresponding area; a, b, c, d are the parameters of the formula; and finally LST value and LST non_value Splicing to generate seamless daily land surface temperature dataset LST seamless .

[0087] Due to the need to fill in the LST data from 2017 to 2022, when filling in the missing results of MODIS LST data, small samples are used in batches for sample training. Statistics of the original MODIS LST data show that the maximum amount of missing data is 213220, while the minimum amount of non-null value data for 5 days is greater than this value. Therefore, during the period of 2017-2022, the data with values ​​every 5 days is used as a training set to generate the missing MODIS LST data results for each day in these 5 days, and then the missing data results generated daily are spliced ​​with the original MODIS LST non-null value area to generate daily spatially continuous MODIS LST results. Figure 5 A comparison of ERA5-LAND, MODIS LST and reconstructed LST is given. It can be seen that there is a large amount of missing data in the central and western parts of the target area of ​​the MODIS product. The ERA5-LAND data covers the entire area, but the resolution is coarse. The blank areas are well filled after LST reconstruction. The spatial distribution of the filled LST is highly consistent with the ERA5-LAND LST and the original MODIS LST, and shows more details than the ERA5-LAND LST product.

[0088] Step 3, construct a multi-layer soil moisture downscaling model and train it: match all spatiotemporal continuous data (including SMAP SM dataset, ERA5-LAND SM dataset, ERA5-LAND LAI dataset, ERA5-LAND LST dataset, precipitation dataset, digital elevation dataset, slope dataset, soil texture dataset, and reconstructed MODIS NDVI / EVI dataset and MODIS LST dataset) with the ground-measured soil moisture station in spatial latitude and longitude, obtain the remote sensing pixel values ​​of the points where the ground-measured soil moisture station is located, use the above-mentioned matched remote sensing pixel values ​​as input data, and use the preprocessed ground-measured soil moisture station data as output data. The input data and the output data form a training data set, and the training data set is divided into a training set and a test set in proportion. Use the random forest (RF) algorithm to construct a multi-layer soil moisture downscaling model in the study area, and use the training set to train the model;

[0089] The constructed multi-layer soil moisture downscaling model is expressed as:

[0090] SSM downscaled =f(SMAP SM ,ERA5 SM ,ERA5 LAI ,LST seamless ,

[0091] NDVI 1d ,EVI 1d ,Precipitaion,Soil texture,DEM,Slope)

[0092] Where: f is the RF algorithm; SMAP SM For SMAP SM data; ERA5 SM ERA5-LAND SM data; ERA5 LAI ERA5-LAND LAI data; LST seamless For seamless daily surface temperature data; NDVI 1d ,EVI 1d is the reconstructed vegetation index data, Precipitation is the precipitation data, Soil texture is the soil texture data, DEM is the digital elevation data, and Slope is the slope data.

[0093] Before evaluating the prediction results, the RF algorithm needs to be hyperparameter tuned. First, the training data set is divided into a training set and a test set in a ratio of 8:2, where the training set data volume is 222990 and the test set data volume is 55748. For the RF algorithm, the hyperparameters that need to be adjusted are the number of decision trees (n_estimators), the depth of the decision tree (max_depth), the maximum number of features selected when establishing the decision tree (max_features), and the minimum number of samples required for segmentation (min_sample_split). After using optuna and GridSearchCV methods to find the optimal (i.e., minimum error) parameter combination of various hyperparameters in the training set, the optimal parameter combination is verified in the test set, and evaluation indicators (including mean absolute error MAE, determination coefficient R 2 When the evaluation index results tend to be stable and reach their respective optimal values, the model training is considered completed.

[0094] The results of RF algorithm parameter adjustment are shown in Table 2. The RF algorithm was adjusted for 20 times, and the RF algorithm had the smallest error in the 13th time. After the RF algorithm was adjusted, the predicted values ​​and measured values ​​of the soil moisture samples in the test set were basically close to the 1:1 line, and R 2 The values ​​are all over 0.940, such as Figure 6 shown.

[0095] Table 2

[0096]

[0097] Step 4. Generate multi-layer high-resolution soil moisture results: Take the results of 2018-06-16, 2019-08-15 and 2020-07-11 as an example to generate 3-day 1km soil moisture results. First, all spatiotemporal continuous data of 2018-06-16, 2019-08-15 and 2020-07-11 (including SMAP SM dataset, ERA5-LAND SM dataset, ERA5-LAND LAI dataset, ERA5-LAND LST dataset, precipitation dataset, digital elevation dataset, slope dataset, soil texture dataset and reconstructed MODIS NDVI / EVI dataset and MODIS LST dataset) are input into the 10cm soil moisture downscaling model trained in step 3 according to the date. The final result generated by the model is the 3-day 1km soil moisture result.

[0098] In order to evaluate the accuracy of the soil moisture downscaling results, the results were verified at two scales, time and space. The time scale was directly compared with the long sequence values ​​of the measured soil moisture stations, and the spatial scale was compared using the spatial distribution of the evaluation indicators. Three verification soil moisture stations were randomly selected and the soil moisture results generated by the ground measured soil moisture station values ​​and the soil moisture downscaling model were compared and analyzed. Taking Tuoyao Village Station as an example, Figure 7 As shown in the figure, the blue bar chart illustrates the distribution of precipitation, which is mainly concentrated in May-September. In general, the soil moisture peak of SMAP soil moisture data is consistent with precipitation events, and SMAP soil moisture data can well reflect the overall trend of soil moisture. There is a significant deviation between SMAP soil moisture and the soil moisture measured at the site in winter (December-January). In contrast, the downscaled SM based on the RF algorithm is closer to ground observations, and the downscaled results are consistent with the R of the measured site. 2 Both are greater than 0.991, and MAE and RMSE are less than 0.1.

[0099] Table 3 shows the comparison between the 3-day downscaling results generated by the traditional downscaling method using only remote sensing products and the measured sites. It can be seen from the table that, regardless of whether it is SMAP or ERA5-LAND soil moisture data, the 1 km × 1 km downscaling results generated by the RF algorithm are R 2 All are less than 0.3. From the results generated on 2018-06-16, we can see that Figure 8 As shown in the figure, the RF algorithm simulates the soil moisture in the western part of Jilin Province poorly and generally overestimates the soil moisture in the western part. The western part of Jilin Province is the main grain producing area in the region. Therefore, the high-resolution soil moisture results generated by the traditional downscaling method may lead to serious misjudgment of the soil moisture in the region, thereby causing a reduction in grain production in the region.

[0100] Table 3

[0101]

[0102]

[0103] Table 4 shows the comparison and verification evaluation results of the downscaling results of the RF algorithm after fusing remote sensing data and measured sites with the results of the measured sites. The results show that the RF algorithm can simulate the soil moisture results well. The R 2 The downscaling method combined with the ground-based soil moisture measurement stations solves the problem of overestimation of soil moisture in the northwest of Jilin Province by the traditional downscaling method, and this method can well fit the peak and low value results of soil moisture in the western region.

[0104] Table 4

[0105]

[0106] The best results from different data sources were selected and compared with the original data sources and the results after site interpolation, e.g. Fig.10 In general, the spatial distribution trend of the fused SMAP and ERA5-LAND soil moisture data is consistent with the interpolation results of the measured sites. The soil moisture in Baicheng, Songyuan and Siping in the northwest is relatively low, while the soil moisture in Tonghua and Yanbian in the east is relatively high.

[0107] It can be seen that the method described in the present invention effectively improves the reliability of soil moisture downscaling results and the accuracy of downscaling in a large-scale study area, and improves the accuracy and efficiency of large-scale spatial monitoring and early warning of soil moisture content in cloudy and rainy areas at a low cost.

[0108] Finally, it should be noted that the above is only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred arrangement scheme, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-layer high-resolution soil moisture generation method based on ensemble learning, characterized in that: The method comprises the following steps: Step 1, data collection and preprocessing: collect multi-source remote sensing data and ground-measured soil moisture site data of different depths in the study area for many years, and preprocess the collected multi-source remote sensing data and ground-measured soil moisture site data respectively to obtain preprocessed multi-source remote sensing data and ground-measured soil moisture site data; The multi-source remote sensing data includes a soil moisture dataset, a vegetation index dataset, a surface temperature dataset, a precipitation dataset, a digital elevation dataset, a slope dataset and a soil texture dataset; the soil moisture dataset includes a SMAP SM dataset and an ERA5-LAND SM dataset of different layer depths; the vegetation index dataset includes a MODIS NDVI / EVI dataset and an ERA5-LAND LAI dataset; the surface temperature dataset includes a MODIS LST dataset and an ERA5-LAND LST dataset; the layer depth of the soil moisture dataset corresponds to the depth of the ground soil moisture station data; Step 2, data spatiotemporal reconstruction: the spatiotemporal continuity of the preprocessed multi-source remote sensing data is judged, the spatiotemporal continuous data and the spatiotemporal discontinuous data are distinguished, and the spatiotemporal discontinuous data are reconstructed to obtain spatiotemporal continuous data; the spatiotemporal continuous data obtained by data reconstruction are merged with the distinguished spatiotemporal continuous data to obtain all spatiotemporal continuous multi-source remote sensing data; Step 3, construct a multi-layer soil moisture downscaling model and train it: match the spatial longitude and latitude of all spatiotemporally continuous multi-source remote sensing data with the ground-measured soil moisture sites, obtain the remote sensing pixel values ​​of the points where the ground-measured soil moisture sites are located, use the matched remote sensing pixel values ​​as input data, and use the preprocessed ground-measured soil moisture site data as output data. The input data and output data constitute the model training data set. Use the integrated learning algorithm to construct a multi-layer soil moisture downscaling model in the study area, and use the model training data set to train the model until the model training is completed; Step 4: Generate multi-layer high-resolution soil moisture results: All spatiotemporally continuous multi-source remote sensing data are used as input data and input into the trained multi-layer soil moisture downscaling model to generate multi-layer high-resolution soil moisture results, that is, generate multi-layer soil moisture results with a spatial resolution of 1 km, and verify the generated results with the measured soil moisture station values ​​on the spatiotemporal scale.

2. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 1 is characterized in that: The step 1 of collecting multi-source remote sensing data and ground-measured soil moisture site data for the study area for many years specifically refers to collecting data for the study area for more than five years.

3. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 1 is characterized in that: The collected multi-source remote sensing data and ground-measured soil moisture site data are preprocessed separately as described in step 1 to obtain the preprocessed multi-source remote sensing data and ground-measured soil moisture site data. Specifically, the multi-source remote sensing data are projected, resampled and cropped to obtain multi-source remote sensing data with consistent projection coordinate system, spatial range and resolution; the ground-measured soil moisture site data are subjected to data quality control to obtain a ground-measured soil moisture site data set with a complete time series and no abnormal values.

4. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 1 is characterized in that: The SMAP SM dataset described in step 1 is a microwave dataset with a temporal resolution of 1d and a spatial resolution of 9km; The MODIS LST dataset and the MODIS NDVI / EVI dataset are both optical remote sensing datasets. The temporal resolution of the MODIS LST dataset is 1 day, and the spatial resolution is 1 km. The temporal resolution of the MODIS NDVI / EVI dataset is 16 days, and the spatial resolution is 1 km. The ERA5-LAND SM dataset, ERA5-LAND LST dataset and ERA5-LAND LAI dataset are reanalysis datasets, with a temporal resolution of 1 day and a spatial resolution of 9 km; the precipitation dataset has a temporal resolution of 1 day and a spatial resolution of 5 km; the digital elevation dataset, slope dataset and soil texture dataset are all static data, with a spatial resolution of 1 km.

5. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 4 is characterized in that: The spatiotemporal discontinuous data distinguished in step 2 include MODIS NDVI / EVI dataset and MODIS LST dataset; The specific process of performing spatiotemporal reconstruction on the spatiotemporal discontinuous data to obtain spatiotemporal continuous data is as follows: For the MODIS NDVI / EVI dataset, the Savitzky-Golay filtering method is used to interpolate the dataset with a time resolution of 16 days into a dataset with a time resolution of 1 day; For the MODIS LST dataset, the multiple regression method is used to fill the missing data in the MODIS LST dataset to generate a spatiotemporally continuous surface temperature dataset in the study area, that is, a seamless daily surface temperature dataset with a spatial resolution of 1 km. The specific formula is: LST value =a×LST ERA5_value +b×NDVI 1d_value +c×DEM value +d (1) LST non_value =a×LST ERA5_non_value +b×NDVI 1d_non_value +c×DEM non_value +d (2) Where: LST value and LST non_value They are the areas with values ​​and areas without values ​​in MODIS LST; ERA5_value and LST ERA5_non_value They are ERA5-LAND LST with and without values ​​in the corresponding areas of MODIS LST; NDVI 1d_value and NDVI 1d_non_value They are the 1-day NDVI data with and without values ​​in the corresponding area of ​​MODIS LST; DEM value and DEM non_value They are the digital elevation data with and without values ​​in the corresponding area of ​​MODIS LST; a, b, c, d are the parameters of the formula; Finally, LST value and LST non_value Splicing to generate seamless daily land surface temperature dataset LST seamless .

6. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 5 is characterized in that: The multi-layer soil moisture downscaling model described in step 3 is: Where: f is the ensemble learning algorithm; SMAP SM For SMAP SM data; ERA5 SM ERA5-LAND SM data; ERA5 LAI ERA5-LAND LAI data; LST seamless For seamless daily surface temperature data; NDVI 1d 、EVI 1d They are the reconstructed NDVI and EVI data respectively; Precipitation is precipitation data; Soil texture is soil texture data; DEM is digital elevation data; Slope is slope data.

7. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 6 is characterized in that: The specific process of using the model training data set to train the model until the model training is completed in step 3 is as follows: the model training data set is divided into a training set and a test set in a ratio of 8:2, and the model is trained using the training set. During the model training process, the optuna and GridSearchCV methods are used to adjust the hyperparameters of the ensemble learning algorithm. After finding the optimal parameter combination of each hyperparameter in the training set, the optimal parameter combination is verified in the test set, and evaluated using evaluation indicators. When the evaluation indicator results tend to be stable and reach their respective optimal values, the model training is completed.

8. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 7 is characterized in that: The evaluation indicators include mean absolute error MAE, determination coefficient R 2 The smaller the mean absolute error MAE and the root mean square error RMSE, the greater the determination coefficient R 2 The larger the value, the smaller the deviation between the predicted result and the measured result, that is, the better the predicted result of the optimal parameter combination, the mean absolute error MAE and the determination coefficient R 2 The calculation formulas for the root mean square error and RMSE are: In the above formula: is the predicted value of soil moisture; i is the true value; is the average value; i is the index value of the data sequence; n is the total length of the data sequence.

9. The multi-layer high-resolution soil moisture generation method based on ensemble learning according to claim 1, characterized in that: The generation of multi-layer high-resolution soil moisture results described in step 4 includes high-resolution soil moisture results at depths of 10, 20, and 40 cm; Using SMAP SM data 0-5cm and ERA5-LAND SM data 0-7cm as model input data and the ground soil moisture measured at a depth of 10cm as model output data, high-resolution soil moisture results at a depth of 10cm are generated; using SMAPSM data 0-100cm and ERA5-LAND SM data 7-28cm as model input data and the ground soil moisture measured at a depth of 20cm are used as model output data, high-resolution soil moisture results at a depth of 20cm are generated; using SMAP SM data 0-100cm and ERA5-LAND SM data 28-100cm as model input data and the ground soil moisture measured at a depth of 40cm are used as model output data, high-resolution soil moisture results at a depth of 40cm are generated.

Images