Soil humidity fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis

Abstract

The invention discloses a soil humidity fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis, and the method comprises the following steps: (1) collecting and sorting multi-source soil humidity related data, and forming a unified data set; (2) constructing an XGB-SHAP-RFE network, and preferably selecting key features from the data set; (3) training a fusion model by using the optimized key features and evaluating the performance of the fusion model; (4) interpretable analysis is carried out on model output based on an SHAP method, and contribution of each feature is quantified; (5) generating multi-source fused high-quality soil humidity data in combination with a model result; the invention aims to efficiently generate a high-precision and high-resolution soil humidity fusion product.

Description

Technical Field

[0001] The present invention relates to the technical field of soil moisture fusion, and in particular to a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis. Background Art

[0002] Soil moisture plays a key role in the terrestrial water cycle. It not only directly affects evapotranspiration and runoff, but also indirectly affects the simulation of regional climate models by regulating soil-atmosphere energy exchange. Therefore, accurate soil moisture data is crucial for drought monitoring, flood forecasting, and land-atmosphere interaction research. With the development of observation technology and land surface models, site observations, satellite remote sensing, and land surface model products have made it possible to obtain high-precision soil moisture estimates. The dynamic changes and spatial distribution of soil moisture are affected by many factors, such as soil properties, climatic conditions, and vegetation changes. Therefore, how to effectively fuse multi-source soil moisture data and make full use of these characteristics to generate high-precision and high-resolution soil moisture data is a current challenge.

[0003] Although traditional data fusion methods such as data assimilation and Bayesian averaging can integrate data from different sources, they are limited in processing complex nonlinear relationships and high-dimensional features, and have high requirements for computing resources. In contrast, machine learning models can learn the complex nonlinear relationships between different soil moisture data sources and multiple features (soil, vegetation, meteorological data, etc.) through training. In recent years, machine learning has made significant progress in soil moisture data fusion. However, the current soil moisture data fusion methods based on machine learning lack reliable feature screening methods and are deficient in machine learning interpretable analysis. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis, which performs feature screening through the XGB-SHAP-RFE method, combines the Optuna automatic hyperparameter optimization method and SHAP interpretable analysis, and solves the problems existing in the background technology.

[0005] Technical solution: The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis described in the present invention includes the following steps: (1) Collect and organize soil moisture data from multiple sources to form a unified data set; (2) Construct the XGB-SHAP-RFE network to select key features from the dataset; (3) Use the selected key features to train the fusion model and evaluate its performance; (4) Perform interpretable analysis of model output based on the SHAP method to quantify the contribution of each feature; (5) Combine the model results to generate high-quality soil moisture data from multiple sources.

[0006] Furthermore, in step (1), the data set sources include feature data and target data; wherein the target data is the soil moisture automatic station observation data from the China Meteorological Administration, and the feature data includes multi-source soil moisture data, meteorological data, vegetation data, terrain data and soil attribute data; wherein the multi-source soil moisture data includes: ERA5, GLDAS land surface model data, satellite fusion data; the meteorological data includes: precipitation, evapotranspiration, soil temperature and runoff.

[0007] Furthermore, step (1) includes the following steps: (11) Quality control of soil moisture observation data: first, the hourly soil moisture observation data were processed into daily average data; based on the ERA5-Land soil temperature data, observation data with soil temperature below 275K were eliminated; on this basis, stations with observation frequency below 0.5 were eliminated, and further stations with observation time less than 180 days were eliminated; (12) Collect characteristic data and unify the resolution to 1 km grid through bilinear interpolation to generate soil moisture fusion data; (13) Based on the 1 km uniform grid data, the nearest neighbor interpolation method is used to extract the grid feature data to the observation site and construct a data set; it is divided into a training set and a test set according to the date of the observation data.

[0008] Furthermore, step (2) includes the following steps: (21) Use the entire training set to train the XGBoost model according to the default hyperparameter combination, perform a five-fold cross validation, and calculate the reliability of each scoring verification model; (22) Based on the SHAP method, 10% of the data is randomly selected from the training set to calculate the feature importance ranking of the XGBoost model: F1, F2, F3, ..., F N , where N is the total number of features; (23) Perform RFE feature screening according to the feature importance ranking table F to find the optimal feature subset.

[0009] Furthermore, step (23) includes the following steps: S1: Initialization; in the first iteration, i=0, the feature subset is the first Ni features in the feature importance ranking table, that is, the entire training set F1…F N ,Right now: ; S2: Based on the default hyperparameter combination and feature subset Z', the XGBoost model is trained on the training set, and a five-fold cross validation is performed to calculate the KGE score; S3: When i=N-3, the iteration stops, that is, the top three features in the feature importance ranking table are retained as the basic features for establishing the soil moisture fusion model, and do not participate in the feature screening iteration; S4: After stopping the iteration, the corresponding functional relationship between the number of features x and the model score KGE index is obtained: KGE=f(x), where KGE max The corresponding number of features x * This is the inflection point defined mathematically, which is the optimal feature subset in theory. In practice, when the number of features is x * When it is near, the KGE index changes very little; set the threshold: ; Among them, KGE max represents the highest value of KGE, and x' satisfies the condition: The minimum number of features; the feature subset corresponding to x' is taken as the actual optimal feature subset.

[0010] Furthermore, step (3) includes the following steps: (31) Using the obtained optimal feature subset, hyperparameter optimization was performed on LightGBM, XGBoost, CatBoost, and RandomForest. Hyperparameter optimization was based on the Optuna method with 50 iterations. The KGE index was calculated through five-fold cross validation in each iteration as the objective function of hyperparameter optimization. (32) Based on the optimal feature subset and the optimal hyperparameter combination, the model was trained on the training set, and the four models were evaluated on the test set. The scoring was based on four indicators: BIAS, RMSE, CORR, and KGE. The optimal model was selected by comprehensively considering various scoring criteria and training time.

[0011] Furthermore, step (4) includes the following steps: (41) Input the test set data and the optimal soil moisture fusion model in step (3) into the SHAP model, calculate and store the Shapely value; (42) Perform global interpretability analysis, draw feature summary graphs, and feature value and SHAP value relationship graphs, and analyze the feature importance ranking and relative contribution of different features to the predictive variables on the test set; (43) Conduct local interpretability analysis, select the soil moisture of the analyzed stations, draw heat maps, and conduct interpretable analysis of the soil moisture time series of a single station.

[0012] Furthermore, step (5) includes the following steps: (51) Based on the optimal feature subset, optimal hyperparameter combination and optimal machine learning model, training is performed on the entire data set and 5-fold cross validation is performed; (52) Interpolate the grid data to any soil moisture automatic station, deploy the trained model at the station, and conduct single-point testing to compare and verify the performance on the test set; (53) The trained model is deployed on a uniform grid for prediction to generate soil moisture fused data.

[0013] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to any one of the items is implemented.

[0014] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the soil moisture fusion methods based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: feature screening based on the XGB-SHAP-RFE method, combined with the Optuna automatic hyperparameter optimization method and SHAP interpretability analysis, aims to efficiently generate high-precision, high-resolution soil moisture fusion products, and improve the interpretability of machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the present invention; Figure 2 is the SHAP interpretable analysis diagram of the present invention; wherein, Figure 2 (a) in the figure is the feature summary diagram; Figure 2 (b) is the relationship between the characteristic values ​​of precipitation and soil temperature and the SHAP value; Figure 2 (c) in the figure is a single site heat map. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0018] The embodiment of the present invention provides a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis, such as Figure 1 As shown, the following steps are included: Step 1: Build a dataset; The dataset to be constructed in this invention includes two parts: label data and feature data. The label data is the observation data from the soil moisture automatic station of the China Meteorological Administration from 2010 to 2017. The feature data includes multi-source soil moisture data (ERA5, ERA5-Land, GLDAS-Noah, etc.), meteorological data (precipitation, evapotranspiration, soil temperature and runoff, etc.), vegetation data and static data (topographic data and soil attribute data).

[0019] The spatial resolution of the target soil moisture fusion product of the present invention is 1 km. Too low resolution of the fusion members may cause unreasonable distribution of the fusion products on the land-sea boundary and lake areas. Therefore, the minimum resolution of the fusion members selected in the present invention is 0.25°, and the maximum resolution of the fusion members should not be lower than the target resolution of the fusion product.

[0020] The specific steps to construct the dataset include: Step 11: Quality control based on observation data: (1) According to the soil moisture observation specifications of the China Meteorological Administration, remove observation data above 0.5 m³ / m³; (2) Since the frequency domain reflectometry (FDR) sensor will provide invalid observation data when the soil temperature is below 0 degrees Celsius, remove the soil moisture observation data with a soil temperature below 0 degrees Celsius; (3) Remove stations with an observation frequency less than 0.5 to ensure that each observation station has at least 365 days of observation time. The original observation data is hourly data, which is processed into daily data through daily averaging.

[0021] Step 12: All feature data sets are interpolated bilinearly to a uniform resolution of 1 km for the generation of soil moisture fusion data.

[0022] Step 13: Further use the nearest interpolation method to extract the grid feature dataset with unified resolution to the observation site according to longitude and latitude, and construct a tabular dataset for training the model.

[0023] Step 14: Divide the training set and the test set: According to the date of the observation data, the present invention uses the data from 2010 to 2015 for training, and sets the data from 2016 and 2017 as the test set.

[0024] Step 2: Feature screening; Using the dataset created in step 1, perform feature screening on the training set to simplify the model and improve the interpretability of the model. The specific steps include: Step 21: Use the entire training set to train the XGBoost model according to the default hyperparameter combination, perform a five-fold cross validation, and calculate the reliability of each scoring verification model.

[0025] Step 22: Based on the SHAP method, 10% of the data is randomly selected from the training set to calculate the feature importance ranking of the XGBoost model: F1, F2, F3, ..., FN, where N is the total number of features. For the xth sample, the Shapley value of the i-th feature is calculated as follows: ; Where S i (f,x) represents the Shapley value of the ith feature for the xth sample, and f is the machine learning model used. Z' represents the feature subset, and X represents the total feature set. M represents the number of features contained in the feature subset. The first term in the product represents the weight term, which means that different weights are assigned according to the size of the feature subset. The second term is the marginal contribution term, which is the deviation of the model prediction between including and excluding the ith feature. On this basis, the sum of all feature combinations is S. i (f, x), it is worth noting that this formula is only a theoretical calculation formula for calculating the Shapley value, which is convenient for understanding the basic calculation idea of ​​SHAP. In fact, the present invention uses the Three-SHAP method to optimize the calculation according to the structure of the tree model to improve the calculation efficiency.

[0026] Step 23: Perform RFE feature screening according to the feature importance ranking table. The core idea is to find the optimal feature subset through iteration. The specific implementation steps are as follows: S1: Initialization. In the first iteration, i=0, the feature subset is the first Ni features in the feature importance ranking table, that is, the entire training set F1…FN, that is: ; S2: Based on the default hyperparameter combination and feature subset Z', the XGBoost model is trained on the training set, and a five-fold cross validation is performed to calculate the KGE coefficient.

[0027] S3: When i=N-3, the iteration is stopped, that is, the top three features in the feature importance ranking table are retained as the basic features for establishing the soil moisture fusion model, and they do not participate in the feature screening iteration.

[0028] S4: So far, the corresponding functional relationship between the number of features x and the model score KGE index is obtained: KGE=f(x). max The corresponding number of features x * This is the inflection point defined mathematically, which is the optimal feature subset in theory. But in reality, when the number of features is x * When it is near, the KGE index changes very little. Therefore, the present invention sets a threshold: ; Among them, KGE max represents the highest value of KGE, and x' satisfies the condition: The minimum number of features; the feature subset corresponding to x' is taken as the actual optimal feature subset.

[0029] Step 3: Model training and evaluation: Step 31: Using the optimal feature subset obtained in step 2, perform hyperparameter optimization based on the Optuna method on the training set. The maximum number of iterations is set to 50, and a five-fold cross validation is performed for each iteration. The KGE index is selected as the objective function.

[0030] Step 32: Based on the optimal feature subset and the optimal hyperparameter combination modeling, train four models: LightGBM, XGBoost, CatBoost, and RandomForest on the training set.

[0031] Step 33: Evaluate on the test set and calculate the comprehensive score, as shown in Table 1. Since the LightGBM model has a faster training speed and the Shapley value calculation complexity is higher, the lightweight and highly accurate LightGBM is finally selected as the soil moisture fusion model.

[0032] Table 1 Scores of the four models (DEF and Optuna represent the scores of the default and optimized hyperparameter combinations, respectively) ; Step 4: Perform interpretability analysis based on the SHAP method; Step 41: Input the test set data and the optimal soil moisture fusion model in step 3 into the SHAP model, calculate and store the Shapely value.

[0033] Step 42: Draw a feature summary diagram based on the Shapley value, and a relationship diagram between the feature value and the SHAP value. Analyze the relative contribution of different features in the overall data set to the prediction results, as well as the evolution trend of different feature Shapley values ​​with the feature value. The main purpose of this step is to diagnose whether the role played by different features in the prediction process of the machine learning model is consistent with prior physical knowledge. Figure 2 As shown in (a), the ERA5 and GLDAS-Noah soil moisture data are ranked first in terms of feature importance, followed by static data based on soil characteristics, and finally meteorological data. Figure 2 As shown in (b), as the precipitation (Prec) increases, the positive contribution to the model prediction becomes stronger, while the soil temperature (LST) has an inflection point. When the soil temperature exceeds a certain threshold, it makes a negative contribution to the model prediction.

[0034] Step 43: Plotting a heat map based on the soil moisture forecast time series for a site can further understand the relative contribution of different features to the time series at the site scale. Figure 2 As shown in (c), when drought occurs, in addition to soil moisture data, soil temperature data also plays an important role in drought prediction.

[0035] Step 5: Deploy the trained model on a uniform grid to produce fused data. This includes the following steps: Step 51: Based on the optimal feature subset obtained in step 2 and the optimal hyperparameter combination obtained in step 3, XGBoost is trained on all data sets from 2010 to 2017, and five-fold cross validation is used to test the model accuracy and reduce the risk of overfitting.

[0036] Step 52: Before producing fused data, it is recommended to first perform a single-point test in step 52. This is because the previous training and testing of the model were performed on the site, while the production of fused data was performed on a uniform grid. In order to quantify the uncertainty caused by unit conversion, interpolation method, and code writing, the grid data is first extracted to any observation site through the nearest interpolation method for prediction, and compared with the performance of the model on the test set in step 3. If the error drops by more than 10%, the data or code needs to be checked.

[0037] Step 53: Deploy the model trained in step 51 on a 1km grid for prediction, and generate daily fusion products of soil moisture in China.

[0038] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the method for fusing soil moisture based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis is implemented according to any one of the items.

[0039] An embodiment of the present invention also provides a storage medium storing a computer program. When the computer program is executed by a processor, it implements any one of the soil moisture fusion methods based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis.

Claims

1. A soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis, characterized in that: The following steps are involved: (1) Collect and organize soil moisture data from multiple sources to form a unified data set; (2) Construct the XGB-SHAP-RFE network to select key features from the dataset; (3) Use the selected key features to train the fusion model and evaluate its performance; (4) Perform interpretable analysis of model output based on the SHAP method to quantify the contribution of each feature; (5) Combine the model results to generate high-quality soil moisture data from multiple sources.

2. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 1 is characterized in that: In step (1), the data set sources include feature data and target data; the target data is soil moisture automatic station observation data, and the feature data includes multi-source soil moisture data, meteorological data, vegetation data, terrain data and soil attribute data; the multi-source soil moisture data includes: ERA5, GLDAS land surface model data, satellite fusion data; the meteorological data includes: precipitation, evapotranspiration, soil temperature and runoff.

3. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 2 is characterized in that: Step (1) includes the following steps: (11) Quality control of soil moisture observation data: first, the hourly soil moisture observation data were processed into daily average data; based on the ERA5-Land soil temperature data, observation data with soil temperature below 275K were eliminated; on this basis, stations with observation frequency below 0.5 were eliminated, and further stations with observation time less than 180 days were eliminated; (12) Collect characteristic data and unify the resolution to 1 km grid through bilinear interpolation to generate soil moisture fusion data; (13) Based on the 1 km uniform grid data, the nearest neighbor interpolation method is used to extract the grid feature data to the observation site and construct a data set; it is divided into a training set and a test set according to the date of the observation data.

4. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 1 is characterized in that: Step (2) includes the following steps: (21) Use the entire training set to train the XGBoost model according to the default hyperparameter combination, perform a five-fold cross validation, and calculate the reliability of each scoring verification model; (22) Based on the SHAP method, 10% of the data is randomly selected from the training set to calculate the feature importance ranking of the XGBoost model: F1, F2, F3, ..., F N , where N is the total number of features; (23) Perform RFE feature screening according to the feature importance ranking table F to find the optimal feature subset.

5. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 4 is characterized in that: The specific process of step (23) includes: S1: Initialization; in the first iteration, i=0, the feature subset is the first Ni features in the feature importance ranking table, that is, the entire training set F1…F N ,Right now: ； S2: Based on the default hyperparameter combination and feature subset Z', the XGBoost model is trained on the training set, and a five-fold cross validation is performed to calculate the KGE score; S3: When i=N-3, the iteration stops, that is, the top three features in the feature importance ranking table are retained as the basic features for establishing the soil moisture fusion model, and do not participate in the feature screening iteration; S4: After stopping the iteration, the corresponding functional relationship between the number of features x and the model score KGE index is obtained: KGE=f(x), where KGE max The corresponding number of features x * That is the inflection point defined mathematically, which is the optimal feature subset; set the threshold: ； Among them, KGE max represents the highest value of KGE, and x' satisfies the condition: The minimum number of features; the feature subset corresponding to x' is taken as the actual optimal feature subset.

6. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 1 is characterized in that: Step (3) includes the following steps: (31) Using the obtained optimal feature subset, hyperparameter optimization was performed on LightGBM, XGBoost, CatBoost, and RandomForest. Hyperparameter optimization was based on the Optuna method with 50 iterations. The KGE index was calculated through five-fold cross validation in each iteration as the objective function of hyperparameter optimization. (32) Based on the optimal feature subset and the optimal hyperparameter combination, the model was trained on the training set, and the four models were evaluated on the test set. The scoring was based on four indicators: BIAS, RMSE, CORR, and KGE. The optimal model was selected by comprehensively considering various scoring criteria and training time.

7. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 1 is characterized in that: Step (4) includes the following steps: (41) Input the test set data and the optimal soil moisture fusion model into the SHAP model, calculate and store the Shapely value; (42) Perform global interpretability analysis, draw feature summary graphs, and feature value and SHAP value relationship graphs, and analyze the feature importance ranking and relative contribution of different features to the predictive variables on the test set; (43) Conduct local interpretability analysis, select the soil moisture of the analyzed stations, draw heat maps, and conduct interpretable analysis of the soil moisture time series of a single station.

8. The soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to claim 1 is characterized in that: Step (5) includes the following steps: (51) Based on the optimal feature subset, optimal hyperparameter combination and optimal machine learning model, training is performed on the entire data set and 5-fold cross validation is performed; (52) Interpolate the grid data to any soil moisture automatic station, deploy the trained model at the station, and conduct single-point testing to compare and verify the performance on the test set; (53) The trained model is deployed on a uniform grid for prediction to generate soil moisture fused data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis according to any one of claims 1 to 8 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis is implemented according to any one of claims 1 to 8.

Images