Method for exploring performance goodness and generalization ability of multiple machine learning models in runoff sediment simulation

Abstract

The invention provides a method for exploring performance goodness and generalization ability of various machine learning models in runoff sediment simulation. According to the method, the time sequence change trend of a river long-time sequence hydrological factor, a meteorological factor, a vegetation factor, an artificial activity factor and a runoff sediment transport change correlation factor is determined. Then identifying the correlation between each factor and the runoff sediment change, and selecting an important influence factor as an input factor of a machine learning model; the performance goodness of different machine learning models in simulation of runoff and sediment transport is evaluated, a sample data set is randomly divided to eliminate the influence of limited sample size and sample sequence on a simulation result, and each model is repeatedly trained and tested to determine the empirical risk, predictive ability and generalization ability of each model. And sorting the contribution degrees of the input factors in the runoff and sediment transport simulation process by applying a local interpretation method so as to reveal the main driving factors influencing the water and sediment change of the drainage basin.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and more specifically to a method for exploring the performance goodness and generalization ability of multiple machine learning models in runoff sediment simulation. Background Art

[0002] With the development of hydrological data collection and computing power, data-driven models have played a huge role in hydrological prediction. In data-driven models, machine learning has been widely used in hydrology and other fields for modeling and predictive analysis in recent years because it can quickly extract trends, rules or features from complex data sets and perform soft computing. Machine learning algorithms have simple structures, few input parameters and short computing time. They are increasingly used in nonlinear hydrological system modeling to accurately and quickly predict hydrological and climate phenomena. At the same time, in basin sediment production simulation, machine learning models can quickly respond to driving factors and quickly build models, and are gradually being used in basin sediment transport simulation. In many cases, machine learning methods can provide better prediction performance than traditional hydrological models. It is worth mentioning that with the development and application of interpretable algorithms, the operation mode and final decision-making mode of black box models such as machine learning have gradually become transparent and interpretable. Machine learning algorithms such as support vector machine algorithms, artificial neural networks, and extreme learning machines are the most commonly used technologies for hydrological time series prediction. At present, the Yellow River Basin faces problems such as large changes in hydrological dynamics and high sediment content in the river. Accurate medium- and long-term runoff and sediment forecasts are closely related to the development and management of water resources in the Yellow River Basin, the governance of sediment problems, and the formulation of development strategies. In recent years, machine learning has become a popular tool for predicting the time series of runoff and sediment in the Yellow River.

[0003] In general, in the existing runoff and sediment transport prediction, mathematical expressions are generally established based on various meteorological factors and human activity factors and the time series of runoff and sediment transport. Nowadays, the research on such mathematical expressions is mostly concentrated in the field of machine learning, but only a single machine learning method is considered and the suitability of other methods for different study areas is ignored. At the same time, due to the one-dimensionality of time series, researchers have not paid attention to the generalization and attribution capabilities of machine learning methods. The contribution of multiple influencing factors to runoff and sediment transport cannot be fully grasped. As a result, the generalization and attribution capabilities of machine learning models in water and sediment change simulation are not explored in depth and comprehensively. This is mainly reflected in the following aspects: (1) Previous studies rarely combined multiple commonly used machine learning models and conducted repeated experiments to explore the empirical risks and generalization capabilities of different machine learning models in runoff and sediment transport prediction, resulting in the generalization ability and reference value of some current research results being weak; (2) Previous studies did not explain traditional machine learning methods well, the direction of the impact of input feature variables on output could not be clearly determined, and the visibility of feature variable importance was poor, which led to many studies avoiding the use of machine learning for attribution analysis. Therefore, there is an urgent need to explore the performance and generalization capabilities of various machine learning models in runoff and sediment transport simulation, and to attribute the contribution of different input factors. Summary of the invention

[0004] This paper develops a comprehensive framework based on input, simulation, evaluation, generalization and attribution to evaluate the performance and generalization ability of multiple popular machine learning models in runoff and sediment simulation, and determine the contribution of input factors. Based on the local interpretation method, the contribution ranking of different input feature variables of the machine learning model in the simulation process is identified to reveal the driving factors of water and sediment changes in the basin.

[0005] In order to achieve the above object, the present invention is implemented by adopting the following technical scheme: the method comprises: Clarify the time series change trends of long-term river hydrological factors, meteorological factors, vegetation factors, human activity factors and factors related to runoff and sediment transport changes; Identify the correlation between each factor and runoff sediment changes, and select important influencing factors, i.e. significant variables, as input factors of the machine learning model; Evaluate the performance of different existing machine learning models in simulating annual runoff and annual sediment transport; Explore the generalization ability of the model by randomly dividing the sample data set to eliminate the influence of limited sample size and sample order on the simulation results, and repeatedly train and test each model to determine the empirical risk, predictive ability and generalization ability of each model; The local interpretation method is used to rank the contribution of each input factor in the annual runoff and annual sediment transport simulation process, revealing the main driving factors affecting the water and sediment changes in the basin.

[0006] In one scheme, the model input data is preprocessed by normalization and scale conversion, and the data set is grouped according to the grouping rule of 80% training set and 20% test set to improve the model's operating efficiency and simulation accuracy.

[0007] In one approach, the selected machine learning models include support vector regression (SVR), random forest regression (RFR), BP neural network (BPNN), extreme gradient boosting regression (XGB), k nearest neighbor regression (kNNR) and bootstrap agglomeration regression (BAR).

[0008] In one embodiment, the method determines the goodness of model performance based on different evaluation indicators, ranks the models, and generates a cumulative score for each criterion.

[0009] In one approach, the method uses randomized groups to repeatedly train and test the data to determine the empirical risk and generalization ability of each model.

[0010] In one scenario, the method uses a local interpretation method to rank the contribution of each input factor in the runoff and sediment transport simulation process to reveal the main driving factors affecting water and sediment changes in the basin.

[0011] In one embodiment, the model generalization capability includes: The sample data set is randomly divided into a training set and a test set; 80% of the samples are divided into the training set, and the remaining 20% ​​of the samples are divided into the test set; Use the model to train the training set samples, and use the model to test the model on the test set samples; Calculate model empirical risk, predictive power, and generalization ability Empirical risk refers to the error of the model on the training set. Specifically, it is calculated by the mean square error: (i from 1 to N) Where N is the number of samples in the training set. is the actual value of the ith sample, is the model's predicted value for the i-th sample; Predictive ability refers to the performance of the model on new, unknown data. The predictive ability of the model is evaluated using the test set, which is calculated by the mean square error on the test set: (i from 1 to M) Where M is the number of samples in the test set, is the actual value of the ith sample, is the model's predicted value for the i-th sample; Through the above steps, the empirical risk and predictive ability of the model are evaluated, and then the generalization ability of the model is evaluated by comprehensively considering the empirical risk and predictive ability of the model.

[0012] In one embodiment, the local interpretation method includes: By training the ML model F and input variables SHAP uses the explanatory model Y to determine the contribution of each variable to model F; its basic principle is as follows:

[0013]

[0014] in, represents the number of input variables, Indicates the degree of simplification of variables, Indicates the contribution of each variable to the machine learning model, Difference notation for representing set operations.

[0015] Beneficial effects of the present invention: 1. This paper proposes for the first time a method based on the performance excellence and generalization ability of multiple machine learning models in runoff and sediment simulation. Under the condition of reasonably considering different combinations of input factors with strong correlation with runoff and sediment transport, this method is generally applicable to different basins around the world to explore the performance of machine learning models in runoff and sediment simulation and attribute the contribution of characteristic factors.

[0016] 2. The present invention provides a calculation method for the performance and generalization ability of different machine learning in runoff sediment simulation, and explains the contribution of different input factors. Compared with other methods in existing research and calculation, it is more scientific and comprehensive, randomly divides samples and conducts training, takes into account the generalization ability of the model, and the results are more generalizable and interpretable.

[0017] 3. Scientifically explore the performance and generalization ability of various machine learning models in runoff and sediment simulation, and attribute the input characteristic factors. On the one hand, it is of great significance to scientifically predict future river water and sediment change trends and identify the driving factors of water and sediment changes in river basins. On the other hand, it also provides a scientific basis for promoting high-quality river development and even water resources management, ecological environmental protection and policy formulation in arid and semi-arid areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is an integrated framework for the present invention. DETAILED DESCRIPTION

[0019] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used in the present invention in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0021] like Figure 1 and Figure 2 As shown, the present invention develops a comprehensive framework based on input, simulation, evaluation, generalization and attribution to evaluate the performance of multiple mainstream machine learning models in runoff and sediment transport simulation; to improve the generalization of the model, the samples are repeatedly trained and tested to evaluate and determine the empirical risk, prediction ability and generalization ability of each model; the contribution of each characteristic factor in the model simulation is identified through the local interpretation method, revealing the driving factors of water and sediment changes in the basin. The specific improvements and implementation plans are as follows: S1. Clarify the time series change trends of river hydrological factors, meteorological factors, vegetation factors, human activity factors and factors related to runoff and sediment transport changes.

[0022] Identify the correlation between each factor and runoff sediment changes, and select important influencing factors, i.e. significant variables, as input factors of the machine learning model. To ensure the operating efficiency and simulation accuracy of the model and improve the universality of the model, the model input data is preprocessed using methods such as normalization and scale conversion, and the data set is grouped according to the grouping rule of 80% training set and 20% test set.

[0023] S2. Identify the correlation between each factor and runoff sediment changes, and select important influencing factors, i.e. significant variables, as input factors of the machine learning model.

[0024] The selected machine learning models are support vector regression (SVR), random forest regression (RFR), BP neural network (BPNN), extreme gradient boosting regression (XGB), k-nearest neighbor regression (kNNR) and guided agglomeration regression (BAR), and more models can be included in the developed comprehensive framework. The performance of different machine learning models in simulating annual runoff and annual sediment transport was evaluated based on commonly used statistical indicators such as R-squared (R 2 ), Nash efficiency coefficient (NSE), root mean square error (RMSE), mean absolute error (MAE), mean square error (MSE) and consistency index (d) are used to evaluate the performance of machine learning models. Subsequently, in order to select the best model, the models are considered to be ranked and each criterion is cumulatively scored. For each evaluation indicator, the training period and validation period of different models are ranked and scored respectively. For example, when six machine learning models are applied for simulation, 1 to 6 points are assigned to different models in different periods. The best model with the best value of each evaluation indicator in each period is assigned 6 points, the second best model is assigned 5 points, and the worst performing model is assigned 1 point.

[0025] S3. Evaluate the performance of different existing machine learning models in simulating annual runoff and annual sediment transport.

[0026] In order to eliminate the influence of limited sample size and sample order on the simulation results, determine the empirical risk and generalization ability of each model, and more comprehensively evaluate the ability of each model, the present invention uses random grouping of sample sets to update the training set and test set of each machine learning. The training set and validation set account for 80% and 20% of the samples, respectively, and each model is repeatedly trained and tested. The NSE coefficient is used as an indicator to evaluate the simulation ability of machine learning models for training sets and test sets. The NSE values ​​of 100 sets of data sets are calculated, and the range and standard deviation of the NSE value of each model are calculated to determine the potential performance and generalization ability of each model. The NSE scores on the training set and test set reflect the empirical risk and predictive ability of each machine learning model, respectively, and the generalization ability of the machine learning model depends on the comprehensive performance of the training set and test set.

[0027] S4. Explore the generalization ability of the model by randomly dividing the sample data set to eliminate the influence of limited sample size and sample order on the simulation results, and repeatedly train and test each model to determine the empirical risk, predictive ability and generalization ability of each model.

[0028] Exploring the generalization ability of the model requires focusing on empirical risk and generalization ability, which we break down into the following steps: S401. Random division of sample data set First, randomly divide the sample data set into a training set and a test set. Assuming that the sample data set has N samples, we usually divide about 80% of the samples into the training set and the remaining 20% ​​of the samples into the test set. This division ratio can also be adjusted according to specific circumstances.

[0029] S402, model training and testing Next, the model is trained using the training set samples, and the model is tested using the test set samples. The training and testing process is repeated multiple times using multiple sets of training sets and test sets obtained by random division to evaluate the performance of the model on different sample sets.

[0030] S403. Calculating empirical risk, predictive power, and generalization ability During the training and testing process, we usually calculate two important indicators: empirical risk and generalization ability.

[0031] Empirical risk: Empirical risk refers to the error of the model on the training set. We hope that the empirical risk is as small as possible. Specifically, the empirical risk can be calculated by the mean square error (MSE): (i from 1 to N) Where N is the number of samples in the training set. is the actual value of the ith sample, is the model's predicted value for the i-th sample.

[0032] Predictive Power: Predictive power refers to how well a model performs on new, unknown data. We usually use a test set to evaluate the predictive power of a model. If the model's error on the test set is also small, then we say that the model has good predictive power. Similarly, if our model is a regression model, then the predictive power can be calculated by the mean squared error (MSE) on the test set: (i from 1 to M) Where M is the number of samples in the test set, is the actual value of the ith sample, is the model's predicted value for the i-th sample.

[0033] The generalization ability of a model depends on the error of the model on the training set and the test set, that is, it depends on the comprehensive performance of the model's empirical risk and predictive ability.

[0034] Through the above steps, we can evaluate the empirical risk, predictive ability and generalization ability of the model, and thus evaluate the overall performance of the model. This is the process of exploring the generalization ability of the model.

[0035] S5. Use local interpretation methods to rank the contribution of each input factor in the annual runoff and annual sediment transport simulation process to reveal the main driving factors affecting the water and sediment changes in the basin.

[0036] The SHAP interpretability method can explain the prediction results of a sample by a local linear approximation of the model output. The advantage of this method is that it is flexible and applicable to any model. The SHAP interpretability method can also explain activity predictions from complex machine learning models and show the positive or negative impact of each variable on the model.

[0037] In this paper, SHAP is used to interpret different machine learning models and identify the contribution of different characteristic factors in the model to the change of runoff sediment. , SHAP can use the explanatory model Y to determine the contribution of each variable to the model F. Its basic principle is as follows:

[0038]

[0039] in, represents the number of input variables, Indicates the degree of simplification of variables, Indicates the contribution of each variable to the machine learning model, Difference notation for representing set operations.

[0040] Example: A 20-year time series from 2000 to 2020 was selected as the research period. First, the data of river hydrological factors (such as rainfall, backflow, etc.), meteorological factors (such as temperature, air pressure, etc.), vegetation factors (such as vegetation coverage, biological productivity, etc.), and human activity factors (such as soil and water conservation measures, land use patterns, population distribution, etc.) were collected, and the time series change trends of each factor were obtained after preprocessing. Through linear regression analysis, several factors with the highest correlation with runoff and sediment transport were determined as input factors of the machine learning model, including annual rainfall, annual average temperature, vegetation coverage, soil and water conservation area, population density, etc.

[0041] Then, machine learning models such as support vector regression (SVR), random forest regression (RFR), BP neural network (BPNN), extreme gradient boosting regression (XGB), k nearest neighbor regression (kNNR) and guided aggregation regression (BAR) were selected for training and testing. The preprocessed data was grouped into 80% training set and 20% test set, and the model was run to obtain the performance of the model on the training set and test set.

[0042] Then, the performance of each model is evaluated using evaluation indicators such as Nash efficiency coefficient (NSE), root mean square error (RMSE), mean absolute error (MAE), mean square error (MSE) and consistency index (d). It is assumed that BP neural network (BPNN) has the highest performance evaluation on the training set and test set, with a cumulative score of 5.8.

[0043] In order to test the robustness and generalization performance of the model, 100 random groupings were performed, and training and testing were repeated. The NSE values ​​of 100 data sets for each model were calculated. It was tentatively determined that the BP neural network (BPNN) model had the strongest generalization ability.

[0044] Finally, the LIME method was used to rank the contribution of each input factor in the annual runoff and annual sediment transport simulation process. The results showed that factors such as annual rainfall, annual average temperature, and vegetation coverage have the greatest impact on the changes in water and sediment in the basin.

[0045] At the same time, the trained BP neural network (BPNN) model was used to predict the runoff and sediment transport in the next five years. The predicted results were consistent with the actual observed values, proving that this method has high accuracy in practical applications.

[0046] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0047] It should be understood that the detailed description of the technical solutions of the present invention by means of the preferred embodiments is illustrative rather than restrictive. A person skilled in the art may modify the technical solutions described in the embodiments, or replace some of the technical features by equivalents, based on reading the specification of the present invention; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation, characterized by: The method includes: Clarify the time series change trends of long-term river hydrological factors, meteorological factors, vegetation factors, human activity factors and factors related to runoff and sediment transport changes; Identify the correlation between each factor and runoff sediment changes, and select important influencing factors, i.e. significant variables, as input factors of the machine learning model; Evaluate the performance of different existing machine learning models in simulating annual runoff and annual sediment transport; Explore the generalization ability of the model by randomly dividing the sample data set to eliminate the influence of limited sample size and sample order on the simulation results, and repeatedly train and test each model to determine the empirical risk, predictive ability and generalization ability of each model; The local interpretation method is used to rank the contribution of each input factor in the annual runoff and annual sediment transport simulation process, revealing the main driving factors affecting the water and sediment changes in the basin.

2. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The model input data is preprocessed by normalization and scale conversion, and the data set is grouped according to the grouping rule of 80% training set and 20% test set to improve the operation efficiency and simulation accuracy of the model.

3. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The selected machine learning models include support vector regression (SVR), random forest regression (RFR), BP neural network (BPNN), extreme gradient boosting regression (XGB), k nearest neighbor regression (kNNR) and bootstrapped agglomeration regression (BAR).

4. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The method described determines the performance of the model based on different evaluation indicators, sorts the models, and gives a cumulative score for each criterion.

5. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The method described uses random grouping to repeatedly train and test the data to determine the empirical risk and generalization ability of each model.

6. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The method described uses a local interpretation method to rank the contribution of each input factor in the runoff and sediment transport simulation process to reveal the main driving factors affecting water and sediment changes in the basin.

7. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: Model generalization capabilities include: The sample data set is randomly divided into a training set and a test set; 80% of the samples are divided into the training set, and the remaining 20% ​​of the samples are divided into the test set; Use the model to train the training set samples, and use the model to test the model on the test set samples; Calculate empirical risk, predictive power, and generalization ability, Empirical risk refers to the error of the model on the training set. Specifically, it is calculated by the mean square error: (i from 1 to N); Where N is the number of samples in the training set. is the actual value of the ith sample, is the model's predicted value for the i-th sample; Predictive ability refers to the performance of the model on new, unknown data. The predictive ability of the model is evaluated using the test set, which is calculated by the mean square error on the test set: (i from 1 to M); Where M is the number of samples in the test set, is the actual value of the ith sample, is the model's predicted value for the i-th sample; Through the above steps, the empirical risk and predictive ability of the model are evaluated, and then the generalization ability of the model is evaluated by comprehensively considering the empirical risk and predictive ability of the model.

8. The method of exploring the performance and generalization ability of multiple machine learning models in runoff and sediment simulation according to claim 1, characterized in that: The local interpretation method includes: By training the ML model F and input variables SHAP uses the explanatory model Y to determine the contribution of each variable to model F; its basic principle is as follows: ； ； in, represents the number of input variables, Indicates the degree of simplification of variables, Indicates the contribution of each variable to the machine learning model, Difference notation for representing set operations.

Images