Mountain torrent prediction method based on interpretable machine learning and mixed runoff production model

Abstract

The invention belongs to the technical field of application hydrology, and particularly relates to a mountain torrent prediction method based on interpretable machine learning and a mixed runoff production model, which comprises the steps of collecting attribute features of a small watershed of a target area, performing feature processing and dimension reduction by using a k-means clustering method and a principal component analysis method, constructing the mixed runoff production model, and predicting the runoff production of the target area. And carrying out parameter transplantation and optimization on the model by utilizing a decision tree, an extreme gradient lifting tree and a lightweight gradient lifting machine learning method. And finally, explaining model output based on a shop value, and determining an optimal parameter transplantation mode, thereby realizing accurate prediction of the mountain torrent risk of the target area. The method not only improves the prediction precision, but also enhances the interpretability and generalization ability of the model, and provides powerful technical support for mountain torrent disaster prevention and control.

Description

Technical Field

[0001] The present invention belongs to the technical field of applied hydrology, and in particular relates to a flash flood prediction method based on interpretable machine learning and a hybrid runoff generation model. Background Art

[0002] Flash flood disaster is a sudden and wide-ranging natural disaster that has brought huge economic losses and casualties to human society. In order to effectively reduce the harm of flash flood disasters, flash flood forecasting technology has become one of the key areas of current research.

[0003] At present, flash flood forecasting mainly relies on two types of models: physical mechanism models and data-driven models. The physical mechanism model is based on the principles of hydrology and hydrodynamics and can accurately describe the physical mechanism of the hydrological process in the basin. However, due to the complexity of flash flood processes, these models often face problems such as difficulty in parameter acquisition, complex calculations, and difficulty in ensuring model accuracy in practical applications. On the other hand, data-driven models, such as models based on machine learning, can use a large amount of historical data for modeling and have high prediction accuracy, but their "black box" characteristics lead to a lack of interpretability of the model, and it is impossible to deeply understand and explain the mechanism of flash floods. Summary of the invention

[0004] The purpose of the present invention is to provide a flash flood prediction method based on interpretable machine learning and a hybrid runoff model to solve the problems raised in the background technology.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The present invention proposes a flash flood prediction method based on interpretable machine learning and a hybrid runoff generation model, the method comprising:

[0007] Collect the attribute characteristics of small watersheds in the target area, combine the k-means clustering method to form K watershed categories, perform dimensionality reduction processing on the attribute characteristics in the watershed categories based on the principal component analysis method, and obtain watershed parameters through parameter transplantation;

[0008] A mixed runoff model is constructed based on the modeling platform, and the parameters of the mixed runoff model are calibrated using the basin parameters. The mixed runoff model after parameter calibration is transplanted based on the decision tree, extreme gradient boosting tree and lightweight gradient boosting machine learning method to obtain a flash flood prediction model under different parameter transplantation conditions;

[0009] The output of the flash flood prediction model is interpreted based on the shap value, the optimal parameter transplantation method is determined based on the contribution of attribute characteristics to the flash flood risk, and the flash flood prediction model under the optimal parameter transplantation method is used to predict flash floods in the target area.

[0010] Furthermore, the k-means clustering method is combined to form K watershed categories, and the attribute features in the watershed categories are reduced in dimension based on the principal component analysis method, and the parameters of each category of watershed are obtained, including:

[0011] The silhouette coefficient is determined based on the intra-class distance and extra-class distance of each small watershed to the center of the category to which it belongs, and the category K value of the best watershed is determined based on the size of the silhouette coefficient;

[0012] The covariance matrix of the attribute characteristics in the K watershed categories is calculated, and the eigenvalue decomposition is performed to obtain the corresponding eigenvectors and eigenvalues. The principal components are sorted according to the size of the eigenvalues, and the sorted principal components are screened based on the set variance contribution rate. The obtained principal components are used for parameter transplantation analysis.

[0013] Furthermore, the hybrid runoff model is constructed based on the modeling platform, and the parameters of the hybrid runoff model are calibrated using the watershed parameters, including:

[0014] A hybrid runoff model including a rainfall-runoff process simulation module and a flood forecasting module is constructed based on the OMS platform, the watershed parameters are used as model input parameters, and the model simulation results are output;

[0015] Combined with the SCE parameter optimization algorithm, the parameters of the hybrid runoff model are optimized based on the actual observation data and simulation results obtained in advance.

[0016] Furthermore, the parameter transplantation of the mixed flow generation model after parameter calibration based on the decision tree includes:

[0017] The CART algorithm is used to construct a binary tree decision tree for the training set. Starting from the root node, based on the principle of maximizing information gain, the optimal input variables and split points are selected to recursively split the training set; wherein the training set is the attribute characteristics of the small watershed in the target area;

[0018] The above process is repeated until the preset stopping condition is met to form a complete decision tree structure. At the leaf nodes of the decision tree, the classification or regression results are output to optimize the parameters of the mixed runoff model.

[0019] Furthermore, the parameter transplantation of the mixed flow generation model after parameter calibration based on the extreme gradient boosting tree includes:

[0020] The grid search optimization method and 3-fold cross validation method in sklearn are used to optimize the parameters of the hybrid flow generation model based on the extreme gradient boosting tree. The specific parameters of the parameter optimization include the maximum tree depth, learning rate and number of decisions of the tree.

[0021] For the classification tree of the extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distances of the first principal component of watershed structure, the fourth principal component of soil texture, the first principal component of meteorological attributes, and the third principal component of soil texture between watersheds were determined in turn;

[0022] For the regression tree of extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distances of the first principal component of watershed structure, the first principal component of soil texture, the second principal component of meteorological attributes, and the absolute distance of watershed space between watersheds were determined;

[0023] The similarity between the watersheds is determined according to the absolute distance between the watersheds, a target watershed is determined based on the similarity, and the maximum tree depth, the learning rate and the number of decisions are applied to the target watershed to achieve parameter transplantation.

[0024] Furthermore, the parameter transplantation of the mixed flow generation model after parameter calibration based on the lightweight gradient boosting algorithm includes:

[0025] (1) Initialize the flash flood prediction model:

[0026]

[0027] (2) When the number of iterations is m∈[1,M], calculate the negative gradient at this time:

[0028]

[0029] (3) Fitting the base learner:

[0030]

[0031] (4) Update flash flood prediction model:

[0032] f m (h) = f m-1 (h)+ρ m h(x i :a m )

[0033] Where: M is the number of base learners, N is the size of the training data set; h(·) represents the expression of the base learner of GBDT, i.e., the decision tree; a m , ρ m Distribution is the hyperparameters and weights of the mth base learner.

[0034] The beneficial effects of the present invention are:

[0035] 1. The present invention uses decision trees, extreme gradient boosting trees and lightweight gradient boosting algorithms to transplant and optimize model parameters. Finally, the model output is interpreted based on the SHAP value to determine the optimal parameter transplantation method, thereby achieving accurate prediction of flash flood disasters in the target area. This method not only improves the prediction accuracy, but also enhances the interpretability and generalization ability of the model, providing strong technical support for early warning and prevention and control of flash flood disasters.

[0036] 2. The present invention applies the extreme gradient boosting tree and lightweight gradient boosting algorithm to parameter transplantation in data-deficient watersheds for the first time, wherein the lightweight gradient boosting algorithm is used to obtain the parameters of the hybrid runoff model from small watersheds in hilly areas with data, which can be used for flood forecasting of this model in small watersheds in hilly areas with lack of data. Moreover, the flood forecasting ability of the hybrid runoff model based on the optimal parameter transplantation is better than that of the parameter transplantation using spatial proximity, physical similarity and multi-contribution methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of a flash flood prediction method based on interpretable machine learning and a hybrid runoff model according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment proposes a flash flood prediction method based on interpretable machine learning and a hybrid runoff model, and the method includes the following steps:

[0041] S1. Collect the attribute characteristics of small watersheds in the target area. These characteristics mainly include land use type, soil texture, watershed topology, climate attributes and other aspects. Combined with the k-means clustering method, K watershed categories are formed. Based on the principal component analysis method, the attribute characteristics in the watershed category are processed to generate the watershed parameters after dimensionality reduction.

[0042] In step S1, K watershed categories are formed by combining the k-means clustering method, and the attribute characteristics in the watershed category are reduced in dimension based on the principal component analysis method, and the parameters of each watershed are obtained, including:

[0043] The silhouette coefficient is determined based on the intra-class distance and extra-class distance from each small watershed to the center of its category, and the category K value of the best watershed is determined based on the size of the silhouette coefficient (e.g., the value range of the silhouette coefficient is -1 to 1, where values ​​close to 1 indicate a better clustering effect. The optimal number of clusters is determined by adjusting the K value and repeated experiments); the covariance matrix of the attribute features in the K watershed categories is calculated, and the eigenvalue decomposition is performed to obtain the corresponding eigenvectors and eigenvalues, and the principal components are sorted according to the size of the eigenvalues. The sorted principal components are screened based on the set variance contribution rate, and the data matrix is ​​used as the watershed parameters.

[0044] As an example, in the principal component analysis method of this embodiment, the most important principal components are then screened out based on the condition that the cumulative variance contribution rate exceeds 80%, and the original 45 attribute features are reduced to 15, thereby retaining the main information in the data and reducing the data dimension, providing a simplified and representative set of attribute factors for subsequent analysis. The covariance matrix C formula of the 45 attribute features is as follows:

[0045]

[0046] in, is the mean vector of the data. Next, perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalues ​​and corresponding eigenvectors. Sort the principal components according to the size of the eigenvalues, and select the first few principal components with cumulative variance contribution exceeding 80%. These principal components represent the most important information in the original data. The final dimensionality reduction process can be expressed by the following formula:

[0047] Z=XW

[0048] Among them, Z is the data matrix after dimension reduction, X is the original data matrix, and W is the matrix composed of the eigenvectors of the selected principal components.

[0049] It should be noted that when selecting watersheds for parameter transplantation, the first consideration is the similarity of watershed structures (including watershed area, total length of river channels in the watershed, average length of river channels in the watershed, and average area of ​​sub-basins); the second consideration is the similarity of watershed meteorological attributes (including multi-year average precipitation, multi-year average temperature, and maximum precipitation in 24 hours); and then the similarity of watershed topological structure (six attributes in total, including watershed average elevation, watershed minimum elevation, watershed maximum elevation, watershed average slope, watershed area ratio with slope greater than 3°, and watershed area ratio with slope greater than 15°), soil texture, and land use type.

[0050] S2. A hybrid runoff model is constructed based on the modeling platform. The parameters of the hybrid runoff model are calibrated using measured data. The parameters of the hybrid runoff model after parameter calibration are transplanted based on decision tree (CART), extreme gradient boosting tree (XGBoost) and lightweight gradient boosting machine learning method (LightGBM) to obtain the flash flood prediction model under different parameter transplantation.

[0051] More specifically, a hybrid runoff model is constructed based on the modeling platform, and the measured data is used to calibrate the parameters of the hybrid runoff model, including: constructing a hybrid runoff model including a rainfall-runoff process simulation module and a flood forecasting module based on the OMS platform, using the measured data as the model input data, and outputting the model simulation results; combining the SCE parameter optimization algorithm, and optimizing the parameters of the hybrid runoff model based on the actual observation data and model simulation results obtained in advance.

[0052] As an example, when calibrating the model parameters of 110 small watersheds, the basic data of these watersheds were first input into the hybrid runoff model system (MRHMS). Then, the model parameters were calibrated by using the optimization algorithm using historical observation data and simulation results. That is, on the basis of the SCE parameter optimization algorithm, the parameters were optimized step by step, and the SCE multi-step optimization algorithm was coupled with the hybrid runoff model within the OMS modular modeling framework. These algorithms usually adjust the parameter values ​​step by step to reduce the error based on the error between the simulation results and the actual observations. This process usually requires multiple iterations to ensure the accuracy and reliability of the model parameters. The data set is divided into two parts: calibration and verification. After optimizing the model parameters on the calibration data set, an independent verification data set is used to verify the simulation effect. The model after parameter calibration was used for flood simulation, and the simulation effect after parameter transplantation was verified by comparing it with the measured data. The median NSE values ​​were 0.74 and 0.72, respectively, and the median flood peak error in the calibration period and the verification period was -0.10. This shows that the simulation effect of MRHMS is very good.

[0053] Further preferably, the hybrid runoff model after parameter calibration is transplanted based on the decision tree, including:

[0054] The CART algorithm is used to construct a binary tree decision tree for the training set. Starting from the root node, based on the principle of maximizing information gain, the optimal input variables and split points are selected to recursively split the training set. The training set is the attribute characteristics of the small watershed in the target area.

[0055] The above process is repeated until the preset stopping condition is met to form a complete decision tree structure. At the leaf nodes of the decision tree, the classification or regression results are output to optimize the parameters of the mixed runoff model.

[0056] Further preferably, parameter transplantation is performed based on the extreme gradient boosting tree, including:

[0057] First, the principal components (PCs) between watersheds are used to measure the similarity between watersheds, so that the model parameters of one watershed are applied to another watershed. This method determines the similarity between different watersheds by calculating the distance of the principal components, thereby realizing the migration of parameters.

[0058] The grid search optimization method and 3-fold cross validation method in sklearn are used to optimize the parameters of the hybrid flow-generating model based on the extreme gradient boosting tree. The specific optimization parameters include the maximum tree depth, learning rate and number of decisions of the tree; more specifically, the three hyperparameters of the maximum tree depth (max_depth, optimization range 1 to 10), learning rate (learning_rate, optimization range 0.01 to 0.15) and number of decisions (n_estimators, i.e. the number of decision trees constructed by the XGBoost model, optimization range 50 to 500) are optimized; the optimal hyperparameters of the XGBoost classification tree obtained are as follows: the maximum depth (max_depth) is 3, the learning rate (learning_rate) is 0.01, and the number of decision trees (n_estimators) is 200; the optimal hyperparameters of the XGBoost regression tree are as follows: the maximum depth (max_depth) is 3, the learning rate (learning_rate) is 0.01, and the number of decision trees (n_estimators) is 50.

[0059] For the classification tree of the extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distances of the first principal component of the watershed structure (PC1_Structure), the fourth principal component of soil texture (PC4_Soil), the first principal component of meteorological attributes (PC1_Meto), and the third principal component of soil texture (PC3_Soil) are judged in turn; more specifically, this embodiment selects the last tree in the classification tree constructed by the XGBoost model, that is, the 200th tree. In the XGBoost tree structure, the larger the value obtained in the leaf, the better the result of the model prediction. That is, when performing parameter transplantation between watersheds, the first thing to judge is whether the absolute distance of PC1_Structure between watersheds is less than 0.744, followed by judging the absolute distances of PC4_Soil and PC1_Meto between watersheds, and finally judging the absolute distance of PC3_Soil between watersheds.

[0060] For the regression tree of the extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distance of the first principal component of the watershed structure, the first principal component of the soil texture, the second principal component of the meteorological attribute, and the absolute distance of the watershed space between the watersheds are determined; more specifically, the last tree in the regression tree constructed by the XGBoost model, that is, the 50th tree. In the XGBoost tree structure, the larger the value obtained in the leaf, the better the model prediction result. That is, when performing parameter transplantation between watersheds, first determine whether the absolute distance of PC1_Structure between watersheds is less than 0.744, then determine the absolute distance of PC1_Soil between watersheds, and finally determine the absolute distance of PC2_Meto between watersheds and the absolute distance of watershed space (DeltaXY).

[0061] The similarity between watersheds is judged according to the absolute distance between watersheds, and the target watershed is determined based on the similarity. The maximum tree depth, learning rate and number of decisions are applied to the target watershed to achieve parameter transplantation.

[0062] Further preferably, the parameters of the mixed flow generation model after parameter calibration are transplanted based on the lightweight gradient boosting algorithm, including:

[0063] First, initialize the flash flood prediction model:

[0064]

[0065] Secondly, when the number of iterations is m∈[1,M], calculate the negative gradient at this time:

[0066]

[0067] Refit the base learner:

[0068]

[0069] Last updated flash flood forecast model:

[0070] f m (x) = f m-1 (x)+ρ m h(x i :a m ).

[0071] Where: M is the number of base learners, N is the size of the training data set; h(·) represents the expression of the base learner of GBDT, i.e., the decision tree; a m , ρ m Distribution is the hyperparameters and weights of the mth base learner.

[0072] S3. Explain the output of the flash flood prediction model based on the shap value, determine the optimal parameter transplantation method based on the contribution of attribute characteristics to flash flood risk, and use the flash flood prediction model under the optimal parameter transplantation method to predict flash floods in the target area.

[0073] In view of the lack of interpretability in machine learning, this embodiment introduces feature contribution analysis (SHAP) to explain the model afterwards; that is, the "benefit" of the combination containing a certain feature is subtracted from the "benefit" of the combination without this feature, so as to obtain the contribution of the feature in the combination; then the contribution of all combinations is calculated, and the weighted average is used to obtain the overall contribution of the feature. After determining the optimal parameter transplantation method, the flash flood prediction model is then combined for forecasting.

[0074] According to the above-mentioned embodiment of the present invention, the present invention first organizes and summarizes the characteristic attributes of small watersheds, and uses them as input variables of different artificial intelligence models. By establishing the relationship between the small watershed attribute characteristics and the output variables, the contribution of each feature to the flash flood risk, that is, the SHAP value, is calculated, so as to perform a post-hoc interpretation of the LSTM "black box" model to reveal the contribution of different characteristic factors to inducing flash floods.

[0075] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can choose different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0076] In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0077] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0078] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A flash flood prediction method based on interpretable machine learning and hybrid runoff model, characterized in that: The method comprises: Collect the attribute characteristics of small watersheds in the target area, combine the k-means clustering method to form K watershed categories, and process the attribute characteristics in the watershed categories based on the principal component analysis method to generate watershed parameters after dimensionality reduction; A mixed runoff model is constructed based on the modeling platform, and the parameters of the mixed runoff model are calibrated using the basin parameters. The parameters of the calibrated mixed runoff model are transplanted based on the decision tree, extreme gradient boosting tree and lightweight gradient boosting algorithm, thereby obtaining flash flood prediction models with different parameters. The contribution of attribute characteristics to flash flood risk is quantitatively characterized, the optimal parameter transplantation method is determined, and the output of the flash flood prediction model is interpreted based on the shap value; and the flash flood prediction model with the optimal parameter transplantation is used to predict flash floods in the target area.

2. A flash flood prediction method based on interpretable machine learning and hybrid runoff model according to claim 1, characterized in that: The k-means clustering method is combined to form K watershed categories, and the attribute features in the watershed categories are reduced in dimension based on the principal component analysis method to generate watershed parameters, including: The silhouette coefficient is determined based on the intra-class distance and extra-class distance of each small watershed to the center of the category to which it belongs, and the category K value of the best watershed is determined based on the size of the silhouette coefficient; The covariance matrix of the attribute features in the K watershed categories is calculated, and the eigenvalue decomposition is performed to obtain the corresponding eigenvectors and eigenvalues. The principal components are sorted according to the feature size, and the sorted principal components are screened based on the set variance contribution rate. The obtained principal components are used for parameter transplantation analysis.

3. A flash flood prediction method based on interpretable machine learning and hybrid runoff model according to claim 1, characterized in that: The method of constructing a mixed runoff model based on the modeling platform and calibrating parameters of the mixed runoff model using the watershed parameters includes: A hybrid runoff model including a rainfall-runoff process simulation module and a flood forecasting module is constructed based on the OMS platform, the watershed parameters are used as model input parameters, and the model simulation results are output; In combination with the SCE parameter optimization algorithm, the parameters of the hybrid runoff model are optimized based on the actual observation data acquired in advance and the model simulation results.

4. The flash flood prediction method based on interpretable machine learning and hybrid runoff model according to claim 1, characterized in that: The parameter transplantation of the mixed flow generation model after parameter calibration based on the decision tree includes: The CART algorithm is used to construct a binary tree decision tree for the training set. Starting from the root node, based on the principle of maximizing information gain, the optimal input variables and split points are selected to recursively split the training set; wherein the training set is the attribute characteristics of the small watershed in the target area; The above process is repeated until the preset stopping condition is met to form a complete decision tree structure. At the leaf nodes of the decision tree, the classification or regression results are output to optimize the parameters of the mixed runoff model.

5. The flash flood prediction method based on interpretable machine learning and hybrid runoff model according to claim 1, characterized in that: The method of performing parameter transplantation on the mixed flow generation model after parameter calibration based on the extreme gradient boosting tree includes: The grid search optimization method and 3-fold cross validation method in sklearn are used to optimize the parameters of the hybrid flow generation model based on the extreme gradient boosting tree. The specific parameters optimized include the maximum tree depth, learning rate and number of decisions. For the classification tree of the extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distances of the first principal component of watershed structure, the fourth principal component of soil texture, the first principal component of meteorological attributes, and the third principal component of soil texture between watersheds were determined in turn; For the regression tree of extreme gradient boosting tree, when performing parameter transplantation between watersheds, the absolute distance of the first principal component of watershed structure, the first principal component of soil texture, the second principal component of meteorological attributes, and the absolute distance of watershed space were determined; The similarity between watersheds is judged according to the absolute distance between watersheds, and the target watershed is determined based on the similarity. The maximum tree depth, learning rate and number of decisions are applied to the target watershed to achieve parameter transplantation.

6. The flash flood prediction method based on interpretable machine learning and hybrid runoff model according to claim 1, characterized in that: The method of performing parameter transplantation on the mixed flow generation model after parameter calibration based on the lightweight gradient boosting algorithm includes: (1) Initialize the flash flood prediction model: (2) When the number of iterations is m∈[1,M], calculate the negative gradient at this time: (3) Fitting the base learner: (4) Update flash flood prediction model: f m (h)=f m-1 (h)+ρ m h(x i :a m ) Where: M is the number of base learners, N is the size of the training data set; h(·) represents the expression of the GBDT base learner, i.e., the decision tree; a m , ρ m Distribution is the hyperparameters and weights of the mth base learner.

Images