A basin runoff forecasting method based on non-negative matrix factorization and machine learning
Through non-negative matrix decomposition and clustering technology, the lack of physical interpretability and data sensitivity in machine learning runoff forecasting is solved, the accuracy and stability of runoff forecasting are improved, and prediction results that are more in line with physical laws are achieved.
Patent Information
- Application Number
- CN202410076588.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-01-18
AI Technical Summary
Existing runoff forecasting methods, especially machine learning methods, lack physical interpretability, high data demands and noise-sensitiveness, resulting in limited prediction accuracy and stability.
The basin runoff forecasting method is adopted using non-negative matrix decomposition and machine learning. The input data set is screened by calculating the correlation of historical hydrological data, and the feature factors are extracted and clustered, reducing the data dimensions, and enhancing the physical explanation of the forecast. and model stability.
It improves the physical interpretability and accuracy of runoff forecasts, reduces the impact of noise data, and enhances the prediction capability and stability of machine learning models.
Smart Images

Figure CN117892162B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of watershed hydrological models, and in particular to a watershed runoff forecasting method based on non-negative matrix decomposition and machine learning. Background Art
[0002] Runoff forecasting is an important research topic in hydrology and is of great significance for water resources management, flood prevention and disaster reduction, etc. Traditional runoff forecasting methods are mainly based on statistical methods, such as regression analysis and time series analysis. These methods can reflect the laws of historical data to a certain extent, but for complex hydrological and meteorological systems, their prediction accuracy and stability are limited. In recent years, with the development of machine learning technology, more and more scholars have begun to apply machine learning methods to runoff forecasting, such as support vector machines and neural networks. These methods can better reflect the nonlinear characteristics of hydrological and meteorological systems and improve the accuracy and stability of runoff forecasting.
[0003] However, there are also some problems with machine learning methods in runoff forecasting. First, the prediction results of machine learning methods lack physical interpretability, and it is difficult to explain the formation process of the prediction results. For example, in some methods of screening runoff forecast characteristic factors, some of the screened characteristic factors are negative, which obviously violates the physical laws of runoff formation. Secondly, machine learning methods require a large amount of data for training, while the acquisition and processing of hydrological and meteorological data is difficult and the amount of data is limited. Finally, machine learning methods are sensitive to noise data, which will affect the prediction accuracy of the model.
[0004] Therefore, a method is needed to improve the interpretability of runoff forecasts based on machine learning models, reduce data dimensions, filter out noisy data, and improve the accuracy of runoff forecasts. Summary of the invention
[0005] The purpose of the present invention is to provide a basin runoff forecasting method based on non-negative matrix decomposition and machine learning. The calculation process and results of this method are more in line with the physical laws of runoff, enhance the interpretability of runoff forecasting, and make the prediction results easier to understand and explain; at the same time, this method effectively reduces noise data and reduces data dimension through non-negative matrix decomposition and clustering process, thereby improving the accuracy and efficiency of runoff forecasting of machine learning models.
[0006] To achieve the above object, the present invention provides a basin runoff forecasting method based on non-negative matrix decomposition and machine learning, comprising:
[0007] Collecting historical hydrological data of the basin, calculating the correlation between the historical hydrological data, and filtering the input data set by the calculation results;
[0008] Performing non-negative matrix decomposition on the input data set, extracting a set of characteristic factors of the historical hydrological data, and obtaining a clustering matrix;
[0009] The clustering matrix is used as input data of the machine learning model, and the corresponding runoff prediction result is output.
[0010] Preferably, the historical hydrological data includes:
[0011] Runoff, precipitation, potential evapotranspiration, soil moisture, air temperature, snow cover, and snow depth.
[0012] Preferably, screening the input data set by calculation results comprises:
[0013] The correlation between different potential factors and runoff data is calculated based on the Pearson coefficient method. Among a group of variables with high correlation, one variable is selected and concatenated with other variables with low correlation as the input variable set of non-negative matrix decomposition, that is, the input data set; wherein, when the absolute value of the Pearson coefficient is higher than a preset threshold, it is determined that the correlation is high.
[0014] Preferably, the method for calculating the correlation between the variables based on the Pearson coefficient method is:
[0015]
[0016] In the formula, r represents the Pearson correlation coefficient between variables M and N, cov(M,N) represents the covariance of variables M and N, and σ M represents the standard deviation of the variable M, σ N represents the standard deviation of variable N.
[0017] Preferably, performing non-negative matrix decomposition on the input data set comprises:
[0018] Randomly decompose the historical hydrological data matrix into a non-negative matrix one and a non-negative matrix two, so that the historical hydrological data matrix is approximately equal to the product of the non-negative matrix one and the non-negative matrix two;
[0019] The non-negative matrix one and the non-negative matrix two are updated based on the historical hydrological data matrix until the convergence condition is met, wherein the historical hydrological data matrix is a matrix obtained by splicing and merging the runoff data and the historical hydrological data.
[0020] Preferably, updating the non-negative matrix 1 and the non-negative matrix 2 comprises:
[0021] The non-negative matrix one and the non-negative matrix two are updated by calculating the difference between the data set and the product of the non-negative matrix one and the non-negative matrix two, and the updating method is to minimize the error function based on the gradient descent method;
[0022] The matrix update formula is:
[0023]
[0024]
[0025] Where W is the non-negative matrix one, H is the non-negative matrix two, E is the error function, and α is the learning rate.
[0026] Preferably, obtaining the clustering matrix includes:
[0027] According to the updated non-negative matrix 1 and the non-negative matrix 2, each sample in the matrix is assigned to the cluster represented by the column where the corresponding maximum value is located, so as to form the cluster matrix;
[0028] The quality index of each cluster is calculated, and the optimal number of clusters is determined by comparing the cluster quality indexes under different numbers of clusters. The historical hydrological data are grouped according to the optimal number of clusters, and the cluster matrix is output.
[0029] Preferably, outputting the corresponding runoff prediction result includes:
[0030] Based on experience, a reasonable model parameter range and training and validation periods are given, and runoff prediction is performed on each set of parameters through a machine learning model. Finally, the model automatically calibrates and outputs a set of parameters and runoff prediction results with the best accuracy during the validation period.
[0031] Preferably, the method for calculating the runoff prediction result is:
[0032]
[0033] Among them, Q F represents the predicted runoff, f represents the activation function, x j represents the jth eigenvalue of the input, ω j represents the weight corresponding to the jth feature, b represents the bias term, and k is the number of cluster centers.
[0034] Compared with the prior art, the present invention has the following advantages and technical effects:
[0035] In the non-negative matrix decomposition method of the present invention, both the decomposition matrix and the obtained clustering matrix are non-negative in the whole process, which is more in line with the law of runoff formation process and enhances the physical interpretability of runoff forecast;
[0036] The present invention extracts characteristic factors of historical hydrological data through non-negative matrix decomposition and completes clustering, effectively reducing noise data, lowering data dimension, and enhancing the predictive ability and stability of the machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0038] Figure 1 This is a flow chart of a method for watershed runoff forecasting based on non-negative matrix decomposition and machine learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0040] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0041] The present invention proposes a basin runoff forecasting method based on non-negative matrix decomposition and machine learning. The present invention performs non-negative matrix decomposition based on historical hydrological data, randomly generates a mixing matrix and a signal matrix, continuously updates the two matrices by reconstructing the difference, splices the final updated signal matrix with the runoff data, and inputs the machine learning model to realize the simulation and prediction of basin runoff. Figure 1 As shown, the technical solution of the present invention specifically includes the following steps:
[0042] Step 1: Collect historical hydrological data, including runoff, precipitation, potential evapotranspiration, soil moisture, temperature, snow cover and snow depth, and calculate the correlation between variables using the Pearson correlation coefficient method, retaining the input data set X with weak correlation:
[0043] In order to better understand the hydrological and meteorological conditions of the basin, it is necessary to collect historical data, including multiple indicators such as runoff, precipitation and potential evapotranspiration. By collecting these data, we can better understand the hydrological and meteorological characteristics of the basin, and then use the Pearson correlation coefficient method to calculate the correlation between different potential influencing factors and runoff data. When the absolute value of the Pearson coefficient is higher than 0.7, it is considered that the potential influencing factor has a high correlation with the runoff data. The influencing factors with higher correlation are spliced as the input variable set of non-negative matrix decomposition. The Pearson correlation coefficient calculation formula is as follows:
[0044]
[0045] In the formula, r represents the Pearson correlation coefficient between variables M and N, cov(M,N) represents the covariance of variables M and N, and σ M represents the standard deviation of the variable M, σ N represents the standard deviation of variable N.
[0046] Step 2: Perform non-negative matrix decomposition on the data set X retained in step 1 to extract the characteristic factor set W' of historical hydrological data.
[0047] The specific operation process includes:
[0048] (1) A non-negative matrix W and a non-negative matrix H are randomly generated from the historical hydrological data, so that the historical hydrological data matrix is approximately equal to the product of the two non-negative matrices W and H, which can be expressed as follows:
[0049]
[0050] Where X is the input historical hydrological data matrix, whose size is m*n; W is the feature matrix of the sample, whose size is m*k; H is the feature weight matrix, whose size is k*n, m is the number of samples, n is the number of features, and k is the number of cluster centers.
[0051] (2) Update W and H by the difference between X and W×H until the error function between X and W×H meets the convergence condition;
[0052] The updating of matrices W and H is mainly divided into two processes. The first process is to update H, fix W, and update H by minimizing the error function; the second process is to update W and fix H. Both processes use the gradient descent method to minimize the error function to update. Repeat these two processes until the error function converges (or reaches the maximum number of iterations). The error function formula and matrix update formula are as follows:
[0053] E=||X-WH|| 2 (3)
[0054]
[0055]
[0056] Where E is the error function, ‖.‖ is the Frobenius norm, and а is the learning rate.
[0057] (3) According to the updated W and H matrices, each sample is assigned to the cluster represented by the column where its corresponding maximum value is located, forming a clustering matrix, that is, the characteristic factor set W':
[0058] During the clustering process, it is necessary to calculate the quality indicators of each cluster, such as the silhouette coefficient, Davies-Bouldin index, etc. By comparing the clustering quality indicators under different cluster numbers, the optimal number of clusters is determined, and the historical hydrological data are grouped according to the optimal number of clusters, and the clustering results are output.
[0059] The silhouette coefficient calculation formula is as follows:
[0060]
[0061] In the formula, i represents the i-th data point, S i is the silhouette coefficient, a i represents the average distance between the i-th data point and other data points in the same cluster, b i represents the minimum value of the average distance between the i-th data point and all data points in other clusters, S i The value range of S is [-1,1]. i The closer it is to 1, the more reasonable the cluster to which data point i belongs, and vice versa.
[0062] Step 3: Use the clustering matrix W' obtained in step (3) of step 2 as the input data of the machine learning model (such as a convolutional neural network), give the machine learning model parameter range, and automatically calibrate the optimal parameters to output the corresponding runoff prediction results:
[0063] The clustering matrix W' is used to effectively reduce the data dimension and simplify the data processing difficulty and parameter setting of the machine learning model. The input data of the machine learning model is divided into the training period and the validation period in a ratio of 7:3, and the Bayesian optimization algorithm is used to optimize the hyperparameters of the machine learning model.
[0064] Finally, the model selects the optimal set of hyperparameters for runoff prediction. Taking the neural network as an example, its calculation formula can be expressed as:
[0065]
[0066] In the formula, Q F represents the predicted runoff, f represents the activation function, x j represents the jth eigenvalue of the input, ω j represents the weight corresponding to the jth feature, and b represents the bias term.
[0067] In the prior art, when extracting characteristic factors, the characteristic factors screened out are generally not non-negative (such as principal component analysis). However, in the non-negative matrix decomposition method of the present invention, both the decomposition matrix and the obtained clustering matrix are non-negative in the entire process, which is more in line with the law of runoff formation process and enhances the physical interpretability of runoff forecast.
[0068] The existing technology generally directly uses physical hydrological models or machine learning models to simulate and predict runoff. The present invention extracts the characteristic factors of historical hydrological data through non-negative matrix decomposition and completes clustering, which effectively reduces noise data, reduces data dimension, and enhances the predictive ability and stability of the machine learning model.
[0069] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A basin runoff forecasting method based on non-negative matrix decomposition and machine learning, characterized in that: include: Collecting historical hydrological data of the basin, calculating the correlation between the historical hydrological data, and filtering the input data set by the calculation results; Performing non-negative matrix decomposition on the input data set, extracting a set of characteristic factors of the historical hydrological data, and obtaining a clustering matrix; The specific operation process of extracting the characteristic factor set W' of historical hydrological data includes: (1) A non-negative matrix W and a non-negative matrix H are randomly generated from the historical hydrological data, so that the historical hydrological data matrix is approximately equal to the product of the two non-negative matrices W and H, which can be expressed as follows: In the formula, X is the input historical hydrological data matrix, whose size is m*n, W is the feature matrix of the sample, whose size is m*k; H is the feature weight matrix, whose size is k*n, m is the number of samples, n is the number of features, and k is the number of cluster centers; (2) Update W and H by the difference between X and W×H until the error function between X and W×H meets the convergence condition; Updating matrices W and H is mainly divided into two processes. The first process is to update H, fix W, and update H by minimizing the error function; the second process is to update W, fix H, and update H by minimizing the error function; The above two processes both use the gradient descent method to minimize the error function to update; repeat the above two processes until the error function meets the convergence condition; The error function formula is: E=||X-WH|| 2 ; The matrix update formula is: In the formula, H u W represents the matrix obtained after H matrix is updated by the formula; u represents the matrix obtained after the W matrix is updated by the formula, E is the error function, ‖.‖ is the Frobenius norm, and α is the learning rate; (3) According to the updated H u , W u Matrix, assign each sample to the cluster represented by the column where its corresponding maximum value is located, forming a clustering matrix, that is, the characteristic factor set W'; Using the clustering matrix as input data of a machine learning model, and outputting corresponding runoff prediction results; Output the corresponding runoff prediction results, including: Based on experience, a reasonable model parameter range, training period and validation period are given, and runoff prediction is performed for each set of parameters through a machine learning model. Finally, the model automatically calibrates and outputs a set of parameters and runoff prediction results with the best accuracy during the validation period. The method for calculating the runoff prediction result is: Among them, Q F represents the predicted runoff, f represents the activation function, x j represents the jth eigenvalue of the input, ω j represents the weight corresponding to the jth feature, b represents the bias term, and k is the number of cluster centers.
2. The basin runoff forecasting method based on non-negative matrix decomposition and machine learning according to claim 1 is characterized in that: The historical hydrological data include: Runoff data, precipitation, potential evapotranspiration, soil moisture, air temperature, snow cover and snow depth.
3. The basin runoff forecasting method based on non-negative matrix decomposition and machine learning according to claim 2 is characterized in that: The input data set is filtered by calculation results, including: Taking precipitation, potential evapotranspiration, soil moisture, air temperature, snow cover and snow depth as the characteristic variable set, the correlation between different characteristic variables and runoff data, as well as the correlation between different characteristic variables are calculated based on the Pearson coefficient method, and several characteristic variables with higher correlation are selected as the input variable set of non-negative matrix decomposition, namely, the input data set; wherein, when the absolute value of the Pearson coefficient is higher than a preset threshold, it is determined that the correlation is high.
4. The basin runoff forecasting method based on non-negative matrix decomposition and machine learning according to claim 3 is characterized in that: The method for calculating the correlation between different feature variables based on the Pearson coefficient method is: In the formula, r represents the Pearson correlation coefficient between variables M and N, cov(M,N) represents the covariance of variables M and N, and σ M represents the standard deviation of the variable M, σ N represents the standard deviation of variable N.
