A water quality monitoring index prediction method based on multiple regression and random forest

Abstract

The application discloses a water quality monitoring index prediction method based on multiple regression and random forest. The application firstly analyzes the correlation degree between monitoring indexes based on various monitoring indexes of water quality monitoring by using Pearson correlation coefficient, so as to obtain the correlation coefficient between the monitoring indexes, further obtains the statistical relationship between the highly correlated parameter indexes by using a multiple linear regression algorithm, uses the form of the regression equation to express the causal relationship between the monitoring variables, and finally realizes the prediction of the dependent variable index by using the independent variable index in water quality monitoring by using a random forest algorithm, so as to reduce the monitoring items and the monitoring cost. The application can effectively improve the efficiency of water quality monitoring and reduce the cost of monitoring index items, and has the characteristics of high prediction accuracy and strong generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data and machine learning, and particularly relates to a water quality monitoring index prediction method based on multiple regression and random forest. BACKGROUND

[0002] Water quality monitoring, as an important means of maintaining water environment health, plays a crucial role in people's life needs and national ecological civilization construction. As one of the effective means of water environment protection in modern society, the application of water quality monitoring is becoming more and more widespread, and it is inevitable to develop towards informatization and automation. Since there are many types of water quality monitoring indicators and a large amount of water quality monitoring data, the data of water quality monitoring indicators have high utilization value. Extracting valuable information from a large amount of water quality monitoring indicator data and finding the essential relationship between the data can effectively improve the information utilization rate of water quality monitoring and reduce the monitoring cost.

[0003] In recent years, water quality monitoring in China has developed rapidly. When monitoring water quality, there are many types of monitoring indicators, the monitoring cost is high, and the information is huge, making it difficult to extract effective information. Effective analysis of data is imperative. To effectively reduce the monitoring cost, the monitoring indicator project is reasonably reduced. The advantage of using Pearson correlation coefficient to analyze the correlation between indicators is that it can effectively measure the strength of the relationship between variables. However, it cannot form a model to solidify the relationship between variables, and cannot use this relationship to predict data. Further regression analysis is needed. In practical applications, a parameter indicator is often affected by multiple parameter indicators. Compared with other regression models, the multiple linear regression model is easier to implement and has more practical significance.

[0004] In the practical application of water quality monitoring of various indicators, the statistical equation between highly correlated monitoring indicators obtained by using multiple linear regression can accurately know the independent variable indicators and dependent variable indicators between the indicators, and the effective prediction of the dependent variable indicators can reduce the monitoring cost. Using the regression model in the random forest algorithm to predict the monitoring indicators, the random forest regression model has the advantages of high prediction accuracy and strong generalization ability compared with other regression models. At present, the random forest algorithm has been applied to the field of water conservancy to predict water quality categories, but it has not been applied to water quality monitoring index prediction. Therefore, the application of random forest regression model in the analysis and prediction of water quality monitoring indicators has important significance. SUMMARY

[0005] Invention purposes: The purpose of the present application is to study the situation that there are too many monitoring indexes, large amount of data, high monitoring cost and it is difficult to quickly obtain effective information in water quality monitoring. A water quality monitoring index prediction method based on multiple regression and random forest is designed, so that the water quality monitoring index is reduced in dimension, the correlation between indexes and the regression model are used to make fewer indexes represent more indexes and predict the indexes, reduce the monitoring cost and improve the information utilization rate.

[0006] Technical scheme: In order to achieve the above invention purposes, the present application adopts the following technical scheme:

[0007] A water quality monitoring index prediction method based on multiple regression and random forest, comprising the following steps:

[0008] (1) obtaining various monitoring index data used in the water quality monitoring project monitoring process;

[0009] (2) cleaning various monitoring index data of water quality monitoring, and eliminating invalid and abnormal data;

[0010] (3) using Pearson correlation coefficient to analyze the correlation degree between various monitoring index variables of the cleaned water quality monitoring index data, obtaining the correlation coefficient between various water quality monitoring indexes, and identifying the dominant factors affecting water quality;

[0011] The formula of Pearson correlation coefficient is:

[0012] In the formula, n is the sample size, x i and y i are the variable values of two monitoring indexes, and are the average values of x i and y i samples respectively. i and y i are the variable values of two monitoring indexes, and are the average values of x i and y i samples respectively.

[0013] (4) using the method of significance test to test the Pearson correlation coefficient, verifying that the correlation between various monitoring indexes is caused by non-chance factors, and the obtained results can represent the correlation degree on the overall index data;

[0014] (5) using the method of constructing multiple linear regression equation to further obtain the specific form of the correlation degree between monitoring indexes. Through the form of regression equation, the statistical relationship between water quality indexes is further analyzed;

[0015] (6) eliminating the abnormal regression coefficient and the insignificant independent variable of the multiple linear regression equation obtained in step (5); repeating steps (5) to (6) until the regression coefficients meet the requirements, at this time the multiple linear regression equation is established;

[0016] (7) Split the entire data set to obtain a training set and a test set; wherein the training set is used to train the random forest regression model, and the test set is used to evaluate the prediction effect of the model;

[0017] (8) Establish a random forest regression model, and train the model using the training set by adopting a 5-fold cross-validation method;

[0018] (9) Evaluate the prediction effect of the random forest regression model using the test set, and the evaluation indexes of the model include root mean square error RMSE, mean absolute error MAE and determination coefficient R 2 These three indexes;

[0019] The formula of the determination coefficient is:

[0020] In the formula, n is the sample size, y is the value to be fitted, is the average value of the sample, i is the fitted value.

[0021] (10) Continuously adjust the number of hyper-parameters of the decision tree and the maximum depth of the tree in the random forest regression model to obtain the best model; repeat steps (8) to (10) so that the determination coefficient R 2 of the random forest regression model meets the standard of being closest to 1;

[0022] (11) According to the best random forest regression model obtained by training, the independent variable indicators are predicted using the independent variable monitoring indicators in the water quality monitoring indicators.

[0023] Further, in step (1), the various monitoring indicator data used in the water quality monitoring project monitoring process generally mainly include chemical oxygen demand parameters, dissolved oxygen parameters, pH value parameters, water temperature parameters, ammonia nitrogen parameters, turbidity parameters and other commonly used water quality monitoring indicators.

[0024] Further, in step (2), the invalid and abnormal data are abnormal data that are not within the measurement range of various water quality monitoring indicator sensors.

[0025] Further, in step (3), the Pearson correlation coefficient represents the correlation degree between data, and the closer the absolute value result is to 1, the stronger the correlation between variables; the closer to 0, the weaker the correlation between variables. The absolute value of the correlation coefficient is between 0 and 0.3, which presents weak correlation; between 0.3 and 0.5, which presents low correlation; between 0.5 and 0.8, which presents significant correlation; between 0.8 and 1, which presents high correlation.

[0026] ​In the preferred embodiment, in step (5), the significance test verifies whether the degree of correlation between variables passes the test according to whether the obtained P value is less than the selected significance level 0.05.

[0027] Further, in step (6), the regression coefficient reflects the parameter in the regression equation representing the size of the influence of the independent variable x on the dependent variable y, the abnormal regression coefficient is the value opposite in sign to the Pearson correlation coefficient, and the insignificant regression coefficient refers to the value that does not meet the regression coefficient test.

[0028] Further, in step (8), the 5-fold cross-validation method refers to: randomly dividing the original data into 5 parts without repeated sampling; each time, selecting 1 part as the test set and the remaining 4 parts as the training set for model training; repeating the step 5 times so that each subset has one opportunity as the test set and the remaining opportunities as the training set; and calculating the average value of the 5 test results as the accuracy of the model.

[0029] Beneficial effects: in the actual process of water quality monitoring, there are various water quality indicators to be monitored, the data volume of the water quality monitoring indicators is huge, and there is strong correlation between the parameter indicators, and the influence between the parameter indicators is obvious, the algorithm in machine learning is introduced, the multivariate linear regression model is used for statistical analysis of the highly correlated water quality monitoring indicators, the independent variables and the dependent variables affected thereby are obtained, and the type of monitoring indicators is effectively reduced. In addition, since the random forest regression model has the outstanding advantages of fast training speed, can be applied to large-scale data sets, and has strong generalization ability, therefore, the method can be effectively applied in different water quality indicator monitoring systems. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flow chart of a water quality monitoring indicator prediction method based on multivariate regression and random forest according to the present application.

[0031] Figure 2 is a Pearson correlation coefficient matrix heat map between various water quality monitoring indicators in an embodiment of the present application.

[0032] Figure 3 , 4 is a comparison chart of the predicted value of the random forest regression model and the actual value of the dependent variable in the water quality monitoring indicators in an example of the present application. DETAILED DESCRIPTION

[0033] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application, and after reading the present application, various equivalent modifications of the present application by those skilled in the art fall within the scope defined by the appended claims.

[0034] As Figure 1 shown, the embodiment of the application discloses a water quality monitoring index prediction method based on multiple regression and random forest, mainly including the following steps:

[0035] (1) Obtain various types of monitoring index data used in the water quality monitoring process. In this embodiment, six types of monitoring index data of water quality monitoring of Qinhuai New River from June 2020 to June 2021 are obtained, including monitoring data of chemical oxygen demand, dissolved oxygen, pH value, water temperature, ammonia nitrogen and turbidity parameter index, and the sampling time interval is 15 minutes.

[0036] (2) Clean the various types of monitoring index data of water quality monitoring, and eliminate invalid and abnormal data. In this embodiment, after eliminating abnormal data, there are 30143 samples.

[0037] (3) Use Pearson correlation coefficient to analyze the correlation degree between the variables of the cleaned water quality monitoring index data, obtain the correlation coefficient between each water quality monitoring index, and identify the dominant factors affecting water quality. The formula of Pearson correlation coefficient is: In the formula, n is the sample size, x i and y i are the variable values of two monitoring indexes, and are the average values of x i and y i samples, respectively. Figure 2 is the Pearson correlation coefficient matrix heat map between each water quality monitoring index in the present application, and the correlation degree between each monitoring index can be obtained according to the color depth of the heat map. The Pearson correlation coefficient matrix is shown in Table 1. In this embodiment, there are two pairs of highly correlated variables, one pair is water temperature index and dissolved oxygen index, and the correlation coefficient of the two monitoring indexes is-0.8965, which shows a high negative correlation, and the other pair is chemical oxygen demand index and ammonia nitrogen index, and their correlation coefficient is 0.9478, which shows a high positive correlation.

[0038] Table 1. Pearson correlation coefficient matrix table

[0039]

[0040] (4) The method of significance test is used to test the Pearson correlation coefficient, to verify that the correlation between the monitoring indicators is not caused by accidental factors, and the results can represent the correlation degree on the overall index data. In this embodiment, the P value of the significance test is less than the selected significance level 0.05, so the correlation between the variables passes the significance test, and the P value of the significance test result is listed in Table 2.

[0041] Table 2. P value list of Pearson correlation coefficient significance test results

[0042]

[0043] (5) The method of constructing a multiple linear regression equation is used to further obtain the specific form of the correlation degree between the monitoring indicators. Through the form of the regression equation, the statistical relationship between the water quality indicators is further analyzed.

[0044] (6) The multiple linear regression equation obtained in the previous step is eliminated from the regression coefficient anomaly and the insignificant independent variable. In this embodiment, the regression coefficient meets the requirements, so the multiple linear regression equation is established, and two multiple linear regression equations are obtained: CODCR = 2.5255 x PH + 23.0224 x NH3N and DOX = -0.1628 x WT - 0.7034 x PH + 0.3417 x TURB. Among them, CODCR is the chemical oxygen demand, NH3N is ammonia nitrogen, DOX is dissolved oxygen, WT is water temperature, and TURB is turbidity.

[0045] (7) The entire data set is split to obtain a training set and a test set; wherein the training set is used to train the random forest regression model, and the test set is used to evaluate the prediction effect of the model. In this embodiment, the number of test set samples is 25% of the total number of data set samples.

[0046] (8) The random forest regression model is established, and the 5-fold cross-validation method is used to train the model with the training set. In this embodiment, the related parameter setting random value random_state=0 is used to find the best parameters of the model through 5-fold cross-validation, and the number of decision trees n_estimators=50 and the maximum depth of the tree max_depth=7 are obtained for the prediction model of the chemical oxygen demand index, and the number of decision trees n_estimators=150 and the maximum depth of the tree max_depth=6 are obtained for the prediction model of the dissolved oxygen index.

[0047] (9) The test set is used to evaluate the prediction effect of the random forest regression model. In this embodiment, the three evaluation indicators of the random forest regression model are: root mean square error RMSE=0.28833, mean absolute error MAE=0.18132, and determination coefficient R2=0.99999.2 = 0.98314.

[0048] (10) continuously adjust the number of hyper-parameters decision trees and the maximum depth of the trees in the random forest regression model to obtain the best model. In this embodiment, since the determination coefficient R 2 of the random forest regression model has reached the closest to 1, the parameter adjustment of the random forest regression model is stopped.

[0049] According to the best random forest regression model obtained by training, the dependent variable indicators in the water quality monitoring indicators are predicted by using the independent variable monitoring indicators. Figure 3 4 is a comparison chart of the predicted value of the random forest regression model and the actual value of the dependent variable indicators in the water quality monitoring indicators in the embodiment of the present application. The comparison chart of the chemical oxygen demand monitoring indicators and the comparison chart of the dissolved oxygen monitoring indicators are obtained in this embodiment. It is obvious from the chart that the predicted value and the actual value of the dependent variable water quality monitoring indicators almost coincide, which effectively shows that the random forest regression model can realize accurate prediction of the dependent variable water quality monitoring indicators.​

Claims

1. A method for predicting water quality monitoring indicators based on multiple regression and random forest, characterized in that, The method includes the following steps: (1) Obtain data on various monitoring indicators used in the water quality monitoring project; (2) Clean the data of various water quality monitoring indicators and remove invalid or abnormal data; (3) Using the Pearson correlation coefficient, the correlation between various monitoring index variables of the cleaned water quality monitoring index data was analyzed to obtain the correlation coefficient between each water quality monitoring index and to identify the dominant factors affecting water quality. The formula for the Pearson correlation coefficient is expressed as follows: In the formula, n is the sample size, x i and y i These are the variable values ​​of the two monitoring indicators, respectively. and x i and y i The sample mean; (4) The Pearson correlation coefficient is tested using the significance test method to verify that the correlation between each monitoring indicator is not caused by accidental factors and that the results can represent the degree of correlation in the overall indicator data. The significance test is based on whether the obtained P value is less than the selected significance level of 0.05 to verify whether the degree of correlation between the indicator variables has passed the test. (5) The method of constructing multiple linear regression equations is used to further obtain the specific form of the correlation between monitoring indicators; the statistical relationship between water quality indicators is further analyzed through the form of regression equations. (6) For the multiple linear regression equation obtained in step (5), remove the abnormal regression coefficients and insignificant independent variables. The regression coefficients reflect the parameters in the regression equation that represent the influence of the independent variable x on the dependent variable y. Abnormal regression coefficients are values ​​whose signs are opposite to the Pearson correlation coefficient. Insignificant regression coefficients are values ​​that do not meet the regression coefficient test. Repeat steps (5) to (6) until all regression coefficients meet the requirements. At this time, the multiple linear regression equation is established. (7) Split the entire dataset to obtain a training set and a test set; the training set is used to train the random forest regression model, and the test set is used to evaluate the model's prediction performance. (8) Establish a random forest regression model and use the 5-fold cross-validation method to train the model using the training set. The 5-fold cross-validation method refers to: randomly dividing the original data into 5 parts without repetition; selecting 1 part as the test set each time, and using the remaining 4 parts as the training set for model training; repeating this step 5 times, so that each subset has one chance to be used as the test set, and the remaining chances are used as the training set; calculating the average of the 5 test results as the accuracy of the model. (9) The predictive performance of the random forest regression model was evaluated using the test set. The evaluation metrics for the model were root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R²). 2 These three indicators; The formula for determining the coefficients is expressed as: In the formula, n is the sample size, and y is the value to be fitted. For y i The sample mean These are the fitted values; (10) Continuously adjust the number of hyperparameter decision trees and the maximum depth of the trees in the random forest regression model to obtain the optimal model; repeat steps (8) to (10) to make the coefficient of determination R of the random forest regression model more stable. 2 The value satisfies the criterion of being closest to 1; (11) Based on the best random forest regression model obtained from training, the dependent variable index is predicted using the independent variable monitoring index in the water quality monitoring index.

2. The method for predicting dependent variable monitoring indicators among water quality monitoring indicators according to claim 1, characterized in that: In step (1), the various monitoring index data used in the water quality monitoring project include six commonly used water quality monitoring indicators: chemical oxygen demand, dissolved oxygen, pH value, water temperature, ammonia nitrogen, and turbidity.

3. The method for predicting dependent variable indicators in water quality monitoring according to claim 1, characterized in that: In step (2), the invalid or abnormal data refers to abnormal data that is outside the measurement range of the various water quality monitoring index sensors used.

4. The method for predicting dependent variable indicators in water quality monitoring according to claim 1, characterized in that: In step (3), the Pearson correlation coefficient represents the degree of correlation between data. The closer the absolute value of the coefficient is to 1, the stronger the correlation between the variables. The closer the correlation coefficient is to 0, the weaker the correlation between the variables; an absolute value between 0 and 0.3 indicates a weak correlation. The correlation is low between 0.3 and 0.5; significant between 0.5 and 0.8; and highly correlated between 0.8 and 1.

Images