Water quality index prediction method, system, equipment and medium

Abstract

The invention discloses a water quality index prediction method, system and equipment and a medium. The method comprises the following steps: constructing a water quality data set; establishing effluent COD, TN and TP soft measurement models based on the XGBoost, and training by adopting the model training set, wherein the effluent COD, TN and TP soft measurement models based on the XGBoost comprise an effluent COD soft measurement model, an effluent TN soft measurement model and an effluent TP soft measurement model; constructing a KNN input missing processing model for performing integrity check and / or missing processing on the input data; combining the effluent COD soft measurement model, the effluent TN soft measurement model, the effluent TP soft measurement model and the KNN input missing processing model which are obtained through training to form a KNN-XGBoost comprehensive soft measurement model; and inputting to-be-predicted data into the KNN-XGBoost comprehensive soft measurement model for prediction to obtain an effluent COD prediction value, an effluent TN prediction value and an effluent TP prediction value. According to the method, the values of COD, TN and TP of the effluent can be quickly obtained, and a basis is provided for whether the quality of the treated effluent is qualified or not.

Description

Technical Field

[0001] The invention belongs to the field of soft measurement of effluent water quality in a wastewater treatment process, and in particular relates to a water quality index prediction method, system, equipment and medium. Background Art

[0002] Data-driven water quality prediction technology has become a key measure for the intelligent upgrade of water treatment. On the one hand, accurate prediction of effluent quality can evaluate the treatment effect of water treatment process and provide a basis for optimizing treatment plan; on the other hand, water quality prediction can detect potential problems in advance, play a role in water quality warning, and avoid environmental damage caused by substandard discharge.

[0003] However, there is a strong coupling relationship between factors such as water quality characteristics, process design and operation management in the water treatment process, and water quality data often show strong nonlinearity and non-stationarity, which makes the accurate prediction of effluent water quality extremely challenging. With the development of artificial intelligence, machine learning provides new ideas and methods to solve these problems. Machine learning can learn the complex and potential relationships between variables from real-world data, and is not limited by human experience and cognition, and can discover potential relationships that have been previously overlooked. By using machine learning models established by using various influent variables related to water treatment, it is possible to accurately predict effluent water quality indicators in advance.

[0004] In the actual data-driven modeling process, researchers currently often only divide the data into training sets and test sets, which will lead to the selection of model algorithms and the optimization of hyperparameters being adjusted directly according to the performance on the test set, which may lead to overly optimistic results of the performance evaluation indicators of the built model and affect the generalization performance of the model. Dividing the data set into training sets, validation sets and test sets or cross-validating training sets and test sets is a more reasonable way of data segmentation, but the performance difference between the models built using these two data segmentation methods is rarely discussed. The selection of K value in the cross-validation training set has a significant impact on the performance of the built model, but current research often limits the selection of K value to within 10, lacks the exploration of the performance impact of larger K values, and the selection of K value often depends on the operator's experience and lacks corresponding reference indicators. In addition, for most machine learning algorithms, when there are missing values ​​in the input features, the established model cannot make predictions (ie, ANN, SVM, etc.). In the actual water treatment process, data loss caused by abnormal data transmission is common, which also indirectly limits the practical application potential of machine learning models. Therefore, it is particularly important to explore a reasonable method to handle missing inputs while ensuring the practicality of the model and minimizing the model performance loss caused by missing inputs.

[0005] In order to improve the prediction performance of the model, the effects of TVT data segmentation and CVT data segmentation on model performance are discussed, and the optimal K value selection in K-fold cross validation is optimized to achieve the optimization of model performance. At the same time, the KNN input missing processing method is adopted to minimize the negative impact of input missing on model performance and improve the practical applicability of the model. Summary of the invention

[0006] In order to solve at least one of the problems existing in the prior art, the present invention proposes an optimization method for a water quality index prediction model under small data sets and missing data, which can effectively improve the prediction performance of the model on small data sets, and enable the constructed model to handle the situation of missing input and greatly reduce the loss of model prediction performance caused by missing input.

[0007] In order to achieve the purpose of the present invention, the present invention provides a method for predicting water quality indicators, comprising the following steps:

[0008] Construct a water quality data set, and divide the water quality data set into a model training set, a model verification set, and a model test set according to the TVT division method, or divide it into a model training set and a model test set according to the CVT division method. When the CVT division method is used, the training set is divided into K non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model verification sets. The optimal value of K is determined by setting different K value gradients and performing deviation and variance analysis;

[0009] Establish an effluent COD, TN, and TP soft-sensing model based on XGBoost and use the model training set for training, use the model verification set to adjust and optimize the hyperparameters, and use the model test set to detect the trained effluent COD, TN, and TP soft-sensing model based on XGBoost. The effluent COD, TN, and TP soft-sensing model based on XGBoost includes an effluent COD soft-sensing model, an effluent TN soft-sensing model, and an effluent TP soft-sensing model.

[0010] Constructing a KNN input missing processing model through a KNN algorithm, wherein the KNN input missing processing model is used to perform integrity check and / or missing processing on input data;

[0011] The trained effluent COD soft measurement model, effluent TN soft measurement model, effluent TP soft measurement model and KNN input missing processing model are combined to form a KNN-XGBoost comprehensive soft measurement model. The trained input missing processing KNN input missing processing model is combined with the effluent water quality prediction XGBoost model to form a KNN-XGBoost comprehensive soft measurement model. While ensuring that the soft measurement model has excellent prediction performance, it greatly reduces the performance degradation or even loss of the model due to missing input features, thereby improving the practical applicability of the model.

[0012] The data to be predicted is input into the KNN-XGBoost comprehensive soft sensor model for prediction, and the predicted values ​​of effluent COD, effluent TN and effluent TP are obtained.

[0013] Furthermore, the data in the water quality data set include data monitored by the water quality online monitoring probe and manual detection data of inlet and outlet water. The data monitored by the water quality online monitoring probe include MLSS concentration of sludge in the sedimentation tank. sedi , Oxidation Reduction Potential (ORP) in Sedimentation Tanks sedi , dissolved oxygen concentration DO in aeration tank area , Oxidation Reduction Potential (ORP) in Aeration Tanks area , dissolved oxygen concentration DO in the biological purification tank ep , conductivity EC in biological purification tank ep , the temperature in the biological purification tank T ep , MLSS sludge concentration in biological purification tank ep and pH in biological purification tanks ep , the manual detection data of inlet and outlet water COD concentration in inlet water inf , TN concentration in influent inf TP concentration in influent inf , COD concentration in effluent eff , TN concentration in effluent eff and TP concentration in effluent eff .

[0014] Furthermore, the input features of the effluent COD, TN, and TP soft sensor model based on XGBoost were screened by mutual information values, the MI value between each variable and the model output was calculated, and the variables with MI values ​​higher than the threshold were selected as the input features of the model.

[0015] Furthermore, for the two data partitioning methods, when training the model, both random search and grid search are used to find and optimize the best hyperparameters of the model. Specifically, random search is first used to determine the approximate range of each hyperparameter of the model with a large step size, and then grid search is used to determine the best value of each hyperparameter within the range with a small step size; after finding the optimal hyperparameter, the model is tested using the model test set data; and this training and testing process is repeated until the trained model meets the predetermined requirements.

[0016] Furthermore, each soft sensor model:

[0017] ① Establishment of effluent COD soft-sensing model under TVT division: The maximum number of trees of the trained effluent COD soft-sensing model based on XGBoost is 325, the maximum depth of trees is 3, and the learning rate is 0.07;

[0018] ② Establishment of the water outlet TN soft sensor model under TVT division: The maximum number of trees of the trained water outlet TN soft sensor model based on XGBoost is 400, the maximum depth of the tree is 7, and the learning rate is 0.02;

[0019] ③ Establishment of the water outlet TP soft sensor model under TVT division: The maximum number of trees of the trained water outlet TP soft sensor model built based on XGBoost is 300, the maximum depth of the trees is 8, and the learning rate is 0.06.

[0020] Furthermore, each soft sensor model:

[0021] ① Establishment of the outlet COD soft-sensing model with the best K value under CVT partitioning: The maximum number of trees of the trained outlet COD soft-sensing model based on XGBoost is 420, the maximum depth of trees is 3, and the learning rate is 0.02;

[0022] ② Establishment of the water outlet TN soft sensor model with the best K value under CVT partitioning: The maximum number of trees of the trained water outlet TN soft sensor model based on XGBoost is 200, the maximum depth of the tree is 4, and the learning rate is 0.03;

[0023] ③ Establishment of the water outlet TP soft sensor model with the optimal K value under CVT partitioning: The maximum number of trees of the trained water outlet TP soft sensor model built based on XGBoost is 625, the maximum depth of the trees is 3, and the learning rate is 0.01.

[0024] Furthermore, the KNN input missing processing model sets different missing degrees and then uses the training set data to establish the KNN input missing processing model, and interpolates and fills the input missing parts at different missing degrees through the correlation between the features and the values ​​of the non-missing features.

[0025] Furthermore, in the KNN input missing processing model, after the input data is input into the KNN input missing processing model, it is first checked whether there are missing values ​​in the input data. If there are missing values, interpolation processing is performed on the missing parts.

[0026] Furthermore, the number of neighbor points of the KNN input missing processing model is 6, the distance weight is uniform, and the p value of the Minkowski distance is 2.

[0027] Furthermore, Python was used to establish the soft-sensing model of effluent COD, TN, and TP based on XGBoost.

[0028] The present invention also provides a water quality index prediction system, comprising the following modules:

[0029] A data set construction module is used to construct a water quality data set, and divide the water quality data set into a model training set, a model verification set and a model test set according to the TVT division method, or divide it into a model training set and a model test set according to the CVT division method. When the CVT division method is used, the training set is divided into K mutually non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model verification sets, wherein the optimal value of K is determined by setting different K value gradients and performing deviation and variance analysis;

[0030] A model building module is used to establish an effluent COD, TN, and TP soft measurement model based on XGBoost and use a model training set for training, use a model verification set for hyperparameter adjustment and optimization, and use a model test set to detect the trained effluent COD, TN, and TP soft measurement model based on XGBoost. The effluent COD, TN, and TP soft measurement model based on XGBoost includes an effluent COD soft measurement model, an effluent TN soft measurement model, and an effluent TP soft measurement model;

[0031] A missing processing module is used to construct a KNN input missing processing model through a KNN algorithm, wherein the KNN input missing processing model is used to perform integrity check and / or missing processing on input data;

[0032] A combination module is used to combine the trained effluent COD soft-sensing model, effluent TN soft-sensing model, effluent TP soft-sensing model and KNN input missing processing model to form a KNN-XGBoost comprehensive soft-sensing model;

[0033] The prediction module is used to input the data to be predicted into the KNN-XGBoost comprehensive soft sensor model for prediction, and obtain the effluent COD prediction value, effluent TN prediction value and effluent TP prediction value.

[0034] The present invention also provides a storage medium storing a program, wherein the program, when executed by a processor, implements the aforementioned water quality index prediction method.

[0035] The present invention also provides a computing device, comprising a processor and a memory for storing a program executable by the processor, wherein the processor implements the aforementioned water quality index prediction method when executing the program stored in the memory.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] Compared with traditional manual detection, this soft measurement method can quickly obtain the values ​​of effluent COD, TN and TP based on the data of online water quality monitoring instruments, and provide a basis for whether the effluent quality after treatment is up to standard. On the other hand, it can effectively improve the prediction performance of the model on small data sets, and enable the built model to handle the situation of missing input, and greatly reduce the loss of model prediction performance caused by missing input;

[0038] In the optimization method technology of the water quality index prediction model with small data set and missing data of the present invention, 12 auxiliary variables related to effluent COD, TN and TP are used to establish an effluent water quality soft measurement model based on XGBoost and combine KNN to form a KNN-XGBoost comprehensive soft measurement method. For various sewage treatment facilities, as long as the relevant variables and methods of the present invention are used to predict effluent COD, TN and TP, and the comprehensive soft measurement method formed by combining the independent soft measurement method with KNN should fall within the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a diagram of the installation locations of the online water quality monitoring probe in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the process of establishing a model in a water quality index prediction method according to an embodiment of the present invention;

[0041] Figure 3 This is a test and prediction result diagram of the effluent COD soft sensor model based on XGBoost under TVT on the test set in an embodiment of the present invention;

[0042] Figure 4 This is a test and prediction result diagram of the water outlet TN soft sensor model based on XGBoost under TVT on the test set in an embodiment of the present invention;

[0043] Figure 5 This is a test and prediction result diagram of the water outlet TP soft sensor model based on XGBoost under TVT on the test set in an embodiment of the present invention;

[0044] Figure 6 It is a test and prediction result diagram of the effluent COD soft sensor model based on XGBoost on the test set under CVT and when K is the optimal value in an embodiment of the present invention;

[0045] Figure 7 It is a test and prediction result diagram of the water outlet TN soft sensor model based on XGBoost on the test set under CVT and when K is the optimal value in an embodiment of the present invention;

[0046] Figure 8 It is a test and prediction result diagram of the water outlet TP soft sensor model based on XGBoost on the test set under CVT and when K is the optimal value in an embodiment of the present invention;

[0047] Fig. 9 It is a t-SNE diagram of the data after the effluent COD soft sensor model in the embodiment of the present invention uses KNN to process the input missing data at different missing degrees;

[0048] Fig.10 This is a t-SNE diagram of the data after the output TN prediction model in the embodiment of the present invention uses KNN to process the input missing data at different missing degrees;

[0049] Fig.11 This is a t-SNE diagram of the data after the water outlet TP prediction model in an embodiment of the present invention uses KNN to perform input missing processing at different missing degrees. DETAILED DESCRIPTION

[0050] The following further describes the implementation of the present invention in conjunction with specific implementation examples. It should be noted that the examples are merely an explanation of the present invention rather than a limitation of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to this application.

[0051] The present invention provides an optimization method for a water quality index prediction model under small data sets and data missing conditions, such as Figure 2 As shown, it includes online water quality monitoring instruments and manual detection data, data partitioning, deviation and variance analysis, optimal model construction, input missing gradient setting, KNN input missing processing model establishment and comprehensive soft sensor model establishment.

[0052] See also Figure 2 The present invention provides a method for optimizing a water quality index prediction model under a small data set or missing data, comprising the following steps:

[0053] Step 1: Selection of online water quality monitoring probes: Select probes that can directly monitor online in real time and are closely related to the effluent water quality variables.

[0054] In this step, the selected probes include: dissolved oxygen (DO) concentration probe, mixed liquor suspended solids (MLSS) concentration probe, oxidation-reduction potential (ORP) probe, electrical conductivity (EC) probe, pH probe and temperature (T) probe.

[0055] In some embodiments of the present invention, 6 types of online water quality monitoring probes, namely MLSS, DO, ORP, EC, pH and T (℃), are selected, of which 2 each of DO, ORP and MLSS online water quality monitoring probes are provided, and 1 each of the other online water quality monitoring probes is provided, for a total of 9 online water quality monitoring probes. Figure 1 This is the installation site diagram of the online water quality monitoring probe. The sludge concentration probe and the redox potential probe are installed in the sedimentation tank to detect the sludge concentration MLSS in the sedimentation tank. sedi And for detecting oxidation-reduction potential (ORP) sedi A dissolved oxygen concentration probe and an oxidation-reduction potential probe were installed in the aeration tank to detect the dissolved oxygen concentration DO in the aeration tank. area And for detecting oxidation-reduction potential (ORP) area , dissolved oxygen concentration probe, conductivity probe, temperature probe, sludge concentration probe and pH probe are installed in the biological purification tank to detect the dissolved oxygen concentration DO in the biological purification tank. ep , electrical conductivity EC ep , Temperature T ep , sludge concentration MLSS ep pH ep .

[0056] Step 2: Establish a water quality data set and divide the data subsets into TVT and CVT methods respectively.

[0057] In this step, the real-time monitoring data of the water quality online monitoring probe and the manual detection data of the inlet and outlet water (including the concentrations of COD, TN and TP in the inlet water and COD, TN and TP in the outlet water, respectively) are used to calculate the concentration of COD in the inlet water and the outlet water. inf TN inf TP inf 、COD eff TN eff TP effRepresented), a set of model input and output data, namely a data set, is constructed, and then the data subsets are divided into two methods, TVT and CVT, respectively. In some embodiments of the present invention, the TVT division method divides the data set into a model training set, a model verification set and a model test set in a ratio of 7:1:2, and the CVT division method divides the data set into a model training set and a model test set in a ratio of 8:2.

[0058] Step 3: Establish effluent COD, TN, and TP soft measurement models based on XGBoost, including effluent COD soft measurement model, effluent TN soft measurement model, and effluent TP soft measurement model, and train the models.

[0059] Among them, for the TVT data partitioning method, the model training set is used to train the effluent COD, TN, and TP soft measurement models based on XGBoost, the model validation set is used to adjust and optimize the model hyperparameters, and the trained model is tested using the model test set to verify the accuracy and stability of the trained model; for the CVT data partitioning method, before the model training set is used for model training, the training set is first divided into K non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model validation sets to evaluate the training performance of the model. This process is repeated K times, and finally the mean of each model evaluation parameter is used as the comprehensive evaluation parameter of the model during the training process, which is used for hyperparameter adjustment and optimization, and the trained model is tested using the model test set to verify the accuracy and stability of the trained model.

[0060] For the two data set division methods, the effluent COD, TN, and TP soft sensor models based on XGBoost are trained by using random search and grid search to find and optimize the best hyperparameters of the model. Specifically, random search is first used to determine the approximate range of each hyperparameter of the model with a large step size, and then grid search is used to determine the best value of each hyperparameter within the range with a small step size; after finding the best hyperparameter, the model is tested using the test set data; this training and testing process is repeated until the trained model meets the predetermined requirements. In some embodiments of the present invention, Figure 3 , Figure 4 and Figure 5 The fitting effect of the soft sensor model of effluent COD, TN and TP on the test set under TVT data division is demonstrated. Figure 6 , Figure 7 and Figure 8 The fitting effect of the effluent COD, TN and TP soft-sensing model on the test set under CVT data partitioning is demonstrated. It can be seen from the figure that the fitted predicted values ​​and the true values ​​are highly consistent, which strongly illustrates the effectiveness of the effluent COD, TN and TP soft-sensing model of the present invention in prediction.

[0061] The input features of the effluent COD, TN, and TP soft-sensing model based on XGBoost are screened by the mutual information value (MI). In some embodiments of the present invention, the MI threshold is set to 0.05, and the variables (i.e., COD inf TN inf TP inf MLSS sedi 、ORP sedi ,DO area 、ORP area ,DO ep ,EC ep 、T ep MLSS ep and pH ep ) and the model output, and the variables with MI values ​​higher than the threshold are selected as the input features of the model.

[0062] After screening, the input features of the effluent COD soft-sensing model based on XGBoost are COD inf TN inf MLSS sedi 、ORP area ,DO area ,DO ep ,EC ep , pH ep , the output feature is COD eff ; The input features of the water outlet soft sensor model based on XGBoost are TN inf MLSS sedi 、ORP sedi ,DO area ,DO ep MLSS ep ,EC ep 、T ep , the output feature is TN eff ; The input features of the water outlet TP soft sensor model based on XGBoost are TP inf 、ORP sedi ,DO area 、ORP area ,DO ep ,EC ep , pH ep 、T ep , the output feature is TP eff .

[0063] Step 4: Determine the optimal K value.

[0064] Since there is no fixed standard for the selection of K value in CVT and it has a great influence on the prediction performance of the established model, it is necessary to determine the optimal K value. In some embodiments of the present invention, K=3, 5, 10, 15, 20, 40, 80 and n are set respectively. train (In other embodiments, other values ​​may also be set), the root mean square error (RMSE) of the trained model on the validation set is plotted as a box plot, and the optimal K value in CVT is determined through deviation and variance analysis to maximize the predictive performance of the model.

[0065] The average RMSE reflects the prediction accuracy (i.e., bias) of the trained model, while the width of the upper and lower edge lines of the box plot reflects the stability (i.e., variance) of the prediction performance of the trained model. The water quality soft sensor model with the best prediction performance should have low bias and low variance, but in the actual modeling process, both cannot be satisfied at the same time. Bias and variance analysis is required to maximize the prediction performance of the soft sensor model.

[0066] That is, step 4 specifically includes:

[0067] Step 4.1: Set different K value gradients: Set K = 3, 5, 10, 15, 20, 40, 80 and n train , n train Represents the number of samples in the training set, when K = n train It is also called leave-one-out cross-validation.

[0068] Step 4.2: Plot the root mean square error (RMSE) of the model on the validation set into a box plot and perform bias and variance analysis. The K value with lower bias and lower variance is the optimal K value.

[0069] Step 5: Construct a KNN input missing processing model through the KNN algorithm, and the KNN input missing processing model is used to perform integrity check and / or missing processing on the input data. Specifically, the KNN input missing processing model sets different missing degrees, and then uses the training set data to establish the KNN input missing processing model. Through the correlation between the features and the values ​​of the non-missing features, the input missing parts under different missing degrees are interpolated and filled to reduce the model prediction performance loss caused by input missing.

[0070] Since the real-time water quality online monitoring probe will inevitably have missing values ​​when collecting and transmitting the collected data to the Internet of Things cloud platform, the input features of the model will have missing values, which will cause the prediction performance of the model to be seriously reduced or even unable to make predictions. This limits the practical application potential of machine learning models. Therefore, the present invention adopts the KNN algorithm to deal with the input missing problem of the water soft measurement model, while ensuring the practicality of the model, and reducing the model performance loss caused by input missing as much as possible.

[0071] Prediction Performance Loss (PPL) is defined as a relative measure of the decline in model prediction performance, and a missing degree is set. In some embodiments of the present invention, the missing degrees are set to 5%, 10%, 20% and 30%, respectively, and the situation in which some real-time data is missing in reality is simulated by randomly removing data points with corresponding missing degrees. The random removal of data points at each missing degree will be repeated multiple times, such as 5 times, to reduce random errors; then the KNN algorithm is used to fit the potential connection between the data in the training set, and a KNN input missing processing model is constructed. In the test set, input missing at different missing degrees is interpolated and filled by the KNN input missing processing model, and the change in the prediction performance of the water outlet soft measurement model before and after the treatment is reflected by the increase or decrease of PPL, and the similarity of the data structure after the input missing processing and when there is no input missing is reflected by the t-SNE diagram, thereby further reflecting the effectiveness of the KNN input missing processing model. Fig. 9 , Fig.10 and Fig.11 The t-SNE plots of the soft sensing models for effluent COD, TN, and TP are shown.

[0072] In some embodiments of the present invention, step 5 specifically includes:

[0073] ① Set the missing degree to 5%, 10%, 20% and 30% respectively, and randomly remove the data points with the corresponding missing degree to simulate the situation where some real-time data is missing in reality. The random removal of data points at each missing degree will be repeated 5 times to reduce random errors;

[0074] ② Define the calculation formula of PPL: R 2 pre and R 2 post Represents the determination coefficient R of the model on the test set before and after the input feature is missing 2 ,PPL is a relative measure of the degradation of the model’s prediction performance;

[0075] ③ Use the KNN algorithm to fit the potential rules in the training set data, and interpolate the missing values ​​in the test set based on the numerical correlation of other features to solve the input missing problem of the model. Compare the increase and decrease of the model PPL before and after the processing and the similarity of the data structure in the t-SNE diagram to determine the effectiveness of the KNN algorithm;

[0076] Through the KNN input missing processing model, you can first perform an input feature integrity check and then perform missing processing. The input feature integrity check is to check whether there are missing values ​​in the data. If there are missing values, the KNN input missing processing model will interpolate the missing values; if there are no missing values, the KNN input missing processing model will not process them.

[0077] Step 6: Combine the trained XGBoost-based effluent COD, TN, and TP soft measurement models with the KNN input missing processing model. The input features are first checked for integrity and processed by the KNN input missing processing model to ensure the integrity of the input features when they are transmitted to each effluent soft measurement model, thereby forming a KNN-XGBoost comprehensive soft measurement model. The comprehensive soft measurement model can significantly reduce the performance degradation or even loss of the model due to the missing input features while ensuring that the soft measurement model has excellent prediction performance, thereby improving the practical applicability of the model.

[0078] Step 6 specifically combines the trained individual effluent COD, TN, and TP soft-sensing models with the KNN input missing processing model to form a KNN-XGBoost comprehensive soft-sensing model.

[0079] By combining these two methods, the soft sensor model can have excellent prediction performance while significantly reducing the performance degradation or even loss of the model due to the lack of input features, thus improving the practical applicability of the model.

[0080] Step 7: Input the data to be predicted into the KNN-XGBoost comprehensive soft sensor model for prediction to obtain the effluent COD prediction value, effluent TN prediction value and effluent TP prediction value.

[0081] The input features of the KNN-XGBoost integrated soft measurement model will first be checked for input feature integrity by the KNN input missing processing model. If the input features are not missing, the input features will be passed to the soft measurement model for effluent water quality index prediction. If the input features are missing, the KNN input missing processing model will first perform missing processing. After processing, the missing features will be passed to the soft measurement model for effluent water quality index prediction to obtain the effluent COD prediction value, effluent TN prediction value and effluent TP prediction value.

[0082] The optimization method of the water quality index prediction model under small data set and missing data provided by the above-mentioned embodiment of the present invention, by respectively using the performance difference comparison of the model built by training-validation-test division and K-fold cross validation division, the determination of the optimal K value in K-fold cross validation, the processing of the missing model input data by KNN and the final establishment of the water quality index prediction model. First, the real-time water quality online monitoring probe reads the real-time data such as DO, MLSS, ORP, EC, pH and T (℃) in the wastewater treatment process, transmits it to the Internet of Things cloud platform through the gateway and stores it, and then analyzes the stored data to obtain the model sample data set. The data set is divided into data subsets by two division methods of training-validation-test and K-fold cross validation test, wherein the TVT division method divides the data set into a model training set, a model validation set and a model test set according to a ratio of 7:1:2, and the CVT division method divides the data set into a model training set and a model test set according to a ratio of 8:2. After screening the input features based on mutual information, the XGBoost algorithm was used to establish soft sensor models for effluent COD, effluent TN, and effluent TP on the data subsets obtained by TVT and CVT division, and the prediction performance of the models was compared. Variance and bias analysis was then used to determine the optimal K value in K-fold cross validation. Considering that the real-time data of the water quality probe will inevitably have varying degrees of missing, the prediction performance loss index is defined, and the KNN algorithm is used to handle the missing values ​​of the input features to minimize the negative impact of the missing input on the model performance. Finally, the KNN algorithm and the XGBoost algorithm are combined to form a water quality index prediction model that can handle missing inputs and has the best performance on a small data set.

[0083] In some embodiments of the present invention, the wastewater treatment system is taken as an example of an actual wastewater treatment system of a breeding tailwater treatment station in Foshan. All data are collected during the daily operation of the treatment system, among which MLSS sedi 、ORP area ,DO area 、ORP area MLSS ep ,DO ep ,EC ep , pH ep , T ep The data are all obtained from the online water quality monitoring instruments installed in the wastewater treatment system. The data of COD, TN and TP in the inlet and outlet water quality are obtained by the water quality laboratory technicians according to the national standard monitoring method.

[0084] The data collection work started after the wastewater treatment system was debugged and started running stably. It took 90 days to complete the data collection work required for the model, and a total of 272 sets of valid original data were obtained.

[0085] The soft sensor models of COD, TN and TP based on XGBoost were established on Python 3.10. The hyperparameters and structures of each soft sensor model were determined by random search and grid search. Finally, the maximum number of trees for the soft sensor model of COD was 420, the maximum depth of trees was 3, and the learning rate was 0.02 when CVT data was partitioned and K was the optimal value; the maximum number of trees for the soft sensor model of TN was 200, the maximum depth of trees was 4, and the learning rate was 0.03; the maximum number of trees for the soft sensor model of TP was 625, the maximum depth of trees was 3, and the learning rate was 0.01. The number of neighbor points of the KNN input missing processing model was 6, the weight of the distance was uniform, and the p value of the Minkowski distance was 2. Subsequently, the trained individual soft sensor models of COD, TN and TP were combined with the KNN input missing processing model to form a KNN-XGBoost comprehensive soft sensor model. At this time, the input features of the model will first be checked for input feature integrity by the KNN input missing processing model. If the input features are not missing, the input features will be passed to the soft measurement model for effluent water quality index prediction. If the input features are missing, the KNN input missing processing model will first perform missing processing, and then pass it to the soft measurement model for effluent water quality index prediction. The comprehensive soft measurement model can greatly reduce the performance degradation or even loss of the model due to missing input features while ensuring that the soft measurement model has excellent prediction performance, thereby improving the practical applicability of the model.

[0086] The original data set was used to train and test each water soft sensor model, of which 218 groups were used as training data and 54 groups were used as test data. The model training and test results are shown in Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 As can be seen from the figure, the correlation between the soft sensor model effluent COD prediction value and the measured true value is 0.83, the correlation between the effluent TN prediction value and the measured true value is 0.87, and the correlation between the effluent TP prediction value and the measured true value is 0.81. The model prediction effect is good.

[0087] The present invention can improve the prediction effect by an average of 9.5%, and reduce the average prediction performance loss of the model to 15.2% at a 30% missing degree.

[0088] In some embodiments of the present invention, a water quality index prediction system is also provided, comprising the following modules:

[0089] A data set construction module is used to construct a water quality data set, and divide the water quality data set into a model training set, a model verification set and a model test set according to the TVT division method, or divide it into a model training set and a model test set according to the CVT division method. When the CVT division method is used, the training set is divided into K mutually non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model verification sets, wherein the optimal value of K is determined by setting different K value gradients and performing deviation and variance analysis;

[0090] A model building module is used to establish an effluent COD, TN, and TP soft measurement model based on XGBoost and use a model training set for training, use a model verification set for hyperparameter adjustment and optimization, and use a model test set to detect the trained effluent COD, TN, and TP soft measurement model based on XGBoost. The effluent COD, TN, and TP soft measurement model based on XGBoost includes an effluent COD soft measurement model, an effluent TN soft measurement model, and an effluent TP soft measurement model;

[0091] A missing processing module is used to construct a KNN input missing processing model through a KNN algorithm, wherein the KNN input missing processing model is used to perform integrity check and / or missing processing on input data;

[0092] A combination module is used to combine the trained effluent COD soft-sensing model, effluent TN soft-sensing model, effluent TP soft-sensing model and KNN input missing processing model to form a KNN-XGBoost comprehensive soft-sensing model;

[0093] The prediction module is used to input the data to be predicted into the KNN-XGBoost comprehensive soft sensor model for prediction, and obtain the effluent COD prediction value, effluent TN prediction value and effluent TP prediction value.

[0094] In some embodiments of the present invention, a computing device is also provided, including a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, a water quality index prediction method provided in the above-mentioned embodiment is implemented.

[0095] In some embodiments of the present invention, a storage medium is further provided, storing a program, and when the program is executed by a processor, a water quality index prediction method provided in the above-mentioned embodiment is implemented.

[0096] The present invention is to solve the obvious shortcomings of the actual sewage treatment process, which is that the amount of data available for modeling is small, the quality is not high, and the missing of model input data will lead to a significant decline in model prediction performance due to the difficulty and high cost of collecting sewage indicator data.

[0097] It should be noted that no matter which computer language environment is used, as long as the method adopted to establish the model is similar to the above, it falls within the scope of the rights of this patent.

[0098] The above embodiment is one of the successful implementation methods of the present invention, but the implementation methods of the present invention are not limited to the above embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for predicting water quality indicators, characterized in that: The following steps are involved: Construct a water quality data set, and divide the water quality data set into a model training set, a model verification set, and a model test set according to the TVT division method, or divide it into a model training set and a model test set according to the CVT division method. When the CVT division method is used, the training set is divided into K non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model verification sets. The optimal value of K is determined by setting different K value gradients and performing deviation and variance analysis; Establish an effluent COD, TN, and TP soft-sensing model based on XGBoost and use the model training set for training, use the model verification set to adjust and optimize the hyperparameters, and use the model test set to detect the trained effluent COD, TN, and TP soft-sensing model based on XGBoost. The effluent COD, TN, and TP soft-sensing model based on XGBoost includes an effluent COD soft-sensing model, an effluent TN soft-sensing model, and an effluent TP soft-sensing model. Constructing a KNN input missing processing model through a KNN algorithm, wherein the KNN input missing processing model is used to perform integrity check and / or missing processing on input data; The trained effluent COD soft-sensing model, effluent TN soft-sensing model, effluent TP soft-sensing model and KNN input missing processing model are combined to form a KNN-XGBoost comprehensive soft-sensing model. The data to be predicted is input into the KNN-XGBoost comprehensive soft sensor model for prediction, and the predicted values ​​of effluent COD, effluent TN and effluent TP are obtained.

2. A water quality index prediction method according to claim 1, characterized in that: The data in the water quality data set include the data monitored by the water quality online monitoring probe and the manual detection data of the inlet and outlet water. The data monitored by the water quality online monitoring probe include the MLSS concentration of sludge in the sedimentation tank. sedi , Oxidation Reduction Potential (ORP) in Sedimentation Tanks sedi , dissolved oxygen concentration DO in aeration tank area , Oxidation Reduction Potential (ORP) in Aeration Tanks area , dissolved oxygen concentration DO in the biological purification tank ep , conductivity EC in biological purification tank ep , the temperature in the biological purification tank T ep , MLSS sludge concentration in biological purification tank ep and pH in biological purification tanks ep , the manual detection data of inlet and outlet water COD concentration in inlet water inf , TN concentration in influent inf TP concentration in influent inf , COD concentration in effluent eff , TN concentration in effluent eff and TP concentration in effluent eff .

3. A water quality index prediction method according to claim 2, characterized in that: The input features of the effluent COD, TN, and TP soft-sensing model based on XGBoost were screened by mutual information values, the MI value between each variable and the model output was calculated, and the variables with MI values ​​higher than the threshold were selected as the input features of the model.

4. A water quality index prediction method according to claim 1, characterized in that: For the two data partitioning methods, when training the model, both random search and grid search are used to find and optimize the best hyperparameters of the model. Specifically, random search is first used to determine the approximate range of each hyperparameter of the model with a large step size, and then grid search is used to determine the best value of each hyperparameter within the range with a small step size; after finding the best hyperparameter, the model is tested using the model test set data; this training and testing process is repeated until the trained model meets the predetermined requirements.

5. A water quality index prediction method according to claim 1, characterized in that: The KNN input missing processing model sets different missing degrees and then uses the training set data to establish the KNN input missing processing model. Through the correlation between the features and the values ​​of the non-missing features, the input missing parts under different missing degrees are interpolated and filled.

6. A water quality index prediction method according to claim 1, characterized in that: In the KNN input missing processing model, after the input data is input into the KNN input missing processing model, it is first checked whether there are missing values ​​in the input data. If there are missing values, interpolation processing is performed on the missing parts.

7. A water quality index prediction method according to any one of claims 1 to 6, characterized in that: Python was used to establish the soft measurement model of effluent COD, TN and TP based on XGBoost.

8. A water quality index prediction system, characterized in that: For implementing the method according to any one of claims 1 to 7, the system comprises the following modules: A data set construction module is used to construct a water quality data set, and divide the water quality data set into a model training set, a model verification set and a model test set according to the TVT division method, or divide it into a model training set and a model test set according to the CVT division method. When the CVT division method is used, the training set is divided into K mutually non-overlapping subsets of the same size, and then K-1 subsets are used as model training sets, and the remaining subsets are used as model verification sets, wherein the optimal value of K is determined by setting different K value gradients and performing deviation and variance analysis; A model building module is used to establish an effluent COD, TN, and TP soft measurement model based on XGBoost and use a model training set for training, use a model verification set for hyperparameter adjustment and optimization, and use a model test set to detect the trained effluent COD, TN, and TP soft measurement model based on XGBoost. The effluent COD, TN, and TP soft measurement model based on XGBoost includes an effluent COD soft measurement model, an effluent TN soft measurement model, and an effluent TP soft measurement model; A missing processing module is used to construct a KNN input missing processing model through a KNN algorithm, wherein the KNN input missing processing model is used to perform integrity check and / or missing processing on input data; A combination module is used to combine the trained effluent COD soft-sensing model, effluent TN soft-sensing model, effluent TP soft-sensing model and KNN input missing processing model to form a KNN-XGBoost comprehensive soft-sensing model; The prediction module is used to input the data to be predicted into the KNN-XGBoost comprehensive soft sensor model for prediction, and obtain the effluent COD prediction value, effluent TN prediction value and effluent TP prediction value.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, a water quality index prediction method according to any one of claims 1 to 7 is implemented.

10. A computing device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, a water quality index prediction method described in any one of claims 1-7 is implemented.

Images