Method for correcting monitoring data errors caused by pretreatment in water quality online monitoring system

Abstract

The present application relates to a kind of water quality online monitoring system in the pre-treatment caused by monitoring data error correction method, comprising the following steps: (1) data acquisition and calculation, (2) the data set needed for building deep learning, (3) model establishment, (4) model parameter optimization training, (5) calculate water quality error influence factor, (6) water quality online detection data calibration, the present application is by collecting the operating state parameters of water quality online monitoring device and water quality pretreatment device, using deep learning model to predict the detection error rate caused by water quality pretreatment device, and through the prediction result, the detection data of water quality online monitoring system is data correction, greatly improve the accuracy of online monitoring instrument detection data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water quality online monitoring, and particularly relates to a method for correcting monitoring data errors caused by pretreatment in a water quality online monitoring system. BACKGROUND

[0002] The conventional detection of surface water quality mainly includes nine parameters: water temperature, pH, turbidity, conductivity, dissolved oxygen, ammonia nitrogen, permanganate index or COD, total phosphorus, and total nitrogen. The surface water quality online monitoring system generally comprises a water quality online monitoring device and a water quality online pretreatment device. The water quality online monitoring device is mainly used for real-time monitoring and continuous testing of the surface water body, and the water quality and subsequent treatment work are determined through the monitoring and testing data. The surface water contains a large amount of suspended matter, which can seriously damage the service life of the online monitoring instrument, and the water quality online pretreatment device is a prerequisite for ensuring the normal operation of the water quality online monitoring device. Therefore, in the water quality online monitoring system, the water scale, silt and other impurities in the water need to be pretreated by the water quality online pretreatment device, and the filtration operation is one of the common pretreatment methods. However, the retained substances in the filter element and the soluble substances dissolved in the water sample will affect the accuracy of the results, and the effect of the filter element will decrease with the extension of the use time.

[0003] In the process of water quality monitoring, in order to ensure the accuracy of the analysis results, most researchers consider adjusting the pretreatment structure and method, but no one has studied the influence of pretreatment on water quality data. This influence is a nonlinear relationship and involves many factors such as water quality, sedimentation time, and filter structure, so that there is a certain error between the pretreated water sample and the original water sample. The existing model and algorithm are mostly linear structures, which result in weak learning ability and cannot model and correct the relatively complex filtration conditions. The deep learning algorithm is more and more applied in water environment modeling due to its good characteristics in nonlinear simulation, and is applied in water quality prediction, pollution early warning, etc. SUMMARY

[0004] In order to solve the above problems, the present application provides a method for correcting monitoring data errors caused by pretreatment in a water quality online monitoring system, which corrects the errors between the monitoring data caused by the water quality pretreatment device and the actual manual laboratory detection in the water quality online monitoring system, reduces the influence of pretreatment filtration on the water quality monitoring results, and improves the accuracy of the online monitoring data.

[0005] In order to achieve the above purpose, the present application provides a method for correcting monitoring data errors caused by pretreatment in a water quality online monitoring system, which is characterized by comprising the following steps:

[0006] (1) Data acquisition and calculation

[0007] In the maintenance period D of the water quality pretreatment device, the water quality pressure p of the water quality pretreatment device is collected i And turbidity S2 i Data; at the same time, the water temperature t of the surface water of the water quality online monitoring device is collected i , water quality q i , turbidity S1 i , running days d in the maintenance period i And sedimentation time TC i ,

[0008] The influent water quality Q of the water quality pretreatment device detected in the laboratory is collected 进水 And effluent water quality Q 出水 , the water quality q i , influent water quality Q 进水 And effluent water quality Q 出水 Any one of CODCR, permanganate index, total phosphorus, ammonia nitrogen and total nitrogen;

[0009] According to the collected data, the maintenance period factor, the sedimentation factor, the suspended matter filtration rate and the water quality error influence factor are calculated;

[0010] The maintenance period factor is d i / D;

[0011] The sedimentation factor is TC i / TC, and TC is the standard sedimentation time of lake water sampling;

[0012] The suspended matter filtration rate is (S1 i -S2 i ) / S1 i ;

[0013] The water quality error influence factor is Q 进水 / (Q 出水 +Q 进水 );

[0014] (2) Construct the data set required for deep learning

[0015] After removing the error data of the pressure p i , water temperature t i , water quality q i , maintenance period factor, sedimentation factor, suspended matter filtration rate and water quality error influence factor data, the data set required for deep learning is constructed, and the data set is divided into training set, validation set and test set, wherein the water quality error influence factor is the true label of the data set;

[0016] (3) Model establishment

[0017] A model is established based on pressure p i , sedimentation factor, water temperature ti , suspended solids filtration rate, water quality q i , the deep learning model with the maintenance cycle factor as input and the water quality error influence factor as output;

[0018] (4) model parameter optimization training

[0019] Set the initial parameters of the deep learning model, train, test and verify the deep learning model using the data set, obtain the available model parameters, and obtain the error correction model;

[0020] (5) Calculate the water quality error influence factor

[0021] Input the pressure p of the sampling time in the error correction model i , turbidity S2 i , water temperature t i , water quality qi, turbidity S1i, and running days d in the maintenance cycle i and sedimentation time TC i Data, calculate the corresponding water quality error influence factor;

[0022] (6) Water quality online detection data calibration

[0023] According to the online detection of the effluent water quality Q 出水 and the water quality error influence factor obtained in step (5), the calibrated influent calibration water quality Q 进水校准 is calculated, and the calculation formula is:

[0024]

[0025] The water quality online monitoring system provided by the application provides a monitoring data error correction method caused by pretreatment, by collecting the operating state parameters of the water quality online monitoring device and the water quality pretreatment device, using a deep learning model to predict the detection error rate caused by the water quality pretreatment device, and correcting the detection data of the water quality online monitoring system through the prediction result, greatly improving the accuracy of the detection data of the online monitoring instrument. DETAILED DESCRIPTION

[0026] The water quality online monitoring system includes a water quality online monitoring device and a water quality pretreatment device, and the surface water body first passes through the water quality pretreatment device for pretreatment before entering the water quality online monitoring device, and is pretreated by filtration and sedimentation, etc., to remove suspended solids and other substances that affect the detection result. The water quality online monitoring device is a device for water quality detection by international law or electrode method, mainly for real-time monitoring and continuous testing of surface water bodies, with a data collector capable of online collecting the water temperature t i , water quality q i , turbidity S1 i , running days d in the maintenance cyclei , sedimentation time TC i , etc. Through various monitoring and inspection data, the water quality and subsequent treatment work are judged.

[0027] A pressure sensor is installed on the pipeline connected with the sampling pump and filter of the water quality pretreatment device, which is used for monitoring the filtration system of the water quality pretreatment device, and the pressure of the filtration system in the water quality pretreatment device can be monitored online. Preferably, the pressure sensor has an accuracy of 0.001 mPa or more, and the output is a 485 digital signal quantity. A turbidity sensor is installed in the sampling pool of the water quality pretreatment device, which is used for monitoring the effluent at the end of the water quality pretreatment device, and the turbidity of the water sample can be monitored online. Preferably, the turbidity sensor has an accuracy of 0.01 NTU or more, and the output is a 485 digital signal quantity. The data collector is connected with the pressure sensor and the turbidity sensor, and the data collected by the pressure sensor and the turbidity sensor are uploaded to the data end of the error correction system. Preferably, the data collector has an edge computing function, which can process the collected data. In this embodiment, the data collector is the same as the data collector of the water quality online monitoring device, that is, the data collector of the water quality online monitoring device simultaneously collects the data of the water quality online monitoring device and the water quality pretreatment device.

[0028] The error correction system includes a data end, an algorithm server, a database, and an error correction platform. The data end sends the data collected by the data collector to the algorithm server and stores it in the database. The database is a MySQL database.

[0029] The monitoring data error correction method caused by pretreatment in the water quality online monitoring system includes the following steps:

[0030] (1) Data collection and calculation

[0031] In the maintenance period D of the water quality pretreatment device, the water quality pressure p i and turbidity S2 i of the water quality pretreatment device are collected every day at a certain interval. i , water quality q i , turbidity S1 i , running days d i in the maintenance period, and sedimentation time TC i of the surface water of the water quality online monitoring device are collected simultaneously. The maintenance period D is the maintenance time for replacing the filter element of the water quality pretreatment device. In this embodiment, the maintenance period D is 60 days, and the collection is performed every 4 hours per day. The running days d i in the maintenance period refer to the number of days that the water sample determined each time runs within the maintenance period of 60 days of the water quality pretreatment device. The sedimentation time TC i is the sedimentation time of the water sample determined each time in the water quality pretreatment device. The pressure p iand turbidity S2 i , respectively detected by the pressure sensor and the turbidity sensor and collected by the data collector and then sent to the data terminal of the error correction system. Water temperature t i , water quality q i , turbidity S1 i , number of days of operation d i in the maintenance period, and sedimentation time TC i collected by the data collector and then sent to the data terminal of the error correction system, the data collector being provided in the existing water quality online monitoring device, and in the embodiment, the data collector is the same as the data collector receiving the data detected by the pressure sensor and the turbidity sensor. The data collector receiving the data detected by the pressure sensor and the turbidity sensor can also use a separate data collector.

[0032] The water samples of the influent and the effluent of the water quality pretreatment device are manually collected, and the influent water quality Q 进水 and the effluent water quality Q 出水 are detected in the laboratory. The water quality q i , the influent water quality Q 进水 , and the effluent water quality Q 出水 are all any one of CODCR, permanganate index, total phosphorus, ammonia nitrogen, and total nitrogen. In the embodiment, the water quality q i , the influent water quality Q 进水 , and the effluent water quality Q 出水 are all the ammonia nitrogen concentration of the water quality, the water quality q i is the ammonia nitrogen concentration of the surface water monitored by the water quality online monitoring device, the influent water quality Q 进水 is the ammonia nitrogen concentration of the influent and the ammonia nitrogen concentration of the effluent of the water quality pretreatment device detected in the laboratory. The data collector collects the influent water quality Q 进水 and the effluent water quality Q 出水 data.

[0033] The data collector uploads the collected data to the data terminal of the error correction system, and the data terminal sends the data to the algorithm server and stores the data in the database. The algorithm server calculates the maintenance period factor, the sedimentation factor, the suspended solids filtration rate, and the water quality error influence factor according to the collected data. The maintenance period factor is d i / D = d i / 60, the sedimentation factor is TC i / TC, in the embodiment, TC is the standard sedimentation time of lake water sampling, which is a constant value of 30 minutes, that is, the sedimentation factor is TC i / TC = TC i / 30. The suspended solids filtration rate is (S1 i -S2 i ) / S1 i , and the water quality error influence factor is Q进水 出水 +Q 进水 ), the water quality error factor represents the difference between the influent water quality and the effluent water quality of the water quality pretreatment device, is designed to calibrate the detection data of the instrument, and is the true label of the data set.

[0034] (2) Constructing a data set required by deep learning

[0035] The above pressure p i , water temperature t i , water quality q i , maintenance cycle factor, sedimentation factor, suspended solids filtration rate, and water quality error influence factor data are removed from the error data, and a data set required by deep learning is constructed. The data set is divided into a training set, a validation set, and a test set, wherein the water quality error influence factor is the true label of the data set.

[0036] Select a total of N groups of the same water quality online monitoring system, the test water quality of the selected system has a wide range of representation, and the total data set is 6*D*N groups. When the data sample of the data set is more than 2000 groups, the training set, the validation set, and the test set each account for 10:2:2. In this embodiment, N is 20, and D is 60, so that the total data set is 7200 groups, the training set, the test set, and the validation set each account for 10:2:2, and the training set, the test set, and the validation set are respectively allocated 5143 groups, 1028 groups, and 1029 groups of data sets. When the data sample of the data set is less than 2000 groups, K-fold cross-validation method is used for division, 20% of the data is selected as the validation set, and the remaining 80% is used as the training set. The K-fold cross-validation is used, and the optimal K value is 10.

[0037] (3) Model establishment

[0038] A deep learning model is established, taking pressure p i , sedimentation factor, water temperature t i , suspended solids filtration rate, water quality q i , and maintenance cycle factor as input, and water quality error influence factor as output.

[0039] The deep learning model at least includes three fully connected layers, and the last layer does not use an activation function. The loss function is:

[0040]

[0041] The deep learning model constructed in this embodiment includes seven fully connected layers, and the last layer does not use an activation function. The loss function is:

[0042]

[0043] (4) Model parameter optimization training ​

[0044] The initial parameters of the deep learning model are set, the deep learning model is trained, tested and verified using the data set, the available model parameters are obtained, and the error correction model is obtained.

[0045] The deep learning model training includes: using the training set and using the stochastic gradient descent method to train the model parameters, the verification set is used to adjust the hyperparameters, the optimal hyperparameter value of the model is determined, and the test set is used to detect the model and evaluate the accuracy of the model.

[0046] The model parameters are trained using the stochastic gradient descent method, and in order to prevent overfitting, L2-norm or dropout method is used for optimization, and the method with better simulation effect can be selected in the two methods. When L2-norm optimization is used, the loss function is:

[0047] (gamma is the regularization coefficient).

[0048] When the dropout method is used for optimization, the dropout parameter is preferably set to 0.45-0.55.

[0049] (5) Calculate the water quality error influence factor

[0050] The obtained error correction model is deployed on the error correction platform, and the error correction platform obtains the pressure p i , turbidity S2 i , water temperature t i , water quality qi, turbidity S1i, running days d i and sedimentation time TC i data in the sampling moment from the database. In the error correction model, input the pressure p i , turbidity S2 i , water temperature t i , water quality qi, turbidity S1i, running days d i and sedimentation time TC i data in the sampling moment, and calculate the corresponding water quality error influence factor.

[0051] (5) Water quality online detection data calibration

[0052] According to the online detection of the effluent water quality Q 出水 and the water quality error influence factor obtained in step (5), the calibrated influent calibration water quality Q 进水校准 is calculated. In step (1), the water quality error influence factor is defined as Q 进水 / (Q 出水 +Q 进水 ), and the water quality error influence factor is calculated in step (5). The water quality of the water quality pretreatment device detected online Q 出水Calibration is performed, and the corresponding calibrated influent calibration water quality Q is obtained by prediction 进水校准 , the influent calibration water quality Q 进水校准 Can be considered as the actual influent water quality. The calculation formula is:

[0053]

[0054] The error correction platform can visualize the water quality correction result.

[0055] Compared with the traditional technical scheme, the error of the monitoring data is reduced by improving the process and structure of the pretreatment system. According to the change of the running state parameter of the water quality online monitoring system as input, the detection error rate caused by the water quality pretreatment device is taken as the expected output label, the initial parameter is set, the model training is carried out by using the training set parameter to obtain the parameter of the model, the output result is tested by the test set, the training is repeated until the prediction ability reaches the requirement, and the applied model is obtained. Finally, the model is applied to the correction of the water quality monitoring data, and the calibration of the data is realized. In addition to considering the running condition of the water quality pretreatment device, the maintenance of the pretreatment device, the water quality of the running water body, the water temperature and other multi-dimensional data are also considered. These will affect the treatment effect of the pretreatment device, and then affect the detection result of the water quality online monitoring instrument. Therefore, considering the parameters outside the working condition can improve the accuracy of data prediction and achieve better detection effect.

Claims

1. A method for correcting monitoring data errors caused by preprocessing in an online water quality monitoring system, characterized in that, Includes the following steps: (1) Data acquisition and calculation During the maintenance cycle D of the water pretreatment device, the water pressure p of the water pretreatment device was collected. i and turbidity S2 i Data; simultaneously, surface water temperature (t) is collected by the online water quality monitoring device. i Water quality q i Turbidity S1 i Number of operating days during the maintenance cycle (d) i and sedimentation time TC i , Collect water quality Q from the influent of the water pretreatment unit for laboratory testing. 进水 and effluent water quality Q 出水 The water quality q i Influent water quality Q 进水 and effluent water quality Q 出水 It is any one of CODCR, permanganate index, total phosphorus, ammonia nitrogen, and total nitrogen; Based on the collected data, the maintenance cycle factor, sedimentation factor, suspended solids filtration rate, and water quality error impact factor are calculated. The maintenance cycle factor is d. i / D; The precipitation factor is TC. i / TC, where TC is the standard sedimentation time for lake water sampling; The suspended solids filtration rate is (S1) i -S2 i ) / S1 i ; The water quality error impact factor is Q. 进水 / (Q 出水 +Q 进水 ); (2) Constructing the dataset required for deep learning The pressure p i Water temperature t i Water quality q i After removing erroneous data from the data on maintenance cycle factor, sedimentation factor, suspended solids filtration rate, and water quality error impact factor, the dataset required for deep learning is constructed. The dataset is divided into training set, validation set, and test set, where the water quality error impact factor is the true label of the dataset. (3) Establishing a model Establish based on pressure p i Sedimentation factor, water temperature t i Filtration rate of suspended solids, water quality q i A deep learning model with maintenance cycle factor as input and water quality error impact factor as output; (4) Model parameter optimization training Set the initial parameters of the deep learning model, use the dataset to train, test and validate the deep learning model, obtain usable model parameters, and obtain an error correction model; (5) Calculate the water quality error impact factor The pressure p at the sampling time is input into the error correction model. i Turbidity S2 i Water temperature t i Water quality (qi), turbidity (S1i), and number of operating days (d) within the maintenance cycle. i and sedimentation time TC i The data was used to calculate the corresponding water quality error impact factors. (6) Calibration of online water quality monitoring data According to the online monitoring of the effluent water quality Q 出水 The water quality error influence factor obtained in step (5) is used to calculate the calibrated influent calibration water quality Q. 进水校准 The calculation formula is:

2. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 1, characterized in that, The deep learning model in step (3) includes at least three fully connected layers, with the last layer not using an activation function, and its loss function is:

3. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 2, characterized in that, In step (4), the deep learning model training includes: using a training set and training the model parameters using stochastic gradient descent; using a validation set to adjust hyperparameters and determine the optimal hyperparameter values ​​of the model; and using a test set to test the model and evaluate the model's accuracy.

4. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 3, characterized in that, The stochastic gradient descent method is used to train the model parameters, which are optimized using the L2-norm method. The loss function is:

5. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 3, characterized in that, The parameters of the model trained using stochastic gradient descent are optimized using the dropout method.

6. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 5, characterized in that, The dropout parameter is set to 0.45-0.

55.

7. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to any one of claims 1-6, characterized in that, In step (2), when the dataset has more than 2,000 data samples, the ratio of the training set, validation set and test set is 10:2:

2.

8. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to any one of claims 1-6, characterized in that, In step (2), the dataset has fewer than 2000 data samples. It is divided using K-fold cross-validation, with 20% of the data selected as the validation set and the remaining 80% used as the training set for K-fold cross-validation.

9. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 8, characterized in that, The water quality q i Influent water quality Q 进水 and effluent water quality Q 出水 All values ​​represent ammonia nitrogen concentrations in the water.

10. The method for correcting monitoring data errors caused by preprocessing in the online water quality monitoring system according to claim 9, characterized in that, The value of K is 10.

Images