A method for measuring water quality with multiple parameters

Abstract

The present invention mainly relates to a multi-parameter water quality measurement method, which belongs to the field of water quality detection technology. It includes the following steps: building an ultraviolet-visible absorption spectrum acquisition device; collecting spectral data of the water sample to be tested; performing chemical analysis of organic matter in accordance with national standards or industry standards; constructing a water quality organic matter prediction model based on the Stacking fusion algorithm; and outputting the predicted value of organic matter. The present invention mainly uses the Stacking fusion algorithm to establish a prediction model for organic matter, and integrates multiple sub-models through an integration strategy, which can effectively avoid the problem that the accuracy of the results obtained by a single model when the sample components are complex is low or unstable, and solves the defects of traditional experimental methods such as complex operation, possible secondary pollution, and low accuracy of prediction models.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of water quality detection, and specifically to a water quality multi-parameter measurement method. Background Art

[0002] With the development of society and the acceleration of urbanization, a large amount of nitrogen-containing wastewater is directly discharged into the water, which seriously threatens the living environment of human beings and is not conducive to the sustainable development of society. The organic pollutants in urban and surrounding surface water mainly come from land-based living sources, surface runoff, industry, service industry, aquaculture and aquatic biological sources, mainly organic pollutants such as protein, amino acids, humic acid and fat. Environmental monitoring technology characterizes organic pollution in water bodies through indicators such as chemical oxygen demand (CODCr), permanganate index (CODMn), ammonia nitrogen (NH3-N), total phosphorus (TP), total nitrogen (TN) and five-day biochemical oxygen demand (BOD5). Among them, CODCr, CODMn and BOD5 are usually used to indicate the total amount of organic pollutants in water bodies; the increase in the content of NH3-N, TN and TP will lead to eutrophication of water bodies, destroy biodiversity and produce odor.

[0003] Traditional chemical methods for water quality monitoring have many interferences, poor selectivity, harsh experimental conditions, long analysis time, and the need to consume chemical reagents, making it difficult to meet the requirements of practical applications. UV-visible absorption spectroscopy is a method for analyzing substances based on spectral data in the ultraviolet and visible bands (200-800nm). Spectroscopy has incomparable advantages over chemical methods, namely, no sample treatment, no pollution, simple operation, and real-time. It has been widely used in environmental monitoring, pollutant analysis, drug testing, food safety and other fields. As optical instruments become more and more sophisticated, the spectral data collected by spectrometers are becoming more and more complete, and can better reflect the composition of complex water bodies.

[0004] Ultraviolet absorption spectroscopy combined with chemometric methods has been widely used in water quality monitoring indicators such as COD, chromaticity, and turbidity. However, with the increase in sample components and the complexity of sample spectral information, the accuracy of the results obtained by predicting only through a single model is usually low or unstable. Ensemble learning can fuse multiple sub-models through a certain strategy. The fusion model has higher stability and better prediction performance than the sub-model. Applying ensemble learning to spectral water quality detection can improve the upper limit of existing models and has great application potential. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-parameter water quality measurement method to solve the problem that the prediction results of a single model for complex water bodies are of low accuracy or are not stable enough, as well as the defects of traditional water body organic matter detection, such as complex operation, high cost, easy to cause secondary pollution, and low prediction model accuracy.

[0006] The technical solution adopted to achieve the purpose of the present invention is:

[0007] 1. Use UV-visible absorption spectroscopy as an alternative to traditional chemical detection methods;

[0008] 2. Establish a regression relationship between UV-visible absorption spectrum data and chemical values;

[0009] 3. Use the Stacking fusion model to ensure the stability of regression prediction.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. This invention illustrates the feasibility of using UV-visible spectroscopy technology to monitor water pollution, and can provide a solution for rapid, in-situ, and efficient monitoring of surface water;

[0012] 2. Solved the defects of traditional experimental methods such as complex operation and possible secondary pollution;

[0013] 3. The problem that the prediction results of water quality organic matter by a single model are usually low in accuracy or unstable has been solved.

[0014] The multi-parameter water quality measurement method described in the present invention can realize the measurement of chemical oxygen demand CODCr, permanganate index CODMn, five-day biochemical oxygen demand BOD5, total organic carbon TOC, dissolved organic carbon DOC, ammonia nitrogen NH3-N, total phosphorus TP, total nitrogen TN, nitrate nitrogen NO3-N, nitrite nitrogen NO2-N, turbidity, chromaticity, total suspended solids, phenol BTX, ozone, hydrogen sulfide or UV254 or simultaneous measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The process of model establishment and use;

[0016] Figure 2 Create a flow chart for Model 1;

[0017] Figure 3 Create a flow chart for Model 2;

[0018] Figure 4 Create a flow chart for Model 3;

[0019] Figure 5 Create a flow chart for Model 4;

[0020] Figure 6 Schematic diagram of the Stacking fusion process;

[0021] Figure 7 The fitting results of the water quality nitrate nitrogen training set and test set model;

[0022] Figure 8 Schematic diagram of model fusion DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] Combination Figures 1 to 7 As shown, taking the establishment of water quality nitrate nitrogen model as an example, the present invention provides the following technical solutions:

[0025] Sample collection:

[0026] Water samples were collected multiple times from 9 surface water monitoring sections. Water from deep below the water surface was collected using an upright sampler and divided into brown glass bottles in parallel according to the water quality sampling specifications. A total of 72 samples were obtained after multiple sampling. After dilution with deionized water, a total of 162 samples were obtained. The supernatant was collected after standing for 30 minutes.

[0027] Chemical analysis:

[0028] To ensure the consistency of samples for chemical analysis and spectral analysis, shake the sample and let it stand for 30 minutes before taking the supernatant for testing. The testing method refers to the relevant national and industry standards. The chemical analysis of nitrate nitrogen in water quality is measured using the method in GB7480-1987.

[0029] S1. UV-visible absorption spectrum measurement:

[0030] The collection device of UV-visible absorption spectrum mainly includes light source, spectrometer, sample slot, optical fiber, etc. The light source adopts deuterium halogen lamp (AvaLight-DHc), and the output light of the light source is coupled to the sample slot through optical fiber. In order to improve the transmission efficiency of incident light and output light, the sample slot adopts 10×10mm quartz cuvette. The generated transmitted light is received by optical fiber spectrometer (Fuxiang Optics, wavelength range is 197~987nm, wavelength resolution is 0.73nm). The collected UV-visible spectrum data is transmitted to the computer through data line for data processing and modeling analysis.

[0031] Data model algorithm establishment:

[0032] The present invention adopts Python to construct a water quality index prediction model. The training set and test set samples are divided into training set and test set samples by random sampling. 30% of the samples are selected as the test set to evaluate the generalization ability and prediction effect of the model, and the remaining samples are used as the training set to establish the prediction model.

[0033] S21. Data preprocessing

[0034] Process the spectral data collected by S1 and use SG filtering and trend correction to process the spectral data in turn.

[0035] S22. Data preprocessing

[0036] Process the spectral data collected by S3, and use SG filtering, constant offset elimination, first-order derivative, and trend correction to process the spectral data in turn;

[0037] S3, S4 data feature engineering and base model establishment

[0038] The flowchart of base model 1 is as follows Figure 2 As shown. KPCA-XGBoost is used to reduce the dimension of the spectral data preprocessed in step S21, and the reduced-dimensional data is input into the algorithm as a feature. The chemical analysis results of various water quality indicators are used as the target values ​​of the algorithm, and the regression prediction model is established using the XGBoost algorithm. The hyperparameters of XGBoost are optimized by combining grid search and cross-validation, and the minimum root mean square error (CVMSE) of the cross-validation set is used as the optimization goal;

[0039] Base model 2 established

[0040] The flowchart of base model 2 is as follows Figure 3 As shown. Use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S21. Input the selected characteristic wavelengths into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, and use the XGBoost algorithm to establish a regression prediction model. Use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error (CVMSE) of the cross-validation set as the optimization goal;

[0041] Base model 3 established

[0042] The flowchart of base model 3 is as follows Figure 4 As shown. KPCA-SVR is used to reduce the dimension of the spectral data preprocessed in step S22, and the reduced data is input into the algorithm as a feature. The chemical analysis results of various water quality indicators are used as the target values ​​of the algorithm, and the regression prediction model is established using the SVR algorithm. The hyperparameters of SVR are optimized by combining the optimization algorithm with cross-validation, and the minimum root mean square error (CVRMSE) of the cross-validation set is used as the optimization goal;

[0043] Base Model 4 Establishment

[0044] The flowchart of base model 4 is as follows Figure 5As shown. Use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S22. Input the selected characteristic wavelengths into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, and use the SVR algorithm to establish a regression prediction model. Use the optimization algorithm and cross-validation to optimize the hyperparameters of SVR, and take the minimum root mean square error (CVRMSE) of the cross-validation set as the optimization goal;

[0045] S5. Establishment of fusion model

[0046] The fusion process diagram is as follows: Figure 6 As shown. The output results of the four base models are integrated, and linear regression is used to establish the regression equation of the output results of the four models and the chemical analysis results. The output of each base model needs to be cross-validated to prevent linear regression overfitting;

[0047] S6. Model prediction output

[0048] The spectral data of the training set were preprocessed and feature engineered and then input into the fusion model established by S5, and the model output results were recorded. The determination coefficient R 2 The root mean square error RMSE is used as the evaluation index of the model effect of the present invention. and the test set determination coefficient The closer it is to 1, the higher the model correlation and the better the prediction effect; the smaller the values ​​of the root mean square error RMSEC of the training set and the root mean square error RMSEP of the test set, the higher the model accuracy and the stronger the generalization ability. The determination coefficient of the training set and the test set is calculated according to Formula 1, and the root mean square error of the training set and the test set is calculated according to Formula 2.

[0049] The formula 1 is:

[0050] The formula 2 is:

[0051] In formula 1 and 2, is the predicted value of the i-th sample model, is the mean of the model's predicted values ​​for all samples, y i is the true value of the i-th sample, and m is the number of samples.

[0052] S5. Combine algorithms and various standard data to establish a water quality index model, so as to conduct real-time detection and monitoring of surface water pollution.

[0053] The prediction results of water quality nitrate nitrogen index based on the full-band spectral model are shown in Table 1. It can be seen that after the stacking fusion of the base model, the performance of the Stacking fusion model has been greatly improved. 2Compared with the other four basic models, the training set determination coefficients of the water quality nitrate nitrogen prediction models were improved by 1.36%, 1.41%, 1.60% and 1.73% respectively. are greater than 0.9, and Close to This shows that the predicted values ​​of the Stacking fusion algorithm have a high degree of fit with the chemical analysis values ​​of water quality nitrate nitrogen. In addition, the gap between RMSCP and RMSEC of each water quality nitrate nitrogen prediction model is small, indicating that the trained model has high accuracy, strong prediction ability and good generalization ability.

[0054] Figure 7 The fitting results of the modeling of the training set and the test set. The determination coefficient R of the fitting curve of the training set and the test set 2 Both are as high as 0.98, indicating that the modeling method used in this paper is correct, and the established model has good robustness and adaptability, and can complete the detection of nitrate nitrogen in water quality. This method can provide a solution for rapid, in-situ, and efficient monitoring of surface water.

[0055] Table 1 Model prediction results

[0056]

[0057] A water quality multi-parameter measurement method of the present invention comprises the following steps:

[0058] S101, UV-visible absorption spectrum measurement: using a spectrometer to collect spectrum data of the water sample to be tested;

[0059] S102, data preprocessing: preprocessing the collected spectral data;

[0060] S103, data feature engineering: performing feature engineering on the preprocessed data;

[0061] S104, input model calculation: the data after feature engineering is fused into the model, and the model outputs the result.

[0062] The data preprocessing in step S102 is specifically as follows:

[0063] S1021, preprocessing scheme 1: process the spectral data collected in S1, and sequentially process the spectral data using SG filtering and trend correction;

[0064] S1022, preprocessing scheme 2: process the spectral data collected in S1, and sequentially process the spectral data using SG filtering, constant offset elimination, first-order derivative, and trend correction.

[0065] The data feature engineering step S103 includes:

[0066] S1031, feature engineering solution 1: use KPCA-XGBoost to reduce the dimension of the spectral data preprocessed in step S1021;

[0067] S1032, feature engineering solution 2: use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S1021;

[0068] S1033, feature engineering solution 3: using KPCA-SVR to reduce the dimension of the spectral data pre-processed in step S1022;

[0069] S1034. Feature engineering solution 4: Use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S1022.

[0070] The base model establishment in step S104 is specifically as follows:

[0071] S1041, base model 1 establishment: input the features after dimension reduction in feature engineering solution 1 in step S1031 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use the XGBoost algorithm to establish a regression prediction model, use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error (CVMSE) of the cross-validation set as the optimization goal;

[0072] S1042, base model 2 establishment: input the features screened out by feature engineering scheme 2 in step S1032 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use XGBoost algorithm to establish a regression prediction model, use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error (CVMSE) of the cross-validation set as the optimization goal;

[0073] S1043, base model 3 establishment: input the features after dimension reduction in feature engineering solution 3 in step S1033 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use the XGBoost algorithm to establish a regression prediction model, use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error (CVMSE) of the cross-validation set as the optimization goal;

[0074] S1044, base model 4 establishment: the features screened out by feature engineering scheme 4 in step S1034 are input into the algorithm, the chemical analysis results of various water quality indicators are used as the target values ​​of the algorithm, the regression prediction model is established using the XGBoost algorithm, and the hyperparameters of XGBoost are optimized by combining grid search and cross-validation, with the minimum root mean square error (CVMSE) of the cross-validation set as the optimization goal.

[0075] The prediction accuracy of a single model can be improved by fusing multiple models. Compared with a single model, the fused model has higher accuracy, stronger robustness, and better generalization ability.

[0076] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A water quality multi-parameter measurement method, characterized in that: The creation method is: S0. Collect sample water and measure the actual concentration value of the parameter according to the national standard method; S1. UV-visible absorption spectrum measurement: using a spectrometer to collect spectrum data of sample water; S2, data preprocessing: preprocessing the collected spectral data; S3. Data feature engineering: Perform feature engineering on the preprocessed data; S4. Base model establishment: Use SVR and XGBoost to establish the base model; S5, fusion model establishment: use Stacking fusion algorithm to establish fusion prediction model; S6. Model evaluation: using the coefficient of determination R 2 And root mean square error RMSE as the evaluation index of model effect; S7, output of measurement results: according to the collected spectral data of unknown concentration, the measurement results are output after steps S2-S5; further, the data preprocessing in step S2 is specifically as follows: S201, preprocessing scheme 1: process the spectral data collected in S1, and sequentially process the spectral data using SG filtering and trend correction; S202, preprocessing scheme 2: process the spectral data collected in S1, and sequentially process the spectral data using SG filtering, constant offset elimination, first-order derivative, and trend correction; Furthermore, step S3 data feature engineering includes: S301, feature engineering solution 1: using KPCA-XGBoost to reduce the dimension of the spectral data pre-processed in step S201; S302, feature engineering solution 2: use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S201; S303, feature engineering solution 3: using KPCA-SVR to reduce the dimension of the spectral data pre-processed in step S202; S304, feature engineering solution 4: use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S202; Furthermore, step S4 of establishing the base model is specifically as follows: S401, base model 1 establishment: input the features after dimension reduction of feature engineering scheme 1 in step S301 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use XGBoost algorithm to establish a regression prediction model, use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error CVMSE of the cross-validation set as the optimization goal; S402, base model 2 establishment: input the features screened out by feature engineering scheme 2 in step S302 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use the XGBoost algorithm to establish a regression prediction model, use grid search and cross-validation to optimize the hyperparameters of XGBoost, and take the minimum root mean square error CVMSE of the cross-validation set as the optimization goal; S403, base model 3 establishment: input the features after dimension reduction of feature engineering solution 3 in step S303 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use the SVR algorithm to establish a regression prediction model, use the optimization algorithm and cross-validation to optimize the hyperparameters of SVR, and take the minimum root mean square error CVMSE of the cross-validation set as the optimization target; S404, base model 4 establishment: input the features screened out by feature engineering solution 4 in step S304 into the algorithm, take the chemical analysis results of various water quality indicators as the target values ​​of the algorithm, use the SVR algorithm to establish a regression prediction model, use the optimization algorithm and cross-validation to optimize the hyperparameters of SVR, and take the minimum root mean square error CVMSE of the cross-validation set as the optimization goal.

2. A water quality multi-parameter measurement method as claimed in claim 1, characterized in that: Step S5: The fusion model is established as follows: linear regression is used as the meta-learner of Stacking, and the output results of base model 1, base model 2, base model 3, and base model 4 are weighted and fused for output, and the output of each base model is cross-validated to avoid overfitting of linear regression.

3. A water quality multi-parameter measurement method as claimed in claim 1, characterized in that: Step S6 model evaluation is specifically as follows: R 2 is the determination coefficient of the training set and the test set, RMSE is the root mean square error of the training set and the test set, Formula 1 is: Formula 2 is: In formula 1 and 2, is the predicted value of the i-th sample model, is the mean of the model's predicted values ​​for all samples, y i is the true value of the i-th sample, and m is the number of samples.

4. A water quality multi-parameter measurement method as claimed in claim 1, characterized in that: S7 includes the following steps: S701, UV-visible absorption spectrum measurement: using a spectrometer to collect spectrum data of the water sample to be tested; S702, data preprocessing: preprocessing the collected spectral data; S703, data feature engineering: performing feature engineering on the preprocessed data; S704, input the base model to obtain the prediction results of each base model; S705, fusion model result output: output the prediction result outputted in S704 to the fusion model to obtain the prediction result; Furthermore, the data preprocessing in step S702 is specifically as follows: S7021, pre-processing scheme 1: process the spectral data collected in S701, and sequentially process the spectral data using SG filtering and trend correction; S7022, preprocessing scheme 2: process the spectral data collected in S701, and sequentially process the spectral data using SG filtering, constant offset elimination, first-order derivative, and trend correction; Furthermore, step S703 data feature engineering is specifically as follows: S7031. Feature engineering solution 1: Use KPCA-XGBoost to reduce the dimension of the spectral data preprocessed in step S7021; S7032. Feature engineering solution 2: Use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S7021; S7033, feature engineering solution 3: use KPCA-SVR to reduce the dimension of the spectral data pre-processed in step S7022; S7034. Feature engineering solution 4: Use RFE-XGBoost to perform feature selection on the spectral data preprocessed in step S7022.

5. A water quality multi-parameter measurement method as claimed in claim 4, characterized in that: Step S704 inputs the base model specifically as follows: S7041, substitute the spectral data output by feature engineering solution 1 in S7031 into base model 1, and output the predicted concentration; S7042, substitute the spectral data output by feature engineering solution 2 in S7032 into base model 2, and output the predicted concentration; S7043, substitute the spectral data output by the feature engineering solution 3 of S7033 into the base model 3, and output the predicted concentration; S7044: Substitute the spectral data output by feature engineering solution 4 of S7034 into base model 4 to output the predicted concentration.

6. A method for measuring multi-parameters of water quality as claimed in claim 5, characterized in that: The fusion model result output in step S705 is specifically as follows: The output results of S7041-S7044 are input into the fusion model to output the prediction results.

7. The multi-parameter water quality measurement method as described in claim 1 can realize the measurement of chemical oxygen demand CODCr, permanganate index CODMn, five-day biochemical oxygen demand BOD5, total organic carbon TOC, dissolved organic carbon DOC, ammonia nitrogen NH3-N, total phosphorus TP, total nitrogen TN, nitrate nitrogen NO3-N, nitrite nitrogen NO2-N, turbidity, chromaticity, total suspended solids, phenol BTX, ozone, hydrogen sulfide or UV254 or simultaneous measurement.

Images