Surface water quality parameter detection method, device, equipment, medium and product

Abstract

The invention discloses a surface water quality parameter detection method, device and equipment, a medium and a product, and relates to the technical field of water quality monitoring. The method comprises the following steps: constructing sample data of each selected water quality parameter index according to ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, sequentially determining a plurality of corresponding optimal water quality parameter estimation models and an optimal model fusion method on the basis of the corresponding sample data for each index, and calculating the optimal water quality parameter estimation model according to the optimal water quality parameter estimation model and the optimal model fusion method. Finally, the ultraviolet-visible absorption spectrum data, collected in real time, of the field surface water are imported into the optimal water quality parameter estimation models of all the indexes in real time, and the final index values of all the indexes of the field surface water are obtained by applying the optimal model fusion method of all the indexes. Therefore, the purpose of quickly, accurately and real-timely monitoring multiple parameters of the surface water quality can be achieved, and the method has the advantages of no reagent, no secondary pollution, real-time performance, high efficiency, multi-parameter monitoring capability and high robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of water quality monitoring, and specifically relates to a method, device, equipment, medium and product for detecting surface water quality parameters. Background Art

[0002] In recent years, the acceleration of industrialization and the increase of human activities have led to increasingly serious surface water pollution problems. The deterioration of water quality not only threatens the ecological environment, but also poses potential risks to human health. Traditional water quality monitoring methods mainly rely on chemical analysis techniques, which usually require sampling and laboratory testing, making them not only time-consuming, but also costly and complicated to operate. In addition, these methods usually rely on chemical reagents and are prone to secondary pollution.

[0003] Spectroscopic technology, as a non-destructive, rapid and real-time detection method, has been widely used in the field of water quality monitoring. By measuring the absorption, emission or scattering characteristics of various substances in water, spectroscopy can be used to analyze various parameters in water quality, such as chemical oxygen demand (COD), total organic carbon (TOC) and ammonia nitrogen (NH3-N). However, traditional spectroscopy usually uses a single model to monitor a single water quality parameter, which is easily affected by the following factors: (1) turbidity interference, that is, water turbidity affects the propagation of light, thereby affecting the accuracy of spectral detection; (2) mutual influence between water quality parameters, that is, there may be mutual interference between different water quality parameters, and a single model is difficult to accurately distinguish and quantify water quality parameters; (3) limitations of a single model, that is, a single model is difficult to cope with complex changes in water quality and lacks sufficient robustness and generalization capabilities.

[0004] Therefore, how to combine spectral technology with advanced data processing methods and artificial intelligence multi-model fusion technology to improve the accuracy and stability of water quality detection is a key challenge facing the current water quality monitoring field. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, computer equipment, computer-readable storage medium and computer program product for detecting surface water quality parameters, so as to solve the problem that when traditional spectroscopy uses a single model to monitor a single water quality parameter, the accuracy and stability of water quality detection results are reduced due to turbidity interference, mutual influence between water quality parameters and the limitations of a single model.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, a method for detecting surface water quality parameters is provided, comprising:

[0008] Obtaining ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data contains detection index values ​​of corresponding samples on multiple water quality parameter indicators;

[0009] For each selected water quality parameter index among the multiple water quality parameter indexes, the corresponding sample data is constructed in the following manner: the ultraviolet-visible absorption spectrum data of the surface water sample is used as a model input item, and then the detection index value of the surface water sample on the corresponding index is used as a model output item, and finally the model input item and the model output item are used as a sample data;

[0010] For each water quality parameter indicator, a training sample set and a first test sample set are extracted from all the corresponding sample data, and then the training sample sets are used to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding to the K artificial intelligence-based learners one by one, and then the first test sample set is used to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding to the K water quality parameter estimation models one by one, and finally, according to the K first verification scores, the corresponding best k water quality parameter estimation models are determined from the K water quality parameter estimation models, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K;

[0011] For each of the water quality parameter indicators, a second test sample set is extracted from all the corresponding sample data, and then the second test sample set is used to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding best k water quality parameter estimation models to obtain a final water quality parameter estimation result, to obtain multiple second verification scores corresponding to the multiple model fusion methods, and finally, according to the multiple second verification scores, the corresponding best model fusion method is determined from the multiple model fusion methods;

[0012] Real-time collection of UV-visible absorption spectrum data of on-site surface water;

[0013] Importing the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and applying the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators;

[0014] The final index value of the on-site surface water on each of the water quality parameter indicators is output in real time.

[0015] Based on the above invention content, a new solution for surface water quality detection based on ultraviolet-visible absorption spectroscopy and combined with artificial intelligence multi-model fusion technology is provided, that is, first, according to the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, sample data of each selected water quality parameter index is constructed, and then for each index, based on the corresponding sample data, the corresponding optimal multiple water quality parameter estimation models and the optimal model fusion method are determined in turn, and finally, the ultraviolet-visible absorption spectrum data of the on-site surface water collected in real time are respectively imported into the optimal multiple water quality parameter estimation models of each index in real time, and the optimal model fusion method of each index is applied to obtain the final index value of the on-site surface water on each index, so that not only the purpose of rapid, accurate and real-time monitoring of multiple parameters of surface water quality can be achieved, but also the following advantages are provided: reagent-free and no secondary pollution, real-time and high efficiency, multi-parameter monitoring capability, high robustness and stability, and thus the problem that the accuracy and stability of water quality detection results are reduced due to turbidity interference, mutual influence between water quality parameters and limitations of single model when the traditional spectroscopy method uses a single model to monitor a single water quality parameter can be solved, which is convenient for practical application and promotion.

[0016] In a possible design, after obtaining the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples and before constructing the sample data, the method further includes:

[0017] Performing data filtering processing on the ultraviolet-visible absorption spectrum data of the surface water sample, and also performing data amplification processing on the water quality detection data of the surface water sample;

[0018] And / or, feature engineering processing is performed on the ultraviolet-visible absorption spectrum data and / or water quality detection data of the surface water sample, wherein the feature engineering processing includes feature screening processing and / or feature construction processing.

[0019] In a possible design, the multiple water quality parameter indicators include COD, TOC, BOD, DOC, permanganate index, nitrate nitrogen, nitrite nitrogen, total nitrogen, UV254, turbidity, total suspended solids and / or chromaticity.

[0020] In one possible design, the K artificial intelligence base learners include base learners based on a multivariate linear regression model, a Lasso regression model, a Ridge regression model, an elastic network regression model, a support vector machine regression model, an XGBoost gradient boosting tree model and / or a LightGBM gradient boosting tree model.

[0021] In one possible design, the multiple model fusion methods include simple averaging, weighted averaging and / or stacked generalization.

[0022] In a possible design, when the first verification score adopts the MAPE indicator, for each water quality parameter indicator, according to the K first verification scores, the corresponding best k water quality parameter estimation models are determined from the K water quality parameter estimation models, including:

[0023] For each of the water quality parameter indicators, according to the K first verification scores, the K water quality parameter estimation models are arranged in order from low to high scores to obtain a corresponding model sequence;

[0024] For each of the water quality parameter indicators, the first k models are selected from the corresponding model sequence as the corresponding best k water quality parameter estimation models, where k represents a positive integer greater than or equal to 2 and less than K.

[0025] In a second aspect, a surface water quality parameter detection device is provided, comprising an active data acquisition unit, a sample data construction unit, an optimal model determination unit, an optimal fusion determination unit, a spectral data acquisition unit, a model fusion application unit and a detection result output unit;

[0026] The source data acquisition unit is used to acquire ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data contains detection index values ​​of corresponding samples on multiple water quality parameter indicators;

[0027] The sample data construction unit is communicatively connected to the source data acquisition unit, and is used to construct corresponding sample data for each selected water quality parameter index among the multiple water quality parameter indexes in the following manner: using the ultraviolet-visible absorption spectrum data of the surface water sample as a model input item, then using the detection index value of the surface water sample on the corresponding index as a model output item, and finally using the model input item and the model output item as a sample data;

[0028] The optimal model determination unit is communicatively connected to the sample data construction unit, and is used to extract a training sample set and a first test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the training sample sets to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding to the K artificial intelligence-based learners, and then use the first test sample set to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding to the K water quality parameter estimation models, and finally determine the corresponding optimal k water quality parameter estimation models from the K water quality parameter estimation models according to the K first verification scores, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K;

[0029] The optimal fusion determination unit is communicatively connected to the optimal model determination unit, and is used to extract a second test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the second test sample set to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding optimal k water quality parameter estimation models to obtain a final water quality parameter estimation result, to obtain multiple second verification scores corresponding to the multiple model fusion methods, and finally determine the corresponding optimal model fusion method from the multiple model fusion methods according to the multiple second verification scores;

[0030] The spectral data acquisition unit is used to collect ultraviolet-visible absorption spectral data of on-site surface water in real time;

[0031] The model fusion application unit is respectively communicatively connected to the optimal model determination unit, the optimal fusion determination unit and the spectral data acquisition unit, and is used to import the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and apply the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators;

[0032] The detection result output unit is communicatively connected to the model fusion application unit, and is used for outputting the final index value of the on-site surface water on each of the water quality parameter indicators in real time.

[0033] In a third aspect, the present invention provides a computer device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the surface water quality parameter detection method as described in the first aspect or any possible design of the first aspect.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, executes the surface water quality parameter detection method as described in the first aspect or any possible design of the first aspect.

[0035] In a fifth aspect, the present invention provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements the surface water quality parameter detection method as described in the first aspect or any possible design of the first aspect.

[0036] Beneficial effects of the above scheme:

[0037] (1) The present invention provides a new solution for surface water quality detection based on ultraviolet-visible absorption spectroscopy and combined with artificial intelligence multi-model fusion technology, that is, firstly, based on the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, sample data of each selected water quality parameter index are constructed, and then for each index, based on the corresponding sample data, the corresponding optimal multiple water quality parameter estimation models and the optimal model fusion method are determined in turn, and finally, the ultraviolet-visible absorption spectrum data of the on-site surface water collected in real time are respectively imported into the optimal multiple water quality parameter estimation models of each index, and the optimal model fusion method of each index is applied to obtain the final index value of the on-site surface water on each index, so that not only the multi-parameter surface water quality can be realized, but also the multi-parameter surface water quality can be detected. For the purpose of rapid, accurate and real-time monitoring, it also has the following advantages: reagent-free and no secondary pollution (i.e. spectral detection does not require chemical reagents, thus avoiding secondary pollution), real-time and high efficiency (i.e. online and real-time monitoring of water quality is realized, with fast response speed), multi-parameter monitoring capability (i.e. the ability to monitor multiple water quality parameters at the same time, thus improving the comprehensiveness and accuracy of monitoring), and high robustness and stability (i.e. through multi-model fusion, the adaptability to complex water quality changes is enhanced, and the error and instability of a single model are reduced). This can solve the problem that when traditional spectroscopy uses a single model to monitor a single water quality parameter, the accuracy and stability of water quality detection results are reduced due to turbidity interference, mutual influence between water quality parameters and the limitations of a single model, which is convenient for practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic flow chart of a method for detecting surface water quality parameters provided in an embodiment of the present application.

[0040] Figure 2 A schematic diagram of the structure of a surface water quality parameter detection device provided in an embodiment of the present application.

[0041] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0043] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0044] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0045] Example

[0046] like Figure 1 As shown, the surface water quality parameter detection method provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device having certain computing resources and communicating with a spectrometer, such as an edge computer communicating with a spectrometer, a personal computer (PC, a multi-purpose computer with a size, price and performance suitable for personal use; desktops, laptops, small laptops, tablets and ultrabooks are all personal computers), smart phones, personal digital assistants (PDA) or wearable devices. Figure 1 As shown, the surface water quality parameter detection method may include but is not limited to the following steps S1 to S7.

[0047] S1. Obtain ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data includes but is not limited to detection index values ​​of corresponding samples on multiple water quality parameter indicators, etc.

[0048] In step S1, the UV-visible absorption spectrum data is used as characteristic data of the corresponding surface water sample, which can be obtained as shown in Table 1 below and at n consecutive frequency points (i.e., ω1, ω2, ..., ω n ) (e.g. x 1,1 ,x 1,2 ,…,x 1,n );The ultraviolet-visible absorption spectrum data of the surface water sample can be specifically obtained by conventional collection of a high-precision spectrometer. The multiple water quality parameter indicators specifically include but are not limited to COD (Chemical Oxygen Demand), TOC (Total Organic Carbon), BOD (Biochemical oxygen demand), DOC (Dissolved organic carbon), permanganate index (CODmm), nitrate nitrogen, nitrite nitrogen, total nitrogen, UV254 (which refers to the absorbance of some organic matter in water under ultraviolet light of 254nm wavelength, reflecting the amount of naturally existing humus-like macromolecular organic matter and aromatic compounds containing C=C double bonds and C=O double bonds in water, in units of AU / cm), turbidity, total suspended solids and / or chromaticity, etc. Their detection index values ​​are used as the true label values ​​of the corresponding samples, and can be obtained by conventional detection in accordance with the relevant standards of existing water and wastewater monitoring and analysis methods. For example, the detection index value of the surface water sample on COD can be obtained by potassium dichromate method experiment to ensure the authenticity and validity of the label value.

[0049] Table 1. Example of the format of UV-Vis absorption spectrum data and water quality test data of multiple surface water samples

[0050]

[0051] S2. For each selected water quality parameter index among the multiple water quality parameter indexes, the corresponding sample data is constructed in the following manner: the ultraviolet-visible absorption spectrum data of the surface water sample is used as a model input item, and then the detection index value of the surface water sample on the corresponding index is used as a model output item, and finally the model input item and the model output item are used as a sample data.

[0052] In step S2, each water quality parameter index can be routinely selected manually based on business needs. In order to ensure the high quality of the sample data, preferably, after obtaining the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples and before constructing the sample data, the method also includes but is not limited to: performing data filtering processing on the ultraviolet-visible absorption spectrum data of the surface water samples, and also performing data amplification processing on the water quality detection data of the surface water samples; and / or, performing feature engineering processing on the ultraviolet-visible absorption spectrum data and / or water quality detection data of the surface water samples, wherein the feature engineering processing includes but is not limited to feature screening processing and / or feature construction processing, etc. The aforementioned data filtering processing specifically but is not limited to the use of existing wavelet transform algorithms or SG (Savitzky-Golay) filtering algorithms, etc. The aforementioned data amplification processing specifically but is not limited to the use of existing linear interpolation algorithms, etc. The aforementioned feature engineering process is used to filter out redundant features through data feature work, improve the quality of the data set, and thereby improve the accuracy and efficiency of subsequent model training; specifically, the feature screening process can specifically but not limited to the use of existing filtering methods, packaging methods, and embedding methods, etc., and the feature construction process can but not limited to the use of existing logarithmic feature construction methods and feature weighted set methods, etc.

[0053] S3. For each water quality parameter indicator, a training sample set and a first test sample set are extracted from all the corresponding sample data, and then the training sample sets are used to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding one-to-one to the K artificial intelligence-based learners, and then the first test sample set is used to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding one-to-one to the K water quality parameter estimation models, and finally, according to the K first verification scores, the corresponding best k water quality parameter estimation models are determined from the K water quality parameter estimation models, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K.

[0054] In the step S3, the aforementioned training sample set and the first test sample set may be extracted in a random manner but is not limited to random extraction. The artificial intelligence base learner is used to construct a detection value inversion model of water quality parameters based on the training set data. Specifically, the K artificial intelligence base learners include but are not limited to base learners (Base Learner, which refers to a single weak learner used to construct an integrated model) based on a multivariate linear regression model, a Lasso regression model, a Ridge regression model, an elastic network regression model, a support vector machine regression model, an XGBoost gradient boosting tree model and / or a LightGBM gradient boosting tree model. The aforementioned models are all existing models, and are now introduced one by one as follows.

[0055] (A) Multiple linear regression model, i.e. multiple linear regression is an extended form of linear regression, which is used to model the relationship between multiple independent variables and one dependent variable. Unlike simple linear regression, multiple linear regression allows multiple independent variables to affect the dependent variable at the same time, thus providing a more flexible and accurate prediction model. This model is relatively simple and has strong interpretability, but it lacks the ability to fit complex data.

[0056] (B) Lasso regression model, namely Lasso regression (Least Absolute Shrinkage and Selection Operator), is a regularization technique for linear regression models. It can not only improve the prediction performance of the model, but also perform feature selection. Lasso regression constrains the size of the regression coefficient by adding an L1 regularization term to the regression model, thereby preventing the model from overfitting.

[0057] (C) Ridge regression model, also known as ridge regression, is a regularization technique for linear regression models that aims to deal with multicollinearity problems and prevent model overfitting. Unlike Lasso regression, Ridge regression constrains the size of the regression coefficient by introducing an L2 regularization term, thereby improving the stability and generalization ability of the model.

[0058] (D) ElasticNet regression model, namely ElasticNet regression, is a linear regression model that combines the advantages of Ridge regression (L2 regularization) and Lasso regression (L1 regularization). It introduces L1 and L2 regularization terms at the same time to solve some problems that may be encountered when using either alone, such as the insufficiency of Lasso regression in dealing with highly correlated features and the limitations of Ridge regression in feature selection.

[0059] (E) Support Vector Machine Regression Model, namely Support Vector Machine Regression (SVR) is an extension of Support Vector Machine (SVM) and is used to solve regression problems. The core idea of ​​SVR is similar to SVM classification. It predicts the target variable by finding a decision boundary that can tolerate a certain error to maximize the interval from the data point to the boundary.

[0060] (F) Xgboost gradient boosting tree model, namely XGBoost (Extreme Gradient Boosting) is a powerful regression model based on the gradient boosting framework. It gradually builds multiple weak prediction models (usually decision trees) and combines them to form a powerful regression prediction model. XGBoost is popular for its efficient computing power, good prediction performance and flexible parameter adjustment options.

[0061] (G) LightGBM Gradient Boosting Tree Model, i.e. LightGBM (Light Gradient Boosting Machine), is an efficient distributed implementation based on the gradient boosting framework developed by Microsoft. It is optimized in terms of speed and memory usage, and is very suitable for processing big data and high-dimensional data. LightGBM has similar functions and application scenarios to XGBoost, but has better performance in some cases, especially when processing large-scale data sets.

[0062] In step S3, the specific process of model training is the process of optimizing the model hyperparameters of the base learner, which can be implemented by, but not limited to, cross-validation and / or Bayesian model hyperparameter optimization. In detail, the specific process of the aforementioned Bayesian model hyperparameter optimization method is as follows ① to ⑦.

[0063] ① Define the objective function: First, define an objective function to evaluate the performance of the hyperparameter combination. This function is usually the error or accuracy of the model on the validation set.

[0064] ②Select a surrogate model: Use Gaussian process or other non-parametric methods to model the posterior distribution of the objective function. The surrogate model estimates the distribution of the objective function in the hyperparameter space by collecting historical evaluation data.

[0065] ③ Prior distribution setting: Set a prior distribution for the hyperparameters, and initially assume that all hyperparameters perform similarly.

[0066] ④ Obtain observation data: Obtain the objective function values ​​of several hyperparameter configurations through experiments and update the proxy model.

[0067] ⑤Update the surrogate model: Use Bayesian reasoning to update the surrogate model and generate the posterior distribution of the current optimal hyperparameters.

[0068] ⑥ Select sampling points: Use acquisition functions (such as expected improvement, probability improvement, etc.) to select the next hyperparameter combination to evaluate. These functions can make a trade-off between balancing exploration and exploitation.

[0069] ⑦Iterative optimization: Repeat the above process ④ to ⑥ until the preset stopping condition is met, such as reaching the maximum number of iterations or finding a hyperparameter combination that meets the requirements.

[0070] In step S3, the specific process of model verification is also the process of verifying the prediction accuracy of the corresponding model, which can be specifically implemented, but not limited to, using the MAPE (Mean Absolute Percentage Error, which is a commonly used evaluation index used to measure the average value of the percentage error between the model prediction value and the actual value, so as to reflect the relative accuracy of the model prediction) index and based on the first verification sample set. Since the lower the MAPE index value, the better the corresponding model, when the first verification score uses the MAPE index, for each water quality parameter index, according to the K first verification scores, the corresponding best k water quality parameter estimation models are determined from the K water quality parameter estimation models, including but not limited to: for each water quality parameter index, firstly according to the K first verification scores, the K water quality parameter estimation models are arranged in order from low to high, and the corresponding model sequence is obtained; then the first k models are selected from the corresponding model sequence as the corresponding best k water quality parameter estimation models, wherein k represents a positive integer greater than or equal to 2 and less than K. In addition, for example, when the K artificial intelligence base learners include base learners based on multivariate linear regression model, Lasso regression model, Ridge regression model, elastic network regression model, support vector machine regression model, XGBoost gradient boosting tree model and LightGBM gradient boosting tree model (ie, K = 7), k can be taken as 5 for example.

[0071] S4. For each of the water quality parameter indicators, a second test sample set is extracted from all the corresponding sample data, and then the second test sample set is used to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding best k water quality parameter estimation models to obtain the final water quality parameter estimation result, and multiple second verification scores corresponding to the multiple model fusion methods are obtained. Finally, based on the multiple second verification scores, the corresponding best model fusion method is determined from the multiple model fusion methods.

[0072] In the step S4, the extraction method of the aforementioned second test sample set can also be but not limited to random extraction. The aforementioned model fusion (Model Ensembling) is an integrated learning technology that improves the overall prediction ability by combining the output results of multiple independent base learner models; its basic assumption is that different base learner models may capture different features or patterns of the data during the learning process, so combining the predictions of multiple models can make up for the shortcomings of a single model and improve generalization ability. Specifically, the multiple model fusion methods include but are not limited to simple averaging, weighted averaging and / or stacked generalization methods. The aforementioned model fusion methods are all existing methods, and are now introduced one by one as follows.

[0073] (H) Simple average method, that is, simple average is the most basic model fusion method, which obtains the final prediction result by averaging the prediction results of multiple regression models:

[0074]

[0075] In the formula, represents the prediction result after the final fusion, N represents the number of base learner models involved in the fusion, i represents a positive integer, Represents the prediction result of the i-th base learner model.

[0076] (I) Weighted average method, that is, weighted average is to assign different weights to the prediction results of different models, adjust the weights according to the performance of each model on the validation set, and calculate the weighted average:

[0077]

[0078] In the formula, r i represents the weight of the i-th base learner model and satisfies Other symbols are the same as the simple averaging method.

[0079] (J) Stacking generalization method, namely stacking generalization method Stacking is a technology that improves prediction performance by combining multiple different types of models. Its basic idea is to input the prediction results of multiple base learners as new features into a meta learner for secondary learning. Stacking usually consists of two or more layers of models:

[0080] The first layer (base learner): consists of multiple different types of models, such as decision trees, linear regression, and support vector machines.

[0081] Second layer (meta-learner): uses the output of the first layer as input and trains a new model (usually a simple linear regression model) to optimize the final prediction result:

[0082]

[0083] In the formula, f meta represents the meta-learner, Represent the prediction results of each base learner model respectively.

[0084] In step S4, the specific process of the aforementioned fusion verification is also the process of verifying the prediction accuracy of the corresponding method, which can also be specifically but not limited to conventionally implemented by using the MAP indicator and based on the second verification sample set. Since the lower the MAPE indicator value, the better the corresponding method, when the second verification score uses the MAPE indicator, for each of the water quality parameter indicators, according to the multiple second verification scores, the corresponding best model fusion method is determined from the multiple model fusion methods, including but not limited to the following steps: for each of the water quality parameter indicators, the lowest score is determined from the multiple second verification scores, and then a certain model fusion method corresponding to the lowest score in the multiple model fusion methods is used as the corresponding best model fusion method.

[0085] S5. Collect UV-visible absorption spectrum data of on-site surface water in real time.

[0086] In step S5, the ultraviolet-visible absorption spectrum data of the on-site surface water can be routinely collected periodically by a high-precision spectrometer arranged on-site (the collection period can be specifically in seconds and can be adjusted to achieve real-time detection purposes).

[0087] S6. Import the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and apply the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators.

[0088] In the step S6, for example, if the permanganate index is selected in the aforementioned step S2, then through the aforementioned steps S2 to S4, the optimal k water quality parameter estimation models of the permanganate index and the optimal model fusion method can be obtained, so that the ultraviolet-visible absorption spectrum data of the on-site surface water can be respectively imported into the optimal k water quality parameter estimation models of the permanganate index in real time, and the optimal model fusion method of the permanganate index is applied to obtain the final index value of the on-site surface water on the permanganate index.

[0089] S7. Output the final index value of the on-site surface water on each of the water quality parameter indicators in real time.

[0090] In the step S7, if the final index value of a certain water quality parameter index of the on-site surface water meets the preset alarm condition, the corresponding alarm action (such as sound and light alarm) can be automatically triggered.

[0091] Therefore, based on the surface water quality parameter detection method described in the aforementioned steps S1 to S7, a new solution for surface water quality detection based on ultraviolet-visible absorption spectroscopy and combined with artificial intelligence multi-model fusion technology is provided, that is, first, according to the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, sample data of each selected water quality parameter index is constructed, and then for each index, based on the corresponding sample data, the corresponding optimal multiple water quality parameter estimation models and the optimal model fusion method are determined in turn, and finally, the ultraviolet-visible absorption spectrum data of the on-site surface water collected in real time are respectively imported into the optimal multiple water quality parameter estimation models of each index, and the optimal model fusion method of each index is applied to obtain the final index value of the on-site surface water on each index, so that not only It can achieve the purpose of rapid, accurate and real-time monitoring of multiple parameters of surface water quality, and has the following advantages: reagent-free and no secondary pollution (i.e. spectral detection does not require chemical reagents, avoiding secondary pollution), real-time and high efficiency (i.e. online and real-time monitoring of water quality is realized, with fast response speed), multi-parameter monitoring capability (i.e. the ability to monitor multiple water quality parameters at the same time, improving the comprehensiveness and accuracy of monitoring), and high robustness and stability (i.e. through multi-model fusion, the adaptability to complex water quality changes is enhanced, and the error and instability of a single model are reduced). It can thus solve the problem that when traditional spectroscopy uses a single model to monitor a single water quality parameter, the accuracy and stability of water quality detection results are reduced due to turbidity interference, mutual influence between water quality parameters and the limitations of a single model, which is convenient for practical application and promotion.

[0092] like Figure 2As shown, the second aspect of this embodiment provides a virtual device for implementing the surface water quality parameter detection method described in the first aspect, including an active data acquisition unit, a sample data construction unit, an optimal model determination unit, an optimal fusion determination unit, a spectral data acquisition unit, a model fusion application unit and a detection result output unit;

[0093] The source data acquisition unit is used to acquire ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data contains detection index values ​​of corresponding samples on multiple water quality parameter indicators;

[0094] The sample data construction unit is communicatively connected to the source data acquisition unit, and is used to construct corresponding sample data for each selected water quality parameter index among the multiple water quality parameter indexes in the following manner: using the ultraviolet-visible absorption spectrum data of the surface water sample as a model input item, then using the detection index value of the surface water sample on the corresponding index as a model output item, and finally using the model input item and the model output item as a sample data;

[0095] The optimal model determination unit is communicatively connected to the sample data construction unit, and is used to extract a training sample set and a first test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the training sample sets to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding to the K artificial intelligence-based learners, and then use the first test sample set to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding to the K water quality parameter estimation models, and finally determine the corresponding optimal k water quality parameter estimation models from the K water quality parameter estimation models according to the K first verification scores, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K;

[0096] The optimal fusion determination unit is communicatively connected to the optimal model determination unit, and is used to extract a second test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the second test sample set to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding optimal k water quality parameter estimation models to obtain a final water quality parameter estimation result, to obtain multiple second verification scores corresponding to the multiple model fusion methods, and finally determine the corresponding optimal model fusion method from the multiple model fusion methods according to the multiple second verification scores;

[0097] The spectral data acquisition unit is used to collect ultraviolet-visible absorption spectral data of on-site surface water in real time;

[0098] The model fusion application unit is respectively communicatively connected to the optimal model determination unit, the optimal fusion determination unit and the spectral data acquisition unit, and is used to import the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and apply the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators;

[0099] The detection result output unit is communicatively connected to the model fusion application unit, and is used for outputting the final index value of the on-site surface water on each of the water quality parameter indicators in real time.

[0100] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be referred to the surface water quality parameter detection method described in the first aspect, and will not be repeated here.

[0101] like Figure 3 As shown, the third aspect of the present embodiment provides a computer device for executing the surface water quality parameter detection method as described in the first aspect, including a memory, a processor and a transceiver that are sequentially connected in communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the surface water quality parameter detection method as described in the first aspect. Specifically, the memory may include, but is not limited to, a random access memory (Random-Access Memory, RAM), a read-only memory (Read-Only Memory, ROM), a flash memory (Flash Memory), a first-in-first-out memory (First Input First Output, FIFO) and / or a first-in-last-out memory (First Input Last Output, FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen and other necessary components.

[0102] The working process, working details and technical effects of the aforementioned computer equipment provided in the third aspect of this embodiment can be referred to the surface water quality parameter detection method described in the first aspect, and will not be repeated here.

[0103] The fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the surface water quality parameter detection method as described in the first aspect, that is, the computer-readable storage medium stores instructions, and when the instructions are run on a computer, the surface water quality parameter detection method as described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memories, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0104] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be referred to the surface water quality parameter detection method described in the first aspect, and will not be repeated here.

[0105] In a fifth aspect, the present embodiment provides a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed by a computer, the surface water quality parameter detection method as described in the first aspect is implemented. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0106] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting surface water quality parameters, characterized in that: include: Obtaining ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data contains detection index values ​​of corresponding samples on multiple water quality parameter indicators; For each selected water quality parameter index among the multiple water quality parameter indexes, the corresponding sample data is constructed in the following manner: the ultraviolet-visible absorption spectrum data of the surface water sample is used as a model input item, and then the detection index value of the surface water sample on the corresponding index is used as a model output item, and finally the model input item and the model output item are used as a sample data; For each water quality parameter indicator, a training sample set and a first test sample set are extracted from all the corresponding sample data, and then the training sample sets are used to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding to the K artificial intelligence-based learners one by one, and then the first test sample set is used to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding to the K water quality parameter estimation models one by one, and finally, according to the K first verification scores, the corresponding best k water quality parameter estimation models are determined from the K water quality parameter estimation models, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K; For each of the water quality parameter indicators, a second test sample set is extracted from all the corresponding sample data, and then the second test sample set is used to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding best k water quality parameter estimation models to obtain a final water quality parameter estimation result, to obtain multiple second verification scores corresponding to the multiple model fusion methods, and finally, according to the multiple second verification scores, the corresponding best model fusion method is determined from the multiple model fusion methods; Real-time collection of UV-visible absorption spectrum data of on-site surface water; Importing the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and applying the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators; The final index value of the on-site surface water on each of the water quality parameter indicators is output in real time.

2. The method for detecting surface water quality parameters according to claim 1, characterized in that: After obtaining the ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples and before constructing the sample data, the method further includes: Performing data filtering processing on the ultraviolet-visible absorption spectrum data of the surface water sample, and also performing data amplification processing on the water quality detection data of the surface water sample; And / or, feature engineering processing is performed on the ultraviolet-visible absorption spectrum data and / or water quality detection data of the surface water sample, wherein the feature engineering processing includes feature screening processing and / or feature construction processing.

3. The method for detecting surface water quality parameters according to claim 1, characterized in that: The multiple water quality parameter indicators include COD, TOC, BOD, DOC, permanganate index, nitrate nitrogen, nitrite nitrogen, total nitrogen, UV254, turbidity, total suspended solids and / or chromaticity.

4. The method for detecting surface water quality parameters according to claim 1, characterized in that: The K artificial intelligence base learners include base learners based on a multivariate linear regression model, a Lasso regression model, a Ridge regression model, an elastic network regression model, a support vector machine regression model, an XGBoost gradient boosting tree model and / or a LightGBM gradient boosting tree model.

5. The method for detecting surface water quality parameters according to claim 1, characterized in that: The multiple model fusion methods include simple averaging method, weighted averaging method and / or stacking generalization method.

6. The method for detecting surface water quality parameters according to claim 1, characterized in that: When the first verification score adopts the MAPE indicator, for each water quality parameter indicator, according to the K first verification scores, determining the corresponding best k water quality parameter estimation models from the K water quality parameter estimation models, including: For each of the water quality parameter indicators, according to the K first verification scores, the K water quality parameter estimation models are arranged in order from low to high scores to obtain a corresponding model sequence; For each of the water quality parameter indicators, the first k models are selected from the corresponding model sequence as the corresponding best k water quality parameter estimation models, where k represents a positive integer greater than or equal to 2 and less than K.

7. A surface water quality parameter detection device, characterized in that: It includes an active data acquisition unit, a sample data construction unit, an optimal model determination unit, an optimal fusion determination unit, a spectral data acquisition unit, a model fusion application unit and a detection result output unit; The source data acquisition unit is used to acquire ultraviolet-visible absorption spectrum data and water quality detection data of different surface water samples, wherein the water quality detection data contains detection index values ​​of corresponding samples on multiple water quality parameter indicators; The sample data construction unit is communicatively connected to the source data acquisition unit, and is used to construct corresponding sample data for each selected water quality parameter index among the multiple water quality parameter indexes in the following manner: using the ultraviolet-visible absorption spectrum data of the surface water sample as a model input item, then using the detection index value of the surface water sample on the corresponding index as a model output item, and finally using the model input item and the model output item as a sample data; The optimal model determination unit is communicatively connected to the sample data construction unit, and is used to extract a training sample set and a first test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the training sample sets to perform model training on K artificial intelligence-based learners respectively to obtain K water quality parameter estimation models corresponding to the K artificial intelligence-based learners, and then use the first test sample set to perform model verification on the K water quality parameter estimation models respectively to obtain K first verification scores corresponding to the K water quality parameter estimation models, and finally determine the corresponding optimal k water quality parameter estimation models from the K water quality parameter estimation models according to the K first verification scores, wherein K represents a positive integer greater than or equal to 3, and k represents a positive integer greater than or equal to 2 and less than K; The optimal fusion determination unit is communicatively connected to the optimal model determination unit, and is used to extract a second test sample set from all the corresponding sample data for each water quality parameter indicator, and then use the second test sample set to perform fusion verification on multiple model fusion methods for fusing the water quality parameter estimation results output by the corresponding optimal k water quality parameter estimation models to obtain a final water quality parameter estimation result, to obtain multiple second verification scores corresponding to the multiple model fusion methods, and finally determine the corresponding optimal model fusion method from the multiple model fusion methods according to the multiple second verification scores; The spectral data acquisition unit is used to collect ultraviolet-visible absorption spectral data of on-site surface water in real time; The model fusion application unit is respectively communicatively connected to the optimal model determination unit, the optimal fusion determination unit and the spectral data acquisition unit, and is used to import the ultraviolet-visible absorption spectrum data of the on-site surface water into the optimal k water quality parameter estimation models of the various water quality parameter indicators in real time, and apply the optimal model fusion method of the various water quality parameter indicators to obtain the final indicator value of the on-site surface water on the various water quality parameter indicators; The detection result output unit is communicatively connected to the model fusion application unit, and is used for outputting the final index value of the on-site surface water on each of the water quality parameter indicators in real time.

8. A computer device, characterized in that: It includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the surface water quality parameter detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the computer, the surface water quality parameter detection method as described in any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or the instruction, when executed by a computer, implements the surface water quality parameter detection method as claimed in any one of claims 1 to 6.

Images