On-line soft measurement method and system for water quality parameters based on optimization algorithm

Abstract

The present invention discloses an online soft measurement method and system for water quality parameters based on an optimization algorithm. The method comprises the following steps: collecting reagent-free water quality index data and chemical reagent water quality index data, analyzing the correlation between the reagent-free water quality index and the chemical water quality index using the Pearson correlation coefficient method, data set division and regularization, model creation, and model training; constructing an optimization function with the minimum MAE of the model on all samples as the objective function, adjusting the model parameters using the optimization algorithm and obtaining the optimal parameters; obtaining a soft measurement model; and applying the soft measurement model to predict water quality parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental water quality monitoring, and in particular to an online soft measurement method and system for water quality parameters based on an optimization algorithm. Background Art

[0002] In the fields of environmental monitoring, sewage treatment, water resources management, and ecological protection, chemical water quality indicators such as permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen are key indicators for measuring water quality. In existing technologies, these key indicators are mainly monitored and analyzed through the following two methods:

[0003] 1. Traditional chemical detection methods: For example, in the monitoring / detection of surface water and industrial pollution sources, chemical reagent detection methods are generally used to determine water quality indicators such as permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen. Although this method has accurate measurement results, it has some significant limitations. For example: First, the need to add additional chemical reagents not only increases the complexity of the operation but may also cause secondary pollution; second, it is time-consuming, and the monitoring frequency and cycle are limited, which can easily affect the real-time and continuity of the data; in addition, it requires high construction and operation and maintenance costs.

[0004] 2. Model analysis methods based on big data and intelligent algorithms; however, existing models have low utilization rates when processing input parameter characteristics and poor performance in nonlinear data fitting, resulting in poor prediction results of existing models.

[0005] Therefore, an online soft measurement method and system for water quality parameters based on an optimization algorithm with high input parameter utilization, good nonlinear fitting state and high prediction accuracy are provided. Summary of the Invention

[0006] In order to solve the above problems in the prior art, the present application provides an online soft measurement method and system for water quality parameters based on an optimization algorithm with high input parameter utilization, good nonlinear fitting state and high prediction accuracy.

[0007] First, the present invention provides an online soft measurement method for water quality parameters based on an optimization algorithm, comprising the following steps:

[0008] S1. Data Collection: Collect data on reagent-free water quality indicators and chemical reagent water quality indicators at preset sampling intervals within a preset period, and form a data sample set for the reagent-free water quality indicators; where the reagent-free water quality indicators include but are not limited to temperature, pH, dissolved oxygen, conductivity, and turbidity, and the chemical reagent water quality indicators include but are not limited to permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen;

[0009] S2. Correlation Analysis: Analyze the correlation between the reagent-free water quality index and the chemical water quality index using the Pearson correlation coefficient method. If the correlation is greater than a preset value, the reagent-free water quality index is selected as the input parameter.

[0010] S3. Dataset Partitioning and Regularization: Regularize the data sample set corresponding to the input parameters selected in S2 and divide it into training and validation sets.

[0011] S4. Model creation: Based on the training set, construct a long short-term memory (LSTM) model whose input is the reagent-free water quality index selected using the Pearson correlation coefficient method and whose output is the corresponding chemical water quality index;

[0012] S5. Model training: Use offline training methods and train the model using the training set. The optimization algorithm used for training the model may include Adam or SGD. Set hyperparameters, including but not limited to the learning rate, batch size, number of neurons, and number of iterations.

[0013] Parameter optimization: Build an optimization function with the objective function of minimizing the MAE of the model on all samples; use the optimization algorithm to adjust the model parameters, i.e., hyperparameters, and obtain the optimal parameters;

[0014] S6. Obtain the final model: Bring the optimal parameters into the model training, use regression analysis to evaluate the model's predictive performance, and evaluate whether it meets the requirements by calculating the RMSE, MAE, and R of the predicted and actual results. 2 , then save the parameters and structure of the model, which is regarded as obtaining the final model, that is, obtaining the soft measurement model;

[0015] S7. Apply the soft sensing model obtained in S6 to predict water quality parameters.

[0016] In S2, the Pearson correlation coefficient is calculated as follows:

[0017] ,

[0018] in, is the i-th measured value of the reagent-free water quality indicator, is the i-th measured value of the chemical water quality indicator, and and They are the corresponding reagent-free water quality indicators and chemical water quality indicators The mean of ; r is the result of Pearson correlation coefficient calculation, and –1≤r≤+1.

[0019] In S3, regularization is done as follows: ,

[0020] in, is the original data, is the mean of the original data, is the standard deviation of the original data.

[0021] In S5, MAE is expressed as: ,

[0022] in, is the sample size, is the actual value of the i-th chemical water quality indicator, is the predicted value of the i-th chemical water quality indicator;

[0023] The objective function is expressed as: ,

[0024] in, ; is the learning rate, is the batch size, is the number of neurons.

[0025] In S5, the transit search algorithm TS, particle swarm optimization PSO, ant colony optimization ACO or imperial competition algorithm ICA are used to adjust the model parameters.

[0026] In S5, the particle swarm optimization algorithm is used to adjust the model parameters; specifically:

[0027] S5.1 Initialize particles and speed: Determine the number of particles in the particle swarm, randomly initialize the positions of the particles, and each particle represents a set of parameters ( ), randomly initialize the particle speed and determine the dimension of the space according to the number of parameters;

[0028] S5.2 Iteration: Perform multiple iterations, each iteration comprising the following steps:

[0029] Update particle position: Update the particle position according to the current speed and position;

[0030] Update particle speed: Update the particle speed based on the particle's current position, historical best position, and global best position;

[0031] Calculate the fitness value: Use the parameters of the current position to train the model and calculate MAE as the fitness value;

[0032] Update the best position of the particle: If the fitness value of the current position is better than the historical best position, then update the best position of the particle;

[0033] Update the global best position: If the fitness value of the current position is better than the global best position, then update the global best position;

[0034] S5.3 Terminate and extract the optimal parameters: When the maximum number of iterations is reached or the fitness value reaches the preset threshold, the corresponding parameters are saved and regarded as the optimal parameters ( ).

[0035] Secondly, the present invention also provides an online soft measurement system for water quality parameters based on an optimization algorithm, which uses the soft measurement method described above to predict water quality parameters.

[0036] The technical solution of the present invention has the following advantages:

[0037] In the present invention, an automated data analysis process and optimization algorithm are used to improve the prediction accuracy and stability of the model, so that the system can quickly respond to changes in water quality; the advanced optimization algorithms used, such as particle swarm optimization, ant colony optimization or imperial competition algorithm, are adaptive, with minimization of mean absolute error (MAE) as the objective function, focusing on optimizing key parameters, improving the prediction accuracy of the model, and being able to automatically adjust parameters based on real-time feedback; at the same time, the method is applicable to various occasions, such as including but not limited to surface water such as rivers and lakes, and is also applicable to water environments such as sewage treatment plant outlets that require water quality monitoring or inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 Schematic diagram of the online soft measurement method for water quality parameters based on the optimization algorithm of the present invention;

[0040] Figure 2 Schematic diagram showing raw data of Example 3 of the present invention;

[0041] Figure 3 Schematic diagram showing the distribution of raw data in Example 3 of the present invention;

[0042] Figure 4 Schematic diagram of the correlation between various parameters in Example 3 of the present invention;

[0043] Figure 5 Schematic diagram of the input variable data set of Example 3 of the present invention;

[0044] Figure 6 Schematic diagram of data set division in Example 3 of the present invention;

[0045] Figure 7Schematic diagram comparing predicted values ​​and actual values ​​of Example 3 of the present invention;

[0046] Figure 8 Schematic diagram comparing the predicted values ​​of Example 3 of the present invention with the predicted values ​​of the prior art. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following content is merely an example and explanation of the concept of the present invention. Those skilled in the art may make various modifications or supplements to the specific implementation cases described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the scope of protection of the present invention. The experimental reagents and materials used in the following examples, unless otherwise specified, can be obtained from commercial channels.

[0048] Example 1: This application provides an online soft measurement method for water quality parameters based on an optimization algorithm, comprising the following steps:

[0049] S1. Data collection: Collect reagent-free water quality index data and chemical reagent water quality index data at preset sampling intervals within a preset period, and form a data sample set for the reagent-free water quality indicators; wherein, the reagent-free water quality indicators include but are not limited to temperature, pH, dissolved oxygen, conductivity and turbidity, and the chemical reagent water quality indicators include but are not limited to permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen; specifically, the preset period is one month, three months, half a year or one year, etc., and the sampling interval is set to 4 hours, 12 hours, 18 hours or 24 hours, etc.; It should be noted here that: a series of parameters such as temperature, pH, dissolved oxygen, conductivity and turbidity that generally do not rely on chemical reagents to monitor in the prior art are called reagent-free water quality indicators, and a series of parameters such as permanganate index, ammonia nitrogen, total phosphorus, total nitrogen that generally rely on chemical reagents to monitor in the prior art are called chemical reagent water quality indicators.

[0050] S2. Correlation Analysis: Analyze the correlation between the reagent-free water quality index and the chemical water quality index using the Pearson correlation coefficient method. If the correlation is greater than a preset value, the reagent-free water quality index is selected as the input parameter for the corresponding chemical water quality index. As a preferred implementation, the preset value is preferably 0.3. The preset correlation value can be selected based on relevant reference materials and water quality parameter experiments.

[0051] Specifically, in S2, the calculation method of the Pearson correlation coefficient is as follows:

[0052] ,

[0053] in, is the i-th measured value of the reagent-free water quality indicator, is the i-th measured value of the chemical water quality indicator, and and They are the corresponding reagent-free water quality indicators and chemical water quality indicators The mean of ; r is the result of Pearson correlation coefficient calculation, and –1≤r≤+1.

[0054] Here, it should be noted that the properties of the Pearson correlation coefficient r are as follows:

[0055] When r>0, it means the two variables are positively correlated, and when r<0, the two variables are negatively correlated;

[0056] When |r|=1, it means that the two variables are completely linearly correlated, that is, a functional relationship;

[0057] When r = 0, it means there is no linear correlation between the two variables;

[0058] When 0 < |r| < 1, there is a certain degree of linear correlation between the two variables. The closer |r| is to 1, the closer the linear relationship between the two variables is; the closer |r| is to 0, the weaker the linear correlation is. This can generally be categorized into three levels: |r| < 0.3 indicates a low degree of linear correlation; 0.3 ≤ |r| < 0.7 indicates a significant correlation; and 0.7 ≤ |r| < 1 indicates a high degree of linear correlation.

[0059] S3. Dataset partitioning and regularization: The data sample set corresponding to the input parameters selected in S2 is regularized and divided into a training set and a validation set; as a preferred embodiment, the input of the training set and the validation set only includes the reagent-free water quality index, that is, the data obtained in S2, and the sample data set is divided into a training set and a validation set in an 8:2 or 9:1 ratio;

[0060] Specifically, in S3, the regularization process is: ,

[0061] in, is the original data, is the mean of the original data, is the standard deviation of the original data.

[0062] S4. Model creation: Based on the training set, a long short-term memory (LSTM) model is constructed whose input is the reagent-free water quality index selected using the Pearson correlation coefficient method, and whose output is the corresponding chemical water quality index, such as total phosphorus (TP);

[0063] Specifically, the long short-term memory model LSTM includes:

[0064] Forget Gate: ,

[0065] Input Gate and candidate memory :

[0066] ,

[0067] ,

[0068] Unit update status: ,

[0069] Output Gate and the final hidden state :

[0070] ,

[0071] tanh( )，

[0072] in ： is the sigmoid activation function, used for the forget gate and output gate; tanh is the hyperbolic tangent activation function, used for candidate memory and final hidden state ； and They are weight and bias, which correspond to different gates; is the hidden state of the water quality input parameter at the previous time step and the input of the current time step; Represents element-wise multiplication.

[0073] S5. Model training: Use offline training to train the model using the training set; the optimization algorithm used to train the model includes Adam or SGD; set hyperparameters, including but not limited to learning rate, batch size, number of neurons, and number of iterations; the learning rate (learning_rate= α ), batch size (batch=k), number of neurons (𝑛= n neurons );

[0074] Specifically, the data processed by S3 is divided into a new training set and a validation set for the model in an 8:2 ratio. The model learns from a large amount of training data with the goal of minimizing the loss function, thereby improving the model's predictive ability. Typically, the training process involves multiple epochs, with each epoch traversing the entire training dataset. During training, the model's parameters are continuously updated to adapt to the training data. Saving the model during training is typically to record the progress of training and prevent loss of training results due to unexpected interruptions. These saved models are typically called "checkpoints" and are then saved using an early stopping mechanism, which is a callback used to stop training early. Specifically, this embodiment stops training when the loss on the training set no longer decreases (i.e., the degree of decrease is less than a certain threshold).

[0075] Parameter optimization: Build an optimization function with the objective function of minimizing the MAE of the model on all samples; use the optimization algorithm to adjust the model parameters, namely the "hyperparameters" in S5, and obtain the optimal parameters;

[0076] Specifically, in S5, MAE is expressed as:

[0077] ,

[0078] in, is the sample size, is the actual value of the i-th chemical water quality indicator, is the predicted value of the i-th chemical water quality indicator;

[0079] The objective function is expressed as: ,

[0080] in, ; learning rate, Batch size, Number of neurons;

[0081] In S5, the transit search algorithm (TS), particle swarm optimization (PSO), ant colony optimization (ACO), or imperial competition algorithm (ICA) is used to adjust the model parameters.

[0082] In S5, the particle swarm optimization algorithm is used to adjust the model parameters, specifically:

[0083] S5.1 Initialize particles and speed: Determine the number of particles in the particle swarm, randomly initialize the positions of the particles, and each particle represents a set of parameters ( ), randomly initialize the particle speed and determine the dimension of the space according to the number of parameters;

[0084] S5.2 Iteration: Perform multiple iterations, each iteration comprising the following steps:

[0085] Update particle position: Update the particle position according to the current speed and position;

[0086] Update particle speed: Update the particle speed based on the particle's current position, historical best position, and global best position;

[0087] Calculate the fitness value: Use the parameters of the current position to train the model and calculate MAE as the fitness value;

[0088] Update the best position of the particle: If the fitness value of the current position is better than the historical best position, then update the best position of the particle;

[0089] Update the global best position: If the fitness value of the current position is better than the global best position, then update the global best position;

[0090] S5.3 Terminate and extract the optimal parameters: When the maximum number of iterations is reached or the fitness value reaches the preset threshold, the corresponding parameters are saved and regarded as the optimal parameters ( ),

[0091] S6. Obtain the final model: Bring the optimal parameters into the model training, use regression analysis to evaluate the model's predictive performance, and calculate the RMSE, MAE, and R of the predicted and actual results. 2 If the conditions are met, the parameters and structure of the model are saved, which is regarded as obtaining the final model, that is, obtaining the soft measurement model. In this embodiment, it is preferred to use the deep learning framework Keras as an aid, and save the trained model as an H5 file. This file format not only records the architecture of the model, but also contains all key weight parameters, preparing for online applications.

[0092] S7. Use the soft sensor model obtained in S6 to predict water quality parameters. This involves deploying the trained soft sensor model to an online monitoring system, receiving easily measurable parameter data in real time, and predicting water quality parameters.

[0093] In this application, reagent-free water quality indicators are used as model inputs, eliminating the need to use traditional chemical reagents; automated data analysis processes and optimization algorithms are used to improve the prediction accuracy and stability of the model, enabling the system to quickly respond to changes in water quality; advanced optimization algorithms such as particle swarm optimization, ant colony optimization, or imperial competition algorithms are adaptive, with minimizing the mean absolute error (MAE) as the objective function, focusing on optimizing key parameters, improving the model's prediction accuracy, and being able to automatically adjust parameters based on real-time feedback.

[0094] Example 2: Based on Example 1, this example provides an online soft measurement system for water quality parameters based on an optimization algorithm. The system uses the soft measurement method described in Example 1 to predict water quality parameters.

[0095] Furthermore, the system described in this embodiment or the method described in Example 1 can also be applied to a mobile monitoring platform. The water quality monitoring steps in specific applications are briefly described as follows:

[0096] Data collection: Use mobile monitoring platforms (such as drones or ship-borne equipment) to collect water samples and water quality parameters;

[0097] Correlation analysis: Analyze data from different locations to determine physical parameters related to TP concentration;

[0098] Input variable selection: Considering that pH, dissolved oxygen, and turbidity have the highest correlation with TP, these three parameters were selected as model inputs;

[0099] Model selection: Determine the recurrent neural network (RNN) model suitable for time series forecasting;

[0100] Parameter optimization: Use grid search and cross-validation to determine the best model parameters;

[0101] Model training and regularization: Train the model on the cloud platform and apply Dropout regularization technology.

[0102] Model output: Integrate the trained model into the data processing unit of the mobile monitoring platform;

[0103] Online application: The model analyzes monitoring data in real time and predicts TP concentration, providing immediate information for pollution tracking.

[0104] The soft sensor analysis model provided in this application has the predictive capability at different river locations, which can quickly respond to water quality changes and provide decision support for pollution control.

[0105] Example 3: The technical solution of the present application will be described below with reference to specific examples, such as the prediction of total phosphorus (TP) of water quality based on an on-site measurement system during actual use.

[0106] The details are as follows:

[0107] Data collection: Deploy on-site monitoring equipment to collect a full year of monitoring data, such as water temperature, pH, dissolved oxygen, conductivity, and turbidity, as well as reagent-free water quality indicators such as permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen; Figure 2 and 3 As shown;

[0108] Correlation analysis: Pearson correlation coefficient analysis was used to determine the physical parameters that were significantly correlated with TP concentration; the correlation between the parameter data after analysis was as follows: Figure 4 As shown in the figure, after analysis, pH, dissolved oxygen and turbidity have the highest correlation with TP, so these three parameters are selected as input variables of the soft sensor model;

[0109] Dataset partitioning and regularization: such as Figure 5 and 6 As shown, the data set is divided and regularized;

[0110] Model selection and optimization: Select a deep learning model, such as the long short-term memory neural network (LSTM), and use particle swarm optimization (PSO) to optimize model parameters;

[0111] Model training: Use the selected input variable data set to perform model training to obtain the final soft sensor model, including data preprocessing, feature selection, etc.

[0112] Application and evaluation: The obtained soft sensor model is applied to the monitoring system or platform for real-time measurement; the soft sensor model receives direct water quality index data in real time and predicts TP concentration; as shown in Table 1, the model accuracy is evaluated by comparing with the experimental analysis results. Figure 7 As shown, the measurement results of the soft measurement model provided in this application are highly consistent with the laboratory analysis results, with a prediction error (MAPE) of less than 10% and high prediction accuracy.

[0113] Furthermore, as shown in Table 1 and Figure 8 Comparing the soft measurement method described in this application with the naive prediction method in the prior art, it can be seen that the soft measurement method described in this application has smaller error and higher prediction accuracy.

[0114] Table 1

[0115]

[0116] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. An online soft measurement method for water quality parameters based on an optimization algorithm is characterized by: The steps include: S1. Data Collection: Collect data on reagent-free water quality indicators and chemical reagent water quality indicators at preset sampling intervals within a preset period, and form a data sample set for the reagent-free water quality indicators; where the reagent-free water quality indicators include but are not limited to temperature, pH, dissolved oxygen, conductivity, and turbidity, and the chemical reagent water quality indicators include but are not limited to permanganate index, ammonia nitrogen, total phosphorus, and total nitrogen; S2. Correlation Analysis: Analyze the correlation between the reagent-free water quality index and the chemical water quality index using the Pearson correlation coefficient method. If the correlation is greater than a preset value, the reagent-free water quality index is selected as the input parameter. S3. Dataset Partitioning and Regularization: Regularize the data sample set corresponding to the input parameters selected in S2 and divide it into training and validation sets. S4. Model creation: Based on the training set, construct a long short-term memory (LSTM) model whose input is the reagent-free water quality index selected using the Pearson correlation coefficient method and whose output is the corresponding chemical water quality index; S5. Model training: Use offline training methods and train the model using the training set. The optimization algorithm used for training the model may include Adam or SGD. Set hyperparameters, including but not limited to the learning rate, batch size, number of neurons, and number of iterations. Parameter optimization: Build an optimization function with the objective function of minimizing the MAE of the model on all samples; use the optimization algorithm to adjust the model parameters, i.e., hyperparameters, and obtain the optimal parameters; S6. Obtain the final model: Bring the optimal parameters into the model training, use regression analysis to evaluate the model's predictive performance, and evaluate whether it meets the requirements by calculating the RMSE, MAE, and R of the predicted and actual results. 2 , then save the parameters and structure of the model, which is regarded as obtaining the final model, that is, obtaining the soft measurement model; S7. Apply the soft sensing model obtained in S6 to predict water quality parameters.

2. The online soft measurement method for water quality parameters based on the optimization algorithm according to claim 1 is characterized in that: In S2, the Pearson correlation coefficient is calculated as follows: ， in, is the i-th measured value of the reagent-free water quality indicator, is the i-th measured value of the chemical water quality indicator, and and They are the corresponding reagent-free water quality indicators and chemical water quality indicators The mean of ; r is the result of Pearson correlation coefficient calculation, and –1≤r≤+1.

3. The water quality parameter online soft measurement method based on the optimization algorithm according to claim 1 is characterized in that: In S3, regularization is done as follows: ， in, is the original data, is the mean of the original data, is the standard deviation of the original data.

4. The online soft measurement method for water quality parameters based on the optimization algorithm according to claim 1 is characterized in that: In S5, MAE is expressed as: ， in, is the sample size, is the actual value of the i-th chemical water quality indicator, is the predicted value of the i-th chemical water quality indicator; The objective function is expressed as: , in, ; is the learning rate, is the batch size, is the number of neurons.

5. The water quality parameter online soft measurement method based on the optimization algorithm according to claim 1 is characterized in that: In S5, the transit search algorithm TS, particle swarm optimization PSO, ant colony optimization ACO or imperial competition algorithm ICA are used to adjust the model parameters.

6. The online soft measurement method for water quality parameters based on the optimization algorithm according to claim 5 is characterized in that: In S5, the particle swarm optimization algorithm is used to adjust the model parameters; specifically: S5.1 Initialize particles and speed: Determine the number of particles in the particle swarm, randomly initialize the positions of the particles, and each particle represents a set of parameters ( ), randomly initialize the particle speed and determine the dimension of the space according to the number of parameters; S5.2 Iteration: Perform multiple iterations, each iteration comprising the following steps: Update particle position: Update the particle position according to the current speed and position; Update particle speed: Update the particle speed based on the particle's current position, historical best position, and global best position; Calculate the fitness value: Use the parameters of the current position to train the model and calculate MAE as the fitness value; Update the best position of the particle: If the fitness value of the current position is better than the historical best position, then update the best position of the particle; Update the global best position: If the fitness value of the current position is better than the global best position, then update the global best position; S5.3 Terminate and extract the optimal parameters: When the maximum number of iterations is reached or the fitness value reaches the preset threshold, the corresponding parameters are saved and regarded as the optimal parameters ( ); in, is the learning rate, is the batch size, is the number of neurons.

7. The online soft measurement system of water quality parameters based on optimization algorithm is characterized by: The system uses the soft sensing method described in any one of claims 1 to 6 to predict water quality parameters.

Images