Water affair index analysis method based on machine learning
By building a water index database and applied machine learning model, the problem of being unable to analyze multi-character water indexes in the existing technology is solved, and effective classification and prediction of water indexes are achieved, and data quality and analysis credibility are improved.
Patent Information
- Application Number
- CN202411506871.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-11
AI Technical Summary
The analysis methods of water indexes in the prior art cannot be analyzed for multi-character water indexes, and the analyzed water indexes cannot be classified in an orderly manner.
By building a water index database, generating multi-feature tables, realizing data standardization; using minimum-maximum normalization to perform feature scaling, completing data preprocessing; calculating water indexes, realizing the classification of different types of water indexes; applying machine learning model prediction Water index, predicting water pollution processes and classification of pollution water index.
Effective analysis and classification of multi-character water indicators has been achieved, data quality has been improved, and the effectiveness and credibility of water indicators have been ensured.
Smart Images
Figure CN119417038A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and more specifically, to a water affairs index analysis method based on machine learning. Background Art
[0002] Water service indicators are usually determined by characteristics such as dissolved oxygen (DO), total coliform (TC), biological oxygen demand (BOD), nitrate, pH and electrical conductivity (EC);
[0003] However, the water affairs index analysis methods in the prior art often only analyze a single index of the water affairs index, and are unable to analyze multi-feature water affairs indexes, and the analyzed water affairs indexes cannot be classified in an orderly manner.
[0004] Therefore, the present invention provides a water index analysis method based on machine learning, which improves the above-mentioned technical problems. Summary of the invention
[0005] The disclosed embodiments aim to address the deficiencies of the prior art and provide a water index analysis method based on machine learning. The present invention constructs a water index database and generates a multi-feature table to achieve data standardization; performs feature scaling through minimum-maximum normalization to complete data preprocessing and improve the data quality of all data sets; calculates water indexes to achieve classification of three different types of water indexes; applies machine learning models to predict water indexes to achieve prediction of water pollution processes and classification of polluted water indexes.
[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A water affairs index analysis method based on machine learning, comprising the following steps: S1, data collection;
[0007] S2, data preprocessing;
[0008] S3. Calculate water service indicators;
[0009] S4. Apply machine learning models to predict water service index.
[0010] As a preferred technical solution of the present invention, the data collection process is: collecting multi-source data through a water affairs website, constructing a water affairs index database based on the multi-source data, retrieving multi-feature fields in the water affairs index database; generating a multi-feature table based on the multi-feature fields;
[0011] The multiple feature fields include: dissolved oxygen, biological oxygen demand, pH, conductivity, nitrate, and coliform bacteria.
[0012] As a preferred technical solution of the present invention, the process of data preprocessing is:
[0013] Feature scaling via min-max normalization:
[0014]
[0015] Where minA and maxA are the minimum and maximum values of feature A; the original value and normalized value of attribute A are represented by v and v′ respectively; from formula (1), it can be seen that the maximum and minimum eigenvalues are mapped to 1 and 0 respectively.
[0016] As a preferred technical solution of the present invention, the process of calculating the water service index is:
[0017] The WQI calculation formula is:
[0018]
[0019] Where N is the complete number of attributes, qi represents the quality rating scale for each parameter I determined by Equation 3, and is the unit weight for each parameter determined by Equation 4.
[0020] As a preferred technical solution of the present invention, q i is the actual value of the parameter in the test water sample:
[0021]
[0022] As a preferred technical solution of the present invention, i The recommended standard value for this parameter is:
[0023]
[0024] If the ratio is consistent, represented by K, it can be determined by the following formula:
[0025]
[0026] As a preferred technical solution of the present invention, the process of applying a machine learning model to predict the water service index is: predicting the water service index through a neural network and a support vector machine.
[0027] As a preferred technical solution of the present invention, the output of the neural network, neuron Y is shown in Formula 6:
[0028] Y=f(W1.X1+W2.X2+b) (6)
[0029] Where X1, X2 = numerical inputs; W1, W2 = weights associated with these inputs, input 1, function of weight b (bias) f = nonlinearity, called activation function; Y = output of the neuron.
[0030] As a preferred technical solution of the present invention, support vector machine (SVM) can analyze data for classification or regression analysis (SVR); given a training data set Where X is the space of input patterns. In SVR, the main goal is to find a function f(x) characterized by the maximum ε between all training data and the actual target value y obtained. i deviation; in addition, f(x) must be as flat as possible; errors less than ε are tolerable, while errors greater than ε are generally unacceptable; to this end, for a linear function of the form:
[0031]
[0032] in and 〈·,·〉 is the dot product in X, the Euclidean norm must be minimized; it can be viewed as a convex optimization problem, as shown in the following equation:
[0033] minimize And meet
[0034] Formula (8) implicitly shows that the function f can be approximated with ε accuracy (x i ,y i ); however, in many cases this is not possible, or a certain error must be tolerated; therefore, slack variables must be introduced in the constraints of the optimization problem Based on the following formula:
[0035] minimize And meet
[0036] Both the flatness of f and the acceptable deviation above ε depend on the constant C>0; the optimization problem defined in equation (9) is usually solved by its dual formulation, using Lagrange multipliers:
[0037]
[0038] in Under the optimal conditions, L has a significant effect on the variable The partial derivative of must be equal to 0;
[0039] The dual optimization problem can be expressed as:
[0040] maximize And meet
[0041] This condition requires that the product between the constraint and the dual variable must vanish at the optimal solution, that is:
[0042]
[0043] As a preferred technical solution of the present invention, in order to make the support vector algorithm nonlinear, the training pattern xi can be preprocessed by mapping Φ:X→F, where F is a certain feature space; the SVR algorithm only relies on the dot product between various patterns, making full use of the kernel k(x i , x j )=<Φ(x i ),Φ(x j )> to explicitly replace the mapping Φ(·); therefore, the support vector algorithm can be rewritten as:
[0044] maximize And meet
[0045] The expansion of f can be expressed as follows:
[0046] although is uniquely defined, but unlike the linear case, is no longer explicitly identified; in the nonlinear case, the optimization problem is to find the flattest function in the feature space, not in the input space; the function k(x i , x j ) must satisfy the Mercer condition, corresponding to the dot product in a certain feature space F;
[0047] The radial basis function (RBF) is selected as the kernel; the form of RBF is:
[0048] k(x i , x j ) = exp(-γ||x i -x j || 2 ),γ>0 (16).
[0049] In summary, the present invention has the following beneficial effects: constructing a water service index database and generating a multi-feature table to achieve data standardization; completing data preprocessing by performing feature scaling through minimum-maximum normalization to improve the data quality of all data sets; classifying three different types of water service indicators by calculating water service indicators; applying a machine learning model to predict water service indexes to achieve the prediction of water pollution processes and the classification of polluted water service indexes. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A flow chart of a water service indicator analysis method based on machine learning provided in an embodiment of the present invention;
[0051] Figure 2 The feature correlation of the grouped data set provided by the embodiment of the present invention. DETAILED DESCRIPTION
[0052] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.
[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0054] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.
[0055] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.
[0056] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.
[0057] The disclosed embodiments are intended to solve the problem that water service index analysis methods often only analyze a single index of water service indexes, are unable to analyze multi-feature water service indexes, and that the analyzed water service indexes cannot be classified in an orderly manner. In view of this, the disclosed embodiments propose a water service index analysis method based on machine learning to construct a water service index database and generate a multi-feature table to achieve data standardization; perform feature scaling through minimum-maximum normalization to complete data preprocessing and improve the data quality of all data sets; classify three different types of water service indicators by calculating water service indicators; and apply machine learning models to predict water service indexes to ensure the effectiveness and credibility of water service indicators.
[0058] Please refer to Figure 1 , Figure 1 The flowchart of a water service indicator analysis method based on machine learning according to an embodiment of the present disclosure is shown. The overall process mainly includes the following 6 steps:
[0059] Step 1: Data collection.
[0060] Collect data from multiple sources through water affairs websites, build a water affairs indicator database based on the data from multiple sources, retrieve multiple feature fields from the water affairs indicator database; generate a multi-feature table based on the multiple feature fields;
[0061] Referring to Table 1, the multiple characteristic fields include: dissolved oxygen (DO), biological oxygen demand (BOD), PH, electrical conductivity (EC), nitrate (NNR), and coliform bacteria (TC).
[0062] Refer to Table 1
[0063] Feature Name Feature fields Dissolved oxygen (DO) The amount of oxygen present in water PH Water Acidity and Alkalinity Electrical conductivity (EC) Can water conduct electricity in the presence of ions? Biological oxygen demand (BOD) Amount of oxygen required for organic waste in water Nitrate (NNR) Nitrate content Coliform bacteria (TC) Is there coliform bacteria in the water?
[0064] Step 2: Data preprocessing.
[0065] Data preprocessing is done by feature scaling via min-max normalization:
[0066] Specifically, the process of feature scaling by minimum-maximum normalization is as follows: the minimum-maximum normalization of features in the range [0,1] is scaled by formula (1):
[0067]
[0068] Where minA and maxA are the minimum and maximum values of feature A. The original value and normalized value of attribute A are represented by v and v′, respectively. From formula (1), it can be seen that the maximum and minimum eigenvalues are mapped to 1 and 0, respectively.
[0069] Step 3: Calculate water service indicators.
[0070] The WQI calculation process is:
[0071]
[0072] Where N is the total number of attributes, qi represents the quality rating scale of each parameter I determined by the following conditions (Formula 3), and is the unit weight of each parameter determined by the conditions (Formula 4).
[0073] In the formula, q i is the actual value of the parameter in the test water sample:
[0074]
[0075] wi The recommended standard value for this parameter is:
[0076]
[0077] If the ratio is consistent, represented by K, it can be determined by the following formula:
[0078]
[0079] Refer to Table 2, which gives the unit weight value of each parameter. Among them, the values of PH, DO, TC, BOD, EC and NNR are 0.16, 0.28, 0.28, 0.23, 0.09 and 0.028 respectively.
[0080] Refer to Table 2
[0081] Feature Name Unit weight Dissolved oxygen (DO) 0.28 PH 0.16 Electrical conductivity (EC) 0.09 Biological oxygen demand (BOD) 0.23 Nitrate (NNR) 0.028 Coliform bacteria (TC) 0.28
[0082] When the evaluation results are substituted into equation (2), the WQI value can be obtained, and the water service indicators can be classified by obtaining different WQIs. Among them, water service indicators can be divided into three categories: good, poor and unsuitable for drinking. The water quality index is 0-50 for good, 51-100 for poor, and greater than 100 for unsuitable for drinking.
[0083] Figure 2 , shows the feature correlation matrix of the features in the dataset after preprocessing. From the figure, it can be seen that TC and DO are positively correlated with WQI, with a value of +0.7, and EC is negatively correlated with PH, with a value of -0.3.
[0084] Step 4: Apply machine learning model to predict water service index.
[0085] Specifically, when the water service index is classified as poor and unsuitable for drinking, the water pollution process is predicted by a neural network and the polluted water service index is classified using a support vector machine.
[0086] In the embodiment, the output of the neural network, neuron Y is shown in Formula 6.
[0087] Y=f(W1.X1+W2.X2+b) (6)
[0088] Where X1, X2 = numerical inputs; W1, W2 = weights associated with these inputs, input 1, weight b (bias) f = nonlinear function, called activation function. Y = output of the neuron.
[0089] In an embodiment, a support vector machine (SVM) can analyze data for classification or regression analysis (SVR). Given a training data set Where X is the space of input patterns. In SVR, the main goal is to find a function f(x) characterized by the maximum ε between all training data and the actual target value y obtained. i In addition, f(x) must be as flat as possible. Errors less than ε are tolerable, while errors greater than ε are generally unacceptable. To this end, for a linear function of the form:
[0090]
[0091] in and 〈·,·〉 is the dot product in X, the Euclidean norm must be minimized. Therefore, the problem can be viewed as a convex optimization problem, as shown in the following equation:
[0092] minimize And meet
[0093] Formula (8) implicitly shows that the function f can be approximated with ε accuracy (x i ,y i ). However, in many cases this is not possible or a certain error must be tolerated. Therefore, slack variables must be introduced in the constraints of the optimization problem. Based on the following formula:
[0094] minimize And meet
[0095] Both the flatness of f and the acceptable deviation above ε depend on the constant C > 0. The optimization problem defined in equation (9) is usually solved by its dual formulation, using Lagrange multipliers:
[0096]
[0097] in Under the optimal conditions, L has a significant effect on the variable The partial derivative of must be equal to 0.
[0098] Therefore, the dual optimization problem can be expressed as:
[0099] maximize And meet
[0100] This condition requires that the product between the constraint and the dual variable must vanish at the optimal solution, that is:
[0101]
[0102] Then, in order to make the support vector algorithm nonlinear, the training pattern xi can be preprocessed by mapping Φ:X→F, where F is a certain feature space. The SVR algorithm only relies on the dot product between various patterns and makes full use of the kernel k(x i , x j )=<Φ(x i ),Φ(x j )> to explicitly replace the mapping Φ(·). Therefore, the support vector algorithm can be rewritten as:
[0103] maximize And meet
[0104] The expansion of f can be expressed as follows:
[0105] Although w is uniquely defined, it is no longer explicitly identified as in the linear case. In the nonlinear case, the optimization problem is to find the flattest function in the feature space, not in the input space. The function k(x i , x j ) must satisfy the Mercer condition, corresponding to a dot product in some feature space F.
[0106] The radial basis function (RBF) is selected as the kernel. The form of RBF is:
[0107] k(x i , x j ) = exp(-γ||x i -x j || 2 ), γ>0 (16)
[0108] The support vector algorithm searches for the best structure of the model and its parameters through a trial-and-error iterative procedure. The deviation between the function found by SVR and the target value is controlled by the ε parameter. Finally, ε was chosen to be equal to 0.01, while the error cost C, which affects the smoothness of the function, was assumed to be equal to 1.
[0109] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A water service index analysis method based on machine learning, characterized in that: The method comprises the following steps: S1. Data collection; S2, data preprocessing; S3. Calculate water service indicators; S4. Apply machine learning models to predict water service index.
2. The water affairs index analysis method based on machine learning according to claim 1 is characterized in that: The data collection process is as follows: collecting multi-source data through water affairs websites, constructing a water affairs index database based on the multi-source data, retrieving multi-feature fields in the water affairs index database; generating a multi-feature table based on the multi-feature fields; The multiple feature fields include: dissolved oxygen, biological oxygen demand, pH, conductivity, nitrate, and coliform bacteria.
3. The water affairs index analysis method based on machine learning according to claim 1 is characterized in that: The data preprocessing process is as follows: Feature scaling via min-max normalization: Where minA and maxA are the minimum and maximum values of feature A; the original value and normalized value of attribute A are represented by v and v′ respectively; from formula (1), it can be seen that the maximum and minimum eigenvalues are mapped to 1 and 0 respectively.
4. The water affairs index analysis method based on machine learning according to claim 1 is characterized in that: The process of calculating water service indicators is as follows: The WQI calculation formula is: Where N is the complete number of attributes, qi represents the quality rating scale for each parameter I determined by Equation 3, and is the unit weight for each parameter determined by Equation 4.
5. The water affairs index analysis method based on machine learning according to claim 4 is characterized in that: q i is the actual value of the parameter in the test water sample:
6. The method for analyzing water service indicators based on machine learning according to claim 4 is characterized in that: w i The recommended standard value for this parameter is: If the ratio is consistent, represented by K, it can be determined by the following formula:
7. The water affairs index analysis method based on machine learning according to claim 1 is characterized in that: The process of applying the machine learning model to predict the water service index is: when the water service index is classified as poor and unsuitable for drinking, the water pollution process is predicted by a neural network and the polluted water service index is classified by a support vector machine.
8. The method for analyzing water service indicators based on machine learning according to claim 7 is characterized in that: The output of the neural network, neuron Y, is shown in Formula 6: Y=f(W1.X1+W2.X2+b) (6) Where X1, X2 = numerical inputs; W1, W2 = weights associated with these inputs, input 1, function of weight b (bias) f = nonlinearity, called activation function; Y = output of the neuron.
9. The method for analyzing water service indicators based on machine learning according to claim 7, characterized in that: Support vector machines (SVM) can analyze data for classification or regression analysis (SVR); given a training data set Where X is the space of input patterns. In SVR, the main goal is to find a function f(x) characterized by the maximum ε between all training data and the actual target value y obtained. i deviation; in addition, f(x) must be as flat as possible; errors less than ε are tolerable, while errors greater than ε are generally unacceptable; to this end, for a linear function of the form: f(x)=<ω,x>+b (7) where w∈X, b∈R and 〈·,·〉 is the dot product in X, and the Euclidean norm ||w|| 2 must be minimized; it can be viewed as a convex optimization problem, as shown in the following equation: minimize And meet Formula (8) implicitly shows that the function f can be approximated with ε accuracy (x i ,y i ); however, in many cases this is not possible or a certain error must be tolerated; therefore, slack variables must be introduced in the constraints of the optimization problem Based on the following formula: minimize And meet Both the flatness of f and the acceptable deviation above ε depend on the constant C>0; the optimization problem defined in equation (9) is usually solved by its dual formulation, using Lagrange multipliers: in Under the optimal conditions, L has a significant effect on the variable The partial derivative of must be equal to 0; The dual optimization problem can be expressed as: maximize And meet This condition requires that the product between the constraint and the dual variable must vanish at the optimal solution, that is:
10. The method for analyzing water service indicators based on machine learning according to claim 7, characterized in that: In order to make the support vector algorithm nonlinear, the training pattern xi can be preprocessed by mapping Φ:X→F, where F is a feature space; the SVR algorithm only relies on the dot product between various patterns and makes full use of the kernel k(x i , x j )=<Φ(x i ),Φ(x j ) to explicitly replace the mapping φ(·); therefore, the support vector algorithm can be rewritten as: maximize And meet The expansion of f can be expressed as follows: Although ω is uniquely defined, it is no longer explicitly identified as in the linear case; in the nonlinear case, the optimization problem is to find the flattest function in the feature space, not in the input space; the function k(x i , x j ) must satisfy the Mercer condition, corresponding to the dot product in a certain feature space F; The radial basis function (RBF) is selected as the kernel; the form of RBF is: k(x i ,x j )=exp(-γ||x i -x j || 2 ),γ>0 (16)。