Method for predicting heavy metal concentration of lake water body based on machine learning
By combining unsupervised machine learning with RPCA and SOM for dimensionality reduction, and combining support vector machine, random forest and multi-layer perceptron models, the problem of accuracy in predicting heavy metal concentrations in lake water was solved, achieving fast and accurate prediction results.
Patent Information
- Application Number
- CN202510906702.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-02
- Publication Date
- 2025-10-17
AI Technical Summary
Existing machine learning models find it difficult to accurately predict heavy metal concentrations in lake water, especially due to the presence of abnormal data and high-dimensional data in the original data, resulting in poor fitting results and difficulty in large-scale promotion.
An unsupervised machine learning method was used, combined with robust principal component analysis (RPCA) and self-organizing map (SOM) for dimensionality reduction, to extract the main feature components. The components were merged with the original data and used as input to support vector machine, random forest and multi-layer perceptron models for training. Finally, the best model was selected for prediction through 10-fold cross validation.
It achieves rapid and accurate prediction of heavy metal concentrations in lake water, improves the generalization ability and prediction accuracy of the model, and provides a simple and fast method for judging the degree of pollution.
Smart Images
Figure CN120804685A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of environment, in particular to a lake water heavy metal concentration prediction method based on machine learning. BACKGROUND
[0002] Human activities have intensified global environmental heavy metal pollution. Compared with other water systems, lakes have poor flowability, and heavy metals are easy to accumulate in lakes for a long time after entering, forming potential pollution threats. The ecosystem of the lake is sensitive and closed, and the concentration can be amplified by thousands of times. Cities, farmland and mining areas are often distributed around the lake, which directly receives industrial wastewater, agricultural runoff and domestic sewage. In addition, lakes are important sources of drinking water. By understanding the heavy metal content of lake water, it is helpful to understand the pollution level of the local area and timely prevent heavy metals. Common detection methods for heavy metal content in lake water include inductively coupled plasma mass spectrometry (ICP-MS), atomic absorption spectrometry (AAS) and inductively coupled plasma emission spectrometry (ICP-OES), etc. However, these methods have the disadvantages of expensive instruments, easy interference, complex operation or low efficiency. At present, many machine learning models have been applied to the prediction of environmental heavy metals, involving air, water, soil and sediments. However, the prediction of lake heavy metals usually only involves raw data of lake physical and chemical properties, which often contain abnormal data and high-dimensional data, making it difficult for machine learning to correctly grasp the key features, so that it has excellent fitting effect at that time, but it is difficult to be popularized on a large scale. Therefore, it is of great application value to explore a method for efficiently and accurately predicting the heavy metal concentration of lake water by using unsupervised machine learning method. SUMMARY
[0003] The present application aims to fill the gap of machine learning model prediction of lake heavy metals, overcome the shortcomings of inaccurate grasping of features by machine learning prediction model and difficult to correctly predict the heavy metal concentration of lake water, and provide a method for predicting the heavy metal concentration of lake water by machine learning, which can realize accurate and rapid prediction of lake heavy metals.
[0004] To achieve the above purpose, the present application realizes the following technical solutions,
[0005] A method for predicting the heavy metal concentration of lake water based on machine learning, comprising the following steps:
[0006] Step S1, collect and detect the physical and chemical properties of a plurality of lake water, and establish a lake database;
[0007] Step S2, standardize the feature data set in step S1 to adapt to the subsequent machine learning method and optimize the numerical stability;
[0008] Step S3, based on unsupervised machine learning, the feature data set is input to robust principal component analysis RPCA and self-organizing map SOM for dimension reduction, extracting the main feature components while retaining the key information, and merging the obtained RPCA features and SOM features with the original data to serve as a new feature data set;
[0009] Step S4, the new feature data set obtained in step S3 is applied to machine learning support vector machine model, random forest model and multilayer perceptron model for training, to obtain a machine learning model that can predict the concentration of heavy metals in the lake water body;
[0010] Step S5, according to the 10-fold cross-validation results of the support vector machine model, random forest model and multilayer perceptron model trained in step S4, the best model with the best prediction result is selected for lake water body heavy metal concentration prediction;
[0011] Step S6, based on the lake heavy metal data published on the academic platform, the prediction effect of the model selected in step S5 is verified.
[0012] Further, in step S1, the physicochemical properties of the lake water body are collected and detected, and a lake database is established, and the specific steps include:
[0013] Step S1.1, detecting the physicochemical properties of the water sample, such as pH value, temperature, conductivity, total organic carbon content and dissolved oxygen DO, and dividing the lake into different basins according to the basin division standard;
[0014] Step S1.2, establishing a lake database based on the physicochemical properties of the lake water body and the data of the basin division.
[0015] Further, in step S2, the feature data set is standardized to match the subsequent machine learning method, and the specific steps include:
[0016] Step S2.1, converting the target variable of the data set from a one-dimensional array to a two-dimensional column vector (x, y), ensuring that the dimensions of the input features x and the target variable y match, and avoiding dimension errors in the subsequent support vector machine model and multilayer perceptron model;
[0017] Step S2.2, converting the data to a distribution with a mean of 0 and a standard deviation of 1, i.e. Z-score standardization, and the formula for standardization is:
[0018]
[0019] Where: χ represents the original data, μ represents the mean of the feature data set, and σ represents the standard deviation of the feature data set.
[0020] Further, step S3 is to input the feature data set into RPCA and SOM based on unsupervised machine learning method for dimension reduction, and merge the obtained RPCA features and SOM features with the original data to form a new feature data set, the specific steps including:
[0021] Step S3.1, the original feature data set is subjected to RPCA, and the data is decomposed into low-dimensional principal components and abnormal data, and through data cleaning, the dimension reduction results of the first four principal components are retained;
[0022] Step S3.2, the original feature data set is subjected to SOM, and the high-dimensional data is mapped to low-dimensional grid while the topological structure is retained, and the parameters of the neurons are adjusted to obtain SOM neuron coordinates and SOM distance;
[0023] Step S3.3, the original data, RPCA features and SOM features are merged as a new feature data set, including 4 columns of original standardized features, 4 columns of PCA dimension reduction features, 2 columns of SOM neuron coordinates and 1 column of SOM distance.
[0024] Further, in step S4, the new feature data set is applied to machine learning support vector machine model, random forest model and multilayer perception model for training to obtain a machine learning model that can predict the concentration of heavy metals in the lake water body, and the specific steps including:
[0025] The 11 groups of data in the new feature data set are taken as variables, and the concentrations of single heavy metal elements As, Cu, Cd, Cr, Pb and Zn are taken as prediction results, which are input into the support vector machine model, the random forest model and the multilayer perception model to obtain the concentration prediction results of each heavy metal.
[0026] Further, step S5 selects the best model in the prediction result by 10-fold cross-validation, and the specific method is: the data set is randomly divided into 10 similar and mutually exclusive subsets, i.e. "folds", each subset maintains the consistency of data distribution as much as possible, and the subsets are divided into training set and test set in the ratio of 9:1, and the rotation verification strategy is adopted, that is, 90% of the data is used for training and 10% of the data is used for verification, ensuring that each fold of data is used as a verification set once, and the evaluation index of each verification is recorded, and the final index is the average value of 10 results, reflecting the generalization ability of the model, and the evaluation index includes the determination coefficient R 2 and the root mean square error RMSE;
[0027] R 2 The formula is:
[0028]
[0029] Where, SS res represents the sum of squares of residuals, i.e. prediction error, SStot denotes the total sum of squares, i.e., the total variance of data, y i denotes the true value, denotes the predicted value, denotes the mean of the true value;
[0030] The formula of RMSE is:
[0031]
[0032] wherein n represents the number of samples, represents the prediction error;
[0033] R 2 and the value range of RMSE is [0, 1] and [0, ∞) respectively, R 2 the closer to 1, the better, and the smaller the RMSE, the better, and the prediction results of different models are compared to determine the prediction superiority of each model.
[0034] Further, in step S6, the prediction effect of the selected model is verified based on the lake heavy metal data published on the academic platform. The specific steps include:
[0035] S6.1, extracting the lake water heavy metal data published in the past 20 years from CNKI, ScienceDirect and Taylor & Francis academic platforms;
[0036] S6.2, after dimension reduction by RPCA and SOM, input into the trained random forest model to verify the prediction effect of the machine learning model on the lake water heavy metal.
[0037] The beneficial effects of the present application are as follows:
[0038] 1. The present application provides a method for predicting the concentration of lake water heavy metal based on machine learning, which takes the detected physicochemical properties of lake water as characteristic input for unsupervised machine learning RPCA and SOM, reduces the dimension of the original feature data set, discovers the potential features, and then trains and learns through machine learning support vector machine model, random forest model and multilayer perception machine model to obtain a random forest model that can predict the concentration of lake water heavy metal, and the R 2 and RMSE of the prediction result are better than those of the support vector machine model and the multilayer perception machine model.
[0039] 2. The model cooperative method for predicting the concentration of lake water heavy metal based on unsupervised machine learning dimension reduction and machine learning model of the present application can simply, quickly and accurately judge the pollution degree of lake water heavy metal, and provides a technical basis for quickly predicting the concentration of lake water heavy metal. BRIEF DESCRIPTION OF DRAWINGS
[0040] The drawings constituting this application are only used to further understand the present application, the embodiments of the present application and their descriptions do not constitute undue limitations on the present application.
[0041] Figure 1 Flow chart for the method of predicting the concentration of heavy metals in lake water bodies based on machine learning of the present application;
[0042] Figure 2 Flow chart for unsupervised machine learning dimension reduction of the present application;
[0043] Figure 3 Model coordination flow chart for the implementation method of the present application;
[0044] Figure 4 Radar chart of the determination coefficient and root mean square error of the 10-fold cross-validation results of the three machine learning models of the present application;
[0045] Figure 5 Scatter plot and distribution plot of the prediction results of the random forest model of the present application. DETAILED DESCRIPTION
[0046] The present application will be further described in detail below in combination with the drawings and specific embodiments.
[0047] The present application provides a method for predicting the concentration of heavy metals in lake water bodies based on machine learning, which can quickly and accurately obtain the concentration of heavy metals in lakes according to the physicochemical characteristic data set of lake water bodies.
[0048] As shown in Figure 1 The present application comprises the following steps:
[0049] S1, collecting and detecting the physicochemical properties of a plurality of lake water bodies to establish a lake database;
[0050] S2, standardizing the characteristic data set in the lake database of step S1 to adapt to subsequent machine learning methods and optimize the numerical stability;
[0051] S3, based on unsupervised machine learning, inputting the characteristic data set into RPCA and SOM for dimension reduction, merging the obtained RPCA features and SOM features with the original data to serve as a new characteristic data set;
[0052] S4, applying the new characteristic data set obtained in step S3 to machine learning support vector machine model, random forest model and multilayer perceptron model for training to obtain machine learning models that can predict the concentration of heavy metals in lake water bodies;
[0053] S5, according to the 10-fold cross-validation results of the support vector machine model, the random forest model and the multilayer perceptron model trained in step S4, the best model for predicting the concentration of heavy metals in the lake water body is screened out, and is used for predicting the concentration of heavy metals in the lake water body.
[0054] Further, the step S1 collects and detects the physicochemical properties of the lake water body, and establishes a lake database, and the specific steps include:
[0055] S1.1, 45 lake water samples in a certain area are collected, and the surface water samples at least 15 m away from the lake shore and 0.5 m deep are collected at a time when there is no rainfall or the rainfall in two weeks is less than 10 mm. The physicochemical properties of the water sample, such as pH value, temperature, conductivity, total organic carbon content and dissolved oxygen (DO), are detected, and the lakes are divided into different basins according to the basin division standard;
[0056] S1.2, a lake database is established by using the physicochemical properties of the lake water body and the data of the basin division.
[0057] Further, the step S2 standardizes the feature data set to match the subsequent machine learning method, and the specific steps include:
[0058] S2.1, the target variable of the feature data set is converted from a one-dimensional array to a two-dimensional column vector (x, y), so as to ensure that the dimensions of the input feature x and the target variable y are matched, and to avoid dimension errors of the subsequent support vector machine model and multilayer perceptron model;
[0059] S2.2, the data is converted into a distribution with a mean of 0 and a standard deviation of 1 (i.e. Z-score standardization), and the formula for standardization processing is:
[0060]
[0061] Wherein: X, μ, σ represent the original data, the mean of the feature data set, and the standard deviation of the feature data set, respectively.
[0062] Further, the step S3 inputs the feature data set into RPCA and SOM based on unsupervised machine learning for dimension reduction, and merges the obtained RPCA features and SOM features with the original data to serve as a new feature data set, and the specific steps include:
[0063] S3.1, the original feature data set is subjected to RPCA, the data is decomposed into low-dimensional principal components and abnormal data, the abnormal data is removed, and the dimension reduction result of the low-dimensional principal components is obtained;
[0064] S3.2, the original feature data set is subjected to SOM, the high-dimensional data is projected onto a low-dimensional grid while the topological structure is preserved, and the parameters of the neurons are adjusted to obtain the SOM neuron coordinates and the SOM distance;
[0065] In steps S3.1 and S3.2 of the present application, the specific dimension reduction method is:
[0066] Data dimension reduction and mining based on Python;
[0067] Import the sklearn.decomposition.PCA library and the minisom.MiniSom library;
[0068] Read the original feature database;
[0069] Set the random number seed;
[0070] Approximate implementation of RPCA using the sklearn.decomposition.PCA library, with the following parameter settings:
[0071] n_components=4;
[0072] Use the minisom.MiniSom library to construct a SOM, set a 5x5 two-dimensional neuron grid, and set the number of training iterations to 1000.
[0073] S3.3, merge the original data, RPCA features and SOM features as a new feature dataset, which includes 4 columns of original standardized features, 4 columns of PCA dimension reduction features, 2 columns of SOM neuron coordinates and 1 column of SOM distance. As shown in Figure 2 , where: the low-rank matrix L is the denoised data, the sparse matrix S is the outlier / noise, W j is the neuron weight, BMU is the best matching unit, and a(t) is the learning rate.
[0074] Further, the step S4 applies the new feature dataset to the machine learning support vector machine model, the random forest model and the multilayer perceptron model for training, to obtain a machine learning model that can predict the concentration of heavy metals in the lake water body. The specific steps include:
[0075] Take 11 groups of data in the new feature dataset as variables, and the concentration of a single heavy metal element (As, Cu, Cd, Cr, Pb, Zn) as the prediction result, and input them into the support vector machine model, the random forest model and the multilayer perceptron model to obtain the prediction result of each heavy metal element.
[0076] In this step, the specific model expression method is:
[0077] Build the model based on Python;
[0078] Import the sklearn.svm.SVR library, sklearn.ensemble.RandomForestRegressor library, and sklearn.neural_network.MLPRegressor library;
[0079] Read the new feature dataset;
[0080] Set the random number seed;
[0081] Use the sklearn.svm.SVR library to build support vector machine models,
[0082] The sklearn.ensemble.RandomForestRegressor library builds a random forest model.
[0083] The sklearn.neural_network.MLPRegressor library builds a multi-layer perceptron model;
[0084] Step S5 uses the 10-fold cross-validation results to select the model with the best prediction results, randomly divide the data set into 10 mutually exclusive subsets of similar size, namely "folds", and keep the consistency of data distribution as much as possible in each subset. The subsets are divided into training sets and test sets in a ratio of 9:1. A rotation validation strategy is adopted, using 90% of the data for training and 10% of the data for validation, ensuring that each fold data is used as a validation set once. The evaluation indicators of each validation are recorded, and the final indicator is the average of the 10 results, which reflects the generalization ability of the model. The evaluation indicators include the coefficient of determination (R 2 ) and root mean square error (RMSE);
[0085] R 2 The formula is:
[0086]
[0087] Among them, SS res Represents the residual sum of squares (prediction error), SS tot represents the total sum of squares (total variance of data), y i Represents the true value, Represents the predicted value, represents the mean of the true values;
[0088] The formula for RMSE is:
[0089]
[0090] Where n represents the number of samples, represents the prediction error.
[0091] R 2 and the value range of RMSE is [0, 1] and [0, ∞) respectively, R 2 The closer to 1, the smaller the RMSE, the better the prediction result is, and the better the prediction of each model is.
[0092] Figure 3 The model collaborative flowchart for the implementation method of the application is shown in the figure;
[0093] Figure 4 As shown in the figure, the determination coefficient and the root mean square error of the ten-fold cross-validation of the three machine learning models of the application are shown, wherein the light blue is the random forest model, the light orange is the support vector machine model, and the light green is the multilayer perception machine model. The determination coefficient (R 2 ) and the root mean square error (RMSE) of the random forest model are better than those of the support vector machine model and the multilayer perception machine model, and are more suitable for application in predicting the heavy metal concentration of lake water.
[0094] Further, the prediction effect of the selected model is verified based on the lake heavy metal data published on the academic platform in step S6. The specific steps include:
[0095] S6.1, extracting the published lake water heavy metal data in the past 20 years from CNKI, ScienceDirect, Taylor & Francis and other academic platforms;
[0096] S6.2, after dimension reduction by RPCA and SOM, inputting into the trained random forest model to verify the prediction effect of the machine learning model on the heavy metal of lake water. As Figure 5 shown, wherein the green ring is the data for verifying the prediction effect of the model, R is the Pearson correlation coefficient, indicating the linear correlation between the predicted value and the true value. P is the significance level, and the statistical significance of R is tested to determine whether it is caused by random noise. In the result, R≥0.815 and p≤0.004, indicating that the prediction effect of the random forest model is excellent, and the result is highly reliable.
[0097] In summary, the application provides a method for predicting the heavy metal concentration of lake water based on machine learning. RPCA and SOM are used to reduce the dimension of the physicochemical properties of 45 lake water bodies, and the potential features are mined. Then, the support vector machine model, the random forest model and the multilayer perception machine model are trained and learned to screen a convenient and accurate random forest model for predicting the heavy metal concentration of lake water. The method for predicting the heavy metal concentration of lake water based on machine learning can quickly and accurately predict the heavy metal of lake, and provides a technical basis for quickly and accurately obtaining the concentration of lake heavy metal.
[0098] The above is only an example method of the present application for predicting heavy metal concentration in a lake water body, and does not limit the protection scope of the present application, and any equivalent replacement or change according to the technical ideas of the present application is within the protection scope of the present application.
Claims
1. A method for predicting heavy metal concentrations in lake waters based on machine learning, characterized in that: The method comprises the following steps: S1. Collect and test the physical and chemical properties of water in multiple lakes and establish a lake database; S2. Standardize the feature dataset in step S1 to adapt to subsequent machine learning methods and optimize numerical stability; S3. Based on the unsupervised machine learning method, the feature dataset is input into the robust principal component analysis (RPCA) and self-organizing map (SOM) for dimensionality reduction, the main feature components are extracted while retaining the key information, and the obtained RPCA features and SOM features are merged with the original data to obtain a new feature dataset; S4, applying the new feature data set obtained in step S3 to a machine learning support vector machine model, a random forest model, and a multi-layer perceptron model for training to obtain a machine learning model that can predict the heavy metal concentration in lake water; S5. Based on the 10-fold cross-validation results of the support vector machine model, random forest model, and multi-layer perceptron model trained in step S4, the model with the best prediction result is finally selected for the prediction of heavy metal concentration in lake water; S6. Based on the lake heavy metal data published on the academic platform, verify the prediction effect of the model selected in step S5.
2. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: Step S1 collects and detects the physical and chemical properties of lake water and establishes a lake database. The specific steps include: S1.
1. Test water samples for their physical and chemical properties: pH, temperature, conductivity, total organic carbon, and dissolved oxygen (DO). Divide the lake into different watersheds according to the watershed classification standards. S1.
2. Establish a lake database using data on the physical and chemical properties of lake water bodies and river basin divisions.
3. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: In step S2, the feature data set is standardized. The specific steps include: S2.
1. Convert the target variable of the feature dataset from a one-dimensional array to a two-dimensional column vector (x, y). Ensure that the dimensions of the input feature x and the target variable y match to avoid dimensional errors in the subsequent support vector machine model and multi-layer perceptron model. S2.
2. Convert the data to a distribution with a mean of 0 and a standard deviation of 1, i.e., Z-score normalization. The formula for normalization is: Where: χ represents the original data, μ represents the mean of the feature data set, and σ represents the standard deviation of the feature data set.
4. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: Step S3 is to input the feature dataset into RPCA and SOM for dimensionality reduction based on the unsupervised machine learning method, and merge the obtained RPCA features and SOM features with the original data to obtain a new feature dataset. The specific steps include: S3.
1. Perform RPCA on the original feature dataset to decompose the data into low-dimensional principal components and abnormal data. After data cleaning, retain the dimensionality reduction results of the first four principal components. S3.
2. Then perform SOM on the original feature dataset to map the high-dimensional data to a low-dimensional grid while preserving the topological structure, adjust the parameters of the neurons, and obtain the SOM neuron coordinates and SOM distance; S3.
3. Merge the original data, RPCA features, and SOM features as a new feature dataset. The new feature dataset includes 4 columns of original standardized features, 4 columns of PCA dimensionality reduction features, 2 columns of SOM neuron coordinates, and 1 column of SOM distance.
5. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: Step S4 applies the new feature dataset to a machine learning support vector machine model, a random forest model, and a multi-layer perceptron model for training to obtain a machine learning model that can predict heavy metal concentrations in lake water. The specific steps include: The 11 groups of data in the new feature dataset were used as variables, and the concentrations of individual heavy metals, As, Cu, Cd, Cr, Pb, and Zn, were used as prediction results. They were input into the support vector machine model, random forest model, and multi-layer perceptron model to obtain the concentration prediction results of each heavy metal.
6. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: Step S5 uses the 10-fold cross-validation results to select the model with the best prediction results. The data set is randomly divided into 10 mutually exclusive subsets of similar size, namely "folds". Each subset maintains the consistency of data distribution as much as possible. The subsets are divided into training sets and test sets in a ratio of 9:
1. A rotation validation strategy is adopted, using 90% of the data for training and 10% for validation, ensuring that each fold data is used as a validation set once. The evaluation indicators of each validation are recorded. The final indicator is the average of the 10 results, which reflects the generalization ability of the model. The evaluation indicators include the determination coefficient R 2 and root mean square error RMSE; R 2 The formula is: Among them, SS res Represents the residual sum of squares, that is, the prediction error, SS tot represents the total sum of squares, that is, the total variance of the data, y i Represents the true value, Represents the predicted value, represents the mean of the true values; The formula for RMSE is: Where n represents the number of samples, represents the prediction error; R 2 The value ranges of and RMSE are [0, 1] and [0, ∞), respectively. The prediction excellence of each model is judged by comparing the prediction results of different models.
7. The method for predicting heavy metal concentration in lake water based on machine learning according to claim 1, characterized in that: Step S6 verifies the prediction effect of the selected model based on the lake heavy metal data published on the academic platform. The specific steps include: S6.
1. Extract lake water heavy metal data published in the past 20 years from CNKI, ScienceDirect, and Taylor & Francis academic platforms; S6.
2. After dimensionality reduction by RPCA and SOM, the data were input into the trained random forest model to verify the prediction effect of the machine learning model on heavy metals in lake water.