Method for predicting content of heavy metals with different occurrence forms in soil based on machine learning

Abstract

The invention discloses a method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning, and relates to the technical field of soil component content analysis, and the method comprises the following steps: (1) collecting soil physicochemical properties, the total amount of soil heavy metals, the types of the heavy metals in the soil and the content of the heavy metals in different occurrence forms in the soil from a literature library, constructing a sample database; (2) preprocessing the sample database to obtain a preprocessed sample database; (3) for each kind of heavy metal, extracting corresponding sample data from the preprocessing sample database, dividing the sample data into a training set and a test set, selecting a machine learning algorithm, establishing a prediction model training model of different occurrence form contents of the heavy metal in the soil, and optimizing hyper-parameters in the training process; and (4) predicting the content of heavy metals in different occurrence forms in the soil to be detected by using the optimized prediction model. The method has a good application prospect in the aspects of heavy metal pollution risk evaluation and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of soil component content analysis, and in particular to a method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning. Background Art

[0002] Soil heavy metal pollution risk assessment is to characterize the harmful consequences of soil heavy metals on the ecosystem, which is the basis for efficient prevention and control of soil heavy metal pollution. Soil heavy metal pollution has become one of the global environmental pollution issues that has attracted much attention, and it poses a serious threat to the ecological environment and human health. With the continuous deepening of research on soil heavy metal pollution, it is found that although the total amount of soil heavy metals can reflect the degree of heavy metal enrichment, the bioavailability, mobility, and toxicity of heavy metals in the soil largely depend on their occurrence form in the soil system. Therefore, it is necessary to evaluate the pollution risk based on the occurrence form of heavy metals.

[0003] Traditional heavy metal pollution research mainly relies on experimental analysis, which has problems such as difficulty in data acquisition and high analysis costs. Using machine learning methods to predict the content of heavy metals in different forms in soil can provide a scientific basis for heavy metal pollution control.

[0004] The Chinese patent document with publication number CN115359185A discloses a method for identifying key factors affecting the accumulation and migration of heavy metals in soil of heavily polluted sites, including: selecting typical heavily polluted sites to collect samples; completing soil physicochemical tests on the samples, and obtaining the site hydrogeological environment at the sampling site; three-dimensional precise construction of the total amount and morphological distribution of heavy metals based on the basic physicochemical and hydrogeological environment of the site soil; and identifying important factors affecting the accumulation and migration of heavy metals of characteristic pollutants in different heavily polluted sites. The invention can identify the key driving factors affecting the accumulation and migration of the total amount of heavy metals and their occurrence forms in different heavily polluted sites based on the basic physicochemical properties of each soil layer and the hydrogeological environmental conditions of different heavily polluted sites.

[0005] The Chinese patent document with publication number CN112394158A discloses a method for assessing the health risk of human beings based on the form of heavy metals in soil, including: sample collection and processing to determine the analyzed heavy metal elements; extracting the heavy metal occurrence forms to determine the heavy metal bioavailable content; determining the heavy metal exposure concentration based on the heavy metal bioavailable content; and conducting health risk assessment analysis based on the heavy metal exposure concentration. This invention combines the bioavailability of heavy metals with the health risk assessment model of heavy metals, which can reduce the uncertainty of the health risk model from the perspective of reasonable estimation of heavy metal exposure dose. Summary of the invention

[0006] The present invention provides a method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning. It deeply integrates machine learning with the research on the occurrence forms of heavy metals in soil, and develops a method and system for efficiently and accurately identifying the content of heavy metals in different occurrence forms in soil by breaking through the key difficulties in the cross-integration of current machine learning and occurrence forms.

[0007] The specific technical solutions adopted are as follows:

[0008] A method for predicting the content of heavy metals in different forms in soil based on machine learning, comprising the following steps:

[0009] (1) Soil physical and chemical properties, total amount of heavy metals in soil, types of heavy metals in soil, and content of heavy metals in different forms in soil were collected from the literature database to construct a sample database; soil physical and chemical properties include pH, total organic carbon content, cation exchange capacity, water content, electrical conductivity, total nitrogen, total phosphorus, clay percentage, silt percentage, and sand percentage; the total amount of heavy metals in soil is the heavy metal content measured after mixed acid digestion;

[0010] (2) preprocessing the sample database constructed in step (1) to obtain a preprocessed sample database;

[0011] (3) For each heavy metal, extract the corresponding sample data from the preprocessed sample database, divide the sample data into a training set and a test set, select a machine learning algorithm, and establish a prediction model for the content of the heavy metal in different forms in the soil. The input of the prediction model is the soil physical and chemical properties and the total amount of heavy metals in the soil, and the output is the predicted value of the content of heavy metals in different forms in the soil. The model is trained under the supervision of the actual value of the content of heavy metals in different forms in the soil, and the hyperparameters are optimized during the training process;

[0012] (4) Use the optimized prediction model to predict the content of heavy metals in different occurrence forms in the tested soil.

[0013] Preferably, in the sample database, the occurrence forms of heavy metals include acid-soluble state, reducible state, oxidizable state and residual state.

[0014] In step (2), the pre-processing step includes:

[0015] S01: testing sample data in a sample database based on a standard database, screening and removing outliers, and obtaining a first sample database; further, the screening method includes a triple standard deviation method, an absolute median difference method, or a box plot method;

[0016] S02: If ≥2 soil physical and chemical properties are missing or the content data of heavy metals in different occurrence forms in the soil are missing in the samples of the first sample database, the sample is deleted, and the other missing data are filled to obtain the second sample database;

[0017] S03: Perform standardization processing on the second sample database to obtain a preprocessed sample database.

[0018] Furthermore, in step S02, the filling method is: using ARCGIS mapping, exporting adjacent and same type soil data of the same longitude and latitude in the standard database for filling, and completing the missing parts in the standard database using mean filling, central measure method or EM algorithm.

[0019] Furthermore, in step S03, the standardization method is the Z-score standardization method.

[0020] Specifically, the machine learning algorithm includes a decision tree, a support vector machine, a random forest or a deep neural network, and more preferably a random forest.

[0021] Furthermore, after the optimized prediction model is obtained in step (3), the input characteristics of the prediction model are analyzed to obtain the soil physical and chemical properties with the highest contribution, thereby determining the key influencing factors on the occurrence form of soil heavy metals.

[0022] The present invention also provides the application of the method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning in heavy metal pollution risk assessment and heavy metal pollution control.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The method of the present invention realizes rapid and high-throughput prediction of the distribution of the occurrence forms of heavy metals in soil. Through the constraint of chemical prior knowledge, the prediction accuracy of the optimized prediction model for the oxidizable state and the reducible state reaches above 0.85, which makes up for the limitations of the traditional experimental extraction method that the time is long and the batch prediction cannot be made quickly. It provides effective information for the migration and transformation behavior of heavy metal elements and has good application prospects in heavy metal pollution risk assessment, heavy metal pollution control and other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning.

[0026] Figure 2 This is a graph showing the feature importance and correlation analysis results of the prediction method of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the examples and drawings. It should be understood that these examples are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. The operating methods in the following examples that do not specify specific conditions are usually carried out under conventional conditions or under conditions recommended by the manufacturer. Contents not described in detail in this specification belong to prior art known to professionals and technicians in the field.

[0028] Existing technologies for assessing the risk of heavy metal pollution in soil rarely take into account the contribution of heavy metal storage forms, resulting in low model accuracy and difficulty in explaining the construction mechanism. The method of the present invention proposes to integrate soil physicochemical properties, the total amount of heavy metals in the soil and the content of heavy metals in different storage forms in the soil, quantify the contribution order of different input characteristics, clarify the intrinsic relationship between soil physicochemical properties, the total amount of heavy metals and the storage forms of heavy metals, reveal the transformation law and storage mechanism of soil heavy metal storage forms under the influence of multi-factor coupling, and improve prediction accuracy.

[0029] Example 1

[0030] like Figure 1 As shown, the method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning in the present invention comprises the following steps:

[0031] (1) Collect soil physical and chemical properties, total amount of heavy metals in soil, types of heavy metals in soil, and content of heavy metals in different forms in soil from the literature database to build a sample database;

[0032] (2) preprocessing the sample database constructed in step (1) to obtain a preprocessed sample database;

[0033] (3) For each heavy metal, extract the corresponding sample data from the preprocessed sample database, divide the sample data into a training set and a test set, select a machine learning algorithm, and establish a prediction model for the content of the heavy metal in different forms in the soil. The input of the prediction model is the soil physical and chemical properties and the total amount of heavy metals in the soil, and the output is the predicted value of the content of heavy metals in different forms in the soil. The model is trained under the supervision of the actual value of the content of heavy metals in different forms in the soil, and the hyperparameters are optimized during the training process;

[0034] (4) Use the optimized prediction model to predict the content of heavy metals in different occurrence forms in the tested soil.

[0035] Specifically, in step (1), a literature survey on basic physical and chemical properties of soil and heavy metal occurrence forms was conducted, and the literature data were extracted, converted and integrated through text mining and manual verification to form a sample database; Chinese and English literature materials were obtained through Web of Science, China National Knowledge Infrastructure, Wanfang Data, and domestic and foreign research institutes. The search keywords were three phrases ["soil" AND "heavy metal" AND ("fraction*" OR "distribution" OR "speciation" OR "sequential extraction")]. Excluding review papers, a total of about 6,000-7,000 Chinese and English literature were retrieved, and each soil-heavy metal occurrence form was one data, roughly estimated to be about 9,000-11,000 data in total.

[0036] For the types of heavy metals, lead, chromium, zinc, arsenic, cadmium, and mercury are the main heavy metal research objects, and other heavy metal data are collected for subsequent modeling. Taking into account the availability of literature data, the regional application of subsequent prediction models, and chemical prior knowledge, for each heavy metal, soil physical and chemical properties and the total amount of soil heavy metals are selected as inputs for subsequent model training. The soil physical and chemical properties include pH, total organic matter content (SOM, %), cation exchange capacity (CEC, cmol / kg), water content (WC, %), conductivity (μS / cm), total nitrogen (TN, g / kg), total phosphorus (TP, g / kg), clay percentage (%), silt percentage (%), sand percentage (%), etc. The total amount of soil heavy metals is the heavy metal content (mg / kg) measured after mixed acid digestion; cadmium and lead with the most abundant literature data are selected as representatives for subsequent research.

[0037] According to preliminary literature research, most of the literature data on the storage forms of heavy metals in soil were obtained through the improved BCR four-step extraction method. Therefore, the project preliminarily determined that the output data y is the content of four forms: acid-soluble state (F1), reducible state (F2), oxidizable state (F3), and residual state (F4).

[0038] Specifically, in step (2), a default data completion method based on data interpolation and standard database deduction is developed to improve the chemical connotation and representativeness of the database; in the preprocessing process, for outliers, pH, total organic matter content (SOM, %), cation exchange capacity (CEC, cmol / kg), water content (WC, %), conductivity (μS / cm), total nitrogen (TN, g / kg), total phosphorus (TP, g / kg), clay percentage (%), silt percentage (%), sand percentage (%) (such as mean, maximum value, minimum value, etc.) are determined as chemical prior knowledge based on the soil physical and chemical property data in the global soil database (HWSD, Harmonized World Soil Database), and then the sample data in the sample database are detected based on the distribution characteristics of the above soil physical and chemical property data, and outliers are identified by the interquartile range (IQR) of the box plot based on the outliers. Sample values ​​outside the range of 1.5 times the upper and lower quartiles can be regarded as outliers and removed to obtain the first sample database.

[0039] If ≥2 soil physical and chemical properties are missing or the data on the content of heavy metals in different forms in the soil are missing in the samples of the first sample database, the sample will be deleted and the other missing data will be filled in. The filling method is: using ARCGIS mapping (with a minimum scale of 250m resolution), adjacent and similar soil data at the same longitude and longitude in the global soil database HWSD are exported for filling. Since the physical and chemical data of environmental covariates present a non-normal distribution, the missing parts in the global soil database HWSD are obtained using the central measure method to obtain the second sample database.

[0040] The second sample database is standardized. For the sample data in the second sample database, Z-score is used to perform standardization based on the mean and standard deviation of the original data. The original data is subtracted from the mean and divided by the standard deviation to obtain the standardized sample data. At this point, the preprocessing step is completed and the preprocessed sample database is obtained.

[0041] Specifically, in step (3), for heavy metal cadmium, the corresponding sample data is extracted from the pre-processed sample database, and the sample data is divided into a training set and a test set at a ratio of 8:2 (heavy metal lead is also processed in the same way), and a prediction model is established based on a variety of machine learning algorithms (decision tree, support vector machine, random forest, deep neural network, etc.), the input of the prediction model is the soil physical and chemical properties and the total amount of heavy metals in the soil, and the output is the predicted value of the heavy metal content in different forms in the soil, the model is trained under the supervision of the actual value of the heavy metal content in different forms in the soil, and the hyperparameters are optimized during the training process; the performance of the prediction models established based on different machine learning algorithms is compared, and the determination coefficient (R 2), root mean square error (RMSE), mean absolute error (MAE), etc. are used as metrics to measure the quality of the prediction model. At the same time, the running time of different prediction models is considered, and the complexity and running efficiency of each algorithm are considered. In order to reduce the fluctuation of prediction performance caused by random data partitioning, the reliability of model comparison is improved by taking the average value through multiple random partitions of the data set (~20 times, the specific number of repetitions is determined by convergence test). The hyperparameters of the machine learning algorithm are optimized and adjusted by the grid search method to determine the optimal hyperparameter combination of each prediction model. Multi-fold cross-validation (k-fold cross-validation) is used on the training set to compare hyperparameters.

[0042] Effective fusion of chemical prior knowledge and data is achieved during the training process to give full play to the advantages of both. Data can provide a large amount of sample information, but there may be noise and uncertainty; chemical prior knowledge can provide theoretical guidance and constraints to reduce unreasonable prediction space. By introducing chemical constraints, the machine learning model can better conform to the actual chemical characteristics of the occurrence forms of heavy metals in soil, and improve the accuracy of the model's prediction of the occurrence forms. In the random forest model, the parameter tuning of the regression prediction is processed by the mean square error, which measures the difference between the predicted value and the true value. The goal is to minimize the sum of the mean square errors of the child nodes after the split. Improving the overall prediction performance by continuously splitting nodes and growing trees is better than mathematically optimizing a single loss function.

[0043] After training to obtain the optimal model, the results show that the optimal machine learning algorithm is random forest. In the parameter-optimized Cd prediction model, the R 2 The R values ​​of F1, F2, F3 and F4 in the Pb prediction model were 0.687, 0.905, 0.998 and 0.872, respectively. 2 They are 0.754, 0.863, 0.872 and 0.771.

[0044] After the optimized prediction model is obtained in step (3), the input characteristics of the prediction model are analyzed to obtain the soil physical and chemical properties with the highest contribution, thereby determining the key influencing factors on the occurrence form of heavy metals in the soil.

[0045] Specifically, the analysis method includes obtaining feature importance values ​​from the model object (methods include deep learning automatic feature extraction, recursive feature elimination, principal component analysis or generation and merging of feature items, etc.), associating feature importance values ​​with feature names, and arranging the data frame in descending order according to the importance value, so as to more intuitively view the features with higher importance, determine the key influencing factors on the storage form of soil heavy metals, explore the intrinsic relationship between soil physical and chemical properties, total heavy metal content and soil heavy metal storage form, and reveal the soil heavy metal storage mechanism under the influence of multiple factors coupling.

[0046] In addition, the Pearson correlation coefficient method was used to conduct correlation analysis on the model input features, and the permutation importance was used to evaluate the importance of the model input features to determine the degree of linear relationship between the variables. Correlation analysis can determine whether there is a positive or negative correlation between the total amount of soil heavy metals and soil pH. The correlation results show that the correlation between the input features is in a low range (below 0.4), indicating that the independence between the features is very strong, effectively avoiding the problem of multicollinearity and reducing the risk of overfitting of the model.

[0047] The permutation importance reveals the order of contribution of different input features to the occurrence forms of heavy metals. The specific steps include: after the model training is completed, select an input feature x, set a random number for the feature, and calculate a new prediction result. The new and old prediction results are compared to obtain the influence of the input feature x on the model prediction. Figure 2 As shown in the figure, importance analysis shows that organic carbon is the most significant factor affecting the total amount of heavy metals in soil during modeling (contribution rate is 0.55), pH value is the most significant factor affecting exchangeable heavy metals (contribution rate is 0.18), organic carbon is the most significant factor affecting oxidizable heavy metals (contribution rate is 0.22), and powder is the most significant factor affecting residual heavy metals (contribution rate is 0.31). The above contribution order is analyzed by integrating chemical principles to clarify the contribution mechanism of different input characteristics to the occurrence form of heavy metals in soil.

[0048] The embodiments described above provide a detailed description of the technical solutions of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements or similar substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting the content of heavy metals in different forms in soil based on machine learning, characterized in that: The following steps are involved: (1) Soil physical and chemical properties, total amount of heavy metals in soil, types of heavy metals in soil, and content of heavy metals in different forms in soil were collected from the literature database to construct a sample database; soil physical and chemical properties include pH, total organic carbon content, cation exchange capacity, water content, electrical conductivity, total nitrogen, total phosphorus, clay percentage, silt percentage, and sand percentage; the total amount of heavy metals in soil is the heavy metal content measured after mixed acid digestion; (2) preprocessing the sample database constructed in step (1) to obtain a preprocessed sample database; (3) For each heavy metal, extract the corresponding sample data from the preprocessed sample database, divide the sample data into a training set and a test set, select a machine learning algorithm, and establish a prediction model for the content of the heavy metal in different forms in the soil. The input of the prediction model is the soil physical and chemical properties and the total amount of heavy metals in the soil, and the output is the predicted value of the content of heavy metals in different forms in the soil. The model is trained under the supervision of the actual value of the content of heavy metals in different forms in the soil, and the hyperparameters are optimized during the training process; (4) Use the optimized prediction model to predict the content of heavy metals in different occurrence forms in the tested soil.

2. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 1, characterized in that: In the sample database, the occurrence forms of heavy metals include acid-soluble state, reducible state, oxidizable state and residual state.

3. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 1, characterized in that: In step (2), the pre-processing step includes: S01: testing sample data in a sample database based on a standard database, screening and removing outliers, and obtaining a first sample database; further, the screening method includes a triple standard deviation method, an absolute median difference method, or a box plot method; S02: If ≥2 soil physical and chemical properties are missing or the content data of heavy metals in different occurrence forms in the soil are missing in the samples of the first sample database, the sample is deleted, and the other missing data are filled to obtain the second sample database; S03: Perform standardization processing on the second sample database to obtain a preprocessed sample database.

4. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 3, characterized in that: In step S02, the filling method is: using ARCGIS mapping, exporting adjacent and same type soil data of the same longitude and latitude in the standard database for filling, and completing the missing parts in the standard database using mean filling, central measure method or EM algorithm.

5. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 3 is characterized in that: In step S03, the standardization method is the Z-score standardization method.

6. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 1, characterized in that: The machine learning algorithm includes a decision tree, a support vector machine, a random forest or a deep neural network, further a random forest.

7. The method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to claim 1, characterized in that: After the optimized prediction model is obtained in step (3), the input characteristics of the prediction model are analyzed to obtain the soil physical and chemical properties with the highest contribution, thereby determining the key influencing factors on the occurrence form of heavy metals in the soil.

8. Application of the method for predicting the content of heavy metals in different occurrence forms in soil based on machine learning according to any one of claims 1-7 in heavy metal pollution risk assessment and heavy metal pollution control.

Images